What this is
Message Queue is a Zeotap CDP source category: it brings data into Zeotap CDP from a queue that your own systems already publish to. You keep publishing messages the way you do today, and Zeotap CDP subscribes to the topic as one more consumer, so the system that produces the data never has to connect to Zeotap CDP directly. A message is a piece of information that another system processes. It can be an actual payload, such as a file, or metadata that triggers processing in another sub-system. A queue receives those messages in a specific order and forwards them to the subscribing application in the same order. Because the queue sits between the two sides, the sender and the recipient stay decoupled: each operates at its own pace, and consumers retrieve messages when they are ready to process them, which makes the processing asynchronous. Message Queue sources carry both Customer Data (CE) and Non-Customer Entity (NCE) data. Message Queue is also the Category you select in the Sources module when you create any of the integrations below.Prerequisites
Before you create a Message Queue source, confirm:
- The topic already exists in your own Google Cloud Pub/Sub or Apache Kafka account, and you can obtain its connection details.
How pub/sub messaging works
In conventional messaging, a message is sent directly from a sender to one specific receiver, and the sender must know the receiver’s address to deliver it. That creates tight coupling between the sender and the receiver, which makes it harder to scale the application or add new recipients. In pub/sub messaging, publishers do not need to know the identity or location of the receivers. Publishers send messages to a topic or channel, subscribers interested in that topic register themselves, and the pub/sub system delivers each message to every interested subscriber. In every Message Queue source you point Zeotap CDP at one of your topics, and Zeotap CDP consumes from it.Supported Message Queue sources
Pub/Sub Stream
Consume Google Pub/Sub messages in real time, as soon as they are published to the topic.
Pub/Sub Batch
Send many messages to a Google Pub/Sub topic in a single request, for large numbers of messages processed together.
Kafka
Ingest from an Apache Kafka topic using a consumer group. AVRO and JSON payloads are supported.
Apache Kafka also exists as a Zeotap CDP destination, which moves data the other way — out of Zeotap CDP and into your Kafka topic. See Apache Kafka. Google Pub/Sub also exists as a Zeotap CDP destination, which sends data out of Zeotap CDP into your Pub/Sub topic through Journeys. See Google Pub/Sub.
Next steps
- Create a Pub/Sub Stream source
- Create a Pub/Sub Batch source
- Kafka — prerequisites and creation steps for the Kafka source
- Sources — all nine source categories and how to choose one