Send Documents to a Kafka Topic

This example pipeline demonstrates how we can send documents to a Kafka topic using the Kafka Producer Snap. We use the Sequence Snap to enable the pipeline to send documents in large numbers.


Kafka Producer pipeline overview

Download this pipeline
  1. Configure the Sequence Snap

    First, we configure the Sequence Snap to send 1000 documents. Hence, we set the Initial value as 1 and Number of documents as 1000.


    Sequence Snap configuration to generate documents

  2. Configure the Kafka Producer Snap

    Next, we configure the Kafka Producer Snap to send the documents to the topic named SampleKafkaTopic. We set the Partition number to 0 to let the broker decide which partition to use.


    Kafka Producer Snap configuration

  3. Validate the output

    Upon successful validation, the Pipeline displays the output preview as shown below:


    Output preview of Kafka Producer Snap after successful validation

  1. Ensure that the Kafka Producer Snap is configured with the correct topic name and partition number.
  2. Validate that the Sequence Snap generates the expected number of documents before sending them to Kafka.
  3. Confirm successful message delivery by reviewing the output preview or monitoring the Kafka topic.