Running Kafka Producer and Consumer from command line

Kafka provided a number of utility tools inside the distribution. In this article, I am going to discuss about some of the most frequently used commands related to Kafka producers and consumers. This tutorial requires you to have Kafka installed. See my previous posts to setup Kafka and Zookeeper, if you haven’t yet installed them. It also requires you to already have a topic created. See my previous post how to create and manage Kafka topics from command line. Let’s start.

Kafka Console Producer:

Let’s publish a few messages to our new topic:

$ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic
my test message 1
my test message 2

We can also send messages stored in a file like this:

$ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic < messages.txt

Sending lines of text will result in messages with null keys. In order to send messages with both keys and values you must set the --parse.key property to true and --key.separator property to a separator (i.e. comma, semicolon, colon etc.) on the command line when running the producer.

$ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic --property "parse.key=true" --property "key.separator=:"
key1:my test message 1
key2:my test message 2

Kafka Console Consumer:

Now let’s consume these messages. Start a new terminal and run:

$ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-topic
my test message 1
my test message 2

If you do not need to consume older messages, then remove --from-beginning. Removing this flag will consume only new messages.

To print key of records in kafka-console-consumer use --property print.key=true property.

$ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-topic --property print.key=true
null my test message 1
null my test message 2
key1 my test message 1
key2 my test message 2

That’t it. These are the most commonly used Kafka commands for running producer and consumer from command line terminal.