• Skip to primary navigation
  • Skip to content
  • Skip to primary sidebar
Clean Programmer

Clean Programmer

Programming & DevOps Resources

  • Home
  • Library
  • About
  • Contact

Running Kafka Producer and Consumer from command line

October 14, 2018 Monzurul Haque Shimul

Kafka provided a number of utility tools inside the distribution. In this article, I am going to discuss about some of the most frequently used commands related to Kafka producers and consumers. This tutorial requires you to have Kafka installed. See my previous posts to setup Kafka and Zookeeper, if you haven’t yet installed them. It also requires you to already have a topic created. See my previous post how to create and manage Kafka topics from command line. Let’s start.

Kafka Console Producer:

Let’s publish a few messages to our new topic:

$ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic
my test message 1
my test message 2

We can also send messages stored in a file like this:

$ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic < messages.txt

Sending lines of text will result in messages with null keys. In order to send messages with both keys and values you must set the --parse.key property to true and --key.separator property to a separator (i.e. comma, semicolon, colon etc.) on the command line when running the producer.

$ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic --property "parse.key=true" --property "key.separator=:"
key1:my test message 1
key2:my test message 2

Kafka Console Consumer:

Now let’s consume these messages. Start a new terminal and run:

$ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-topic
my test message 1
my test message 2

If you do not need to consume older messages, then remove --from-beginning. Removing this flag will consume only new messages.

To print key of records in kafka-console-consumer use --property print.key=true property.

$ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-topic --property print.key=true
null my test message 1
null my test message 2
key1 my test message 1
key2 my test message 2

That’t it. These are the most commonly used Kafka commands for running producer and consumer from command line terminal.

Apache Kafka kafka

Reader Interactions

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Primary Sidebar

Categories

  • Apache Kafka
  • Druid
  • Git
  • Java
  • Java EE
  • Redis
  • Spring
  • Uncategorized
  • Weblogic
  • Wildfly

Featured Posts

How to Configure Druid to Use Minio as Deep Storage

Count Unique in Druid Using DataSketches HLL Sketch

Configure Druid to Use Lookups from CSV Files

Loading data into Druid from TSV file

Handling Nested Json During Ingestion Into Druid

Tags

bash bitbucket cassandra cloudserver curl docker druid.io eclipselink ejb git imply.io java java-ee jaxws jboss jboss-cli jdbc jdk jms kafka maven minio mssql mysql ojdbc oracle postgresql redis rest rest-template S3 scality sdk sdkman soap spring sqlserver stream stream api weblogic web services wildfly wsdl zenko zookeeper

Archives

  • January 2019
  • December 2018
  • November 2018
  • October 2018
  • September 2018
  • August 2018
  • July 2018
  • June 2018
  • May 2018

Copyright © 2019 · CLEAN PROGRAMMER

  • Privacy Policy
We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.OKNoRead more
Revoke Cookies