kcat

Generic command line non-JVM Apache Kafka producer and consumer

5,390

483

5,390

153

View on GitHub

Top Related Projects

sarama

11,473

Sarama is a Go library for Apache Kafka.

Quick Overview

kcat (formerly known as kafkacat) is a generic command-line non-JVM Apache Kafka producer and consumer. It's a versatile tool that allows users to produce, consume, and list topic and partition information for Kafka clusters. kcat is particularly useful for debugging and testing Kafka setups.

Pros

Lightweight and fast, with no JVM dependency
Supports both producer and consumer modes
Offers advanced features like metadata listing and Avro message format support
Can be used as a standalone tool or integrated into shell scripts

Cons

Limited compared to full-featured Kafka clients for complex operations
May require additional setup for certain authentication methods
Not suitable for production-level message processing
Documentation could be more comprehensive for advanced use cases

Code Examples

Consuming messages from a topic:

kcat -b localhost:9092 -t my_topic -C

This command connects to a Kafka broker at localhost:9092 and consumes messages from the "my_topic" topic.

Producing messages to a topic:

echo "Hello, Kafka!" | kcat -b localhost:9092 -t my_topic -P

This example sends the message "Hello, Kafka!" to the "my_topic" topic.

Listing topic metadata:

kcat -b localhost:9092 -L

This command lists metadata for all topics in the Kafka cluster.

Getting Started

To get started with kcat:

Install kcat using your package manager or build from source:

# On macOS with Homebrew
brew install kcat

# On Ubuntu/Debian
apt-get install kcat

Verify the installation:
```
kcat -V
```

Use kcat to interact with your Kafka cluster:

# Consume messages
kcat -b your_broker:9092 -t your_topic -C

# Produce messages
echo "Test message" | kcat -b your_broker:9092 -t your_topic -P

Replace your_broker and your_topic with your actual Kafka broker address and topic name.

Competitor Comparisons

kafka

28,601

Mirror of Apache Kafka

Pros of Kafka

Full-featured distributed streaming platform with high scalability and fault tolerance
Supports complex stream processing and real-time data pipelines
Extensive ecosystem with connectors, clients, and tools

Cons of Kafka

Heavier resource requirements and more complex setup
Steeper learning curve for configuration and management
Overkill for simple message production/consumption scenarios

Code Comparison

Kafka (Java):

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);

kcat (Command-line):

echo "Hello, Kafka!" | kcat -b localhost:9092 -t my-topic

Key Differences

Kafka is a full-fledged streaming platform, while kcat is a lightweight command-line utility
Kafka offers more advanced features and scalability, kcat focuses on simplicity and ease of use
Kafka requires Java and additional setup, kcat is a single binary with minimal dependencies
Kafka is suitable for production environments, kcat is ideal for debugging and quick tests

Use Cases

Choose Kafka for building robust, scalable streaming applications
Opt for kcat when you need a quick, easy-to-use tool for Kafka interaction and troubleshooting

kafka-go

7,464

Kafka library in Go

Pros of kafka-go

Written in Go, providing native integration with Go applications
Offers a comprehensive client library for Kafka operations
Supports both producer and consumer functionalities

Cons of kafka-go

Limited to Go programming language
May have a steeper learning curve for those unfamiliar with Go
Lacks some advanced features and tools provided by kcat

Code Comparison

kafka-go (Producer example):

writer := kafka.NewWriter(kafka.WriterConfig{
    Brokers: []string{"localhost:9092"},
    Topic:   "my-topic",
})
err := writer.WriteMessages(context.Background(),
    kafka.Message{Value: []byte("Hello, Kafka!")},
)

kcat (Producer example):

echo "Hello, Kafka!" | kcat -P -b localhost:9092 -t my-topic

kafka-go is a Go library for Kafka, offering native integration with Go applications and comprehensive client functionality. However, it's limited to Go and may have a steeper learning curve. kcat, on the other hand, is a versatile command-line tool that supports multiple programming languages and offers a simpler interface for quick Kafka operations. The code comparison shows the difference in complexity between using kafka-go in a Go application versus using kcat in a shell command for producing messages to Kafka.

sarama

11,473

Sarama is a Go library for Apache Kafka.

Pros of Sarama

Full-featured Kafka client library for Go, offering more comprehensive functionality
Supports both consumer and producer operations with extensive configuration options
Actively maintained with regular updates and a large community

Cons of Sarama

Higher learning curve due to its extensive API and configuration options
Requires more setup and code to perform basic Kafka operations
May be overkill for simple use cases or quick debugging tasks

Code Comparison

Sarama (producer example):

producer, err := sarama.NewSyncProducer([]string{"localhost:9092"}, nil)
message := &sarama.ProducerMessage{Topic: "test", Value: sarama.StringEncoder("test message")}
partition, offset, err := producer.SendMessage(message)

kcat (equivalent command):

echo "test message" | kcat -b localhost:9092 -t test

Summary

Sarama is a comprehensive Go library for Kafka, offering extensive features and flexibility. It's ideal for building robust Kafka-based applications but may be complex for simple tasks. kcat, on the other hand, is a command-line utility focused on quick and easy Kafka interactions, making it more suitable for debugging and simple operations. The choice between them depends on the specific use case and development requirements.

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

logo by @dtrapezoid

kcat

kcat is the project formerly known as as kafkacat

https://github.com/edenhill/kcat

kcat logo by @dtrapezoid

What is kcat

kcat is a generic non-JVM producer and consumer for Apache Kafka >=0.8, think of it as a netcat for Kafka.

In producer mode kcat reads messages from stdin, delimited with a configurable delimiter (-D, defaults to newline), and produces them to the provided Kafka cluster (-b), topic (-t) and partition (-p).

In consumer mode kcat reads messages from a topic and partition and prints them to stdout using the configured message delimiter.

There's also support for the Kafka >=0.9 high-level balanced consumer, use the -G <group> switch and provide a list of topics to join the group.

kcat also features a Metadata list (-L) mode to display the current state of the Kafka cluster and its topics and partitions.

Supports Avro message deserialization using the Confluent Schema-Registry, and generic primitive deserializers (see examples below).

kcat is fast and lightweight; statically linked it is no more than 150Kb.

What happened to kafkacat?

kcat is kafkacat. The kafkacat project was renamed to kcat in August 2021 to adhere to the Apache Software Foundation's (ASF) trademark policies. Apart from the name, nothing else was changed.

Try it out with docker

# List brokers and topics in cluster
$ docker run -it --network=host edenhill/kcat:1.7.1 -b YOUR_BROKER -L

See Examples for usage options, and Running in Docker for more information on how to properly run docker-based clients with Kafka.

Install

On recent enough Debian systems:

apt-get install kafkacat

On recent openSUSE systems:

zypper addrepo https://download.opensuse.org/repositories/network:utilities/openSUSE_Factory/network:utilities.repo
zypper refresh
zypper install kafkacat

(see this page for instructions to install with openSUSE LEAP)

On Mac OS X with homebrew installed:

brew install kcat

On Fedora

# dnf copr enable bvn13/kcat
# dnf update
# dnf install kafkacat

See this blog for how to build from sources and install kafkacat/kcat on recent Fedora systems.

Otherwise follow directions below

Requirements

librdkafka - https://github.com/edenhill/librdkafka
libyajl (for JSON support, optional)
libavro-c and libserdes (for Avro support, optional. See https://github.com/confluentinc/libserdes)

On Ubuntu or Debian: sudo apt-get install librdkafka-dev libyajl-dev

Build

./configure <usual-configure-options>
make
sudo make install

Build for Windows

cd win32
nuget restore
msbuild

NOTE: Requires Build Tools for Visual Studio 2017 with components Windows 8.1 SDK and VC++ 2015.3 v14.00 (v140) toolset to be installed.

Quick build

The bootstrap.sh build script will download and build the required dependencies, providing a quick and easy means of building kcat. Internet connectivity and wget/curl is required by this script. The resulting kcat binary will be linked statically to avoid runtime dependencies. NOTE: Requires curl and cmake (for yajl) to be installed.

./bootstrap.sh

Configuration

Any librdkafka configuration property can be set on the command line using -X property=value, or in a configuration file specified by -F <config-file>.

If no configuration file was specified with -F .. on the command line, kcat will try the $KCAT_CONFIG or (deprecated) $KAFKACAT_CONFIG environment variable, and then the default configuration file ~/.config/kcat.conf or the (deprecated) ~/.config/kafkacat.conf.

Configuration files are optional.

Examples

High-level balanced KafkaConsumer: subscribe to topic1 and topic2 (requires broker >=0.9.0 and librdkafka version >=0.9.1)

$ kcat -b mybroker -G mygroup topic1 topic2

Read messages from stdin, produce to 'syslog' topic with snappy compression

$ tail -f /var/log/syslog | kcat -b mybroker -t syslog -z snappy

Read messages from Kafka 'syslog' topic, print to stdout

$ kcat -b mybroker -t syslog

Produce messages from file (one file is one message)

$ kcat -P -b mybroker -t filedrop -p 0 myfile1.bin /etc/motd thirdfile.tgz

Produce messages transactionally (one single transaction for all messages):

$ kcat -P -b mybroker -t mytopic -X transactional.id=myproducerapp

Read the last 2000 messages from 'syslog' topic, then exit

$ kcat -C -b mybroker -t syslog -p 0 -o -2000 -e

Consume from all partitions from 'syslog' topic

$ kcat -C -b mybroker -t syslog

Output consumed messages in JSON envelope:

$ kcat -b mybroker -t syslog -J

Decode Avro key (-s key=avro), value (-s value=avro) or both (-s avro) to JSON using schema from the Schema-Registry:

$ kcat -b mybroker -t ledger -s avro -r http://schema-registry-url:8080

Decode Avro message value and extract Avro record's "age" field:

$ kcat -b mybroker -t ledger -s value=avro -r http://schema-registry-url:8080 | jq .payload.age

Decode key as 32-bit signed integer and value as 16-bit signed integer followed by an unsigned byte followed by string:

$ kcat -b mybroker -t mytopic -s key='i$' -s value='hB s'

Hint: see kcat -h for all available deserializer options.

Output consumed messages according to format string:

$ kcat -b mybroker -t syslog -f 'Topic %t[%p], offset: %o, key: %k, payload: %S bytes: %s\n'

Read the last 100 messages from topic 'syslog' with librdkafka configuration parameter 'broker.version.fallback' set to '0.8.2.1' :

$ kcat -C -b mybroker -X broker.version.fallback=0.8.2.1 -t syslog -p 0 -o -100 -e

Produce a tombstone (a "delete" for compacted topics) for key "abc" by providing an empty message value which -Z interpretes as NULL:

$ echo "abc:" | kcat -b mybroker -t mytopic -Z -K:

Produce with headers:

$ echo "hello there" | kcat -b mybroker -P -t mytopic -H "header1=header value" -H "nullheader" -H "emptyheader=" -H "header1=duplicateIsOk"

Print headers in consumer:

$ kcat -b mybroker -C -t mytopic -f 'Headers: %h: Message value: %s\n'

Enable the idempotent producer, providing exactly-once and strict-ordering producer guarantees:

$ kcat -b mybroker -X enable.idempotence=true -P -t mytopic ....

Connect to cluster using SSL and SASL PLAIN authentication:

$ kcat -b mybroker -X security.protocol=SASL_SSL -X sasl.mechanism=PLAIN -X sasl.username=myapikey -X sasl.password=myapisecret ...

Metadata listing:

$ kcat -L -b mybroker
Metadata for all topics (from broker 1: mybroker:9092/1):
 3 brokers:
  broker 1 at mybroker:9092
  broker 2 at mybrokertoo:9092
  broker 3 at thirdbroker:9092
 16 topics:
  topic "syslog" with 3 partitions:
    partition 0, leader 3, replicas: 1,2,3, isrs: 1,2,3
    partition 1, leader 1, replicas: 1,2,3, isrs: 1,2,3
    partition 2, leader 1, replicas: 1,2, isrs: 1,2
  topic "rdkafkatest1_auto_49f744a4327b1b1e" with 2 partitions:
    partition 0, leader 3, replicas: 3, isrs: 3
    partition 1, leader 1, replicas: 1, isrs: 1
  topic "rdkafkatest1_auto_e02f58f2c581cba" with 2 partitions:
    partition 0, leader 3, replicas: 3, isrs: 3
    partition 1, leader 1, replicas: 1, isrs: 1
  ....

JSON metadata listing

$ kcat -b mybroker -L -J

Pretty-printed JSON metadata listing

$ kcat -b mybroker -L -J | jq .

Query offset(s) by timestamp(s)

$ kcat -b mybroker -Q -t mytopic:3:2389238523 -t mytopic2:0:18921841

Consume messages between two timestamps

$ kcat -b mybroker -C -t mytopic -o s@1568276612443 -o e@1568276617901

Running in Docker

The latest kcat docker image is edenhill/kcat:1.7.1, there's also Confluent's kafkacat docker images on Docker Hub.

If you are connecting to Kafka brokers also running on Docker you should specify the network name as part of the docker run command using the --network parameter. For more details of networking with Kafka and Docker see this post.

Here are two short examples of using kcat from Docker. See the Docker Hub listing and kafkacat docs for more details:

Send messages using here doc:

docker run -it --rm \
        edenhill/kcat \
                -b kafka-broker:9092 \
                -t test \
                -K: \
                -P <<EOF

1:{"order_id":1,"order_ts":1534772501276,"total_amount":10.50,"customer_name":"Bob Smith"}
2:{"order_id":2,"order_ts":1534772605276,"total_amount":3.32,"customer_name":"Sarah Black"}
3:{"order_id":3,"order_ts":1534772742276,"total_amount":21.00,"customer_name":"Emma Turner"}
EOF

Consume messages:

docker run -it --rm \
        edenhill/kcat \
           -b kafka-broker:9092 \
           -C \
           -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\n\Partition: %p\tOffset: %o\n--\n' \
           -t test

Key (1 bytes): 1
Value (88 bytes): {"order_id":1,"order_ts":1534772501276,"total_amount":10.50,"customer_name":"Bob Smith"}
Partition: 0    Offset: 0
--

Key (1 bytes): 2
Value (89 bytes): {"order_id":2,"order_ts":1534772605276,"total_amount":3.32,"customer_name":"Sarah Black"}
Partition: 0    Offset: 1
--

Key (1 bytes): 3
Value (90 bytes): {"order_id":3,"order_ts":1534772742276,"total_amount":21.00,"customer_name":"Emma Turner"}
Partition: 0    Offset: 2
--
% Reached end of topic test [0] at offset 3

Run a mock Kafka cluster

With kcat you can spin up an ephemeral in-memory mock Kafka cluster that you you can connect your Kafka applications to for quick testing. The mock cluster supports a reasonable subset of the Kafka protocol, such as:

Producer
Idempotent Producer
Transactional Producer
Low-level consumer
High-level balanced consumer groups with offset commits
Topic Metadata and auto creation

Spin the cluster by running kcat in the -M (for mock) mode:


# Create mock cluster with 3 brokers
$ kcat -M 3
...
BROKERS=localhost:12345,localhost:46346,localhost:23599
...

While kcat runs, let your Kafka applications connect to the mock cluster by configuring them with the bootstrap.servers emitted in the BROKERS line above.

Let kcat run for as long as you need the cluster, then terminate it by pressing Ctrl-D.

Since the cluster runs all in memory, with no disk IO, it is quite suitable for performance testing.

Top Related Projects

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot