Top Related Projects
Dockerfile for Apache Kafka
Kafka Docker for development. Kafka, Zookeeper, Schema Registry, Kafka-Connect, Landoop Tools, 20+ connectors
Open-Source Web UI for Apache Kafka Management
Quick Overview
The spotify/docker-kafka repository is a Docker image for Apache Kafka, maintained by Spotify. It provides a containerized version of Kafka, making it easier to deploy and run Kafka in various environments, including development and production setups.
Pros
- Easy deployment and scaling of Kafka clusters using Docker
- Consistent environment across different machines and platforms
- Simplified setup process for development and testing purposes
- Regularly maintained and updated by Spotify
Cons
- May require additional configuration for production-grade setups
- Docker overhead might impact performance in some scenarios
- Limited customization options compared to manual Kafka installations
- Potential security concerns if not properly configured in a containerized environment
Getting Started
To use the Spotify Docker Kafka image, follow these steps:
-
Pull the Docker image:
docker pull spotify/kafka
-
Run a Kafka broker:
docker run -d --name kafka -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`docker-machine ip \`docker-machine active\`` --env ADVERTISED_PORT=9092 spotify/kafka
-
Create a topic:
docker run --rm spotify/kafka kafka-topics.sh --create --topic test --partitions 1 --replication-factor 1 --if-not-exists --zookeeper $(docker-machine ip $(docker-machine active)):2181
-
Produce messages:
docker run --rm -it spotify/kafka kafka-console-producer.sh --broker-list $(docker-machine ip $(docker-machine active)):9092 --topic test
-
Consume messages:
docker run --rm -it spotify/kafka kafka-console-consumer.sh --bootstrap-server $(docker-machine ip $(docker-machine active)):9092 --topic test --from-beginning
These instructions provide a basic setup for running Kafka in a Docker container and interacting with it using the provided tools. For more advanced configurations and usage, refer to the repository's documentation.
Competitor Comparisons
Dockerfile for Apache Kafka
Pros of kafka-docker
- More actively maintained with frequent updates
- Supports multiple Kafka and Scala versions
- Offers more configuration options and flexibility
Cons of kafka-docker
- Slightly more complex setup process
- May require additional configuration for production use
- Less opinionated, which can lead to more decision-making for users
Code Comparison
kafka-docker:
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
build: .
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: localhost
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
docker-kafka:
version: '2'
services:
kafka:
image: spotify/kafka
ports:
- "9092:9092"
- "2181:2181"
environment:
ADVERTISED_HOST: localhost
ADVERTISED_PORT: 9092
The kafka-docker example shows separate services for Zookeeper and Kafka, offering more granular control. The docker-kafka example combines both into a single service, which is simpler but less flexible.
kafka-docker provides more configuration options through environment variables, while docker-kafka offers a more streamlined setup with fewer options out-of-the-box.
Kafka Docker for development. Kafka, Zookeeper, Schema Registry, Kafka-Connect, Landoop Tools, 20+ connectors
Pros of fast-data-dev
- Includes a comprehensive set of Kafka ecosystem tools (Kafka, ZooKeeper, Schema Registry, Kafka Connect, Landoop Tools)
- Provides a web UI for easier management and monitoring
- Offers pre-configured settings for quick setup and development
Cons of fast-data-dev
- Larger image size due to additional components
- May be overkill for simple Kafka setups or testing
- Potentially more complex to customize or extend
Code Comparison
fast-data-dev:
version: '2'
services:
fast-data-dev:
image: lensesio/fast-data-dev
ports:
- "2181:2181"
- "9092:9092"
- "8081-8083:8081-8083"
docker-kafka:
version: '2'
services:
kafka:
image: spotify/kafka
ports:
- "9092:9092"
- "2181:2181"
The fast-data-dev example exposes more ports for additional services, while docker-kafka focuses on core Kafka and ZooKeeper ports. fast-data-dev provides a more comprehensive environment out-of-the-box, whereas docker-kafka offers a simpler, more focused Kafka setup that may be easier to customize for specific needs.
Open-Source Web UI for Apache Kafka Management
Pros of kafka-ui
- Provides a user-friendly web interface for managing Kafka clusters
- Offers more comprehensive monitoring and management features
- Actively maintained with regular updates and new features
Cons of kafka-ui
- Requires additional setup and configuration compared to docker-kafka
- May have a steeper learning curve for users new to Kafka management tools
Code Comparison
kafka-ui (Docker Compose example):
version: '2'
services:
kafka-ui:
image: provectuslabs/kafka-ui
container_name: kafka-ui
ports:
- "8080:8080"
environment:
- KAFKA_CLUSTERS_0_NAME=local
- KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092
docker-kafka (Docker Compose example):
version: '2'
services:
kafka:
image: spotify/kafka
ports:
- "9092:9092"
environment:
ADVERTISED_HOST: localhost
ADVERTISED_PORT: 9092
kafka-ui offers a more feature-rich solution for managing Kafka clusters through a web interface, while docker-kafka provides a simpler Docker image for running Kafka instances. The choice between the two depends on the specific needs of the project and the level of management capabilities required.
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual CopilotREADME
Kafka in Docker
This repository provides everything you need to run Kafka in Docker.
For convenience also contains a packaged proxy that can be used to get data from a legacy Kafka 7 cluster into a dockerized Kafka 8.
Why?
The main hurdle of running Kafka in Docker is that it depends on Zookeeper. Compared to other Kafka docker images, this one runs both Zookeeper and Kafka in the same container. This means:
- No dependency on an external Zookeeper host, or linking to another container
- Zookeeper and Kafka are configured to work together out of the box
Run
docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`docker-machine ip \`docker-machine active\`` --env ADVERTISED_PORT=9092 spotify/kafka
export KAFKA=`docker-machine ip \`docker-machine active\``:9092
kafka-console-producer.sh --broker-list $KAFKA --topic test
export ZOOKEEPER=`docker-machine ip \`docker-machine active\``:2181
kafka-console-consumer.sh --zookeeper $ZOOKEEPER --topic test
Running the proxy
Take the same parameters as the spotify/kafka image with some new ones:
CONSUMER_THREADS
- the number of threads to consume the source kafka 7 withTOPICS
- whitelist of topics to mirrorZK_CONNECT
- the zookeeper connect string of the source kafka 7GROUP_ID
- the group.id to use when consuming from kafka 7
docker run -p 2181:2181 -p 9092:9092 \
--env ADVERTISED_HOST=`boot2docker ip` \
--env ADVERTISED_PORT=9092 \
--env CONSUMER_THREADS=1 \
--env TOPICS=my-topic,some-other-topic \
--env ZK_CONNECT=kafka7zookeeper:2181/root/path \
--env GROUP_ID=mymirror \
spotify/kafkaproxy
In the box
-
spotify/kafka
The docker image with both Kafka and Zookeeper. Built from the
kafka
directory. -
spotify/kafkaproxy
The docker image with Kafka, Zookeeper and a Kafka 7 proxy that can be configured with a set of topics to mirror.
Public Builds
https://registry.hub.docker.com/u/spotify/kafka/
https://registry.hub.docker.com/u/spotify/kafkaproxy/
Build from Source
docker build -t spotify/kafka kafka/
docker build -t spotify/kafkaproxy kafkaproxy/
Todo
- Not particularily optimzed for startup time.
- Better docs
Top Related Projects
Dockerfile for Apache Kafka
Kafka Docker for development. Kafka, Zookeeper, Schema Registry, Kafka-Connect, Landoop Tools, 20+ connectors
Open-Source Web UI for Apache Kafka Management
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual Copilot