Convert Figma logo to code with AI

spotify logodocker-kafka

Kafka (and Zookeeper) in Docker

1,399
643
1,399
62

Top Related Projects

Dockerfile for Apache Kafka

Kafka Docker for development. Kafka, Zookeeper, Schema Registry, Kafka-Connect, Landoop Tools, 20+ connectors

Open-Source Web UI for Apache Kafka Management

Quick Overview

The spotify/docker-kafka repository is a Docker image for Apache Kafka, maintained by Spotify. It provides a containerized version of Kafka, making it easier to deploy and run Kafka in various environments, including development and production setups.

Pros

  • Easy deployment and scaling of Kafka clusters using Docker
  • Consistent environment across different machines and platforms
  • Simplified setup process for development and testing purposes
  • Regularly maintained and updated by Spotify

Cons

  • May require additional configuration for production-grade setups
  • Docker overhead might impact performance in some scenarios
  • Limited customization options compared to manual Kafka installations
  • Potential security concerns if not properly configured in a containerized environment

Getting Started

To use the Spotify Docker Kafka image, follow these steps:

  1. Pull the Docker image:

    docker pull spotify/kafka
    
  2. Run a Kafka broker:

    docker run -d --name kafka -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`docker-machine ip \`docker-machine active\`` --env ADVERTISED_PORT=9092 spotify/kafka
    
  3. Create a topic:

    docker run --rm spotify/kafka kafka-topics.sh --create --topic test --partitions 1 --replication-factor 1 --if-not-exists --zookeeper $(docker-machine ip $(docker-machine active)):2181
    
  4. Produce messages:

    docker run --rm -it spotify/kafka kafka-console-producer.sh --broker-list $(docker-machine ip $(docker-machine active)):9092 --topic test
    
  5. Consume messages:

    docker run --rm -it spotify/kafka kafka-console-consumer.sh --bootstrap-server $(docker-machine ip $(docker-machine active)):9092 --topic test --from-beginning
    

These instructions provide a basic setup for running Kafka in a Docker container and interacting with it using the provided tools. For more advanced configurations and usage, refer to the repository's documentation.

Competitor Comparisons

Dockerfile for Apache Kafka

Pros of kafka-docker

  • More actively maintained with frequent updates
  • Supports multiple Kafka and Scala versions
  • Offers more configuration options and flexibility

Cons of kafka-docker

  • Slightly more complex setup process
  • May require additional configuration for production use
  • Less opinionated, which can lead to more decision-making for users

Code Comparison

kafka-docker:

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    build: .
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

docker-kafka:

version: '2'
services:
  kafka:
    image: spotify/kafka
    ports:
     - "9092:9092"
     - "2181:2181"
    environment:
      ADVERTISED_HOST: localhost
      ADVERTISED_PORT: 9092

The kafka-docker example shows separate services for Zookeeper and Kafka, offering more granular control. The docker-kafka example combines both into a single service, which is simpler but less flexible.

kafka-docker provides more configuration options through environment variables, while docker-kafka offers a more streamlined setup with fewer options out-of-the-box.

Kafka Docker for development. Kafka, Zookeeper, Schema Registry, Kafka-Connect, Landoop Tools, 20+ connectors

Pros of fast-data-dev

  • Includes a comprehensive set of Kafka ecosystem tools (Kafka, ZooKeeper, Schema Registry, Kafka Connect, Landoop Tools)
  • Provides a web UI for easier management and monitoring
  • Offers pre-configured settings for quick setup and development

Cons of fast-data-dev

  • Larger image size due to additional components
  • May be overkill for simple Kafka setups or testing
  • Potentially more complex to customize or extend

Code Comparison

fast-data-dev:

version: '2'
services:
  fast-data-dev:
    image: lensesio/fast-data-dev
    ports:
      - "2181:2181"
      - "9092:9092"
      - "8081-8083:8081-8083"

docker-kafka:

version: '2'
services:
  kafka:
    image: spotify/kafka
    ports:
      - "9092:9092"
      - "2181:2181"

The fast-data-dev example exposes more ports for additional services, while docker-kafka focuses on core Kafka and ZooKeeper ports. fast-data-dev provides a more comprehensive environment out-of-the-box, whereas docker-kafka offers a simpler, more focused Kafka setup that may be easier to customize for specific needs.

Open-Source Web UI for Apache Kafka Management

Pros of kafka-ui

  • Provides a user-friendly web interface for managing Kafka clusters
  • Offers more comprehensive monitoring and management features
  • Actively maintained with regular updates and new features

Cons of kafka-ui

  • Requires additional setup and configuration compared to docker-kafka
  • May have a steeper learning curve for users new to Kafka management tools

Code Comparison

kafka-ui (Docker Compose example):

version: '2'
services:
  kafka-ui:
    image: provectuslabs/kafka-ui
    container_name: kafka-ui
    ports:
      - "8080:8080"
    environment:
      - KAFKA_CLUSTERS_0_NAME=local
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092

docker-kafka (Docker Compose example):

version: '2'
services:
  kafka:
    image: spotify/kafka
    ports:
      - "9092:9092"
    environment:
      ADVERTISED_HOST: localhost
      ADVERTISED_PORT: 9092

kafka-ui offers a more feature-rich solution for managing Kafka clusters through a web interface, while docker-kafka provides a simpler Docker image for running Kafka instances. The choice between the two depends on the specific needs of the project and the level of management capabilities required.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Kafka in Docker

This repository provides everything you need to run Kafka in Docker.

For convenience also contains a packaged proxy that can be used to get data from a legacy Kafka 7 cluster into a dockerized Kafka 8.

Why?

The main hurdle of running Kafka in Docker is that it depends on Zookeeper. Compared to other Kafka docker images, this one runs both Zookeeper and Kafka in the same container. This means:

  • No dependency on an external Zookeeper host, or linking to another container
  • Zookeeper and Kafka are configured to work together out of the box

Run

docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`docker-machine ip \`docker-machine active\`` --env ADVERTISED_PORT=9092 spotify/kafka
export KAFKA=`docker-machine ip \`docker-machine active\``:9092
kafka-console-producer.sh --broker-list $KAFKA --topic test
export ZOOKEEPER=`docker-machine ip \`docker-machine active\``:2181
kafka-console-consumer.sh --zookeeper $ZOOKEEPER --topic test

Running the proxy

Take the same parameters as the spotify/kafka image with some new ones:

  • CONSUMER_THREADS - the number of threads to consume the source kafka 7 with
  • TOPICS - whitelist of topics to mirror
  • ZK_CONNECT - the zookeeper connect string of the source kafka 7
  • GROUP_ID - the group.id to use when consuming from kafka 7
docker run -p 2181:2181 -p 9092:9092 \
    --env ADVERTISED_HOST=`boot2docker ip` \
    --env ADVERTISED_PORT=9092 \
    --env CONSUMER_THREADS=1 \
    --env TOPICS=my-topic,some-other-topic \
    --env ZK_CONNECT=kafka7zookeeper:2181/root/path \
    --env GROUP_ID=mymirror \
    spotify/kafkaproxy

In the box

  • spotify/kafka

    The docker image with both Kafka and Zookeeper. Built from the kafka directory.

  • spotify/kafkaproxy

    The docker image with Kafka, Zookeeper and a Kafka 7 proxy that can be configured with a set of topics to mirror.

Public Builds

https://registry.hub.docker.com/u/spotify/kafka/

https://registry.hub.docker.com/u/spotify/kafkaproxy/

Build from Source

docker build -t spotify/kafka kafka/
docker build -t spotify/kafkaproxy kafkaproxy/

Todo

  • Not particularily optimzed for startup time.
  • Better docs