fast-data-dev
Kafka Docker for development. Kafka, Zookeeper, Schema Registry, Kafka-Connect, , 20+ connectors
Top Related Projects
docker-compose.yml files for cp-all-in-one , cp-all-in-one-community, cp-all-in-one-cloud, Apache Kafka Confluent Platform
Open-Source Web UI for Apache Kafka Management
docker compose files to create a fully working kafka stack
Dockerfile for Apache Kafka
Quick Overview
Fast-data-dev is a Docker image that provides a complete Kafka development environment. It includes Kafka, Zookeeper, Schema Registry, Kafka-Connect, Landoop's Topic UI, Schema Registry UI, and Kafka-Connect UI, all pre-configured and ready to use. This project aims to simplify the setup process for developers working with Kafka and related technologies.
Pros
- Easy setup: One-command deployment of a full Kafka ecosystem
- Comprehensive: Includes multiple tools and UIs for Kafka management
- Customizable: Allows configuration through environment variables
- Educational: Provides a sandbox environment for learning and experimenting with Kafka
Cons
- Resource-intensive: Running the full stack may require significant system resources
- Not suitable for production: Designed for development and testing purposes only
- Limited scalability: Not intended for distributed setups or high-load scenarios
- Potential version conflicts: May not always have the latest versions of all included components
Getting Started
To start using fast-data-dev, follow these steps:
- Ensure Docker is installed on your system.
- Run the following command to start the container:
docker run --rm -it -p 2181:2181 -p 3030:3030 -p 8081-8083:8081-8083 \
-p 9581-9585:9581-9585 -p 9092:9092 -e ADV_HOST=127.0.0.1 \
lensesio/fast-data-dev:latest
-
Access the web UIs:
- Kafka Topics UI: http://localhost:3030
- Schema Registry UI: http://localhost:8081
- Kafka Connect UI: http://localhost:8083
-
Use the Kafka broker at
localhost:9092
in your applications.
For more advanced configurations and usage, refer to the project's GitHub repository.
Competitor Comparisons
docker-compose.yml files for cp-all-in-one , cp-all-in-one-community, cp-all-in-one-cloud, Apache Kafka Confluent Platform
Pros of cp-all-in-one
- Offers a more comprehensive Confluent Platform ecosystem, including additional tools like Schema Registry and KSQL
- Provides a closer representation of a production-like Confluent environment
- Regularly updated with the latest Confluent Platform versions
Cons of cp-all-in-one
- More complex setup and configuration compared to fast-data-dev
- Requires more system resources due to the full suite of Confluent tools
- May have a steeper learning curve for beginners
Code Comparison
fast-data-dev:
version: '2'
services:
fast-data-dev:
image: lensesio/fast-data-dev
ports:
- "2181:2181"
- "9092:9092"
cp-all-in-one:
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
kafka:
image: confluentinc/cp-kafka:latest
schema-registry:
image: confluentinc/cp-schema-registry:latest
Both repositories provide Docker Compose configurations for setting up Kafka environments. fast-data-dev offers a more streamlined setup with a single service, while cp-all-in-one separates components into individual services, allowing for more granular control but requiring more configuration.
Open-Source Web UI for Apache Kafka Management
Pros of kafka-ui
- Modern, user-friendly web interface for managing Kafka clusters
- Supports multiple clusters in a single UI
- Actively maintained with frequent updates and new features
Cons of kafka-ui
- Focused solely on Kafka management, lacking additional data processing tools
- May require separate setup for Kafka and other components
- Less comprehensive out-of-the-box solution compared to fast-data-dev
Code Comparison
kafka-ui (Docker Compose example):
version: '2'
services:
kafka-ui:
image: provectuslabs/kafka-ui
container_name: kafka-ui
ports:
- "8080:8080"
environment:
- KAFKA_CLUSTERS_0_NAME=local
- KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092
fast-data-dev (Docker Compose example):
version: '2'
services:
fast-data-dev:
image: lensesio/fast-data-dev
environment:
ADV_HOST: 127.0.0.1
ports:
- "2181:2181"
- "3030:3030"
- "8081-8083:8081-8083"
- "9581-9585:9581-9585"
- "9092:9092"
Both projects provide Docker-based solutions for Kafka management, but fast-data-dev offers a more comprehensive environment with additional components like Schema Registry and Kafka Connect, while kafka-ui focuses on a modern UI for Kafka cluster management.
docker compose files to create a fully working kafka stack
Pros of kafka-stack-docker-compose
- More customizable and flexible, allowing users to configure individual components
- Includes additional tools like Schema Registry and Kafka Connect
- Regularly updated with the latest Kafka versions
Cons of kafka-stack-docker-compose
- Requires more setup and configuration compared to the all-in-one solution
- May have a steeper learning curve for beginners
- Lacks some of the built-in monitoring and management tools
Code Comparison
fast-data-dev:
version: '2'
services:
fast-data-dev:
image: lensesio/fast-data-dev
environment:
- ADV_HOST=127.0.0.1
kafka-stack-docker-compose:
version: '3'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
kafka:
image: confluentinc/cp-kafka:latest
schema-registry:
image: confluentinc/cp-schema-registry:latest
The fast-data-dev repository provides a single container solution with pre-configured components, making it easier to set up and run quickly. On the other hand, kafka-stack-docker-compose offers more granular control over individual services, allowing users to customize their Kafka environment according to specific needs. While fast-data-dev may be more suitable for rapid prototyping and development, kafka-stack-docker-compose is better suited for production-like environments and advanced use cases.
Dockerfile for Apache Kafka
Pros of kafka-docker
- Lightweight and focused solely on Kafka and ZooKeeper
- Highly customizable through environment variables
- Widely adopted and well-maintained
Cons of kafka-docker
- Limited to core Kafka functionality
- Requires more manual setup for additional tools
- Less suitable for beginners or quick prototyping
Code Comparison
kafka-docker:
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
kafka:
image: wurstmeister/kafka
environment:
KAFKA_ADVERTISED_HOST_NAME: localhost
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
fast-data-dev:
version: '2'
services:
fast-data-dev:
image: lensesio/fast-data-dev
environment:
ADV_HOST: 127.0.0.1
RUNTESTS: 0
Key Differences
- fast-data-dev provides a comprehensive Kafka ecosystem with additional tools and UI
- kafka-docker focuses on core Kafka and ZooKeeper services
- fast-data-dev is easier to set up for beginners and rapid prototyping
- kafka-docker offers more granular control over Kafka configuration
- fast-data-dev includes a web UI for management and monitoring
- kafka-docker requires separate setup for monitoring and management tools
Both repositories serve different use cases, with fast-data-dev being more suitable for development and learning environments, while kafka-docker is better for production-like setups and custom configurations.
Convert
designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual CopilotREADME
Fast Data Dev
Join the Slack Lenses.io Community!
Overview
Fast Data Dev is an Apache Kafka development environment packaged as a Docker container. It provides a Kafka ecosystem including a Kafka Broker, Confluent Schema Registry, and Kafka Connect with Lenses.io's Stream Reactor open source connectors (with optional enterprise support) pre-installed. On top of that there are data generators available, a web interface so you can access the various services logs and optionally TLS keystores and truststores, and a ton of settings so you can tailor it to your needs.
Quick Start
Our recommendation is to combine fast-data-dev with Lenses CE (community edition) using our docker-compose, available here. Lenses provides industry-leading Developer Experience for Apache Kafka and Kafka Connect and among many features, you may particularly enjoy our SQL Studio that allows you to run SQL queries on your Kafka topics and the SQL Processors for creating Streaming SQL queries.
Our docker-compose will setup fast-data-dev and Lenses for you.
Basic Usage
Start a complete Kafka environment with one command:
docker run --rm \
-p 9092:9092 \
-p 8081:8081 \
-p 8083:8083 \
-p 3030:3030 \
lensesio/fast-data-dev
Access the web interface at: http://localhost:3030 The Kafka Broker will be accessible at localhost:9092, the Schema Registry at http://localhost:8081, and Kafka Connect at http://localhost:8083.
With a Custom Host for Remote Access
docker run --rm \
-e ADV_HOST=<YOUR_IP_ADDRESS or DNS name> \
-p 9092:9092 \
-p 8081:8081 \
-p 8083:8083 \
-p 3030:3030 \
lensesio/fast-data-dev
This setup may require to allow connections to fast-data-dev from itself. For example if you are running in a Cloud VM, you may need to setup the firewall to not only accept connections from your computer, but also from the VM itself.
Within docker-compose
This setup allows for both access from your host and the rest of the
docker-compose services. The Broker address from your host is localhost:9092
,
whilst for docker-compose services it is kafka:19092
.
kafka:
image: lensesio/fast-data-dev:3.9.0
hostname: kafka
environment:
ADV_HOST: kafka
RUNNING_SAMPLEDATA: 1
RUNTESTS: 0
# The next three variables are required if you want to have Kafka
# available at the host for local development. They are tailored to KRaft
# fast-data-dev (3.9.x or later). The broker will be available at localhost:9092.
KAFKA_LISTENERS: PLAINTEXT://:9092,DOCKERCOMPOSE://:19092,CONTROLLER://:16062
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,DOCKERCOMPOSE://demo-kafka:19092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: >
DOCKERCOMPOSE:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,
SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
# Makes things a little lighter
DISABLE: debezium-mongodb,debezium-mysql,debezium-postgres,debezium-sqlserver,debezium-jdbc
# These are the ports for Kafka and Schema Registry. You can disable them if
# you don't need to access Kafka from your host.
ports:
- 9092:9092
- 8081:8081
- 8083:8083
- 3030:3030
# Optional, to allow to resume work (persistence)
# volumes:
# - kafka-data-volume:/data
SSL/TLS Setup
# Enable SSL with automatic certificate generation
docker run --rm -p 9092:9092 -p 8081:8081 -p 3030:3030 -p 8083:8083 \
-e ENABLE_SSL=1 \
-e SSL_EXTRA_HOSTS=myhost.example.com,192.168.1.100 \
lensesio/fast-data-dev
# Access certificates at http://localhost:3030/certs/
# Password for keystores: fastdata
No data generators and Kafka Connect
docker run --rm -p 9092:9092 -p 8081:8081 -p 3030:3030 \
-e SAMPLEDATA=0 \
-e RUNTESTS=0 \
-e CONNECT_PORT=0 \
lensesio/fast-data-dev
Connector Development
# Enable specific connectors only
docker run --rm --net=host \
-e CONNECTORS=file \
-e CONNECT_HEAP=2G \
-v /path/to/custom/connector.jar:/connectors/connector.jar \
lensesio/fast-data-dev
Architecture and Components
Fast Data Dev was first introduced in 2016, back when Kafka was getting traction but was not yet mainstream. We wanted to make it easy for everyone to get streaming within a couple minutes. People starting with Kafka and even experienced engineers hugged our effort and we keep maintaining it since then with upgrades and new features.
We do use vanilla Apache Kafka built from source, and Confluent's Schema Registry also built from source.
Components
The most reliable way to see the current components versions is to check /build.info
:
docker run --rm lensesio/fast-data-dev:latest cat /build.info
- Apache Kafka 3.9.0
- Confluent Schema Registry 7.7.1
- Stream Reactor Connectors 9.0.2
- AWS S3
- Azure Datalake
- Azure DocumentDB
- Azure EventHubs
- Azure ServiceBus
- Cassandra
- ElasticSearch
- FTP
- Google Cloud Platform PubSub
- Google Cloud Platform Cloud Storage
- HTTP
- JMS
- MQTT
- and more
- Debezium Connectors 2.7.4
Services
Enabled by default services:
- Kafka Broker
- Default address:
PLAINTEXT://localhost:9092
- Setup in KRaft mode
- JMX available at
localhost:9581
- Optional TLS listener with
-e ENABLE_TLS=1
- Optional advertised listener with
-e ADV_HOST=[HOST]
. If TLS is enabled,ADV_HOST
will be added to the self-signed certificates. You can also add extra hosts with-e EXTRA_HOSTS=[HOST1],[HOST2],...
- Configuration under
/var/run/broker/server.properties
- Restart with
supervisorctl restart broker
- Default address:
- Schema Registry
- Default address:
http://localhost:8081
- Configuration under
/var/run/schema-registry
- Restart with
supervisorctl restart schema-registry
- Default address:
- Kafka Connect
- Default address:
http://localhost:8083
- JMX available at
localhost:9584
- Get a list of all available connectors:
docker run --rm lensesio/fast-data-dev:latest \ find /opt/lensesio/connectors -name "kafka-connect-*" -type d -exec basename '{}' \;
- Disable connectors to improve RAM usage or avoid conflicts with user-added
plugins with
-e DISABLE=aws-s3,debezium-jdbc
- Explicitly enable connectors with
-e CONNECTORS=aws-s3,file
- Add your own connectors by mounting them under
/connectors
- Disable Connect entirely:
-e CONNECT_PORT=0
- Configuration under
/var/run/connect/connect-avro-distributed.properties
- Restart with
supervisorctl restart connect-distributed
- Default address:
- Caddy Web Server
Serves a web interface where you can see logs, configuration files, and download self-signed keystore and truststore files.- Default address: http://localhost:3030
- Disable entirely with
-e WEB_PORT=0
- Password protect by setting
-e USER=[USER] -e PASSWORD=[PASSWORD]
- Data Generators
Create a few topics with complex data in both AVRO and JSON formats.- By default will run once create and populate the topics, then exit.
- To keep the generators running (loop over the datasets) set
-e RUNNING_SAMPLEDATA=1
- To disable them set
-e SAMPLEDATA=0
- If enabled, a demo file connector will also be created reading the broker's
logs and producing them into a topic. To disable set
-e FORWARDLOGS=0
- Smoke Tests
Basic functionality tests to confirm the services started correctly- To disable set
-e RUNTESTS=0
- To disable set
Disabled by default services:
- Web terminal
- Enable with
-e WEB_TERMINAL_PORT=[PORT]
- Default credentials: admin/admin
- Change credentials with
-e WEB_TERMINAL_CREDS=[USER]:[PASS]
- Enable with
- Supervisord Web UI
- Enable with
-e SUPERVISORWEB=1
- Default address at http://localhost:9001
- If the web server is setup with credentials, they will also apply here
- Enable with
Advanced Configuration
Memory Configuration Options
-e CONNECT_HEAP_OPTS="-Xmx640M -Xms128M"
-e BROKER_HEAP_OPTS="-Xmx320M -Xms320M"
-e CONNECT_HEAP_OPTS="-Xmx256M -Xms128M"
Security and Authentication Options
-e USER=admin # Basic auth username for Web UI (default: kafka)
-e PASSWORD=secret # Basic auth password for Web UI
-e ENABLE_SSL=1 # Enable SSL/TLS for Broker
-e SSL_EXTRA_HOSTS=host1,host2 # Additional SSL certificate hosts
-e WEB_TERMINAL_CREDS=admin:admin
Data and Testing
-e SAMPLEDATA=0 # Disable sample data generation
-e RUNNING_SAMPLEDATA=1 # Enable continuous sample data flow
-e RUNTESTS=0 # Disable integration tests
-e FORWARDLOGS=0 # Disable log forwarding to Kafka topics
Connector Management
-e CONNECTORS=jdbc,elastic,hbase # Enable specific connectors only
-e DISABLE=hbase,mongodb # Disable specific connectors
Debug and Development
-e DEBUG=1 # Enable debug logging
-e RUN_AS_ROOT=1 # Run services as root user
-e BROWSECONFIGS=1 # Expose service configs in web UI
-e SUPERVISORWEB=1 # Enable supervisor web interface (port 9001)
-e WEB_TERMINAL_PORT=9002 # Enable web terminal
Kafka Component Configuration
Configure any Kafka component by converting properties to environment variables:
- Replace dots with underscores
- Convert to uppercase
- Prefix with service name
Examples:
# Broker: log.retention.bytes -> KAFKA_LOG_RETENTION_BYTES
-e KAFKA_LOG_RETENTION_BYTES=1073741824
# Schema Registry: kafkastore.topic -> SCHEMA_REGISTRY_KAFKASTORE_TOPIC
-e SCHEMA_REGISTRY_KAFKASTORE_TOPIC=_schemas
# Connect: plugin.path -> CONNECT_PLUGIN_PATH
-e CONNECT_PLUGIN_PATH=/custom/connectors
Pre/Post Setup Scripts
Execute custom scripts during startup:
# Inline script
-e PRE_SETUP="echo 'Pre-setup script'"
# Script file
-e PRE_SETUP_FILE="/path/to/script.sh"
-v /local/script.sh:/path/to/script.sh
# Remote script
-e PRE_SETUP_URL="https://example.com/setup.sh"
Working with the Environment
Command Line Tools
Access Kafka command line tools:
# Run commands directly
docker run --rm -it --net=host lensesio/fast-data-dev \
kafka-topics --bootstrap-server localhost:9092 --list
# Enter container for interactive use
docker run --rm -it --net=host lensesio/fast-data-dev bash
Troubleshooting
Common Issues
-
Container fails to start with hostname errors
- Solution: Add your hostname to
/etc/hosts
:127.0.0.1 YourHostname localhost
- Solution: Add your hostname to
-
Services not accessible from other machines
- Solution: Use
-e ADV_HOST=<YOUR_IP>
and ensure firewall allows connections
- Solution: Use
-
Out of memory errors
- Solution: Increase Docker memory limit (minimum 4GB recommended)
- Adjust heap sizes:
-e CONNECT_HEAP=2G -e BROKER_HEAP_OPTS="-Xmx1G"
-
SSL connection issues
- Download certificates from http://localhost:3030/certs/
- Use password
fastdata
for all keystores
Memory Requirements
- Minimum: 2GB RAM
- Recommended: 4GB+ RAM
- For heavy connector usage: 6GB+ RAM
Building from Source
Requirements
- Docker with multi-stage build support
- Optional: Docker buildx for multi-architecture builds
Build Commands
# Basic build
docker build -t local/fast-data-dev .
# Multi-architecture build
docker buildx build --platform linux/amd64,linux/arm64 -t local/fast-data-dev .
Top Related Projects
docker-compose.yml files for cp-all-in-one , cp-all-in-one-community, cp-all-in-one-cloud, Apache Kafka Confluent Platform
Open-Source Web UI for Apache Kafka Management
docker compose files to create a fully working kafka stack
Dockerfile for Apache Kafka
Convert
designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual Copilot