CMAK

CMAK is a tool for managing Apache Kafka clusters

11,820

2,504

11,820

521

View on GitHub

Top Related Projects

cruise-control

2,749

Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a Kafka cluster. It provides great value to Kafka users by simplifying the operation of Kafka clusters.

kafka

28,601

Mirror of Apache Kafka

kafka-ui

9,640

Open-Source Web UI for Apache Kafka Management

kafdrop

5,516

Kafka Web UI

Quick Overview

CMAK (Cluster Manager for Apache Kafka), previously known as Kafka Manager, is an open-source tool developed by Yahoo for managing and monitoring Apache Kafka clusters. It provides a web interface for performing cluster operations, monitoring broker health, and managing topics and partitions.

Pros

User-friendly web interface for Kafka cluster management
Supports multiple cluster configurations
Provides detailed metrics and monitoring capabilities
Allows for easy topic and partition management

Cons

Limited documentation and community support compared to other Kafka tools
Some users report occasional stability issues
May require additional setup for security features
Not actively maintained (last commit was over a year ago)

Getting Started

To get started with CMAK, follow these steps:

Ensure you have Java 8 or later installed
Download the latest release from the GitHub repository
Extract the downloaded archive
Configure the application.conf file with your Kafka cluster details
Run the application using the following command:

bin/cmak -Dconfig.file=conf/application.conf -Dhttp.port=8080

Access the web interface at http://localhost:8080

Note: For production use, it's recommended to set up proper authentication and SSL configuration.

Competitor Comparisons

cruise-control

2,749

Pros of Cruise Control

Provides automated load balancing and self-healing capabilities for Kafka clusters
Offers more advanced monitoring and anomaly detection features
Supports multi-tenant Kafka clusters with resource isolation

Cons of Cruise Control

Steeper learning curve and more complex setup compared to CMAK
Less focus on UI-based management and visualization tools
May require additional infrastructure and resources to run effectively

Code Comparison

CMAK (Scala):

def getClusterList: Action[AnyContent] = Action.async { implicit request: RequestHeader =>
  kafkaManager.getClusterList.map { errorOrClusterList =>
    errorOrClusterList.fold(
      error => BadRequest(views.html.common.resultOfCommand(
        views.html.navigation.defaultMenu(),
        models.navigation.Menus.clusterListMenu,
        models.navigation.QuickRoutes.clusters,
        "Error",
        error.msg
      )),
      clusterList => Ok(views.html.cluster.clusterList(clusterList))
    )
  }
}

Cruise Control (Java):

public void addBroker(Collection<Integer> brokerIds, boolean dryRun, boolean throttleAddedBrokers,
                      List<String> goals, boolean allowCapacityEstimation, boolean concurrentLeaderMovements,
                      boolean skipHardGoalCheck, boolean excludeRecentlyDemotedBrokers,
                      boolean excludeRecentlyRemovedBrokers, String uuid,
                      Supplier<String> reasonSupplier) {
  sanityCheckBrokerPresence(brokerIds);
  _asyncKafkaCruiseControl.addBroker(brokerIds, dryRun, throttleAddedBrokers, goals,
                                     allowCapacityEstimation, concurrentLeaderMovements,
                                     skipHardGoalCheck, excludeRecentlyDemotedBrokers,
                                     excludeRecentlyRemovedBrokers, uuid, reasonSupplier);
}

kafka

28,601

Mirror of Apache Kafka

Pros of Kafka

Core distributed streaming platform with high throughput and scalability
Robust ecosystem with wide industry adoption and extensive documentation
Supports complex event processing and stream processing applications

Cons of Kafka

Steeper learning curve and more complex setup compared to CMAK
Requires more resources and infrastructure to run effectively
Less focused on management and monitoring aspects than CMAK

Code Comparison

Kafka (Producer example):

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
Producer<String, String> producer = new KafkaProducer<>(props);

CMAK (Configuration example):

cmak.zkhosts="my.zookeeper.host.com:2181"
cmak.zkhosts=${?ZK_HOSTS}
cmak.zkMaxRetry=100
cmak.pollConsumers=true
cmak.pollConsumersInterval=30

Summary

Kafka is a powerful distributed streaming platform, while CMAK (formerly Kafka Manager) is a tool for managing and monitoring Kafka clusters. Kafka offers more extensive features for building scalable streaming applications, but CMAK provides an easier-to-use interface for cluster management and monitoring tasks.

kafka-ui

9,640

Open-Source Web UI for Apache Kafka Management

Pros of kafka-ui

Modern, lightweight, and user-friendly interface
Built-in Kafka Connect and Schema Registry management
Active development with frequent updates and new features

Cons of kafka-ui

Less mature and battle-tested compared to CMAK
Fewer advanced features for large-scale cluster management
Limited support for older Kafka versions

Code comparison

CMAK (Scala):

def getClusterContext(clusterName: String): KafkaAdminClientContext = {
  kafkaAdminClientContextMap.getOrElseUpdate(clusterName, {
    KafkaAdminClientContext(kafkaManager.getClusterConfig(clusterName))
  })
}

kafka-ui (Java):

@Override
public Mono<ClusterMetrics> getClusterMetrics(String clusterId) {
  return clusterRepository.getClusterById(clusterId)
    .flatMap(cluster -> kafkaService.getClusterMetrics(cluster));
}

Both projects use different programming languages and frameworks, making direct code comparison challenging. CMAK is built with Scala and Play Framework, while kafka-ui uses Java and Spring Boot. The code snippets demonstrate typical operations in each project, such as retrieving cluster information and metrics.

kafdrop

5,516

Kafka Web UI

Pros of Kafdrop

Lightweight and easy to set up, with a single Docker image deployment option
Modern, responsive web interface with a clean and intuitive design
Supports viewing message contents directly in the UI

Cons of Kafdrop

Limited functionality compared to CMAK, focusing primarily on monitoring and basic management
Lacks advanced features like ACL management and multi-cluster support
Less mature project with fewer contributors and a smaller community

Code Comparison

CMAK (Scala):

def getClusterContext(clusterName: String): Future[Option[CMKClusterContext]] = {
  clusterContextMap.get(clusterName) match {
    case Some(cc) => Future.successful(Some(cc))
    case None => Future.successful(None)
  }
}

Kafdrop (Java):

@GetMapping("/topic/{name:.+}")
public String topicDetails(@PathVariable("name") String topicName, Model model) {
    final var topic = kafkaMonitor.getTopic(topicName)
        .orElseThrow(() -> new TopicNotFoundException(topicName));
    model.addAttribute("topic", topic);
    return "topic-detail";
}

Both repositories provide Kafka cluster management tools, but CMAK offers more comprehensive features and scalability for large deployments, while Kafdrop focuses on simplicity and ease of use for smaller setups or quick monitoring needs.

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

CMAK (Cluster Manager for Apache Kafka, previously known as Kafka Manager)

CMAK (previously known as Kafka Manager) is a tool for managing Apache Kafka clusters. See below for details about the name change.

CMAK supports the following:

Manage multiple clusters
Easy inspection of cluster state (topics, consumers, offsets, brokers, replica distribution, partition distribution)
Run preferred replica election
Generate partition assignments with option to select brokers to use
Run reassignment of partition (based on generated assignments)
Create a topic with optional topic configs (0.8.1.1 has different configs than 0.8.2+)
Delete topic (only supported on 0.8.2+ and remember set delete.topic.enable=true in broker config)
Topic list now indicates topics marked for deletion (only supported on 0.8.2+)
Batch generate partition assignments for multiple topics with option to select brokers to use
Batch run reassignment of partition for multiple topics
Add partitions to existing topic
Update config for existing topic
Optionally enable JMX polling for broker level and topic level metrics.
Optionally filter out consumers that do not have ids/ owners/ & offsets/ directories in zookeeper.

Cluster Management

cluster

Topic List

topic

Topic View

topic

Consumer List View

consumer

Consumed Topic View

consumer

Broker List

broker

Broker View

broker

Requirements

Kafka 0.8.. or 0.9.. or 0.10.. or 0.11..
Java 11+

Configuration

The minimum configuration is the zookeeper hosts which are to be used for CMAK (pka kafka manager) state. This can be found in the application.conf file in conf directory. The same file will be packaged in the distribution zip file; you may modify settings after unzipping the file on the desired server.

cmak.zkhosts="my.zookeeper.host.com:2181"

You can specify multiple zookeeper hosts by comma delimiting them, like so:

cmak.zkhosts="my.zookeeper.host.com:2181,other.zookeeper.host.com:2181"

Alternatively, use the environment variable ZK_HOSTS if you don't want to hardcode any values.

ZK_HOSTS="my.zookeeper.host.com:2181"

You can optionally enable/disable the following functionality by modifying the default list in application.conf :

application.features=["KMClusterManagerFeature","KMTopicManagerFeature","KMPreferredReplicaElectionFeature","KMReassignPartitionsFeature"]

KMClusterManagerFeature - allows adding, updating, deleting cluster from CMAK (pka Kafka Manager)
KMTopicManagerFeature - allows adding, updating, deleting topic from a Kafka cluster
KMPreferredReplicaElectionFeature - allows running of preferred replica election for a Kafka cluster
KMReassignPartitionsFeature - allows generating partition assignments and reassigning partitions

Consider setting these parameters for larger clusters with jmx enabled :

cmak.broker-view-thread-pool-size=< 3 * number_of_brokers>
cmak.broker-view-max-queue-size=< 3 * total # of partitions across all topics>
cmak.broker-view-update-seconds=< cmak.broker-view-max-queue-size / (10 * number_of_brokers) >

Here is an example for a kafka cluster with 10 brokers, 100 topics, with each topic having 10 partitions giving 1000 total partitions with JMX enabled :

cmak.broker-view-thread-pool-size=30
cmak.broker-view-max-queue-size=3000
cmak.broker-view-update-seconds=30

The follow control consumer offset cache's thread pool and queue :

cmak.offset-cache-thread-pool-size=< default is # of processors>
cmak.offset-cache-max-queue-size=< default is 1000>
cmak.kafka-admin-client-thread-pool-size=< default is # of processors>
cmak.kafka-admin-client-max-queue-size=< default is 1000>

You should increase the above for large # of consumers with consumer polling enabled. Though it mainly affects ZK based consumer polling.

Kafka managed consumer offset is now consumed by KafkaManagedOffsetCache from the "__consumer_offsets" topic. Note, this has not been tested with large number of offsets being tracked. There is a single thread per cluster consuming this topic so it may not be able to keep up on large # of offsets being pushed to the topic.

Authenticating a User with LDAP

Warning, you need to have SSL configured with CMAK (pka Kafka Manager) to ensure your credentials aren't passed unencrypted. Authenticating a User with LDAP is possible by passing the user credentials with the Authorization header. LDAP authentication is done on first visit, if successful, a cookie is set. On next request, the cookie value is compared with credentials from Authorization header. LDAP support is through the basic authentication filter.

Configure basic authentication

basicAuthentication.enabled=true
basicAuthentication.realm=< basic authentication realm>

Encryption parameters (optional, otherwise randomly generated on startup) :

basicAuthentication.salt="some-hex-string-representing-byte-array"
basicAuthentication.iv="some-hex-string-representing-byte-array"
basicAuthentication.secret="my-secret-string"

Configure LDAP / LDAP + StartTLS / LDAPS authentication

Note: LDAP is unencrypted and insecure. LDAPS is a commonly implemented extension that implements an encryption layer in a manner similar to how HTTPS adds encryption to an HTTP. LDAPS has not been documented, and the specification is not formally defined anywhere. LDAP + StartTLS is the currently recommended way to start an encrypted channel, and it upgrades an existing LDAP connection to achieve this encryption.

basicAuthentication.ldap.enabled=< Boolean flag to enable/disable ldap authentication >
basicAuthentication.ldap.server=< fqdn of LDAP server >
basicAuthentication.ldap.port=< port of LDAP server (typically 389 for LDAP and LDAP + StartTLS and typically 636 for LDAPS) >
basicAuthentication.ldap.username=< LDAP search username >
basicAuthentication.ldap.password=< LDAP search password >
basicAuthentication.ldap.search-base-dn=< LDAP search base >
basicAuthentication.ldap.search-filter=< LDAP search filter >
basicAuthentication.ldap.connection-pool-size=< maximum number of connection to LDAP server >
basicAuthentication.ldap.ssl=< Boolean flag to enable/disable LDAPS (usually incompatible with StartTLS) >
basicAuthentication.ldap.starttls=< Boolean flat to enable StartTLS (usually incompatible with SSL) >

(Optional) Limit access to a specific LDAP Group

basicAuthentication.ldap.group-filter=< LDAP group filter >
basicAuthentication.ldap.ssl-trust-all=< Boolean flag to allow non-expired invalid certificates >

Example (Online LDAP Test Server):

basicAuthentication.ldap.enabled=true
basicAuthentication.ldap.server="ldap.forumsys.com"
basicAuthentication.ldap.port=389
basicAuthentication.ldap.username="cn=read-only-admin,dc=example,dc=com"
basicAuthentication.ldap.password="password"
basicAuthentication.ldap.search-base-dn="dc=example,dc=com"
basicAuthentication.ldap.search-filter="(uid=$capturedLogin$)"
basicAuthentication.ldap.group-filter="cn=allowed-group,ou=groups,dc=example,dc=com"
basicAuthentication.ldap.connection-pool-size=10
basicAuthentication.ldap.ssl=false
basicAuthentication.ldap.ssl-trust-all=false
basicAuthetication.ldap.starttls=false

Deployment

The command below will create a zip file which can be used to deploy the application.

./sbt clean dist

Please refer to play framework documentation on production deployment/configuration.

If java is not in your path, or you need to build against a specific java version, please use the following (the example assumes zulu java11):

$ PATH=/usr/lib/jvm/zulu-11-amd64/bin:$PATH \
  JAVA_HOME=/usr/lib/jvm/zulu-11-amd64 \
  /path/to/sbt -java-home /usr/lib/jvm/zulu-11-amd64 clean dist

This ensures that the 'java' and 'javac' binaries in your path are first looked up in the correct location. Next, for all downstream tools that only listen to JAVA_HOME, it points them to the java11 location. Lastly, it tells sbt to use the java11 location as well.

Starting the service

After extracting the produced zipfile, and changing the working directory to it, you can run the service like this:

$ bin/cmak

By default, it will choose port 9000. This is overridable, as is the location of the configuration file. For example:

$ bin/cmak -Dconfig.file=/path/to/application.conf -Dhttp.port=8080

Again, if java is not in your path, or you need to run against a different version of java, add the -java-home option as follows:

$ bin/cmak -java-home /usr/lib/jvm/zulu-11-amd64

Starting the service with Security

To add JAAS configuration for SASL, add the config file location at start:

$ bin/cmak -Djava.security.auth.login.config=/path/to/my-jaas.conf

NOTE: Make sure the user running CMAK (pka kafka manager) has read permissions on the jaas config file

Packaging

If you'd like to create a Debian or RPM package instead, you can run one of:

sbt debian:packageBin

sbt rpm:packageBin

Credits

Most of the utils code has been adapted to work with Apache Curator from Apache Kafka.

Name and Management

CMAK was renamed from its previous name due to this issue. CMAK is designed to be used with Apache Kafka and is offered to support the needs of the Kafka community. This project is currently managed by employees at Verizon Media and the community who supports this project.

License

Licensed under the terms of the Apache License 2.0. See accompanying LICENSE file for terms.

Consumer/Producer Lag

Producer offset is polled. Consumer offset is read from the offset topic for Kafka based consumers. This means the reported lag may be negative since we are consuming offset from the offset topic faster then polling the producer offset. This is normal and not a problem.

Migration from Kafka Manager to CMAK

Copy config files from old version to new version (application.conf, consumer.properties)
Change start script to use bin/cmak instead of bin/kafka-manager

Top Related Projects

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot