pulsar

Apache Pulsar - distributed pub-sub messaging system

14,104

3,565

14,104

1,390

View on GitHub

Top Related Projects

rabbitmq-server

12,046

Open source RabbitMQ: core server and tier 1 (built-in) plugins

rocketmq

21,052

Apache RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications.

nats-server

15,495

High-Performance server for NATS.io, the cloud and edge native messaging system.

Quick Overview

Apache Pulsar is a cloud-native, distributed messaging and streaming platform. It provides a unified messaging model and API for both streaming and queuing use cases, with built-in features for geo-replication, multi-tenancy, and durability. Pulsar is designed to handle high-throughput, low-latency messaging at scale.

Pros

Unified messaging model for both streaming and queuing
Built-in geo-replication and multi-tenancy support
Scalable and high-performance architecture
Strong durability guarantees with tiered storage

Cons

Steeper learning curve compared to some simpler messaging systems
Requires more resources to run and manage compared to lightweight alternatives
Less mature ecosystem compared to Apache Kafka
Configuration and setup can be complex for advanced features

Code Examples

Producing messages:

PulsarClient client = PulsarClient.builder()
    .serviceUrl("pulsar://localhost:6650")
    .build();

Producer<String> producer = client.newProducer(Schema.STRING)
    .topic("my-topic")
    .create();

producer.send("Hello, Pulsar!");

Consuming messages:

Consumer<String> consumer = client.newConsumer(Schema.STRING)
    .topic("my-topic")
    .subscriptionName("my-subscription")
    .subscribe();

while (true) {
    Message<String> msg = consumer.receive();
    System.out.println("Received message: " + msg.getValue());
    consumer.acknowledge(msg);
}

Using Pulsar Functions:

public class WordCountFunction implements Function<String, Void> {
    @Override
    public Void process(String input, Context context) throws Exception {
        String[] words = input.split(" ");
        for (String word : words) {
            String counterKey = word.toLowerCase();
            context.incrCounter(counterKey, 1);
        }
        return null;
    }
}

Getting Started

Download and start Pulsar:

wget https://archive.apache.org/dist/pulsar/pulsar-2.8.1/apache-pulsar-2.8.1-bin.tar.gz
tar xvfz apache-pulsar-2.8.1-bin.tar.gz
cd apache-pulsar-2.8.1
bin/pulsar standalone

Add Pulsar client dependency to your project:

<dependency>
    <groupId>org.apache.pulsar</groupId>
    <artifactId>pulsar-client</artifactId>
    <version>2.8.1</version>
</dependency>

Create a client and start producing/consuming messages using the code examples provided above.

Competitor Comparisons

kafka

28,317

Mirror of Apache Kafka

Pros of Kafka

Higher throughput and lower latency for simple use cases
More mature ecosystem with extensive tooling and community support
Simpler architecture, easier to set up and manage for small-scale deployments

Cons of Kafka

Limited multi-tenancy support
Lacks built-in geo-replication and disaster recovery features
Storage and compute are tightly coupled, making scaling less flexible

Code Comparison

Kafka producer example:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
Producer<String, String> producer = new KafkaProducer<>(props);

Pulsar producer example:

PulsarClient client = PulsarClient.builder()
    .serviceUrl("pulsar://localhost:6650")
    .build();
Producer<byte[]> producer = client.newProducer()
    .topic("my-topic")
    .create();

Both Kafka and Pulsar are powerful distributed messaging systems, but they have different strengths. Kafka excels in high-throughput scenarios and has a more established ecosystem, while Pulsar offers better multi-tenancy, geo-replication, and flexible scaling options. The choice between them depends on specific use cases and requirements.

rabbitmq-server

12,046

Open source RabbitMQ: core server and tier 1 (built-in) plugins

Pros of RabbitMQ

Simpler setup and configuration, making it easier for beginners
Mature ecosystem with extensive plugins and integrations
Lower resource consumption, suitable for smaller-scale deployments

Cons of RabbitMQ

Limited scalability compared to Pulsar's multi-layer architecture
Lacks built-in multi-tenancy support, which Pulsar provides out-of-the-box
Less efficient for high-throughput scenarios with large message volumes

Code Comparison

RabbitMQ (Erlang):

basic_publish(Channel, <<"my_exchange">>, <<"routing_key">>, <<"Hello, World!">>),
{#'basic.get_ok'{}, Content} = basic_get(Channel, <<"my_queue">>, no_ack),
io:format("Received: ~p~n", [Content]).

Pulsar (Java):

producer.send("Hello, World!".getBytes());
Message<byte[]> msg = consumer.receive();
System.out.println("Received: " + new String(msg.getData()));
consumer.acknowledge(msg);

Both examples demonstrate basic message publishing and consuming. RabbitMQ uses exchanges and routing keys, while Pulsar uses a simpler topic-based approach. Pulsar requires explicit message acknowledgment, which is optional in RabbitMQ.

rocketmq

21,052

Apache RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications.

Pros of RocketMQ

Simpler architecture and easier to deploy
Lower latency for small messages
Better support for traditional messaging patterns

Cons of RocketMQ

Limited support for streaming use cases
Less scalable for very large deployments
Fewer advanced features like tiered storage

Code Comparison

RocketMQ producer example:

DefaultMQProducer producer = new DefaultMQProducer("ProducerGroupName");
producer.setNamesrvAddr("127.0.0.1:9876");
producer.start();

Message msg = new Message("TopicTest", "TagA", "Hello RocketMQ".getBytes(RemotingHelper.DEFAULT_CHARSET));
SendResult sendResult = producer.send(msg);

Pulsar producer example:

PulsarClient client = PulsarClient.builder()
    .serviceUrl("pulsar://localhost:6650")
    .build();
Producer<byte[]> producer = client.newProducer()
    .topic("my-topic")
    .create();
producer.send("Hello Pulsar".getBytes());

Both RocketMQ and Pulsar are powerful distributed messaging systems, but they have different strengths. RocketMQ is often favored for its simplicity and performance with small messages, while Pulsar offers more advanced features and better scalability for large-scale deployments. The choice between them depends on specific use cases and requirements.

activemq

2,294

Mirror of Apache ActiveMQ

Pros of ActiveMQ

Mature and well-established project with a large user base
Supports multiple protocols (AMQP, MQTT, STOMP) out of the box
Easier to set up and configure for simple use cases

Cons of ActiveMQ

Limited scalability compared to Pulsar's multi-layer architecture
Lacks built-in geo-replication capabilities
Less efficient in handling high-throughput scenarios

Code Comparison

ActiveMQ (Java):

ConnectionFactory factory = new ActiveMQConnectionFactory("tcp://localhost:61616");
Connection connection = factory.createConnection();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Destination destination = session.createQueue("myQueue");
MessageProducer producer = session.createProducer(destination);

Pulsar (Java):

PulsarClient client = PulsarClient.builder()
    .serviceUrl("pulsar://localhost:6650")
    .build();
Producer<byte[]> producer = client.newProducer()
    .topic("my-topic")
    .create();

Both ActiveMQ and Pulsar are robust messaging systems, but they cater to different use cases. ActiveMQ is more suitable for traditional messaging scenarios, while Pulsar excels in high-throughput, scalable environments with features like geo-replication and multi-tenancy. The code comparison shows that Pulsar's API is slightly more concise, reflecting its modern design approach.

nats-server

15,495

High-Performance server for NATS.io, the cloud and edge native messaging system.

Pros of NATS Server

Lightweight and simple architecture, resulting in lower resource usage and easier deployment
Extremely high performance, capable of handling millions of messages per second
Supports multiple messaging patterns (pub/sub, request/reply, queueing) out of the box

Cons of NATS Server

Limited built-in persistence options compared to Pulsar's tiered storage
Less robust multi-tenancy features and resource isolation
Fewer advanced streaming capabilities, such as delayed message delivery or message replay

Code Comparison

NATS Server (Go):

nc, err := nats.Connect(nats.DefaultURL)
if err != nil {
    log.Fatal(err)
}
nc.Publish("foo", []byte("Hello World"))

Pulsar (Java):

PulsarClient client = PulsarClient.builder()
    .serviceUrl("pulsar://localhost:6650")
    .build();
Producer<byte[]> producer = client.newProducer()
    .topic("my-topic")
    .create();
producer.send("Hello World".getBytes());

Both examples demonstrate basic message publishing, but NATS Server's API is more concise and straightforward, while Pulsar's API offers more configuration options and type safety.

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

logo

Pulsar is a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API.

Learn more about Pulsar at https://pulsar.apache.org

Main features

Horizontally scalable (Millions of independent topics and millions of messages published per second)
Strong ordering and consistency guarantees
Low latency durable storage
Topic and queue semantics
Load balancer
Designed for being deployed as a hosted service:
- Multi-tenant
- Authentication
- Authorization
- Quotas
- Support mixing very different workloads
- Optional hardware isolation
Keeps track of consumer cursor position
REST API for provisioning, admin and stats
Geo replication
Transparent handling of partitioned topics
Transparent batching of messages

Repositories

This repository is the main repository of Apache Pulsar. Pulsar PMC also maintains other repositories for components in the Pulsar ecosystem, including connectors, adapters, and other language clients.

Pulsar Core

Helm Chart

Pulsar Helm Chart

Ecosystem

Pulsar Adapters

Clients

Pulsar Runtime Java Version Recommendation

pulsar ver > 2.10 and master branch

Components	Java Version
Broker	17
Functions / IO	17
CLI	17
Java Client	8 or 11 or 17

2.8 <= pulsar ver <= 2.10

Components	Java Version
Broker	11
Functions / IO	11
CLI	8 or 11
Java Client	8 or 11

pulsar ver < 2.8

Components	Java Version
All	8 or 11

Build Pulsar

Requirements

JDK

Pulsar Version JDK Version
master and 2.11 + JDK 17
2.8 / 2.9 / 2.10 JDK 11
2.7 - JDK 8
Maven 3.6.1+
zip

Pulsar Version	JDK Version
master and 2.11 +	JDK 17
2.8 / 2.9 / 2.10	JDK 11
2.7 -	JDK 8

Note:

This project includes a Maven Wrapper that can be used instead of a system-installed Maven. Use it by replacing mvn by ./mvnw on Linux and mvnw.cmd on Windows in the commands below.

Build

Compile and install:

$ mvn install -DskipTests

Compile and install individual module

$ mvn -pl module-name (e.g: pulsar-broker) install -DskipTests

Minimal build (This skips most of external connectors and tiered storage handlers)

mvn install -Pcore-modules,-main -DskipTests

Run Unit Tests:

$ mvn test

Run Individual Unit Test:

$ mvn -pl module-name (e.g: pulsar-client) test -Dtest=unit-test-name (e.g: ConsumerBuilderImplTest)

Run Selected Test packages:

$ mvn test -pl module-name (for example, pulsar-broker) -Dinclude=org/apache/pulsar/**/*.java

Start standalone Pulsar service:

$ bin/pulsar standalone

Check https://pulsar.apache.org for documentation and examples.

Build custom docker images

The commands used in the Apache Pulsar release process can be found in the release process documentation.

Here are some general instructions for building custom docker images:

Docker images must be built with Java 8 for branch-2.7 or previous branches because of ISSUE-8445.
Java 11 is the recommended JDK version in branch-2.8, branch-2.9 and branch-2.10.
Java 17 is the recommended JDK version in master.

The following command builds the docker images apachepulsar/pulsar-all:latest and apachepulsar/pulsar:latest:

mvn clean install -DskipTests
# setting DOCKER_CLI_EXPERIMENTAL=enabled is required in some environments with older docker versions
export DOCKER_CLI_EXPERIMENTAL=enabled
mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests

After the images are built, they can be tagged and pushed to your custom repository. Here's an example of a bash script that tags the docker images with the current version and git revision and pushes them to localhost:32000/apachepulsar.

image_repo_and_project=localhost:32000/apachepulsar
pulsar_version=$(mvn initialize help:evaluate -Dexpression=project.version -pl . -q -DforceStdout)
gitrev=$(git rev-parse HEAD | colrm 10)
tag="${pulsar_version}-${gitrev}"
echo "Using tag $tag"
docker tag apachepulsar/pulsar-all:latest ${image_repo_and_project}/pulsar-all:$tag
docker push ${image_repo_and_project}/pulsar-all:$tag
docker tag apachepulsar/pulsar:latest ${image_repo_and_project}/pulsar:$tag
docker push ${image_repo_and_project}/pulsar:$tag

Setting up your IDE

Read https://pulsar.apache.org/contribute/setup-ide for setting up IntelliJ IDEA or Eclipse for developing Pulsar.

Documentation

Note:

For how to make contributions to Pulsar documentation, see Pulsar Documentation Contribution Guide.

Contact

Mailing lists

Name	Scope	Subscribe	Unsubscribe	Archives
users@pulsar.apache.org	User-related discussions	Subscribe	Unsubscribe	Archives
dev@pulsar.apache.org	Development-related discussions	Subscribe	Unsubscribe	Archives

Slack

Pulsar slack channel at https://apache-pulsar.slack.com/

You can self-register at https://communityinviter.com/apps/apache-pulsar/apache-pulsar

Security Policy

If you find a security issue with Pulsar then please read the security policy. It is critical to avoid public disclosure.

Reporting a security vulnerability

To report a vulnerability for Pulsar, contact the Apache Security Team. When reporting a vulnerability to security@apache.org, you can copy your email to private@pulsar.apache.org to send your report to the Apache Pulsar Project Management Committee. This is a private mailing list.

https://github.com/apache/pulsar/security/policy contains more details.

License

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

Crypto Notice

This distribution includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country's laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See The Wassenaar Arrangement for more information.

The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS), has classified this software as Export Commodity Control Number (ECCN) 5D002.C.1, which includes information security software using or performing cryptographic functions with asymmetric algorithms. The form and manner of this Apache Software Foundation distribution makes it eligible for export under the License Exception ENC Technology Software Unrestricted (TSU) exception (see the BIS Export Administration Regulations, Section 740.13) for both object code and source code.

The following provides more details on the included cryptographic software: Pulsar uses the SSL library from Bouncy Castle written by http://www.bouncycastle.org.

Top Related Projects

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

Top Related Projects

Quick Overview

Pros

Cons

Code Examples

Getting Started

Competitor Comparisons

Pros of Kafka

Cons of Kafka

Code Comparison

Pros of RabbitMQ

Cons of RabbitMQ

Code Comparison

Pros of RocketMQ

Cons of RocketMQ

Code Comparison

Pros of ActiveMQ

Cons of ActiveMQ

Code Comparison

Pros of NATS Server

Cons of NATS Server

Code Comparison

Convert designs to code with AI

README

Main features

Repositories

Helm Chart

Ecosystem

Clients

Dashboard & Management Tools

Website

CI/CD

Archived/Halted

Pulsar Runtime Java Version Recommendation

Build Pulsar

Requirements

Build

Minimal build (This skips most of external connectors and tiered storage handlers)

Build custom docker images

Setting up your IDE

Documentation

Contact

Mailing lists

Slack

Security Policy

Reporting a security vulnerability

License

Crypto Notice

Top Related Projects

Convert designs to code with AI