bullmq
BullMQ - Message Queue and Batch processing for NodeJS and Python based on Redis
Top Related Projects
Premium Queue package for handling distributed jobs and messages in NodeJS.
A simple, fast, robust job/task queue for Node.js, backed by Redis.
High performance Node.js/PostgreSQL job queue (also suitable for getting jobs generated by PostgreSQL triggers/functions out into a different work queue)
Lightweight job scheduling for Node.js
Quick Overview
BullMQ is a Node.js job and message queue library built on top of Redis. It provides a robust, scalable, and feature-rich solution for handling distributed job processing in Node.js applications. BullMQ is designed to be high-performance and reliable, making it suitable for both small and large-scale applications.
Pros
- High performance and scalability due to its Redis-based architecture
- Rich feature set including job prioritization, retries, and rate limiting
- Supports both job queues and message queues for different use cases
- Active development and community support
Cons
- Requires Redis as a dependency, which may increase infrastructure complexity
- Learning curve for advanced features and configurations
- Limited built-in support for non-Node.js consumers (though possible with custom implementations)
Code Examples
- Creating a queue and adding a job:
import { Queue } from 'bullmq';
const myQueue = new Queue('myQueue');
await myQueue.add('myJobName', { foo: 'bar' });
- Processing jobs with a worker:
import { Worker } from 'bullmq';
const worker = new Worker('myQueue', async job => {
console.log(job.data);
// Process the job
});
- Using job events:
import { Queue } from 'bullmq';
const myQueue = new Queue('myQueue');
myQueue.on('completed', job => {
console.log(`Job ${job.id} has completed!`);
});
myQueue.on('failed', (job, err) => {
console.log(`Job ${job.id} has failed with ${err.message}`);
});
Getting Started
To get started with BullMQ, first install it via npm:
npm install bullmq
Then, create a basic queue and worker:
import { Queue, Worker } from 'bullmq';
const myQueue = new Queue('myQueue');
const worker = new Worker('myQueue', async job => {
console.log('Processing job:', job.data);
// Add your job processing logic here
});
// Add a job to the queue
await myQueue.add('myJobName', { foo: 'bar' });
Make sure you have Redis running and accessible. BullMQ will connect to Redis using default settings (localhost:6379) unless specified otherwise.
Competitor Comparisons
Premium Queue package for handling distributed jobs and messages in NodeJS.
Pros of Bull
- More mature and established project with a larger community
- Extensive documentation and examples available
- Wider range of third-party integrations and plugins
Cons of Bull
- Less performant for high-throughput scenarios
- Limited TypeScript support compared to BullMQ
- Lacks some advanced features like rate limiting and prioritization
Code Comparison
Bull:
const Queue = require('bull');
const myQueue = new Queue('my-queue');
myQueue.add({ data: 'example' });
myQueue.process(async (job) => {
console.log(job.data);
});
BullMQ:
const { Queue, Worker } = require('bullmq');
const myQueue = new Queue('my-queue');
await myQueue.add('job-name', { data: 'example' });
const worker = new Worker('my-queue', async (job) => {
console.log(job.data);
});
Both Bull and BullMQ are popular job queue libraries for Node.js, built on top of Redis. While Bull has been around longer and has a larger ecosystem, BullMQ is a more modern rewrite with improved performance and TypeScript support. BullMQ offers more advanced features and is better suited for high-throughput applications, while Bull may be easier to get started with due to its extensive documentation and community resources.
A simple, fast, robust job/task queue for Node.js, backed by Redis.
Pros of bee-queue
- Simpler API and easier to set up for basic use cases
- Lightweight with fewer dependencies
- Better performance for high-throughput scenarios with simple job processing
Cons of bee-queue
- Limited features compared to BullMQ (e.g., no job prioritization, rate limiting, or repeatable jobs)
- Less active development and community support
- Lacks advanced error handling and retry mechanisms
Code Comparison
bee-queue:
const Queue = require('bee-queue');
const queue = new Queue('example');
queue.createJob({x: 2, y: 3}).save();
queue.process(async (job) => {
return job.data.x + job.data.y;
});
BullMQ:
const { Queue, Worker } = require('bullmq');
const queue = new Queue('example');
await queue.add('myJob', {x: 2, y: 3});
const worker = new Worker('example', async (job) => {
return job.data.x + job.data.y;
});
Both libraries offer similar basic functionality for creating and processing jobs. However, BullMQ provides a more modular approach with separate Queue and Worker classes, allowing for more complex setups and distributed processing. bee-queue's API is more straightforward for simple use cases but lacks the flexibility and advanced features of BullMQ.
High performance Node.js/PostgreSQL job queue (also suitable for getting jobs generated by PostgreSQL triggers/functions out into a different work queue)
Pros of Worker
- Built specifically for PostgreSQL, leveraging its features for efficient job processing
- Supports advanced job scheduling and dependencies out of the box
- Integrates seamlessly with GraphQL APIs through Graphile ecosystem
Cons of Worker
- Limited to PostgreSQL as the backend, less flexible for different database choices
- Steeper learning curve for developers not familiar with PostgreSQL or Graphile
- Smaller community and ecosystem compared to BullMQ
Code Comparison
Worker:
await worker.addJob('send-email', {
to: 'user@example.com',
subject: 'Welcome',
body: 'Hello, welcome to our service!'
});
BullMQ:
const queue = new Queue('email');
await queue.add('send', {
to: 'user@example.com',
subject: 'Welcome',
body: 'Hello, welcome to our service!'
});
Both libraries offer similar job creation syntax, but Worker's approach is more tightly integrated with PostgreSQL operations. BullMQ, being Redis-based, provides a more traditional queue-centric API.
Worker excels in PostgreSQL-centric environments and complex job scheduling scenarios, while BullMQ offers broader database compatibility and a larger ecosystem. The choice between them depends on your specific project requirements and existing technology stack.
Lightweight job scheduling for Node.js
Pros of Agenda
- Simpler API and easier to set up for basic job scheduling
- Built-in support for MongoDB, which can be advantageous for projects already using MongoDB
- Supports repeatable jobs out of the box without additional configuration
Cons of Agenda
- Limited to MongoDB as the only storage option
- Less robust handling of high-concurrency scenarios compared to BullMQ
- Fewer advanced features and less active development
Code Comparison
Agenda:
const Agenda = require('agenda');
const agenda = new Agenda({db: {address: mongoConnectionString}});
agenda.define('send email', async job => {
await sendEmail(job.attrs.data.to, job.attrs.data.subject);
});
await agenda.start();
await agenda.schedule('in 5 minutes', 'send email', {to: 'user@example.com', subject: 'Hello'});
BullMQ:
const { Queue, Worker } = require('bullmq');
const queue = new Queue('email');
const worker = new Worker('email', async job => {
await sendEmail(job.data.to, job.data.subject);
});
await queue.add('send email', {to: 'user@example.com', subject: 'Hello'}, {delay: 5 * 60 * 1000});
Both libraries offer job scheduling capabilities, but BullMQ provides more advanced features and better scalability for complex use cases, while Agenda offers simplicity and MongoDB integration.
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual CopilotREADME
The fastest, most reliable, Redis-based distributed queue for Node.
Carefully written for rock solid stability and atomicity.
Follow @manast for *important* Bull/BullMQ/BullMQ-Pro news and updates!
ð Tutorials
You can find tutorials and news in this blog: https://blog.taskforce.sh/
News ð
ð Language agnostic BullMQ
Do you need to work with BullMQ on platforms other than Node.js? If so, check out the BullMQ Proxy
ð Rediscover Scale Conference 2024
Discover the latest in in-memory and real-time data technologies at Rediscover Scale 2024. Ideal for engineers, architects, and technical leaders looking to push technological boundaries. Connect with experts and advance your skills at The Foundry SF, San Francisco.
Official FrontEnd
Supercharge your queues with a professional front end:
- Get a complete overview of all your queues.
- Inspect jobs, search, retry, or promote delayed jobs.
- Metrics and statistics.
- and many more features.
Sign up at Taskforce.sh
ð Sponsors ð
Dragonfly is a new Redis⢠drop-in replacement that is fully compatible with BullMQ and brings some important advantages over Redis⢠such as massive better performance by utilizing all CPU cores available and faster and more memory efficient data structures. Read more here on how to use it with BullMQ. |
Used by
Some notable organizations using BullMQ:
The gist
Install:
$ yarn add bullmq
Add jobs to the queue:
import { Queue } from 'bullmq';
const queue = new Queue('Paint');
queue.add('cars', { color: 'blue' });
Process the jobs in your workers:
import { Worker } from 'bullmq';
const worker = new Worker('Paint', async job => {
if (job.name === 'cars') {
await paintCar(job.data.color);
}
});
Listen to jobs for completion:
import { QueueEvents } from 'bullmq';
const queueEvents = new QueueEvents('Paint');
queueEvents.on('completed', ({ jobId }) => {
console.log('done painting');
});
queueEvents.on(
'failed',
({ jobId, failedReason }: { jobId: string; failedReason: string }) => {
console.error('error painting', failedReason);
},
);
This is just scratching the surface, check all the features and more in the official documentation
Feature Comparison
Since there are a few job queue solutions, here is a table comparing them:
Feature | BullMQ-Pro | BullMQ | Bull | Kue | Bee | Agenda |
---|---|---|---|---|---|---|
Backend | redis | redis | redis | redis | redis | mongo |
Observables | â | |||||
Group Rate Limit | â | |||||
Group Support | â | |||||
Batches Support | â | |||||
Parent/Child Dependencies | â | â | ||||
Debouncing | â | â | â | |||
Priorities | â | â | â | â | â | |
Concurrency | â | â | â | â | â | â |
Delayed jobs | â | â | â | â | â | |
Global events | â | â | â | â | ||
Rate Limiter | â | â | â | |||
Pause/Resume | â | â | â | â | ||
Sandboxed worker | â | â | â | |||
Repeatable jobs | â | â | â | â | ||
Atomic ops | â | â | â | â | ||
Persistence | â | â | â | â | â | â |
UI | â | â | â | â | â | |
Optimized for | Jobs / Messages | Jobs / Messages | Jobs / Messages | Jobs | Messages | Jobs |
Contributing
Fork the repo, make some changes, submit a pull-request! Here is the contributing doc that has more details.
Thanks
Thanks for all the contributors that made this library possible, also a special mention to Leon van Kammen that kindly donated his npm bullmq repo.
Top Related Projects
Premium Queue package for handling distributed jobs and messages in NodeJS.
A simple, fast, robust job/task queue for Node.js, backed by Redis.
High performance Node.js/PostgreSQL job queue (also suitable for getting jobs generated by PostgreSQL triggers/functions out into a different work queue)
Lightweight job scheduling for Node.js
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual Copilot