Convert Figma logo to code with AI

taskforcesh logobullmq

BullMQ - Message Queue and Batch processing for NodeJS and Python based on Redis

5,898
381
5,898
333

Top Related Projects

15,398

Premium Queue package for handling distributed jobs and messages in NodeJS.

A simple, fast, robust job/task queue for Node.js, backed by Redis.

1,775

High performance Node.js/PostgreSQL job queue (also suitable for getting jobs generated by PostgreSQL triggers/functions out into a different work queue)

9,360

Lightweight job scheduling for Node.js

Quick Overview

BullMQ is a Node.js job and message queue library built on top of Redis. It provides a robust, scalable, and feature-rich solution for handling distributed job processing in Node.js applications. BullMQ is designed to be high-performance and reliable, making it suitable for both small and large-scale applications.

Pros

  • High performance and scalability due to its Redis-based architecture
  • Rich feature set including job prioritization, retries, and rate limiting
  • Supports both job queues and message queues for different use cases
  • Active development and community support

Cons

  • Requires Redis as a dependency, which may increase infrastructure complexity
  • Learning curve for advanced features and configurations
  • Limited built-in support for non-Node.js consumers (though possible with custom implementations)

Code Examples

  1. Creating a queue and adding a job:
import { Queue } from 'bullmq';

const myQueue = new Queue('myQueue');

await myQueue.add('myJobName', { foo: 'bar' });
  1. Processing jobs with a worker:
import { Worker } from 'bullmq';

const worker = new Worker('myQueue', async job => {
  console.log(job.data);
  // Process the job
});
  1. Using job events:
import { Queue } from 'bullmq';

const myQueue = new Queue('myQueue');

myQueue.on('completed', job => {
  console.log(`Job ${job.id} has completed!`);
});

myQueue.on('failed', (job, err) => {
  console.log(`Job ${job.id} has failed with ${err.message}`);
});

Getting Started

To get started with BullMQ, first install it via npm:

npm install bullmq

Then, create a basic queue and worker:

import { Queue, Worker } from 'bullmq';

const myQueue = new Queue('myQueue');

const worker = new Worker('myQueue', async job => {
  console.log('Processing job:', job.data);
  // Add your job processing logic here
});

// Add a job to the queue
await myQueue.add('myJobName', { foo: 'bar' });

Make sure you have Redis running and accessible. BullMQ will connect to Redis using default settings (localhost:6379) unless specified otherwise.

Competitor Comparisons

15,398

Premium Queue package for handling distributed jobs and messages in NodeJS.

Pros of Bull

  • More mature and established project with a larger community
  • Extensive documentation and examples available
  • Wider range of third-party integrations and plugins

Cons of Bull

  • Less performant for high-throughput scenarios
  • Limited TypeScript support compared to BullMQ
  • Lacks some advanced features like rate limiting and prioritization

Code Comparison

Bull:

const Queue = require('bull');
const myQueue = new Queue('my-queue');

myQueue.add({ data: 'example' });
myQueue.process(async (job) => {
  console.log(job.data);
});

BullMQ:

const { Queue, Worker } = require('bullmq');
const myQueue = new Queue('my-queue');

await myQueue.add('job-name', { data: 'example' });
const worker = new Worker('my-queue', async (job) => {
  console.log(job.data);
});

Both Bull and BullMQ are popular job queue libraries for Node.js, built on top of Redis. While Bull has been around longer and has a larger ecosystem, BullMQ is a more modern rewrite with improved performance and TypeScript support. BullMQ offers more advanced features and is better suited for high-throughput applications, while Bull may be easier to get started with due to its extensive documentation and community resources.

A simple, fast, robust job/task queue for Node.js, backed by Redis.

Pros of bee-queue

  • Simpler API and easier to set up for basic use cases
  • Lightweight with fewer dependencies
  • Better performance for high-throughput scenarios with simple job processing

Cons of bee-queue

  • Limited features compared to BullMQ (e.g., no job prioritization, rate limiting, or repeatable jobs)
  • Less active development and community support
  • Lacks advanced error handling and retry mechanisms

Code Comparison

bee-queue:

const Queue = require('bee-queue');
const queue = new Queue('example');

queue.createJob({x: 2, y: 3}).save();

queue.process(async (job) => {
  return job.data.x + job.data.y;
});

BullMQ:

const { Queue, Worker } = require('bullmq');
const queue = new Queue('example');

await queue.add('myJob', {x: 2, y: 3});

const worker = new Worker('example', async (job) => {
  return job.data.x + job.data.y;
});

Both libraries offer similar basic functionality for creating and processing jobs. However, BullMQ provides a more modular approach with separate Queue and Worker classes, allowing for more complex setups and distributed processing. bee-queue's API is more straightforward for simple use cases but lacks the flexibility and advanced features of BullMQ.

1,775

High performance Node.js/PostgreSQL job queue (also suitable for getting jobs generated by PostgreSQL triggers/functions out into a different work queue)

Pros of Worker

  • Built specifically for PostgreSQL, leveraging its features for efficient job processing
  • Supports advanced job scheduling and dependencies out of the box
  • Integrates seamlessly with GraphQL APIs through Graphile ecosystem

Cons of Worker

  • Limited to PostgreSQL as the backend, less flexible for different database choices
  • Steeper learning curve for developers not familiar with PostgreSQL or Graphile
  • Smaller community and ecosystem compared to BullMQ

Code Comparison

Worker:

await worker.addJob('send-email', {
  to: 'user@example.com',
  subject: 'Welcome',
  body: 'Hello, welcome to our service!'
});

BullMQ:

const queue = new Queue('email');
await queue.add('send', {
  to: 'user@example.com',
  subject: 'Welcome',
  body: 'Hello, welcome to our service!'
});

Both libraries offer similar job creation syntax, but Worker's approach is more tightly integrated with PostgreSQL operations. BullMQ, being Redis-based, provides a more traditional queue-centric API.

Worker excels in PostgreSQL-centric environments and complex job scheduling scenarios, while BullMQ offers broader database compatibility and a larger ecosystem. The choice between them depends on your specific project requirements and existing technology stack.

9,360

Lightweight job scheduling for Node.js

Pros of Agenda

  • Simpler API and easier to set up for basic job scheduling
  • Built-in support for MongoDB, which can be advantageous for projects already using MongoDB
  • Supports repeatable jobs out of the box without additional configuration

Cons of Agenda

  • Limited to MongoDB as the only storage option
  • Less robust handling of high-concurrency scenarios compared to BullMQ
  • Fewer advanced features and less active development

Code Comparison

Agenda:

const Agenda = require('agenda');
const agenda = new Agenda({db: {address: mongoConnectionString}});

agenda.define('send email', async job => {
  await sendEmail(job.attrs.data.to, job.attrs.data.subject);
});

await agenda.start();
await agenda.schedule('in 5 minutes', 'send email', {to: 'user@example.com', subject: 'Hello'});

BullMQ:

const { Queue, Worker } = require('bullmq');
const queue = new Queue('email');

const worker = new Worker('email', async job => {
  await sendEmail(job.data.to, job.data.subject);
});

await queue.add('send email', {to: 'user@example.com', subject: 'Hello'}, {delay: 5 * 60 * 1000});

Both libraries offer job scheduling capabilities, but BullMQ provides more advanced features and better scalability for complex use cases, while Agenda offers simplicity and MongoDB integration.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README




The fastest, most reliable, Redis-based distributed queue for Node.
Carefully written for rock solid stability and atomicity.

Read the documentation

Follow @manast for *important* Bull/BullMQ/BullMQ-Pro news and updates!

🛠 Tutorials

You can find tutorials and news in this blog: https://blog.taskforce.sh/

News 🚀

🌐 Language agnostic BullMQ

Do you need to work with BullMQ on platforms other than Node.js? If so, check out the BullMQ Proxy

🌟 Rediscover Scale Conference 2024

Discover the latest in in-memory and real-time data technologies at Rediscover Scale 2024. Ideal for engineers, architects, and technical leaders looking to push technological boundaries. Connect with experts and advance your skills at The Foundry SF, San Francisco.

Learn more and register here!

Official FrontEnd

Taskforce.sh, Inc

Supercharge your queues with a professional front end:

  • Get a complete overview of all your queues.
  • Inspect jobs, search, retry, or promote delayed jobs.
  • Metrics and statistics.
  • and many more features.

Sign up at Taskforce.sh

🚀 Sponsors 🚀

Dragonfly Dragonfly is a new Redis™ drop-in replacement that is fully compatible with BullMQ and brings some important advantages over Redis™ such as massive better performance by utilizing all CPU cores available and faster and more memory efficient data structures. Read more here on how to use it with BullMQ.

Used by

Some notable organizations using BullMQ:

Microsoft Vendure Datawrapper Nest
Curri Novu NoCodeDB Infisical

The gist

Install:

$ yarn add bullmq

Add jobs to the queue:

import { Queue } from 'bullmq';

const queue = new Queue('Paint');

queue.add('cars', { color: 'blue' });

Process the jobs in your workers:

import { Worker } from 'bullmq';

const worker = new Worker('Paint', async job => {
  if (job.name === 'cars') {
    await paintCar(job.data.color);
  }
});

Listen to jobs for completion:

import { QueueEvents } from 'bullmq';

const queueEvents = new QueueEvents('Paint');

queueEvents.on('completed', ({ jobId }) => {
  console.log('done painting');
});

queueEvents.on(
  'failed',
  ({ jobId, failedReason }: { jobId: string; failedReason: string }) => {
    console.error('error painting', failedReason);
  },
);

This is just scratching the surface, check all the features and more in the official documentation

Feature Comparison

Since there are a few job queue solutions, here is a table comparing them:

FeatureBullMQ-ProBullMQBullKueBeeAgenda
Backendredisredisredisredisredismongo
Observables✓
Group Rate Limit✓
Group Support✓
Batches Support✓
Parent/Child Dependencies✓✓
Debouncing✓✓✓
Priorities✓✓✓✓✓
Concurrency✓✓✓✓✓✓
Delayed jobs✓✓✓✓✓
Global events✓✓✓✓
Rate Limiter✓✓✓
Pause/Resume✓✓✓✓
Sandboxed worker✓✓✓
Repeatable jobs✓✓✓✓
Atomic ops✓✓✓✓
Persistence✓✓✓✓✓✓
UI✓✓✓✓✓
Optimized forJobs / MessagesJobs / MessagesJobs / MessagesJobsMessagesJobs

Contributing

Fork the repo, make some changes, submit a pull-request! Here is the contributing doc that has more details.

Thanks

Thanks for all the contributors that made this library possible, also a special mention to Leon van Kammen that kindly donated his npm bullmq repo.

NPM DownloadsLast 30 Days