Convert Figma logo to code with AI

hibiken logoasynq

Simple, reliable, and efficient distributed task queue in Go

10,236
735
10,236
213

Top Related Projects

13,221

Simple, efficient background processing for Ruby

9,441

Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later.

Beanstalk is a simple, fast work queue.

Machinery is an asynchronous task queue/job queue based on distributed message passing.

2,443

Process background jobs in Go

1,569

Message queue system written in Go and backed by Redis

Quick Overview

Asynq is a Go library for distributed task queue processing. It provides a simple, reliable, and efficient way to process background tasks asynchronously in Go applications, using Redis as a message broker.

Pros

  • Simple and easy-to-use API for task queue management
  • Built-in support for task scheduling and retries
  • Scalable architecture with support for multiple workers and queues
  • Comprehensive monitoring and observability features

Cons

  • Requires Redis as a dependency
  • Limited to Go programming language
  • May have a learning curve for developers new to task queue systems
  • Lacks some advanced features found in more mature task queue solutions

Code Examples

  1. Defining and enqueueing a task:
import "github.com/hibiken/asynq"

// Define a task
task := asynq.NewTask("email:send", map[string]interface{}{
    "user_id": 123,
    "template_id": "welcome_email",
})

// Enqueue the task
client := asynq.NewClient(asynq.RedisClientOpt{Addr: "localhost:6379"})
info, err := client.Enqueue(task)
if err != nil {
    log.Fatalf("could not enqueue task: %v", err)
}
fmt.Printf("enqueued task: id=%s queue=%s\n", info.ID, info.Queue)
  1. Processing tasks with a worker:
import "github.com/hibiken/asynq"

func main() {
    srv := asynq.NewServer(
        asynq.RedisClientOpt{Addr: "localhost:6379"},
        asynq.Config{Concurrency: 10},
    )

    mux := asynq.NewServeMux()
    mux.HandleFunc("email:send", handleSendEmailTask)

    if err := srv.Run(mux); err != nil {
        log.Fatalf("could not run server: %v", err)
    }
}

func handleSendEmailTask(ctx context.Context, t *asynq.Task) error {
    userID, _ := t.Payload.GetInt("user_id")
    templateID, _ := t.Payload.GetString("template_id")
    // Implement email sending logic here
    return nil
}
  1. Scheduling a periodic task:
import "github.com/hibiken/asynq"

scheduler := asynq.NewScheduler(
    asynq.RedisClientOpt{Addr: "localhost:6379"},
    &asynq.SchedulerOpts{},
)

task := asynq.NewTask("cleanup:expired_data", nil)
entryID, err := scheduler.Register("@daily", task)
if err != nil {
    log.Fatal(err)
}

if err := scheduler.Run(); err != nil {
    log.Fatal(err)
}

Getting Started

To get started with Asynq, follow these steps:

  1. Install Asynq:

    go get -u github.com/hibiken/asynq
    
  2. Set up Redis (ensure it's running on your system)

  3. Create a client to enqueue tasks and a server to process them (see code examples above)

  4. Run your application and start processing tasks

For more detailed information and advanced usage, refer to the official documentation at https://github.com/hibiken/asynq.

Competitor Comparisons

13,221

Simple, efficient background processing for Ruby

Pros of Sidekiq

  • Mature and battle-tested with a large community and ecosystem
  • Rich feature set including web UI, cron jobs, and batches
  • Extensive documentation and support options

Cons of Sidekiq

  • Ruby-specific, limiting language choices
  • Commercial version required for some advanced features
  • Higher resource consumption compared to Asynq

Code Comparison

Sidekiq (Ruby):

class HardWorker
  include Sidekiq::Worker
  def perform(name, count)
    # do something
  end
end

HardWorker.perform_async('bob', 5)

Asynq (Go):

type EmailDeliveryTask struct {
    UserID int
    Email  string
}

asynq.NewTask("email:deliver", map[string]interface{}{
    "user_id": 123,
    "email":   "user@example.com",
})

Both Sidekiq and Asynq provide simple APIs for defining and enqueuing background jobs. Sidekiq uses a class-based approach with Ruby, while Asynq uses structs and interfaces in Go. Sidekiq's syntax is more concise, but Asynq offers type safety and better performance characteristics due to Go's compiled nature.

Asynq is a newer project that aims to bring Sidekiq-like functionality to Go applications. It offers similar core features like Redis-backed queues, retries, and scheduled jobs. Asynq benefits from Go's performance and concurrency model but has a smaller ecosystem compared to Sidekiq's mature Ruby-based offering.

9,441

Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later.

Pros of Resque

  • Mature and battle-tested, with a large community and extensive ecosystem
  • Built-in web interface for monitoring and managing jobs
  • Supports multiple queues and prioritization out of the box

Cons of Resque

  • Ruby-specific, limiting its use in polyglot environments
  • Relies on Redis for job storage, which may not be ideal for all use cases
  • Can be more resource-intensive compared to newer alternatives

Code Comparison

Resque job definition:

class ImageProcessor
  @queue = :image_processing

  def self.perform(image_id)
    image = Image.find(image_id)
    image.process
  end
end

Asynq task definition:

func ProcessImage(ctx context.Context, imageID int64) error {
    image, err := findImage(imageID)
    if err != nil {
        return err
    }
    return image.Process()
}

Asynq offers a more Go-idiomatic approach with context support and error handling, while Resque uses a class-based structure typical of Ruby. Asynq's design allows for better integration with Go's concurrency model and error handling patterns.

Beanstalk is a simple, fast work queue.

Pros of Beanstalkd

  • Language-agnostic: Supports multiple programming languages through various client libraries
  • Simple and lightweight: Easy to set up and use with minimal configuration
  • Proven reliability: Has been in use for many years with a stable codebase

Cons of Beanstalkd

  • Limited features: Lacks advanced functionality like periodic tasks or result storage
  • No built-in monitoring: Requires additional tools for monitoring and management
  • Single-node architecture: Doesn't natively support distributed processing

Code Comparison

Beanstalkd (using Go client):

c, _ := beanstalk.Dial("tcp", "127.0.0.1:11300")
id, _ := c.Put([]byte("hello"), 1, 0, 120*time.Second)

Asynq:

client := asynq.NewClient(asynq.RedisClientOpt{Addr: "localhost:6379"})
task := asynq.NewTask("email:send", []byte("hello"))
_, err := client.Enqueue(task)

Summary

Beanstalkd is a simple, language-agnostic job queue that's easy to set up and use. It's reliable and has been around for a long time. However, it lacks some advanced features and built-in monitoring capabilities. Asynq, on the other hand, is a Go-specific solution that offers more advanced features like periodic tasks and result storage, as well as built-in monitoring and distributed processing support. The choice between the two depends on your specific requirements and the programming languages you're using in your project.

Machinery is an asynchronous task queue/job queue based on distributed message passing.

Pros of Machinery

  • Supports multiple brokers (Redis, AMQP, MongoDB)
  • Offers more advanced features like rate limiting and chaining tasks
  • Has a longer development history and potentially more mature codebase

Cons of Machinery

  • More complex setup and configuration
  • Heavier resource usage due to broader feature set
  • Less focused on performance optimization compared to Asynq

Code Comparison

Machinery task definition:

var AddTask = &tasks.Signature{
    Name: "add",
    Args: []tasks.Arg{
        {Type: "int64", Value: 1},
        {Type: "int64", Value: 1},
    },
}

Asynq task definition:

task := asynq.NewTask("add", map[string]interface{}{
    "x": 1,
    "y": 1,
})

Both libraries offer distributed task processing for Go applications, but they have different focuses. Machinery provides a more feature-rich solution with support for multiple brokers, while Asynq emphasizes simplicity and performance with Redis as its primary backend. Asynq's design is more streamlined, making it easier to set up and use for simpler use cases. However, Machinery's flexibility and advanced features may be preferable for more complex distributed systems or those requiring specific broker integrations.

2,443

Process background jobs in Go

Pros of Work

  • Mature and battle-tested, with a longer history of production use
  • Supports job priorities out of the box
  • Offers more granular control over worker concurrency

Cons of Work

  • Less active development and community support
  • Limited to Redis as the backend storage
  • Lacks some modern features like periodic jobs and middleware support

Code Comparison

Work:

job := work.NewJob()
job.ArgString("name", "John")
job.ArgInt("age", 30)
enqueuer.Enqueue("send_email", job)

Asynq:

task := asynq.NewTask("send_email", map[string]interface{}{
    "name": "John",
    "age":  30,
})
client.Enqueue(task)

Both libraries offer similar functionality for enqueueing jobs, but Asynq uses a more modern and flexible approach with a map for task payload. Work provides a more structured way to add arguments to jobs, which can be beneficial for type safety but may be less flexible for complex data structures.

While Work has been around longer and offers some unique features, Asynq is gaining popularity due to its active development, broader feature set, and more modern API design. The choice between the two depends on specific project requirements and preferences for API style and feature set.

1,569

Message queue system written in Go and backed by Redis

Pros of rmq

  • Lightweight and simple implementation
  • Supports both synchronous and asynchronous processing
  • Easy to integrate with existing Ruby applications

Cons of rmq

  • Less feature-rich compared to asynq
  • Limited documentation and community support
  • Lacks built-in monitoring and management tools

Code Comparison

rmq:

RMQ.configure do |config|
  config.redis = Redis.new
end

RMQ.enqueue("my_queue", "my_job", arg1: "value1", arg2: "value2")

asynq:

client := asynq.NewClient(asynq.RedisClientOpt{Addr: "localhost:6379"})
task := asynq.NewTask("my_job", map[string]interface{}{"arg1": "value1", "arg2": "value2"})
_, err := client.Enqueue(task)

Both libraries provide simple ways to enqueue tasks, but asynq offers more configuration options and type safety due to Go's static typing. rmq's syntax is more concise, reflecting Ruby's simplicity, while asynq's approach is more verbose but provides better control over task creation and execution.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Asynq logo

Simple, reliable & efficient distributed task queue in Go

GoDoc Go Report Card Build Status License: MIT Gitter chat

Asynq is a Go library for queueing tasks and processing them asynchronously with workers. It's backed by Redis and is designed to be scalable yet easy to get started.

Highlevel overview of how Asynq works:

  • Client puts tasks on a queue
  • Server pulls tasks off queues and starts a worker goroutine for each task
  • Tasks are processed concurrently by multiple workers

Task queues are used as a mechanism to distribute work across multiple machines. A system can consist of multiple worker servers and brokers, giving way to high availability and horizontal scaling.

Example use case

Task Queue Diagram

Features

Stability and Compatibility

Status: The library relatively stable and is currently undergoing moderate development with less frequent breaking API changes.

☝️ Important Note: Current major version is zero (v0.x.x) to accommodate rapid development and fast iteration while getting early feedback from users (feedback on APIs are appreciated!). The public API could change without a major version update before v1.0.0 release.

Redis Cluster Compatibility

Some of the lua scripts in this library may not be compatible with Redis Cluster.

Sponsoring

If you are using this package in production, please consider sponsoring the project to show your support!

Quickstart

Make sure you have Go installed (download). The last two Go versions are supported (See https://go.dev/dl).

Initialize your project by creating a folder and then running go mod init github.com/your/repo (learn more) inside the folder. Then install Asynq library with the go get command:

go get -u github.com/hibiken/asynq

Make sure you're running a Redis server locally or from a Docker container. Version 4.0 or higher is required.

Next, write a package that encapsulates task creation and task handling.

package tasks

import (
    "context"
    "encoding/json"
    "fmt"
    "log"
    "time"
    "github.com/hibiken/asynq"
)

// A list of task types.
const (
    TypeEmailDelivery   = "email:deliver"
    TypeImageResize     = "image:resize"
)

type EmailDeliveryPayload struct {
    UserID     int
    TemplateID string
}

type ImageResizePayload struct {
    SourceURL string
}

//----------------------------------------------
// Write a function NewXXXTask to create a task.
// A task consists of a type and a payload.
//----------------------------------------------

func NewEmailDeliveryTask(userID int, tmplID string) (*asynq.Task, error) {
    payload, err := json.Marshal(EmailDeliveryPayload{UserID: userID, TemplateID: tmplID})
    if err != nil {
        return nil, err
    }
    return asynq.NewTask(TypeEmailDelivery, payload), nil
}

func NewImageResizeTask(src string) (*asynq.Task, error) {
    payload, err := json.Marshal(ImageResizePayload{SourceURL: src})
    if err != nil {
        return nil, err
    }
    // task options can be passed to NewTask, which can be overridden at enqueue time.
    return asynq.NewTask(TypeImageResize, payload, asynq.MaxRetry(5), asynq.Timeout(20 * time.Minute)), nil
}

//---------------------------------------------------------------
// Write a function HandleXXXTask to handle the input task.
// Note that it satisfies the asynq.HandlerFunc interface.
//
// Handler doesn't need to be a function. You can define a type
// that satisfies asynq.Handler interface. See examples below.
//---------------------------------------------------------------

func HandleEmailDeliveryTask(ctx context.Context, t *asynq.Task) error {
    var p EmailDeliveryPayload
    if err := json.Unmarshal(t.Payload(), &p); err != nil {
        return fmt.Errorf("json.Unmarshal failed: %v: %w", err, asynq.SkipRetry)
    }
    log.Printf("Sending Email to User: user_id=%d, template_id=%s", p.UserID, p.TemplateID)
    // Email delivery code ...
    return nil
}

// ImageProcessor implements asynq.Handler interface.
type ImageProcessor struct {
    // ... fields for struct
}

func (processor *ImageProcessor) ProcessTask(ctx context.Context, t *asynq.Task) error {
    var p ImageResizePayload
    if err := json.Unmarshal(t.Payload(), &p); err != nil {
        return fmt.Errorf("json.Unmarshal failed: %v: %w", err, asynq.SkipRetry)
    }
    log.Printf("Resizing image: src=%s", p.SourceURL)
    // Image resizing code ...
    return nil
}

func NewImageProcessor() *ImageProcessor {
	return &ImageProcessor{}
}

In your application code, import the above package and use Client to put tasks on queues.

package main

import (
    "log"
    "time"

    "github.com/hibiken/asynq"
    "your/app/package/tasks"
)

const redisAddr = "127.0.0.1:6379"

func main() {
    client := asynq.NewClient(asynq.RedisClientOpt{Addr: redisAddr})
    defer client.Close()

    // ------------------------------------------------------
    // Example 1: Enqueue task to be processed immediately.
    //            Use (*Client).Enqueue method.
    // ------------------------------------------------------

    task, err := tasks.NewEmailDeliveryTask(42, "some:template:id")
    if err != nil {
        log.Fatalf("could not create task: %v", err)
    }
    info, err := client.Enqueue(task)
    if err != nil {
        log.Fatalf("could not enqueue task: %v", err)
    }
    log.Printf("enqueued task: id=%s queue=%s", info.ID, info.Queue)


    // ------------------------------------------------------------
    // Example 2: Schedule task to be processed in the future.
    //            Use ProcessIn or ProcessAt option.
    // ------------------------------------------------------------

    info, err = client.Enqueue(task, asynq.ProcessIn(24*time.Hour))
    if err != nil {
        log.Fatalf("could not schedule task: %v", err)
    }
    log.Printf("enqueued task: id=%s queue=%s", info.ID, info.Queue)


    // ----------------------------------------------------------------------------
    // Example 3: Set other options to tune task processing behavior.
    //            Options include MaxRetry, Queue, Timeout, Deadline, Unique etc.
    // ----------------------------------------------------------------------------

    task, err = tasks.NewImageResizeTask("https://example.com/myassets/image.jpg")
    if err != nil {
        log.Fatalf("could not create task: %v", err)
    }
    info, err = client.Enqueue(task, asynq.MaxRetry(10), asynq.Timeout(3 * time.Minute))
    if err != nil {
        log.Fatalf("could not enqueue task: %v", err)
    }
    log.Printf("enqueued task: id=%s queue=%s", info.ID, info.Queue)
}

Next, start a worker server to process these tasks in the background. To start the background workers, use Server and provide your Handler to process the tasks.

You can optionally use ServeMux to create a handler, just as you would with net/http Handler.

package main

import (
    "log"

    "github.com/hibiken/asynq"
    "your/app/package/tasks"
)

const redisAddr = "127.0.0.1:6379"

func main() {
    srv := asynq.NewServer(
        asynq.RedisClientOpt{Addr: redisAddr},
        asynq.Config{
            // Specify how many concurrent workers to use
            Concurrency: 10,
            // Optionally specify multiple queues with different priority.
            Queues: map[string]int{
                "critical": 6,
                "default":  3,
                "low":      1,
            },
            // See the godoc for other configuration options
        },
    )

    // mux maps a type to a handler
    mux := asynq.NewServeMux()
    mux.HandleFunc(tasks.TypeEmailDelivery, tasks.HandleEmailDeliveryTask)
    mux.Handle(tasks.TypeImageResize, tasks.NewImageProcessor())
    // ...register other handlers...

    if err := srv.Run(mux); err != nil {
        log.Fatalf("could not run server: %v", err)
    }
}

For a more detailed walk-through of the library, see our Getting Started guide.

To learn more about asynq features and APIs, see the package godoc.

Web UI

Asynqmon is a web based tool for monitoring and administrating Asynq queues and tasks.

Here's a few screenshots of the Web UI:

Queues view

Web UI Queues View

Tasks view

Web UI TasksView

Metrics view Screen Shot 2021-12-19 at 4 37 19 PM

Settings and adaptive dark mode

Web UI Settings and adaptive dark mode

For details on how to use the tool, refer to the tool's README.

Command Line Tool

Asynq ships with a command line tool to inspect the state of queues and tasks.

To install the CLI tool, run the following command:

go install github.com/hibiken/asynq/tools/asynq@latest

Here's an example of running the asynq dash command:

Gif

For details on how to use the tool, refer to the tool's README.

Contributing

We are open to, and grateful for, any contributions (GitHub issues/PRs, feedback on Gitter channel, etc) made by the community.

Please see the Contribution Guide before contributing.

License

Copyright (c) 2019-present Ken Hibino and Contributors. Asynq is free and open-source software licensed under the MIT License. Official logo was created by Vic Shóstak and distributed under Creative Commons license (CC0 1.0 Universal).