kala

Modern Job Scheduler

2,140

188

2,140

View on GitHub

Top Related Projects

celery

25,320

Distributed Task Queue (development branch)

flower

6,518

Real-time monitor and web admin for Celery distributed task queue

Quick Overview

Kala is a modern job scheduler written in Go, designed to be a simpler alternative to cron. It offers a REST API for job management, supports various job types including recurring jobs, and provides features like job dependencies and retry mechanisms.

Pros

Simple and intuitive REST API for job management
Supports various job types, including recurring jobs and one-off jobs
Offers job dependencies and retry mechanisms
Written in Go, providing good performance and easy deployment

Cons

Limited ecosystem compared to more established job schedulers
May lack some advanced features found in enterprise-grade schedulers
Documentation could be more comprehensive
Relatively small community, which may impact support and contributions

Code Examples

Creating a new job:

job := &kala.Job{
    Name:    "My Job",
    Command: "echo 'Hello, World!'",
    Schedule: "0 30 * * * *",
}
client.CreateJob(job)

Retrieving job details:

jobId := "your-job-id"
job, err := client.GetJob(jobId)
if err != nil {
    log.Fatal(err)
}
fmt.Printf("Job Name: %s\n", job.Name)

Deleting a job:

jobId := "your-job-id"
err := client.DeleteJob(jobId)
if err != nil {
    log.Fatal(err)
}
fmt.Println("Job deleted successfully")

Getting Started

Install Kala:
```
go get github.com/ajvb/kala
```
Start the Kala server:
```
kala serve
```

Create a new Go file (e.g., main.go) and add the following code:

package main

import (
    "fmt"
    "github.com/ajvb/kala/client"
)

func main() {
    c := client.New("http://127.0.0.1:8000")
    job := &client.Job{
        Name:    "Hello World",
        Command: "echo 'Hello, World!'",
        Schedule: "0 30 * * * *",
    }
    jobId, err := c.CreateJob(job)
    if err != nil {
        fmt.Println("Error creating job:", err)
        return
    }
    fmt.Println("Job created with ID:", jobId)
}

Run the Go file:
```
go run main.go
```

This will create a new job that runs every hour at 30 minutes past the hour, printing "Hello, World!" to the console.

Competitor Comparisons

schedule

11,889

Python job scheduling for humans.

Pros of Schedule

Simpler and more lightweight, focused solely on job scheduling in Python
Easier to integrate into existing Python projects
More intuitive API for basic scheduling tasks

Cons of Schedule

Less feature-rich compared to Kala's distributed job scheduling capabilities
Limited to running jobs within a single Python process
Lacks built-in persistence and job status tracking

Code Comparison

Schedule:

import schedule
import time

def job():
    print("I'm working...")

schedule.every(10).minutes.do(job)

while True:
    schedule.run_pending()
    time.sleep(1)

Kala:

job := &Job{
    Name:    "test_job",
    Command: "bash -c 'date'",
    Schedule: Schedule{
        RepeatInterval: "P1DT",
    },
}
client.CreateJob(job)

Schedule offers a more straightforward approach for simple scheduling tasks within Python applications. Kala, on the other hand, provides a more robust solution for distributed job scheduling across multiple machines, with features like persistence and job status tracking. The choice between the two depends on the specific requirements of your project and the complexity of your scheduling needs.

celery

25,320

Distributed Task Queue (development branch)

Pros of Celery

More mature and widely adopted project with extensive documentation
Supports a broader range of message brokers and result backends
Offers more advanced features like task routing and periodic tasks

Cons of Celery

Can be complex to set up and configure, especially for simpler use cases
Requires a separate message broker (e.g., RabbitMQ, Redis) to function
Higher resource consumption due to its comprehensive feature set

Code Comparison

Kala task definition:

@kala.task(name="example_task")
def example_task(arg1, arg2):
    return arg1 + arg2

Celery task definition:

@app.task(name="example_task")
def example_task(arg1, arg2):
    return arg1 + arg2

Summary

Celery is a more feature-rich and mature distributed task queue system, suitable for complex applications with diverse requirements. It offers extensive functionality but comes with increased complexity and resource overhead. Kala, on the other hand, is a simpler, standalone job scheduler that may be more appropriate for lightweight use cases or projects that don't require the full feature set of Celery. The code comparison shows that both systems use similar decorator syntax for defining tasks, making it relatively easy to switch between them if needed.

flower

6,518

Real-time monitor and web admin for Celery distributed task queue

Pros of Flower

More extensive documentation and examples
Supports multiple message brokers (Redis, RabbitMQ, etc.)
Larger community and more frequent updates

Cons of Flower

Steeper learning curve due to more complex architecture
Heavier resource usage for smaller projects
Requires additional dependencies for full functionality

Code Comparison

Kala job definition:

{
  "name": "test_job",
  "command": "echo 'Hello World'",
  "schedule": "0 30 * * * *"
}

Flower task definition:

@app.task
def hello_world():
    print('Hello World')

# In celery beat schedule
'hello-world-task': {
    'task': 'tasks.hello_world',
    'schedule': crontab(minute=30, hour='*')
}

Summary

Kala is a lightweight, standalone job scheduler with a simple API, making it ideal for smaller projects or those requiring minimal dependencies. Flower, on the other hand, is a more robust task queue and worker system built on top of Celery, offering greater flexibility and scalability for larger, more complex applications. While Kala provides a straightforward approach to job scheduling, Flower offers more advanced features and integrations at the cost of increased complexity and resource usage.

apscheduler

6,370

Task scheduling library for Python

Pros of APScheduler

More mature and widely used project with extensive documentation
Supports various job stores (e.g., memory, SQLAlchemy, MongoDB) for persistence
Offers more scheduling options, including cron-style scheduling

Cons of APScheduler

Can be more complex to set up and configure for simple use cases
Requires additional dependencies for certain features (e.g., SQLAlchemy for database job stores)

Code Comparison

APScheduler:

from apscheduler.schedulers.background import BackgroundScheduler

scheduler = BackgroundScheduler()
scheduler.add_job(my_job, 'interval', minutes=30)
scheduler.start()

Kala:

job := &Job{
    Name:    "My Job",
    Command: "echo 'Hello, World!'",
    Schedule: Schedule{
        TimesString: "R/PT30M",
    },
}
client.CreateJob(job)

Key Differences

APScheduler is a Python library, while Kala is written in Go
Kala focuses on simplicity and distributed scheduling, while APScheduler offers more advanced features
APScheduler provides in-process scheduling, whereas Kala is designed as a standalone service

Both projects have their strengths, with APScheduler being more suitable for complex Python applications and Kala offering a lightweight, distributed solution for various environments.

rq

9,995

Simple job queues for Python

Pros of RQ

More mature and widely adopted project with a larger community
Supports Redis clusters for better scalability
Offers a web-based dashboard for monitoring and management

Cons of RQ

Limited to Python-only environments
Requires Redis as a dependency, which may add complexity

Code Comparison

RQ job definition:

from rq import Queue
from redis import Redis

q = Queue(connection=Redis())
result = q.enqueue(my_function, args=(1, 2), kwargs={'c': 3})

Kala job definition:

job := &kala.Job{
    Name:    "test_job",
    Command: "echo 'Hello World'",
    Schedule: "R/2015-06-04T19:25:16.828696-07:00/PT10S",
}
client.CreateJob(job)

Key Differences

RQ is Python-specific, while Kala is language-agnostic
Kala uses a built-in scheduler, whereas RQ relies on external scheduling
RQ focuses on queuing and worker management, while Kala emphasizes job scheduling

Use Cases

RQ: Best for Python-centric applications requiring robust queuing and worker management
Kala: Suitable for multi-language environments needing a standalone job scheduler

Both projects offer valuable features for different scenarios, with RQ excelling in Python ecosystems and Kala providing flexibility across languages.

cron

13,186

a cron library for go

Pros of cron

Lightweight and simple to use, with a clean API
Widely adopted and battle-tested in production environments
Supports standard cron syntax, making it familiar to many developers

Cons of cron

Limited built-in features compared to Kala (e.g., no persistence, API, or dashboard)
Lacks advanced scheduling options like timezone support or job dependencies
No built-in distributed execution capabilities

Code Comparison

Kala example:

job := kala.NewJob()
job.Name = "test_job"
job.Command = "echo 'Hello World'"
job.Schedule = "0 30 * * * *"
client.CreateJob(job)

cron example:

c := cron.New()
c.AddFunc("30 * * * *", func() {
    fmt.Println("Hello World")
})
c.Start()

Summary

cron is a lightweight, widely-used Go library for scheduling tasks using cron syntax. It's simple to use and integrate into existing projects. However, it lacks some advanced features that Kala provides, such as persistence, an API, and a dashboard. Kala offers more robust scheduling options and distributed execution capabilities, making it suitable for more complex use cases. The choice between the two depends on the specific requirements of your project and the level of functionality needed.

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Kala

Dashboard

Kala is a simplistic, modern, and performant job scheduler written in Go. Features:

Single binary
No dependencies
JSON over HTTP API
Job Stats
Configurable Retries
Scheduling with ISO 8601 Date and Interval notation
Dependent Jobs
Persistent with several database drivers
Web UI

Note that it is not battle-tested. Use at your own risk.

Kala was inspired by the desire for a simpler Chronos (developed by Airbnb). Kala is Chronos for the rest of us.

If you need fault tolerance, distributed features, massive scale, then I recommend checking out Chronos. This is designed to be the Chronos for start-ups.

Installing Kala

Kala uses Go Modules

Get Kala
```
go get github.com/ajvb/kala
```
Run Kala
```
kala serve
```

Getting Started

Once you have installed Kala onto the machine you would like to use, you can follow the below steps to start using it.

To run Kala as a server:

$ kala serve
INFO[0000] Preparing cache
INFO[0000] Starting server on port :8000

$ kala serve -p 2222
INFO[0000] Preparing cache
INFO[0000] Starting server on port :2222

Kala uses BoltDB by default for the job database by using jobdb and boltpath params:

kala serve --jobdb=boltdb --boltpath=/path/to/dir

use Redis by using the jobdb, jobdb-address and jobdb-password params:

kala serve --jobdb=redis --jobdb-address=127.0.0.1:6379 --jobdb-password=password

use Consul by using the jobdb and jobdb-address params:

kala serve --jobdb=consul --jobdb-address=127.0.0.1:8500

use Mongo by using the jobdb, jobdb-address, jobdb-username, and jobdb-password params:

kala serve --jobdb=mongo --jobdb-address=server1.example.com,server2.example.com --jobdb-username=admin --jobdb-password=password

use Postgres by using the jobdb, jobdb-address params:

kala serve --jobdb=postgres --jobdb-address=server1.example.com/kala --jobdb-username=admin --jobdb-password=password

use MariaDB, MySQL by using the jobdb, jobdb-address, jobdb-tls-capath, jobdb-tls-certpath, jobdb-tls-keypath, jobdb-tls-servername params:

kala serve --jobdb=mariadb --jobdb-address=(server1.example.com)/kala --jobdb-username=admin --jobdb-password=password

kala serve --jobdb=mysql --jobdb-address="tcp(server1.example.com:3306)/kala?tls=custom" --jobdb-username=admin --jobdb-password=password --jobdb-tls-capath=/path/to/server-ca.pem --jobdb-tls-certpath=/path/to/client-cert.pem --jobdb-tls-keypath=/path/to/client-key.pem --jobdb-tls-servername=server1.example.com

Kala runs on 127.0.0.1:8000 by default. You can easily test it out by curling the metrics path.

$ curl http://127.0.0.1:8000/api/v1/stats/
{"Stats":{"ActiveJobs":2,"DisabledJobs":0,"Jobs":2,"ErrorCount":0,"SuccessCount":0,"NextRunAt":"2015-06-04T19:25:16.82873873-07:00","LastAttemptedRun":"0001-01-01T00:00:00Z","CreatedAt":"2015-06-03T19:58:21.433668791-07:00"}}

Once it's up in running, you can utilize curl or the official go client to interact with Kala. Also check out the examples directory.

Examples of Usage

There are more examples in the examples directory within this repo. Currently its pretty messy. Feel free to submit a new example if you have one.

Deployment

Supervisord

After installing supervisord, open its config file (/etc/supervisor/supervisord.conf is the default usually) and add something like:

[program:kala]
command=kala serve
autorestart=true
stdout_logfile=/var/log/kala.stdout.log
stderr_logfile=/var/log/kala.stderr.log

Docker

If you have docker installed, you can build the dockerfile in this directory with docker build -t kala . and run it as a daemon with: docker run -it -d -p 8000:8000 kala

API v1 Docs

All routes have a prefix of /api/v1

Client Libraries

Official:

Go - Docs: http://godoc.org/github.com/ajvb/kala/client
```
go get github.com/ajvb/kala/client
```

Contrib:

Node.js
```
npm install kala-node
```

Python

pip install git+https://github.com/dmajere/kala-python.git

Job Data Struct

Docs can be found here

Things to Note

If schedule is omitted, the job will run immediately.

Job JSON Example

{
        "name":"test_job",
        "id":"93b65499-b211-49ce-57e0-19e735cc5abd",
        "command":"bash /home/ajvb/gocode/src/github.com/ajvb/kala/examples/example-kala-commands/example-command.sh",
        "owner":"",
        "disabled":false,
        "dependent_jobs":null,
        "parent_jobs":null,
        "schedule":"R2/2015-06-04T19:25:16.828696-07:00/PT10S",
        "retries":0,
        "epsilon":"PT5S",
        "success_count":0,
        "last_success":"0001-01-01T00:00:00Z",
        "error_count":0,
        "last_error":"0001-01-01T00:00:00Z",
        "last_attempted_run":"0001-01-01T00:00:00Z",
        "next_run_at":"2015-06-04T19:25:16.828794572-07:00"
}

Breakdown of schedule string. (ISO 8601 Notation)

Example schedule string:

R2/2017-06-04T19:25:16.828696-07:00/PT10S

This string can be split into three parts:

Number of times to repeat/Start Datetime/Interval Between Runs

Number of times to repeat

This is designated with a number, prefixed with an R. Leave out the number if it should repeat forever.

Examples:

R - Will repeat forever
R1 - Will repeat once
R231 - Will repeat 231 times.

Start Datetime

This is the datetime for the first time the job should run.

Kala will return an error if the start datetime has already passed.

Examples:

2017-06-04T19:25:16
2017-06-04T19:25:16.828696
2017-06-04T19:25:16.828696-07:00
2017-06-04T19:25:16-07:00

To Note: It is recommended to include a timezone within your schedule parameter.

Interval Between Runs

This is defined by the ISO8601 Interval Notation.

It starts with a P, then you can specify years, months, or days, then a T, preceded by hours, minutes, and seconds.

Lets break down a long interval: P1Y2M10DT2H30M15S

P - Starts the notation
1Y - One year
2M - Two months
10D - Ten days
T - Starts the time second
2H - Two hours
30M - Thirty minutes
15S - Fifteen seconds

Now, there is one alternative. You can optionally use just weeks. When you use the week operator, you only get that. An example of using the week operator for an interval of every two weeks is P2W.

Examples:

P1DT1M - Interval of one day and one minute
P1W - Interval of one week
PT1H - Interval of one hour.

More Information on ISO8601

Wikipedia's Article

Overview of routes

Task	Method	Route
Creating a Job	POST	/api/v1/job/
Getting a list of all Jobs	GET	/api/v1/job/
Getting a Job	GET	/api/v1/job/{id}/
Deleting a Job	DELETE	/api/v1/job/{id}/
Deleting all Jobs	DELETE	/api/v1/job/all/
Getting metrics about a certain Job	GET	/api/v1/job/stats/{id}/
Starting a Job manually	POST	/api/v1/job/start/{id}/
Disabling a Job	POST	/api/v1/job/disable/{id}/
Enabling a Job	POST	/api/v1/job/enable/{id}/
Getting app-level metrics	GET	/api/v1/stats/

/job

This route accepts both a GET and a POST. Performing a GET request will return a list of all currently running jobs. Performing a POST (with the correct JSON) will create a new Job.

Note: When creating a Job, the only fields that are required are the Name and the Command field. But, if you omit the Schedule field, the job will be ran immediately.

Example:

$ curl http://127.0.0.1:8000/api/v1/job/
{"jobs":{}}
$ curl http://127.0.0.1:8000/api/v1/job/ -d '{"epsilon": "PT5S", "command": "bash /home/ajvb/gocode/src/github.com/ajvb/kala/examples/example-kala-commands/example-command.sh", "name": "test_job", "schedule": "R2/2017-06-04T19:25:16.828696-07:00/PT10S"}'
{"id":"93b65499-b211-49ce-57e0-19e735cc5abd"}
$ curl http://127.0.0.1:8000/api/v1/job/
{
    "jobs":{
        "93b65499-b211-49ce-57e0-19e735cc5abd":{
            "name":"test_job",
            "id":"93b65499-b211-49ce-57e0-19e735cc5abd",
            "command":"bash /home/ajvb/gocode/src/github.com/ajvb/kala/examples/example-kala-commands/example-command.sh",
            "owner":"",
            "disabled":false,
            "dependent_jobs":null,
            "parent_jobs":null,
            "schedule":"R2/2017-06-04T19:25:16.828696-07:00/PT10S",
            "retries":0,
            "epsilon":"PT5S",
            "success_count":0,
            "last_success":"0001-01-01T00:00:00Z",
            "error_count":0,
            "last_error":"0001-01-01T00:00:00Z",
            "last_attempted_run":"0001-01-01T00:00:00Z",
            "next_run_at":"2017-06-04T19:25:16.828794572-07:00"
        }
    }
}

/job/{id}

This route accepts both a GET and a DELETE, and is based off of the id of the Job. Performing a GET request will return a full JSON object describing the Job. Performing a DELETE will delete the Job.

Example:

$ curl http://127.0.0.1:8000/api/v1/job/93b65499-b211-49ce-57e0-19e735cc5abd/
{"job":{"name":"test_job","id":"93b65499-b211-49ce-57e0-19e735cc5abd","command":"bash /home/ajvb/gocode/src/github.com/ajvb/kala/examples/example-kala-commands/example-command.sh","owner":"","disabled":false,"dependent_jobs":null,"parent_jobs":null,"schedule":"R2/2017-06-04T19:25:16.828696-07:00/PT10S","retries":0,"epsilon":"PT5S","success_count":0,"last_success":"0001-01-01T00:00:00Z","error_count":0,"last_error":"0001-01-01T00:00:00Z","last_attempted_run":"0001-01-01T00:00:00Z","next_run_at":"2017-06-04T19:25:16.828737931-07:00"}}
$ curl http://127.0.0.1:8000/api/v1/job/93b65499-b211-49ce-57e0-19e735cc5abd/ -X DELETE
$ curl http://127.0.0.1:8000/api/v1/job/93b65499-b211-49ce-57e0-19e735cc5abd/

/job/stats/{id}

Example:

$ curl http://127.0.0.1:8000/api/v1/job/stats/5d5be920-c716-4c99-60e1-055cad95b40f/
{"job_stats":[{"JobId":"5d5be920-c716-4c99-60e1-055cad95b40f","RanAt":"2017-06-03T20:01:53.232919459-07:00","NumberOfRetries":0,"Success":true,"ExecutionDuration":4529133}]}

/job/start/{id}

Example:

$ curl http://127.0.0.1:8000/api/v1/job/start/5d5be920-c716-4c99-60e1-055cad95b40f/ -X POST

/job/disable/{id}

Example:

$ curl http://127.0.0.1:8000/api/v1/job/disable/5d5be920-c716-4c99-60e1-055cad95b40f/ -X POST

/job/enable/{id}

Example:

$ curl http://127.0.0.1:8000/api/v1/job/enable/5d5be920-c716-4c99-60e1-055cad95b40f/ -X POST

/stats

Example:

$ curl http://127.0.0.1:8000/api/v1/stats/
{"Stats":{"ActiveJobs":2,"DisabledJobs":0,"Jobs":2,"ErrorCount":0,"SuccessCount":0,"NextRunAt":"2017-06-04T19:25:16.82873873-07:00","LastAttemptedRun":"0001-01-01T00:00:00Z","CreatedAt":"2017-06-03T19:58:21.433668791-07:00"}}

Debugging Jobs

There is a command within Kala called run which will immediately run a command as Kala would run it live, and then gives you a response on whether it was successful or not. Allows for easier and quicker debugging of commands.

$ kala run "ruby /home/user/ruby/my_ruby_script.rb"
Command Succeeded!
$ kala run "ruby /home/user/other_dir/broken_script.rb"
FATA[0000] Command Failed with err: exit status 1

Dependent Jobs

How to add a dependent job

Check out this example for how to add dependent jobs within a python script.

Notes on Dependent Jobs

Dependent jobs follow a rule of First In First Out
A child will always have to wait until a parent job finishes before it runs
A child will not run if its parent job does not.
If a child job is disabled, it's parent job will still run, but it will not.
If a child job is deleted, it's parent job will continue to stay around.
If a parent job is deleted, unless its child jobs have another parent, they will be deleted as well.

Original Contributors and Contact

Original Author and Core Maintainer:

AJ Bahnken / @ajvbahnken / aj@ajvb.me

Original Reviewers:

Sam Dolan / @samdolan
Steve Phillips / @elimisteve

Donate

Top Related Projects

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot