Convert Figma logo to code with AI

cadence-workflow logocadence

Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.

8,383
803
8,383
91
View on GitHub

Top Related Projects

cadence-workflow's avatar

cadence

8,375

Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.

temporalio's avatar

temporal

12,316

Temporal service

apache's avatar

airflow

36,684

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

PrefectHQ's avatar

prefect

16,099

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

dagster-io's avatar

dagster

11,381

An orchestration platform for the development, production, and observation of data assets.

apache's avatar

dolphinscheduler

13,140

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

Quick Overview

Cadence is a distributed, scalable, durable, and highly available orchestration engine developed by Uber Technologies. It's designed to execute asynchronous long-running business logic in a scalable and resilient way. Cadence solves the complexities around distributed systems and provides developers with a simple programming model to write fault-tolerant applications.

Pros

  • Provides a simple and intuitive programming model for complex, long-running workflows
  • Offers automatic scaling and fault-tolerance out of the box
  • Supports multiple programming languages including Go, Java, and Python
  • Includes features like versioning, retries, and timeouts for robust application development

Cons

  • Has a steeper learning curve compared to simpler task queue systems
  • Requires additional infrastructure setup and maintenance
  • May be overkill for simple, short-running tasks or small-scale applications
  • Limited community support compared to some other workflow engines

Code Examples

  1. Defining a simple workflow:
func SimpleWorkflow(ctx workflow.Context, name string) (string, error) {
    greeting := "Hello " + name + "!"
    return greeting, nil
}
  1. Implementing an activity:
func GreetingActivity(ctx context.Context, name string) (string, error) {
    return fmt.Sprintf("Hello %s!", name), nil
}
  1. Starting a workflow execution:
workflowOptions := client.StartWorkflowOptions{
    ID:        "greeting_workflow",
    TaskQueue: "greeting-tasks",
}
we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, SimpleWorkflow, "World")
if err != nil {
    log.Fatalln("Unable to execute workflow", err)
}

Getting Started

To get started with Cadence, follow these steps:

  1. Install Cadence server:

    docker run --network host --name cadence ubercadence/server:master-auto-setup

  2. Install Cadence client library:

    go get go.uber.org/cadence

  3. Create a new workflow in your Go project:

    import "go.uber.org/cadence/workflow"

func MyWorkflow(ctx workflow.Context, input string) (string, error) {
    // Your workflow logic here
}

  4. Register and start the worker:

    worker := worker.New(service, "your-task-queue", worker.Options{})
worker.RegisterWorkflow(MyWorkflow)
err := worker.Start()
if err != nil {
    log.Fatalln("Unable to start worker", err)
}

For more detailed instructions and advanced usage, refer to the official Cadence documentation.

Competitor Comparisons

cadence-workflow logo

cadence

8,375

Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.

Pros of Cadence

  • More mature and stable project with a longer development history
  • Larger community and ecosystem support
  • Better documentation and resources for developers

Cons of Cadence

  • Potentially more complex to set up and configure
  • May have a steeper learning curve for newcomers
  • Larger codebase might be harder to navigate for contributors

Code Comparison

Cadence workflow definition:

workflow.Go(ctx, func(ctx workflow.Context) error {
    var result string
    err := workflow.ExecuteActivity(ctx, MyActivity, "input").Get(ctx, &result)
    if err != nil {
        return err
    }
    logger.Info("Activity result: " + result)
    return nil
})

Cadence> workflow definition:

func MyWorkflow(ctx workflow.Context, input string) error {
    ctx = workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
        StartToCloseTimeout: time.Minute,
    })
    var result string
    err := workflow.ExecuteActivity(ctx, MyActivity, input).Get(ctx, &result)
    if err != nil {
        return err
    }
    workflow.GetLogger(ctx).Info("Activity result: " + result)
    return nil
}

Note: The code comparison is based on available information and may not reflect the most recent changes in either project.

temporalio logo

temporal

12,316

Temporal service

Pros of Temporal

  • More active development and larger community support
  • Better documentation and learning resources
  • Enhanced features like native support for multiple languages

Cons of Temporal

  • Potentially more complex setup and configuration
  • Steeper learning curve for newcomers to workflow engines
  • Some compatibility issues when migrating from Cadence

Code Comparison

Cadence workflow definition:

func MyWorkflow(ctx workflow.Context, input string) error {
    activity.ExecuteActivity(ctx, MyActivity, input)
    return nil
}

Temporal workflow definition:

func MyWorkflow(ctx workflow.Context, input string) error {
    ctx = workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
        StartToCloseTimeout: time.Minute,
    })
    return workflow.ExecuteActivity(ctx, MyActivity, input).Get(ctx, nil)
}

Both Cadence and Temporal use similar concepts for defining workflows and activities. However, Temporal provides more granular control over activity execution options and error handling. The Temporal example showcases the use of ActivityOptions and explicit error handling with the Get method.

apache logo

airflow

36,684

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Pros of Airflow

  • More mature and widely adopted in the data engineering community
  • Extensive ecosystem with a large number of pre-built operators and integrations
  • User-friendly web UI for monitoring and managing workflows

Cons of Airflow

  • Can be complex to set up and maintain, especially for large-scale deployments
  • Potential scalability issues with the centralized scheduler architecture
  • Less suitable for event-driven workflows compared to Cadence

Code Comparison

Airflow DAG definition:

from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import datetime

def my_task():
    print("Hello, Airflow!")

dag = DAG('example_dag', start_date=datetime(2023, 1, 1))
task = PythonOperator(task_id='my_task', python_callable=my_task, dag=dag)

Cadence workflow definition:

func MyWorkflow(ctx workflow.Context) error {
    logger := workflow.GetLogger(ctx)
    logger.Info("Hello, Cadence!")
    return nil
}

Both Airflow and Cadence offer powerful workflow orchestration capabilities, but they cater to different use cases. Airflow excels in data pipeline management and scheduled tasks, while Cadence is better suited for long-running, event-driven workflows with strong consistency guarantees.

PrefectHQ logo

prefect

16,099

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

Pros of Prefect

  • More user-friendly with a focus on Python-native workflows
  • Offers both cloud-hosted and self-hosted options
  • Provides a rich UI for monitoring and managing workflows

Cons of Prefect

  • Less mature ecosystem compared to Cadence
  • May have limitations for complex, long-running workflows
  • Potentially higher learning curve for non-Python users

Code Comparison

Prefect workflow example:

from prefect import task, Flow

@task
def say_hello(name):
    print(f"Hello, {name}!")

with Flow("My First Flow") as flow:
    say_hello("World")

Cadence workflow example:

func HelloWorld(ctx workflow.Context, name string) error {
    activity.ExecuteActivity(ctx, SayHello, name)
    return nil
}

func SayHello(name string) (string, error) {
    return fmt.Sprintf("Hello, %s!", name), nil
}

Both Prefect and Cadence offer powerful workflow orchestration capabilities, but they cater to different use cases and programming paradigms. Prefect is more Python-centric and user-friendly, while Cadence provides robust support for long-running, complex workflows across multiple languages.

dagster-io logo

dagster

11,381

An orchestration platform for the development, production, and observation of data assets.

Pros of Dagster

  • More flexible and extensible architecture, allowing for easier integration with various data tools and platforms
  • Better support for data testing and data quality checks within pipelines
  • More comprehensive documentation and tutorials for beginners

Cons of Dagster

  • Steeper learning curve due to its more complex architecture
  • Less mature ecosystem compared to Cadence, with fewer production-ready integrations

Code Comparison

Dagster pipeline definition:

@pipeline
def my_pipeline():
    result = process_data()
    store_result(result)

Cadence workflow definition:

func MyWorkflow(ctx workflow.Context) error {
    result := workflow.ExecuteActivity(ctx, ProcessData)
    return workflow.ExecuteActivity(ctx, StoreResult, result)
}

Both Dagster and Cadence provide ways to define and orchestrate workflows, but Dagster's Python-based approach may be more familiar to data engineers and scientists. Cadence's Go-based workflows offer strong type safety and concurrency support, which can be advantageous for certain use cases.

apache logo

dolphinscheduler

13,140

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

Pros of DolphinScheduler

  • More comprehensive UI for workflow management and monitoring
  • Built-in support for a wider range of data processing tasks and integrations
  • Active development with frequent updates and community contributions

Cons of DolphinScheduler

  • Steeper learning curve due to more complex features and configurations
  • Potentially higher resource requirements for deployment and operation
  • Less focus on long-running workflows compared to Cadence

Code Comparison

DolphinScheduler (Python API example):

from pydolphinscheduler.tasks import Shell
from pydolphinscheduler.core.workflow import Workflow

with Workflow(name="simple_workflow") as workflow:
    Shell(name="shell_task", command="echo hello dolphinscheduler")

workflow.submit()

Cadence (Go SDK example):

func SimpleWorkflow(ctx workflow.Context) error {
    ctx = workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
        ScheduleToStartTimeout: time.Minute,
        StartToCloseTimeout:    time.Minute,
    })
    return workflow.ExecuteActivity(ctx, SimpleActivity).Get(ctx, nil)
}

Both repositories offer workflow orchestration capabilities, but DolphinScheduler provides a more comprehensive solution for data processing and scheduling, while Cadence focuses on distributed system workflows with strong consistency guarantees.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Cadence

Build Status Coverage Slack Status

Github release License

GitHub stars GitHub forks

This repo contains the source code of the Cadence server and other tooling including CLI, schema tools, bench and canary.

You can implement your workflows with one of our client libraries. The Go and Java libraries are officially maintained by the Cadence team, while the Python and Ruby client libraries are developed by the community.

You can also use iWF as a DSL framework on top of Cadence.

See Maxim's talk at Data@Scale Conference for an architectural overview of Cadence.

Visit cadenceworkflow.io to learn more about Cadence. Join us in Cadence Documentation project. Feel free to raise an Issue or Pull Request there.

Community

  • Github Discussion
    • Best for Q&A, support/help, general discusion, and annoucement
  • StackOverflow
    • Best for Q&A and general discusion
  • Github Issues
    • Best for reporting bugs and feature requests
  • Slack
    • Best for contributing/development discussion

Getting Started

Start the cadence-server

To run Cadence services locally, we highly recommend that you use Cadence service docker to run the service. You can also follow the instructions to build and run it.

Please visit our documentation site for production/cluster setup.

Run the Samples

Try out the sample recipes for Go or Java to get started.

Use Cadence CLI

Cadence CLI can be used to operate workflows, tasklist, domain and even the clusters.

You can use the following ways to install Cadence CLI:

  • Use brew to install CLI: brew install cadence-workflow
    • Follow the instructions if you need to install older versions of CLI via homebrew. Usually this is only needed when you are running a server of a too old version.
  • Use docker image for CLI: docker run --rm ubercadence/cli:<releaseVersion> or docker run --rm ubercadence/cli:master . Be sure to update your image when you want to try new features: docker pull ubercadence/cli:master
  • Build the CLI binary yourself, check out the repo and run make cadence to build all tools. See CONTRIBUTING for prerequisite of make command.
  • Build the CLI image yourself, see instructions

Cadence CLI is a powerful tool. The commands are organized by tabs. E.g. workflow->batch->start, or admin->workflow->describe.

Please read the documentation and always try out --help on any tab to learn & explore.

Use Cadence Web

Try out Cadence Web UI to view your workflows on Cadence. (This is already available at localhost:8088 if you run Cadence with docker compose)

Contributing

We'd love your help in making Cadence great. Please review our contribution guide.

If you'd like to propose a new feature, first join the Slack channel to start a discussion and check if there are existing design discussions. Also peruse our design docs in case a feature has been designed but not yet implemented. Once you're sure the proposal is not covered elsewhere, please follow our proposal instructions.

Other binaries in this repo

Bench/stress test workflow tools

See bench documentation.

Periodical feature health check workflow tools(aka Canary)

See canary documentation.

Schema tools for SQL and Cassandra

The tools are for manual setup or upgrading database schema

The easiest way to get the schema tool is via homebrew.

brew install cadence-workflow also includes cadence-sql-tool and cadence-cassandra-tool.

  • The schema files are located at /usr/local/etc/cadence/schema/.
  • To upgrade, make sure you remove the old ElasticSearch schema first: mv /usr/local/etc/cadence/schema/elasticsearch /usr/local/etc/cadence/schema/elasticsearch.old && brew upgrade cadence-workflow. Otherwise ElasticSearch schemas may not be able to get updated.
  • Follow the instructions if you need to install older versions of schema tools via homebrew. However, easier way is to use new versions of schema tools with old versions of schemas. All you need is to check out the older version of schemas from this repo. Run git checkout v0.21.3 to get the v0.21.3 schemas in the schema folder.

Stargazers over time

Stargazers over time

License

MIT License, please see LICENSE for details.