Convert Figma logo to code with AI

IDSIA logosacred

Sacred is a tool to help you configure, organize, log and reproduce experiments developed at IDSIA.

4,208
380
4,208
102
View on GitHub

Top Related Projects

pymc-devs's avatar

pymc

8,623

Bayesian Modeling and Probabilistic Programming in Python

mlflow's avatar

mlflow

18,287

Open source platform for the machine learning lifecycle

wandb's avatar

wandb

8,869

The AI developer platform. Use Weights & Biases to train and fine-tune models, and manage models from experimentation to production.

optuna's avatar

optuna

10,484

A hyperparameter optimization framework

microsoft's avatar

nni

13,973

An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.

Quick Overview

Sacred is a Python framework designed to help configure, organize, log, and reproduce computational experiments. It's particularly useful for machine learning and scientific computing projects, providing tools to track parameters, random seeds, and results, making experiments more reproducible and easier to manage.

Pros

  • Excellent experiment configuration management with minimal boilerplate
  • Automatic parameter logging and result capturing
  • Supports various observers for storing experiment information (e.g., MongoDB, FileStorage)
  • Integrates well with popular machine learning libraries and frameworks

Cons

  • Learning curve can be steep for beginners
  • Some features may feel overly complex for simple projects
  • Documentation, while comprehensive, can be difficult to navigate at times
  • Limited built-in visualization tools for experiment results

Code Examples

  1. Basic experiment setup:
from sacred import Experiment

ex = Experiment('my_experiment')

@ex.config
def config():
    learning_rate = 0.01
    batch_size = 32

@ex.automain
def run(learning_rate, batch_size):
    print(f"Running experiment with lr={learning_rate} and batch_size={batch_size}")
    # Your experiment code here
  1. Using observers to log results:
from sacred.observers import FileStorageObserver

ex = Experiment('my_experiment')
ex.observers.append(FileStorageObserver('my_runs'))

@ex.automain
def run():
    for i in range(10):
        loss = 1 / (i + 1)
        ex.log_scalar('loss', loss, i)
  1. Capturing results:
@ex.capture
def compute_accuracy(data, model):
    # Compute accuracy
    return accuracy

@ex.main
def run(_run):
    accuracy = compute_accuracy()
    _run.result = accuracy

Getting Started

To get started with Sacred, follow these steps:

  1. Install Sacred:
pip install sacred
  1. Create a new Python file (e.g., experiment.py) and import Sacred:
from sacred import Experiment

ex = Experiment('my_first_experiment')

@ex.config
def config():
    param1 = 10
    param2 = 'default'

@ex.automain
def run(param1, param2):
    print(f"Running with param1={param1} and param2={param2}")
    # Your experiment code here
  1. Run the experiment:
python experiment.py

This will execute the experiment with the default configuration. You can modify parameters from the command line or use configuration files for more complex setups.

Competitor Comparisons

pymc-devs logo

pymc

8,623

Bayesian Modeling and Probabilistic Programming in Python

Pros of PyMC

  • Specialized for probabilistic programming and Bayesian inference
  • Extensive documentation and tutorials for statistical modeling
  • Integration with popular scientific Python libraries (NumPy, SciPy, Pandas)

Cons of PyMC

  • Steeper learning curve for users not familiar with Bayesian statistics
  • Less flexible for general experiment tracking and configuration management
  • More focused on statistical modeling, less suitable for general ML experiments

Code Comparison

PyMC example:

import pymc as pm

with pm.Model() as model:
    mu = pm.Normal('mu', mu=0, sigma=1)
    obs = pm.Normal('obs', mu=mu, sigma=1, observed=data)
    trace = pm.sample(1000)

Sacred example:

from sacred import Experiment

ex = Experiment('my_experiment')

@ex.config
def config():
    param1 = 10
    param2 = 'default'

@ex.main
def run_experiment(param1, param2):
    # Experiment code here
    pass

PyMC is tailored for statistical modeling and Bayesian inference, while Sacred is more general-purpose for experiment tracking and configuration management in machine learning projects.

mlflow logo

mlflow

18,287

Open source platform for the machine learning lifecycle

Pros of MLflow

  • More comprehensive ecosystem with tracking, project packaging, and model management
  • Better integration with popular ML frameworks and cloud platforms
  • Active development and larger community support

Cons of MLflow

  • Steeper learning curve due to more features and complexity
  • Heavier infrastructure requirements for full functionality

Code Comparison

MLflow:

import mlflow

mlflow.start_run()
mlflow.log_param("learning_rate", 0.01)
mlflow.log_metric("accuracy", 0.85)
mlflow.end_run()

Sacred:

from sacred import Experiment

ex = Experiment()

@ex.config
def config():
    learning_rate = 0.01

@ex.main
def run(learning_rate):
    accuracy = 0.85
    return accuracy

ex.run()

Both frameworks offer experiment tracking and configuration management, but MLflow provides a more streamlined API for logging parameters and metrics. Sacred uses decorators and a configuration function, which can be more flexible for complex setups but requires more boilerplate code.

MLflow's approach is generally more intuitive for beginners and integrates well with existing ML workflows, while Sacred offers finer control over experiment configuration and is well-suited for researchers who need extensive customization.

wandb logo

wandb

8,869

The AI developer platform. Use Weights & Biases to train and fine-tune models, and manage models from experimentation to production.

Pros of wandb

  • More comprehensive visualization and experiment tracking features
  • Better support for collaborative work and team projects
  • Seamless integration with popular deep learning frameworks

Cons of wandb

  • Requires internet connection for full functionality
  • Potential privacy concerns with cloud-based storage of experiment data

Code comparison

wandb:

import wandb

wandb.init(project="my-project")
wandb.config.hyperparameters = {
    "learning_rate": 0.01,
    "epochs": 100
}
wandb.log({"accuracy": 0.9, "loss": 0.1})

sacred:

from sacred import Experiment

ex = Experiment("my_experiment")

@ex.config
def config():
    learning_rate = 0.01
    epochs = 100

@ex.automain
def run(_run):
    _run.log_scalar("accuracy", 0.9)
    _run.log_scalar("loss", 0.1)

Both wandb and sacred provide experiment tracking and logging capabilities. wandb offers a more user-friendly interface and extensive visualization tools, while sacred focuses on reproducibility and local experiment management. wandb is better suited for collaborative projects and cloud-based workflows, whereas sacred is ideal for researchers who prefer complete control over their experiment data and storage.

optuna logo

optuna

10,484

A hyperparameter optimization framework

Pros of Optuna

  • More focused on hyperparameter optimization with advanced algorithms like Bayesian optimization
  • Provides visualization tools for analyzing optimization results
  • Integrates well with popular machine learning frameworks like PyTorch and TensorFlow

Cons of Optuna

  • Less comprehensive experiment management features compared to Sacred
  • Primarily designed for hyperparameter tuning, while Sacred offers broader experiment tracking capabilities
  • May require additional tools for full experiment reproducibility

Code Comparison

Sacred:

from sacred import Experiment

ex = Experiment()

@ex.config
def config():
    learning_rate = 0.01
    batch_size = 32

@ex.automain
def train(_run, learning_rate, batch_size):
    # Training code here

Optuna:

import optuna

def objective(trial):
    learning_rate = trial.suggest_loguniform('learning_rate', 1e-5, 1e-1)
    batch_size = trial.suggest_int('batch_size', 16, 128)
    # Training code here

study = optuna.create_study()
study.optimize(objective, n_trials=100)
microsoft logo

nni

13,973

An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.

Pros of NNI

  • Offers a wider range of features for hyperparameter tuning and neural architecture search
  • Provides a user-friendly web UI for experiment management and visualization
  • Supports distributed training and multiple frameworks (TensorFlow, PyTorch, etc.)

Cons of NNI

  • Steeper learning curve due to its extensive feature set
  • May be overkill for simpler experiments or smaller projects
  • Requires more setup and configuration compared to Sacred

Code Comparison

Sacred:

from sacred import Experiment

ex = Experiment()

@ex.config
def config():
    learning_rate = 0.001
    batch_size = 32

@ex.automain
def train(_run, learning_rate, batch_size):
    # Training code here

NNI:

import nni

@nni.get_next_parameter()
def get_params():
    return {
        'learning_rate': 0.001,
        'batch_size': 32
    }

def train(params):
    # Training code here

if __name__ == '__main__':
    params = get_params()
    train(params)

Both frameworks allow for easy configuration and experiment tracking, but NNI offers more advanced features for hyperparameter tuning and neural architecture search. Sacred provides a simpler setup for basic experiment management, while NNI requires more configuration but offers greater flexibility for complex machine learning projects.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Sacred

| *Every experiment is sacred*
| *Every experiment is great*
| *If an experiment is wasted*
| *God gets quite irate*

|pypi| |py_versions| |license| |rtfd| |doi|

|build| |coverage| |code_quality| |black|

Sacred is a tool to help you configure, organize, log and reproduce experiments. It is designed to do all the tedious overhead work that you need to do around your actual experiment in order to:

  • keep track of all the parameters of your experiment
  • easily run your experiment for different settings
  • save configurations for individual runs in a database
  • reproduce your results

Sacred achieves this through the following main mechanisms:

  • Config Scopes A very convenient way of the local variables in a function to define the parameters your experiment uses.
  • Config Injection: You can access all parameters of your configuration from every function. They are automatically injected by name.
  • Command-line interface: You get a powerful command-line interface for each experiment that you can use to change parameters and run different variants.
  • Observers: Sacred provides Observers that log all kinds of information about your experiment, its dependencies, the configuration you used, the machine it is run on, and of course the result. These can be saved to a MongoDB, for easy access later.
  • Automatic seeding helps controlling the randomness in your experiments, such that the results remain reproducible.

Example

+------------------------------------------------+--------------------------------------------+ | Script to train an SVM on the iris dataset | The same script as a Sacred experiment | +------------------------------------------------+--------------------------------------------+ | .. code:: python | .. code:: python | | | | | from numpy.random import permutation | from numpy.random import permutation | | from sklearn import svm, datasets | from sklearn import svm, datasets | | | from sacred import Experiment | | | ex = Experiment('iris_rbf_svm') | | | | | | @ex.config | | | def cfg(): | | C = 1.0 | C = 1.0 | | gamma = 0.7 | gamma = 0.7 | | | | | | @ex.automain | | | def run(C, gamma): | | iris = datasets.load_iris() | iris = datasets.load_iris() | | perm = permutation(iris.target.size) | per = permutation(iris.target.size) | | iris.data = iris.data[perm] | iris.data = iris.data[per] | | iris.target = iris.target[perm] | iris.target = iris.target[per] | | clf = svm.SVC(C=C, kernel='rbf', | clf = svm.SVC(C=C, kernel='rbf', | | gamma=gamma) | gamma=gamma) | | clf.fit(iris.data[:90], | clf.fit(iris.data[:90], | | iris.target[:90]) | iris.target[:90]) | | print(clf.score(iris.data[90:], | return clf.score(iris.data[90:], | | iris.target[90:])) | iris.target[90:]) | +------------------------------------------------+--------------------------------------------+

Documentation

The documentation is hosted at ReadTheDocs <http://sacred.readthedocs.org/>_.

Installing

You can directly install it from the Python Package Index with pip:

pip install sacred

Or if you want to do it manually you can checkout the current version from git and install it yourself:

| git clone https://github.com/IDSIA/sacred.git | cd sacred | python setup.py install

You might want to also install the numpy and the pymongo packages. They are optional dependencies but they offer some cool features:

pip install numpy pymongo

Tests

The tests for sacred use the pytest <http://pytest.org/latest/>_ package. You can execute them by running pytest in the sacred directory like this:

pytest

There is also a config file for tox <https://tox.readthedocs.io/en/latest/>_ so you can automatically run the tests for various python versions like this:

tox

Update pytest version +++++++++++++++++++++

If you update or change the pytest version, the following files need to be changed:

  • dev-requirements.txt
  • tox.ini
  • test/test_utils.py
  • setup.py

Contributing

If you find a bug, have a feature request or want to discuss something general you are welcome to open an issue <https://github.com/IDSIA/sacred/issues>. If you have a specific question related to the usage of sacred, please ask a question on StackOverflow under the python-sacred tag <https://stackoverflow.com/questions/tagged/python-sacred>. We value documentation a lot. If you find something that should be included in the documentation please document it or let us know whats missing. If you are using Sacred in one of your projects and want to share your code with others, put your repo in the Projects using Sacred <docs/projects_using_sacred.rst>_ list. Pull requests are highly welcome!

Frontends

At this point there are three frontends to the database entries created by sacred (that I'm aware of). They are developed externally as separate projects.

Omniboard <https://github.com/vivekratnavel/omniboard>_ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ .. image:: docs/images/omniboard-table.png .. image:: docs/images/omniboard-metric-graphs.png

Omniboard is a web dashboard that helps in visualizing the experiments and metrics / logs collected by sacred. Omniboard is written with React, Node.js, Express and Bootstrap.

Incense <https://github.com/JarnoRFB/incense>_ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ .. image:: docs/images/incense-artifact.png .. image:: docs/images/incense-metric.png

Incense is a Python library to retrieve runs stored in a MongoDB and interactively display metrics and artifacts in Jupyter notebooks.

Sacredboard <https://github.com/chovanecm/sacredboard>_ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ .. image:: docs/images/sacredboard.png

Sacredboard is a web-based dashboard interface to the sacred runs stored in a MongoDB.

Neptune <https://neptune.ai/>_ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ .. image:: docs/images/neptune-compare.png .. image:: docs/images/neptune-collaboration.png

Neptune is a metadata store for MLOps, built for teams that run a lot of experiments. It gives you a single place to log, store, display, organize, compare, and query all your model-building metadata via API available for both Python and R programming languages:

.. image:: docs/images/neptune-query-api.png

In order to log your sacred experiments to Neptune, all you need to do is add an observer:

.. code-block:: python

from neptune.new.integrations.sacred import NeptuneObserver
ex.observers.append(NeptuneObserver(api_token='<YOUR_API_TOKEN>',
                                    project='<YOUR_WORKSPACE/YOUR_PROJECT>'))

For more info, check the Neptune + Sacred integration guide <https://docs.neptune.ai/integrations-and-supported-tools/experiment-tracking/sacred>_.

SacredBrowser <https://github.com/michaelwand/SacredBrowser>_ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ .. image:: docs/images/sacred_browser.png

SacredBrowser is a PyQt4 application to browse the MongoDB entries created by sacred experiments. Features include custom queries, sorting of the results, access to the stored source-code, and many more. No installation is required and it can connect to a local database or over the network.

Prophet <https://github.com/Qwlouse/prophet>_ +++++++++++++++++++++++++++++++++++++++++++++++ Prophet is an early prototype of a webinterface to the MongoDB entries created by sacred experiments, that is discontinued. It requires you to run RestHeart <http://restheart.org>_ to access the database.

Related Projects

Sumatra <https://pythonhosted.org/Sumatra/>_ ++++++++++++++++++++++++++++++++++++++++++++++ | Sumatra is a tool for managing and tracking projects based on numerical | simulation and/or analysis, with the aim of supporting reproducible research. | It can be thought of as an automated electronic lab notebook for | computational projects.

Sumatra takes a different approach by providing commandline tools to initialize a project and then run arbitrary code (not just python). It tracks information about all runs in a SQL database and even provides a nice browser tool. It integrates less tightly with the code to be run, which makes it easily applicable to non-python experiments. But that also means it requires more setup for each experiment and configuration needs to be done using files. Use this project if you need to run non-python experiments, or are ok with the additional setup/configuration overhead.

Future Gadget Laboratory <https://github.com/Kaixhin/FGLab>_ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ | FGLab is a machine learning dashboard, designed to make prototyping | experiments easier. Experiment details and results are sent to a database, | which allows analytics to be performed after their completion. The server | is FGLab, and the clients are FGMachines.

Similar to Sumatra, FGLab is an external tool that can keep track of runs from any program. Projects are configured via a JSON schema and the program needs to accept these configurations via command-line options. FGLab also takes the role of a basic scheduler by distributing runs over several machines.

License

This project is released under the terms of the MIT license <http://opensource.org/licenses/MIT>_.

Citing Sacred

K. Greff, A. Klein, M. Chovanec, F. Hutter, and J. Schmidhuber, ‘The Sacred Infrastructure for Computational Research’, in Proceedings of the 15th Python in Science Conference (SciPy 2017), Austin, Texas, 2017, pp. 49–56 <http://conference.scipy.org/proceedings/scipy2017/klaus_greff.html>_.

.. |pypi| image:: https://img.shields.io/pypi/v/sacred.svg :target: https://pypi.python.org/pypi/sacred :alt: Current PyPi Version

.. |py_versions| image:: https://img.shields.io/pypi/pyversions/sacred.svg :target: https://pypi.python.org/pypi/sacred :alt: Supported Python Versions

.. |license| image:: https://img.shields.io/badge/license-MIT-blue.png :target: http://choosealicense.com/licenses/mit/ :alt: MIT licensed

.. |rtfd| image:: https://readthedocs.org/projects/sacred/badge/?version=latest&style=flat :target: https://sacred.readthedocs.io/en/stable/ :alt: ReadTheDocs

.. |doi| image:: https://zenodo.org/badge/doi/10.5281/zenodo.16386.svg :target: http://dx.doi.org/10.5281/zenodo.16386 :alt: DOI for this release

.. |build| image:: https://github.com/IDSIA/sacred/actions/workflows/test.yml/badge.svg :target: https://github.com/IDSIA/sacred/actions/workflows/test.yml/badge.svg :alt: Github Actions PyTest

.. |coverage| image:: https://coveralls.io/repos/IDSIA/sacred/badge.svg :target: https://coveralls.io/r/IDSIA/sacred :alt: Coverage Report

.. |code_quality| image:: https://scrutinizer-ci.com/g/IDSIA/sacred/badges/quality-score.png?b=master :target: https://scrutinizer-ci.com/g/IDSIA/sacred/ :alt: Code Scrutinizer Quality

.. |black| image:: https://img.shields.io/badge/code%20style-black-000000.svg :target: https://github.com/ambv/black :alt: Code style: black