Convert Figma logo to code with AI

pytorch logobotorch

Bayesian optimization in PyTorch

3,055
390
3,055
72
View on GitHub

Top Related Projects

cornellius-gp's avatar

gpytorch

3,519

A highly efficient implementation of Gaussian Processes in PyTorch

huggingface's avatar

optimum

2,447

🚀 Accelerate training and inference of 🤗 Transformers and 🤗 Diffusers with easy to use hardware optimization tools

optuna's avatar

optuna

10,708

A hyperparameter optimization framework

microsoft's avatar

FLAML

3,881

A fast library for AutoML and tuning. Join our Discord: https://discord.gg/Cppx2vSPVP.

Quick Overview

BoTorch is an open-source library for Bayesian Optimization built on PyTorch. It provides a modular and extensible framework for developing and deploying state-of-the-art Bayesian optimization algorithms, with a focus on supporting composite and multi-objective optimization problems.

Pros

  • Built on PyTorch, allowing for seamless integration with deep learning workflows and GPU acceleration
  • Highly modular and extensible architecture, making it easy to implement custom acquisition functions and models
  • Supports advanced features like multi-objective optimization and constrained optimization
  • Well-documented with extensive tutorials and examples

Cons

  • Steeper learning curve compared to simpler Bayesian optimization libraries
  • Requires familiarity with PyTorch for advanced usage
  • May be overkill for simple optimization problems
  • Limited support for certain specialized optimization scenarios

Code Examples

  1. Basic single-objective optimization:
import torch
from botorch.models import SingleTaskGP
from botorch.fit import fit_gpytorch_model
from botorch.acquisition import UpperConfidenceBound
from botorch.optim import optimize_acqf

# Define objective function
def objective(x):
    return -(x ** 2).sum(dim=-1)

# Generate initial data
train_X = torch.rand(10, 2)
train_Y = objective(train_X).unsqueeze(-1)

# Define and fit model
model = SingleTaskGP(train_X, train_Y)
fit_gpytorch_model(model)

# Optimize acquisition function
UCB = UpperConfidenceBound(model, beta=0.1)
bounds = torch.stack([torch.zeros(2), torch.ones(2)])
candidate, _ = optimize_acqf(UCB, bounds=bounds, q=1, num_restarts=5, raw_samples=20)
  1. Multi-objective optimization:
from botorch.models import ModelListGP
from botorch.acquisition import qExpectedHypervolumeImprovement
from botorch.utils.multi_objective import box_decomposition, get_chebyshev_scalarization

# Define multi-objective function
def multi_objective(X):
    return torch.stack([-X.pow(2).sum(dim=-1), -(X - 1.5).pow(2).sum(dim=-1)], dim=-1)

# Generate initial data
train_X = torch.rand(10, 2)
train_Y = multi_objective(train_X)

# Define and fit model
model = ModelListGP(train_X, train_Y)
fit_gpytorch_model(model)

# Define acquisition function
ref_point = torch.tensor([-2.0, -2.0])
qEHVI = qExpectedHypervolumeImprovement(model, ref_point, partitioning=box_decomposition)

# Optimize acquisition function
bounds = torch.stack([torch.zeros(2), torch.ones(2)])
candidate, _ = optimize_acqf(qEHVI, bounds=bounds, q=1, num_restarts=5, raw_samples=20)
  1. Constrained optimization:
from botorch.acquisition import ConstrainedExpectedImprovement

# Define objective and constraint functions
def objective(X):
    return -(X ** 2).sum(dim=-1)

def constraint(X):
    return X.sum(dim=-1) - 1

# Generate initial data
train_X = torch.rand(10, 2)
train_Y = objective(train_X).unsqueeze(-1)
train_C = constraint(train_X).unsqueeze(-1)

# Define and fit models
obj_model = SingleTaskGP(train_X, train_Y)
con_model = SingleTaskGP(train_X, train_C)
fit_gpytorch_model(obj_model)
fit_gpytorch_model(con_model)

# Define acquisition function
CEI = ConstrainedExpectedImprovement(obj_model, con_model, best_f=train_Y.max())

# Optimize acquisition function
bounds = torch.stack([torch.zeros

Competitor Comparisons

cornellius-gp logo

gpytorch

3,519

A highly efficient implementation of Gaussian Processes in PyTorch

Pros of GPyTorch

  • More flexible and customizable for general Gaussian process models
  • Better suited for advanced GP techniques and research applications
  • Offers a wider range of kernels and likelihood functions

Cons of GPyTorch

  • Steeper learning curve for beginners in Bayesian optimization
  • Less focused on Bayesian optimization-specific features
  • Requires more manual implementation for optimization tasks

Code Comparison

GPyTorch:

class ExactGPModel(gpytorch.models.ExactGP):
    def __init__(self, train_x, train_y, likelihood):
        super().__init__(train_x, train_y, likelihood)
        self.mean_module = gpytorch.means.ConstantMean()
        self.covar_module = gpytorch.kernels.ScaleKernel(gpytorch.kernels.RBFKernel())

BoTorch:

class SingleTaskGP(gpytorch.models.ExactGP):
    def __init__(self, train_X, train_Y):
        likelihood = gpytorch.likelihoods.GaussianLikelihood()
        super().__init__(train_X, train_Y, likelihood)
        self.mean_module = gpytorch.means.ConstantMean()
        self.covar_module = gpytorch.kernels.ScaleKernel(gpytorch.kernels.MaternKernel())

GPyTorch is more general-purpose and flexible for GP modeling, while BoTorch is specifically designed for Bayesian optimization tasks, offering a more streamlined experience for that use case.

huggingface logo

optimum

2,447

🚀 Accelerate training and inference of 🤗 Transformers and 🤗 Diffusers with easy to use hardware optimization tools

Pros of Optimum

  • Broader focus on optimizing and accelerating various ML models, not just Bayesian optimization
  • Integrates well with popular Hugging Face libraries and ecosystem
  • Supports a wider range of hardware accelerators and deployment targets

Cons of Optimum

  • Less specialized for Bayesian optimization tasks compared to BoTorch
  • May have a steeper learning curve for users not familiar with the Hugging Face ecosystem
  • Potentially less performant for specific Bayesian optimization use cases

Code Comparison

BoTorch example:

from botorch.models import SingleTaskGP
from botorch.fit import fit_gpytorch_model
from botorch.acquisition import ExpectedImprovement

model = SingleTaskGP(train_X, train_Y)
mll = ExactMarginalLogLikelihood(model.likelihood, model)
fit_gpytorch_model(mll)
EI = ExpectedImprovement(model, best_f=train_Y.max())

Optimum example:

from optimum.intel import OVModelForSequenceClassification
from transformers import AutoTokenizer

model = OVModelForSequenceClassification.from_pretrained("distilbert-base-uncased")
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
optimized_model = model.optimize(optimization_level="O1")
optuna logo

optuna

10,708

A hyperparameter optimization framework

Pros of Optuna

  • More user-friendly and easier to get started with for beginners
  • Supports a wider range of optimization algorithms beyond Bayesian optimization
  • Integrates well with popular machine learning frameworks like TensorFlow and scikit-learn

Cons of Optuna

  • Less specialized for Bayesian optimization compared to BoTorch
  • May not be as performant for complex, high-dimensional optimization problems
  • Lacks some advanced features available in BoTorch, such as multi-objective optimization

Code Comparison

Optuna example:

import optuna

def objective(trial):
    x = trial.suggest_float('x', -10, 10)
    return (x - 2) ** 2

study = optuna.create_study()
study.optimize(objective, n_trials=100)

BoTorch example:

import torch
from botorch.models import SingleTaskGP
from botorch.fit import fit_gpytorch_model
from botorch.acquisition import ExpectedImprovement
from botorch.optim import optimize_acqf

train_X = torch.rand(10, 1)
train_Y = (train_X - 2).pow(2)
model = SingleTaskGP(train_X, train_Y)
fit_gpytorch_model(model)
microsoft logo

FLAML

3,881

A fast library for AutoML and tuning. Join our Discord: https://discord.gg/Cppx2vSPVP.

Pros of FLAML

  • Designed for automated machine learning (AutoML) with a focus on efficiency and ease of use
  • Supports a wider range of ML tasks, including classification, regression, and time series forecasting
  • Includes built-in hyperparameter tuning and model selection capabilities

Cons of FLAML

  • Less specialized for Bayesian optimization compared to BoTorch
  • May not offer as advanced features for complex optimization problems
  • Smaller community and ecosystem compared to BoTorch's PyTorch backing

Code Comparison

FLAML example:

from flaml import AutoML
automl = AutoML()
automl.fit(X_train, y_train, task="classification")
predictions = automl.predict(X_test)

BoTorch example:

from botorch.models import SingleTaskGP
from botorch.fit import fit_gpytorch_model
model = SingleTaskGP(train_X, train_Y)
fit_gpytorch_model(model)

Both libraries offer concise APIs, but FLAML's AutoML approach provides a higher-level abstraction for general machine learning tasks, while BoTorch focuses on more specialized Bayesian optimization workflows.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

BoTorch Logo

Support Ukraine Lint Test Docs Nightly Codecov

Conda PyPI License

BoTorch is a library for Bayesian Optimization built on PyTorch.

BoTorch is currently in beta and under active development!

Why BoTorch ?

BoTorch

  • Provides a modular and easily extensible interface for composing Bayesian optimization primitives, including probabilistic models, acquisition functions, and optimizers.
  • Harnesses the power of PyTorch, including auto-differentiation, native support for highly parallelized modern hardware (e.g. GPUs) using device-agnostic code, and a dynamic computation graph.
  • Supports Monte Carlo-based acquisition functions via the reparameterization trick, which makes it straightforward to implement new ideas without having to impose restrictive assumptions about the underlying model.
  • Enables seamless integration with deep and/or convolutional architectures in PyTorch.
  • Has first-class support for state-of-the art probabilistic models in GPyTorch, including support for multi-task Gaussian Processes (GPs) deep kernel learning, deep GPs, and approximate inference.

Target Audience

The primary audience for hands-on use of BoTorch are researchers and sophisticated practitioners in Bayesian Optimization and AI. We recommend using BoTorch as a low-level API for implementing new algorithms for Ax. Ax has been designed to be an easy-to-use platform for end-users, which at the same time is flexible enough for Bayesian Optimization researchers to plug into for handling of feature transformations, (meta-)data management, storage, etc. We recommend that end-users who are not actively doing research on Bayesian Optimization simply use Ax.

Installation

Installation Requirements

  • Python >= 3.10
  • PyTorch >= 2.0.1
  • gpytorch == 1.13
  • linear_operator == 0.5.3
  • pyro-ppl >= 1.8.4
  • scipy
  • multiple-dispatch

Prerequisite only for MacOS users with Intel processors:

Before installing BoTorch, we recommend first manually installing PyTorch, a required dependency of BoTorch. Installing it according to the PyTorch installation instructions ensures that it is properly linked against MKL, a library that optimizes mathematical computation for Intel processors. This will result in up to an order-of-magnitude speed-up for Bayesian optimization, as at the moment, installing PyTorch from pip does not link against MKL.

The PyTorch installation instructions currently recommend:

  1. Install Anaconda. Note that there are different installers for Intel and M1 Macs.
  2. Install PyTorch following the PyTorch installation instructions. Currently, this suggests running conda install pytorch torchvision -c pytorch.

If you want to customize your installation, please follow the PyTorch installation instructions to build from source.

Option 1: Installing the latest release

The latest release of BoTorch is easily installed either via Anaconda (recommended) or pip.

To install BoTorch from Anaconda, run

conda install botorch -c pytorch -c gpytorch -c conda-forge

The above command installs BoTorch and any needed dependencies. -c pytorch -c gpytorch -c conda-forge means that the most preferred source to install from is the PyTorch channel, the next most preferred is the GPyTorch channel, and the least preferred is conda-forge.

Alternatively, to install with pip, do

pip install botorch

Note: Make sure the pip being used is actually the one from the newly created Conda environment. If you're using a Unix-based OS, you can use which pip to check.

Option 2: Installing from latest main branch

If you would like to try our bleeding edge features (and don't mind potentially running into the occasional bug here or there), you can install the latest development version directly from GitHub. If you want to also install the current gpytorch and linear_operator development versions, you will need to ensure that the ALLOW_LATEST_GPYTORCH_LINOP environment variable is set:

pip install --upgrade git+https://github.com/cornellius-gp/linear_operator.git
pip install --upgrade git+https://github.com/cornellius-gp/gpytorch.git
export ALLOW_LATEST_GPYTORCH_LINOP=true
pip install --upgrade git+https://github.com/pytorch/botorch.git

Option 3: Editable/dev install

If you want to contribute to BoTorch, you will want to install editably so that you can change files and have the changes reflected in your local install.

If you want to install the current gpytorch and linear_operator development versions, as in Option 2, do that before proceeding.

Option 3a: Bare-bones editable install

git clone https://github.com/pytorch/botorch.git
cd botorch
pip install -e .

Option 3b: Editable install with development and tutorials dependencies

git clone https://github.com/pytorch/botorch.git
cd botorch
export ALLOW_BOTORCH_LATEST=true
pip install -e ".[dev, tutorials]"
  • dev: Specifies tools necessary for development (testing, linting, docs building; see Contributing below).
  • tutorials: Also installs all packages necessary for running the tutorial notebooks.
  • You can also install either the dev or tutorials dependencies without installing both, e.g. by changing the last command to pip install -e ".[dev]".

Getting Started

Here's a quick run down of the main components of a Bayesian optimization loop. For more details see our Documentation and the Tutorials.

  1. Fit a Gaussian Process model to data
import torch
from botorch.models import SingleTaskGP
from botorch.models.transforms import Normalize, Standardize
from botorch.fit import fit_gpytorch_mll
from gpytorch.mlls import ExactMarginalLogLikelihood

# Double precision is highly recommended for GPs.
# See https://github.com/pytorch/botorch/discussions/1444
train_X = torch.rand(10, 2, dtype=torch.double) * 2
Y = 1 - (train_X - 0.5).norm(dim=-1, keepdim=True)  # explicit output dimension
Y += 0.1 * torch.rand_like(Y)

gp = SingleTaskGP(
    train_X=train_X,
    train_Y=Y,
    input_transform=Normalize(d=2),
    outcome_transform=Standardize(m=1),
)
mll = ExactMarginalLogLikelihood(gp.likelihood, gp)
fit_gpytorch_mll(mll)
  1. Construct an acquisition function
from botorch.acquisition import LogExpectedImprovement

logEI = LogExpectedImprovement(model=gp, best_f=Y.max())
  1. Optimize the acquisition function
from botorch.optim import optimize_acqf

bounds = torch.stack([torch.zeros(2), torch.ones(2)]).to(torch.double)
candidate, acq_value = optimize_acqf(
    logEI, bounds=bounds, q=1, num_restarts=5, raw_samples=20,
)

Citing BoTorch

If you use BoTorch, please cite the following paper:

M. Balandat, B. Karrer, D. R. Jiang, S. Daulton, B. Letham, A. G. Wilson, and E. Bakshy. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. Advances in Neural Information Processing Systems 33, 2020.

@inproceedings{balandat2020botorch,
  title={{BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization}},
  author={Balandat, Maximilian and Karrer, Brian and Jiang, Daniel R. and Daulton, Samuel and Letham, Benjamin and Wilson, Andrew Gordon and Bakshy, Eytan},
  booktitle = {Advances in Neural Information Processing Systems 33},
  year={2020},
  url = {http://arxiv.org/abs/1910.06403}
}

See here for an incomplete selection of peer-reviewed papers that build off of BoTorch.

Contributing

See the CONTRIBUTING file for how to help out.

License

BoTorch is MIT licensed, as found in the LICENSE file.