Convert Figma logo to code with AI

meta-pytorch logobotorch

Bayesian optimization in PyTorch

3,363
443
3,363
89
View on GitHub

Top Related Projects

pytorch's avatar

pytorch

93,668

Tensors and Dynamic neural networks in Python with strong GPU acceleration

pytorch's avatar

ignite

4,682

High-level library to help with training and evaluating neural networks in PyTorch flexibly and transparently.

huggingface's avatar

transformers

150,567

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

scikit-optimize's avatar

scikit-optimize

2,794

Sequential model-based optimization with a `scipy.optimize` interface

optuna's avatar

optuna

12,406

A hyperparameter optimization framework

Quick Overview

BoTorch is an open-source library for Bayesian Optimization built on PyTorch. It provides a modular and extensible interface for creating and optimizing probabilistic models, with a focus on supporting advanced models and acquisition functions for Bayesian optimization.

Pros

  • Built on PyTorch, allowing for GPU acceleration and automatic differentiation
  • Highly modular and extensible architecture
  • Supports advanced models and acquisition functions
  • Integrates well with other PyTorch-based libraries

Cons

  • Steeper learning curve compared to some simpler Bayesian optimization libraries
  • Requires familiarity with PyTorch
  • Documentation can be complex for beginners
  • May be overkill for simple optimization tasks

Code Examples

  1. Creating a simple Bayesian optimization loop:
import torch
from botorch.models import SingleTaskGP
from botorch.fit import fit_gpytorch_model
from botorch.acquisition import ExpectedImprovement
from botorch.optim import optimize_acqf

# Define the objective function
def objective(x):
    return -(x ** 2).sum()

# Initialize data
train_X = torch.rand(10, 2)
train_Y = objective(train_X).unsqueeze(-1)

# Create and fit the model
model = SingleTaskGP(train_X, train_Y)
fit_gpytorch_model(model)

# Define the acquisition function
EI = ExpectedImprovement(model, best_f=train_Y.max().item())

# Optimize the acquisition function
new_x, _ = optimize_acqf(
    EI, bounds=torch.stack([torch.zeros(2), torch.ones(2)]),
    q=1, num_restarts=5, raw_samples=20,
)
  1. Using a custom acquisition function:
from botorch.acquisition import AcquisitionFunction

class CustomAcquisition(AcquisitionFunction):
    def forward(self, X):
        mean, _ = self.model.posterior(X).mvn.mean_var()
        return mean.squeeze(-1)

custom_acq = CustomAcquisition(model)
  1. Multi-objective optimization:
from botorch.models import MultiTaskGP
from botorch.acquisition import qExpectedHypervolumeImprovement

# Assume train_X and train_Y are multi-objective data
model = MultiTaskGP(train_X, train_Y)
fit_gpytorch_model(model)

ref_point = torch.zeros(2)
EHVI = qExpectedHypervolumeImprovement(model, ref_point, maximize=True)

Getting Started

To get started with BoTorch, follow these steps:

  1. Install BoTorch:
pip install botorch
  1. Import necessary modules:
import torch
from botorch.models import SingleTaskGP
from botorch.fit import fit_gpytorch_model
from botorch.acquisition import ExpectedImprovement
from botorch.optim import optimize_acqf
  1. Define your objective function and create initial data.
  2. Create and fit a model, define an acquisition function, and optimize it.
  3. Refer to the BoTorch documentation for more advanced usage and examples.

Competitor Comparisons

pytorch logo

pytorch

93,668

Tensors and Dynamic neural networks in Python with strong GPU acceleration

Pros of PyTorch

  • Broader scope and functionality for general deep learning tasks
  • Larger community and ecosystem with more resources and third-party libraries
  • More frequent updates and contributions from a diverse set of developers

Cons of PyTorch

  • Steeper learning curve for beginners due to its extensive features
  • Larger codebase and installation size, which may be overkill for specialized tasks
  • Less focused on Bayesian optimization and related techniques

Code Comparison

PyTorch (general tensor operations):

import torch

x = torch.tensor([1, 2, 3])
y = torch.tensor([4, 5, 6])
z = torch.matmul(x, y)

BoTorch (Bayesian optimization):

from botorch.models import SingleTaskGP
from botorch.fit import fit_gpytorch_model

train_X = torch.rand(10, 2)
train_Y = torch.sin(train_X).sum(dim=1, keepdim=True)
model = SingleTaskGP(train_X, train_Y)
fit_gpytorch_model(model)

BoTorch is built on top of PyTorch and focuses specifically on Bayesian optimization and related techniques, while PyTorch is a more general-purpose deep learning framework. BoTorch provides specialized tools for tasks like Gaussian process modeling and acquisition functions, making it more suitable for certain optimization problems. PyTorch, on the other hand, offers a wider range of capabilities for various machine learning tasks beyond Bayesian optimization.

pytorch logo

ignite

4,682

High-level library to help with training and evaluating neural networks in PyTorch flexibly and transparently.

Pros of Ignite

  • Simpler and more lightweight framework for general PyTorch training loops
  • Easier to get started with for beginners in deep learning
  • Provides built-in metrics and logging functionality

Cons of Ignite

  • Less specialized for Bayesian optimization tasks
  • Fewer advanced features for hyperparameter tuning and experimentation
  • May require more custom code for complex optimization scenarios

Code Comparison

BoTorch example:

qEI = qExpectedImprovement(model, best_f=best_f)
candidate, acq_value = optimize_acqf(
    acq_function=qEI,
    bounds=bounds,
    q=q,
    num_restarts=num_restarts,
    raw_samples=raw_samples,
)

Ignite example:

trainer = Engine(update_model)
RunningAverage(output_transform=lambda x: x).attach(trainer, "loss")
ProgressBar().attach(trainer)
trainer.run(train_loader, max_epochs=10)

BoTorch is specifically designed for Bayesian optimization and provides more specialized functions for acquisition and optimization. Ignite offers a more general-purpose training loop with built-in utilities for common deep learning tasks. BoTorch is better suited for advanced optimization scenarios, while Ignite is more accessible for standard neural network training.

huggingface logo

transformers

150,567

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

Pros of Transformers

  • Broader scope, covering a wide range of NLP tasks and models
  • Larger community and more frequent updates
  • Extensive documentation and examples for various use cases

Cons of Transformers

  • Can be overwhelming for users focused on specific tasks
  • Potentially higher resource requirements due to its comprehensive nature
  • May have a steeper learning curve for beginners

Code Comparison

Transformers:

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")
inputs = tokenizer("Hello world!", return_tensors="pt")
outputs = model(**inputs)

BoTorch:

import torch
from botorch.models import SingleTaskGP
from botorch.fit import fit_gpytorch_model

train_X = torch.rand(10, 2)
train_Y = torch.sin(train_X).sum(dim=1, keepdim=True)
model = SingleTaskGP(train_X, train_Y)
fit_gpytorch_model(model)

Both repositories are powerful tools in their respective domains. Transformers excels in NLP tasks with a wide range of pre-trained models, while BoTorch focuses on Bayesian optimization and provides a more specialized set of tools for this purpose. The choice between them depends on the specific requirements of your project.

scikit-optimize logo

scikit-optimize

2,794

Sequential model-based optimization with a `scipy.optimize` interface

Pros of scikit-optimize

  • Simpler API and easier to use for beginners
  • Broader range of optimization algorithms, including non-Bayesian methods
  • Better integration with scikit-learn ecosystem

Cons of scikit-optimize

  • Less focus on Bayesian optimization compared to BoTorch
  • Fewer advanced features for multi-objective and constrained optimization
  • Limited support for GPU acceleration

Code Comparison

scikit-optimize:

from skopt import gp_minimize

def objective(x):
    return (x[0] - 3)**2 + (x[1] + 2)**2

res = gp_minimize(objective, [(-5.0, 5.0), (-5.0, 5.0)], n_calls=50)

BoTorch:

import torch
from botorch.models import SingleTaskGP
from botorch.fit import fit_gpytorch_model
from botorch.acquisition import ExpectedImprovement
from botorch.optim import optimize_acqf

def objective(x):
    return -(x[0] - 3)**2 - (x[1] + 2)**2

train_X = torch.rand(10, 2) * 10 - 5
train_Y = objective(train_X).unsqueeze(-1)

gp = SingleTaskGP(train_X, train_Y)
fit_gpytorch_model(gp)

Both libraries offer powerful optimization capabilities, but BoTorch is more focused on Bayesian optimization and provides more advanced features for complex scenarios, while scikit-optimize offers a broader range of algorithms and easier integration with the scikit-learn ecosystem.

optuna logo

optuna

12,406

A hyperparameter optimization framework

Pros of Optuna

  • More versatile, supporting various ML frameworks beyond PyTorch
  • Easier to use with a simpler API, especially for beginners
  • Extensive documentation and examples for various use cases

Cons of Optuna

  • Less specialized for Bayesian optimization compared to BoTorch
  • May be less efficient for complex, high-dimensional optimization problems
  • Fewer advanced features for expert users in Bayesian optimization

Code Comparison

Optuna example:

import optuna

def objective(trial):
    x = trial.suggest_float('x', -10, 10)
    return (x - 2) ** 2

study = optuna.create_study()
study.optimize(objective, n_trials=100)

BoTorch example:

import torch
from botorch.models import SingleTaskGP
from botorch.fit import fit_gpytorch_model
from botorch.acquisition import ExpectedImprovement
from botorch.optim import optimize_acqf

train_X = torch.rand(10, 1)
train_Y = (train_X - 2).pow(2)
model = SingleTaskGP(train_X, train_Y)
fit_gpytorch_model(model)

Both libraries offer powerful optimization capabilities, but Optuna is more user-friendly and versatile, while BoTorch provides more advanced features for Bayesian optimization specialists.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

BoTorch Logo

Support Ukraine Lint Test Docs Nightly Codecov

PyPI License

BoTorch is a library for Bayesian Optimization built on PyTorch.

BoTorch is currently in beta and under active development!

Why BoTorch ?

BoTorch

  • Provides a modular and easily extensible interface for composing Bayesian optimization primitives, including probabilistic models, acquisition functions, and optimizers.
  • Harnesses the power of PyTorch, including auto-differentiation, native support for highly parallelized modern hardware (e.g. GPUs) using device-agnostic code, and a dynamic computation graph.
  • Supports Monte Carlo-based acquisition functions via the reparameterization trick, which makes it straightforward to implement new ideas without having to impose restrictive assumptions about the underlying model.
  • Enables seamless integration with deep and/or convolutional architectures in PyTorch.
  • Has first-class support for state-of-the art probabilistic models in GPyTorch, including support for multi-task Gaussian Processes (GPs) deep kernel learning, deep GPs, and approximate inference.

Target Audience

The primary audience for hands-on use of BoTorch are researchers and sophisticated practitioners in Bayesian Optimization and AI. We recommend using BoTorch as a low-level API for implementing new algorithms for Ax. Ax has been designed to be an easy-to-use platform for end-users, which at the same time is flexible enough for Bayesian Optimization researchers to plug into for handling of feature transformations, (meta-)data management, storage, etc. We recommend that end-users who are not actively doing research on Bayesian Optimization simply use Ax.

Installation

Installation Requirements

  • Python >= 3.10
  • PyTorch >= 2.0.1
  • gpytorch >= 1.14
  • linear_operator >= 0.6
  • pyro-ppl >= 1.8.4
  • scipy
  • multiple-dispatch

Option 1: Installing the latest release

The latest release of BoTorch is easily installed via pip:

pip install botorch

Note: Make sure the pip being used is actually the one from the newly created Conda environment. If you're using a Unix-based OS, you can use which pip to check.

BoTorch stopped publishing an official Anaconda package to the pytorch channel after the 0.12 release. However, users can still use the package published to the conda-forge channel and install botorch via

conda install botorch -c gpytorch -c conda-forge

Option 2: Installing from latest main branch

If you would like to try our bleeding edge features (and don't mind potentially running into the occasional bug here or there), you can install the latest development version directly from GitHub. You may also want to install the current gpytorch and linear_operator development versions:

pip install --upgrade git+https://github.com/cornellius-gp/linear_operator.git
pip install --upgrade git+https://github.com/cornellius-gp/gpytorch.git
pip install --upgrade git+https://github.com/meta-pytorch/botorch.git

Option 3: Editable/dev install

If you want to contribute to BoTorch, you will want to install editably so that you can change files and have the changes reflected in your local install.

If you want to install the current gpytorch and linear_operator development versions, as in Option 2, do that before proceeding.

Option 3a: Bare-bones editable install

git clone https://github.com/meta-pytorch/botorch.git
cd botorch
pip install -e .

Option 3b: Editable install with development and tutorials dependencies

git clone https://github.com/meta-pytorch/botorch.git
cd botorch
pip install -e ".[dev, tutorials]"
  • dev: Specifies tools necessary for development (testing, linting, docs building; see Contributing below).
  • tutorials: Also installs all packages necessary for running the tutorial notebooks.
  • You can also install either the dev or tutorials dependencies without installing both, e.g. by changing the last command to pip install -e ".[dev]".

Getting Started

Here's a quick run down of the main components of a Bayesian optimization loop. For more details see our Documentation and the Tutorials.

  1. Fit a Gaussian Process model to data
import torch
from botorch.models import SingleTaskGP
from botorch.models.transforms import Normalize
from botorch.fit import fit_gpytorch_mll
from gpytorch.mlls import ExactMarginalLogLikelihood

# Double precision is highly recommended for GPs.
# See https://github.com/meta-pytorch/botorch/discussions/1444
train_X = torch.rand(10, 2, dtype=torch.double) * 2
Y = 1 - (train_X - 0.5).norm(dim=-1, keepdim=True)  # explicit output dimension
Y += 0.1 * torch.rand_like(Y)

gp = SingleTaskGP(
    train_X=train_X,
    train_Y=Y,
    input_transform=Normalize(d=2),
)
mll = ExactMarginalLogLikelihood(gp.likelihood, gp)
fit_gpytorch_mll(mll)
  1. Construct an acquisition function
from botorch.acquisition import LogExpectedImprovement

logEI = LogExpectedImprovement(model=gp, best_f=Y.max())
  1. Optimize the acquisition function
from botorch.optim import optimize_acqf

bounds = torch.stack([torch.zeros(2), torch.ones(2)]).to(torch.double)
candidate, acq_value = optimize_acqf(
    logEI, bounds=bounds, q=1, num_restarts=5, raw_samples=20,
)

Citing BoTorch

If you use BoTorch, please cite the following paper:

M. Balandat, B. Karrer, D. R. Jiang, S. Daulton, B. Letham, A. G. Wilson, and E. Bakshy. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. Advances in Neural Information Processing Systems 33, 2020.

@inproceedings{balandat2020botorch,
  title={{BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization}},
  author={Balandat, Maximilian and Karrer, Brian and Jiang, Daniel R. and Daulton, Samuel and Letham, Benjamin and Wilson, Andrew Gordon and Bakshy, Eytan},
  booktitle = {Advances in Neural Information Processing Systems 33},
  year={2020},
  url = {http://arxiv.org/abs/1910.06403}
}

See here for an incomplete selection of peer-reviewed papers that build off of BoTorch.

Contributing

See the CONTRIBUTING file for how to help out.

License

BoTorch is MIT licensed, as found in the LICENSE file.