guided-diffusion

No description available

6,045

807

6,045

102

View on GitHub

Top Related Projects

diffusers

25,061

🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch and FLAX.

denoising-diffusion-pytorch

7,878

Implementation of Denoising Diffusion Probabilistic Model in Pytorch

k-diffusion

2,255

Karras et al. (2022) diffusion models for PyTorch

Quick Overview

The openai/guided-diffusion repository is a PyTorch implementation of a diffusion-based generative model, which can be used for tasks such as image generation and audio synthesis. The project provides a flexible and extensible framework for training and using diffusion models, with a focus on high-quality and stable generation.

Pros

Flexible and Extensible: The project is designed to be easily extensible, allowing users to experiment with different model architectures, training techniques, and applications.
High-Quality Generation: The diffusion models trained using this framework have been shown to produce high-quality, realistic-looking images and audio samples.
Stable Training: The project includes techniques for stabilizing the training of diffusion models, which can be challenging due to the iterative nature of the generation process.
Active Development: The project is actively maintained by the OpenAI team, with regular updates and improvements.

Cons

Computational Complexity: Training and using diffusion models can be computationally intensive, requiring significant GPU resources and training time.
Limited Documentation: While the project is well-designed, the documentation could be more comprehensive, especially for users new to diffusion models or PyTorch.
Specialized Knowledge Required: Effectively using and extending the project requires a good understanding of diffusion models, PyTorch, and machine learning concepts in general.
Limited Pre-Trained Models: The project does not currently provide a wide range of pre-trained models, which can make it more difficult for users to get started quickly.

Code Examples

Here are a few code examples from the openai/guided-diffusion repository:

Training a Diffusion Model:

from guided_diffusion.script_util import create_model_and_diffusion, model_and_diffusion_defaults

# Set up the model and diffusion
model, diffusion = create_model_and_diffusion(**model_and_diffusion_defaults())

# Train the model
model.train(data_loader)

This code sets up a diffusion model and diffusion process, and then trains the model on a dataset.

Generating Images:

from guided_diffusion.script_util import create_model_and_diffusion, model_and_diffusion_defaults

# Set up the model and diffusion
model, diffusion = create_model_and_diffusion(**model_and_diffusion_defaults())

# Load a pre-trained model
model.load_state_dict(torch.load("path/to/model.pt"))

# Generate an image
image = diffusion.sample(model, (1, 3, 64, 64))

This code loads a pre-trained diffusion model and uses it to generate a new image.

Evaluating Model Performance:

from guided_diffusion.script_util import create_model_and_diffusion, model_and_diffusion_defaults
from guided_diffusion.evaluation import create_classifier, load_classifier

# Set up the model and diffusion
model, diffusion = create_model_and_diffusion(**model_and_diffusion_defaults())

# Load a pre-trained model
model.load_state_dict(torch.load("path/to/model.pt"))

# Create and load a classifier
classifier = create_classifier()
classifier.load_state_dict(torch.load("path/to/classifier.pt"))

# Evaluate the model's performance
fid_score = diffusion.compute_fid(model, classifier, dataset)

This code creates a classifier, loads a pre-trained diffusion model, and uses the classifier to compute the Fréchet Inception Distance (FID) score of the model's generated images.

Getting Started

To get started with the openai/guided-diffusion project, follow these steps:

Clone the repository:

git clone https://github.com/openai/guided-diffusion.git

Install the required dependencies:

cd guided-diffusion
pip install -r requirements.txt

Prepare your dataset:
- The project supports various image and audio datasets. You'll need to ensure your dataset is in a format compatible with the project.
Train a diffusion model:

from guided_diffusion.script_util import create_model

Competitor Comparisons

stable-diffusion

1,029

Pros of Stable Diffusion

More efficient and faster image generation
Supports a wider range of image sizes and aspect ratios
Better at handling complex prompts and generating coherent images

Cons of Stable Diffusion

Requires more GPU memory for training and inference
Less flexibility in controlling the diffusion process
May produce less diverse outputs in some cases

Code Comparison

Guided Diffusion:

def p_sample_loop(model, shape, noise=None, clip_denoised=True, denoised_fn=None,
                  model_kwargs=None, device=None, progress=False):
    if device is None:
        device = next(model.parameters()).device
    if noise is None:
        noise = th.randn(*shape, device=device)
    x = noise

Stable Diffusion:

@torch.no_grad()
def p_sample_plms(
    model,
    x,
    t,
    old_eps,
    t_next=None,
    x_self_cond=None,
    clip_denoised=True,
    denoised_fn=None,
    model_kwargs=None,
    order=2,
):
    if model_kwargs is None:
        model_kwargs = {}

diffusers

25,061

🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch and FLAX.

Pros of diffusers

Broader scope, supporting multiple diffusion models and techniques
More active development and community support
Easier integration with other Hugging Face libraries and ecosystems

Cons of diffusers

Higher complexity due to supporting multiple models
Potentially slower inference for specific models compared to specialized implementations

Code Comparison

guided-diffusion:

def p_sample(model, x, t, clip_denoised=True, denoised_fn=None, cond_fn=None, model_kwargs=None):
    out = p_mean_variance(model, x, t, clip_denoised=clip_denoised, denoised_fn=denoised_fn, model_kwargs=model_kwargs)
    noise = th.randn_like(x)
    nonzero_mask = (t != 0).float().view(-1, *([1] * (len(x.shape) - 1)))
    sample = out["mean"] + nonzero_mask * th.exp(0.5 * out["log_variance"]) * noise
    return {"sample": sample, "pred_xstart": out["pred_xstart"]}

diffusers:

def step(self, model_output: torch.FloatTensor, timestep: int, sample: torch.FloatTensor):
    t = timestep
    prev_t = self.previous_timestep(t)
    alpha_prod_t = self.alphas_cumprod[t]
    alpha_prod_t_prev = self.alphas_cumprod[prev_t] if prev_t >= 0 else self.final_alpha_cumprod
    beta_prod_t = 1 - alpha_prod_t
    pred_original_sample = (sample - beta_prod_t ** 0.5 * model_output) / alpha_prod_t ** 0.5
    pred_sample_direction = (1 - alpha_prod_t_prev) ** 0.5 * model_output
    prev_sample = alpha_prod_t_prev ** 0.5 * pred_original_sample + pred_sample_direction
    return prev_sample

denoising-diffusion-pytorch

7,878

Implementation of Denoising Diffusion Probabilistic Model in Pytorch

Pros of denoising-diffusion-pytorch

More lightweight and easier to understand implementation
Focuses on core diffusion concepts without additional complexities
Actively maintained with frequent updates and community contributions

Cons of denoising-diffusion-pytorch

Less comprehensive feature set compared to guided-diffusion
May require additional implementation for advanced techniques
Limited pre-trained models and datasets provided

Code Comparison

guided-diffusion:

def p_sample_loop(self, model, shape, noise=None, clip_denoised=True, denoised_fn=None,
                  model_kwargs=None, device=None, progress=False):
    final = None
    for sample in self.p_sample_loop_progressive(
        model,
        shape,
        noise=noise,
        clip_denoised=clip_denoised,
        denoised_fn=denoised_fn,
        model_kwargs=model_kwargs,
        device=device,
        progress=progress,
    ):
        final = sample
    return final["sample"]

denoising-diffusion-pytorch:

@torch.no_grad()
def p_sample_loop(self, shape, return_all_timesteps = False):
    batch, device = shape[0], self.betas.device
    img = torch.randn(shape, device = device)
    imgs = [img]

    x_start = None

    for t in tqdm(reversed(range(0, self.num_timesteps)), desc = 'sampling loop time step', total = self.num_timesteps):
        self_cond = x_start if self.self_condition else None
        img, x_start = self.p_sample(img, t, self_cond)
        imgs.append(img)

    ret = img if not return_all_timesteps else torch.stack(imgs, dim = 1)
    ret = self.unnormalize(ret)
    return ret

k-diffusion

2,255

Karras et al. (2022) diffusion models for PyTorch

Pros of k-diffusion

More flexible and customizable diffusion models
Supports a wider range of sampling methods
Better performance on certain tasks, especially image generation

Cons of k-diffusion

Less documentation and community support
May require more expertise to use effectively
Not as well-integrated with other AI frameworks

Code Comparison

k-diffusion:

model = diffusion.get_model(config)
x = torch.randn(4, 3, 64, 64)
samples = diffusion.sample(model, x, steps=20)

guided-diffusion:

model = create_model(image_size=64, num_channels=128, num_res_blocks=2)
diffusion = create_gaussian_diffusion()
samples = diffusion.p_sample_loop(model, (4, 3, 64, 64))

Both repositories provide implementations of diffusion models, but k-diffusion offers more flexibility in terms of sampling methods and model architectures. guided-diffusion, on the other hand, provides a more straightforward API and better integration with other OpenAI tools. The code comparison shows that k-diffusion allows for more customization in the sampling process, while guided-diffusion offers a simpler interface for basic usage.

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

guided-diffusion

This is the codebase for Diffusion Models Beat GANS on Image Synthesis.

This repository is based on openai/improved-diffusion, with modifications for classifier conditioning and architecture improvements.

Download pre-trained models

We have released checkpoints for the main models in the paper. Before using these models, please review the corresponding model card to understand the intended use and limitations of these models.

Here are the download links for each model checkpoint:

64x64 classifier: 64x64_classifier.pt
64x64 diffusion: 64x64_diffusion.pt
128x128 classifier: 128x128_classifier.pt
128x128 diffusion: 128x128_diffusion.pt
256x256 classifier: 256x256_classifier.pt
256x256 diffusion: 256x256_diffusion.pt
256x256 diffusion (not class conditional): 256x256_diffusion_uncond.pt
512x512 classifier: 512x512_classifier.pt
512x512 diffusion: 512x512_diffusion.pt
64x64 -> 256x256 upsampler: 64_256_upsampler.pt
128x128 -> 512x512 upsampler: 128_512_upsampler.pt
LSUN bedroom: lsun_bedroom.pt
LSUN cat: lsun_cat.pt
LSUN horse: lsun_horse.pt
LSUN horse (no dropout): lsun_horse_nodropout.pt

Sampling from pre-trained models

To sample from these models, you can use the classifier_sample.py, image_sample.py, and super_res_sample.py scripts. Here, we provide flags for sampling from all of these models. We assume that you have downloaded the relevant model checkpoints into a folder called models/.

For these examples, we will generate 100 samples with batch size 4. Feel free to change these values.

SAMPLE_FLAGS="--batch_size 4 --num_samples 100 --timestep_respacing 250"

Classifier guidance

Note for these sampling runs that you can set --classifier_scale 0 to sample from the base diffusion model. You may also use the image_sample.py script instead of classifier_sample.py in that case.

64x64 model:

MODEL_FLAGS="--attention_resolutions 32,16,8 --class_cond True --diffusion_steps 1000 --dropout 0.1 --image_size 64 --learn_sigma True --noise_schedule cosine --num_channels 192 --num_head_channels 64 --num_res_blocks 3 --resblock_updown True --use_new_attention_order True --use_fp16 True --use_scale_shift_norm True"
python classifier_sample.py $MODEL_FLAGS --classifier_scale 1.0 --classifier_path models/64x64_classifier.pt --classifier_depth 4 --model_path models/64x64_diffusion.pt $SAMPLE_FLAGS

128x128 model:

MODEL_FLAGS="--attention_resolutions 32,16,8 --class_cond True --diffusion_steps 1000 --image_size 128 --learn_sigma True --noise_schedule linear --num_channels 256 --num_heads 4 --num_res_blocks 2 --resblock_updown True --use_fp16 True --use_scale_shift_norm True"
python classifier_sample.py $MODEL_FLAGS --classifier_scale 0.5 --classifier_path models/128x128_classifier.pt --model_path models/128x128_diffusion.pt $SAMPLE_FLAGS

256x256 model:

MODEL_FLAGS="--attention_resolutions 32,16,8 --class_cond True --diffusion_steps 1000 --image_size 256 --learn_sigma True --noise_schedule linear --num_channels 256 --num_head_channels 64 --num_res_blocks 2 --resblock_updown True --use_fp16 True --use_scale_shift_norm True"
python classifier_sample.py $MODEL_FLAGS --classifier_scale 1.0 --classifier_path models/256x256_classifier.pt --model_path models/256x256_diffusion.pt $SAMPLE_FLAGS

256x256 model (unconditional):

MODEL_FLAGS="--attention_resolutions 32,16,8 --class_cond False --diffusion_steps 1000 --image_size 256 --learn_sigma True --noise_schedule linear --num_channels 256 --num_head_channels 64 --num_res_blocks 2 --resblock_updown True --use_fp16 True --use_scale_shift_norm True"
python classifier_sample.py $MODEL_FLAGS --classifier_scale 10.0 --classifier_path models/256x256_classifier.pt --model_path models/256x256_diffusion_uncond.pt $SAMPLE_FLAGS

512x512 model:

MODEL_FLAGS="--attention_resolutions 32,16,8 --class_cond True --diffusion_steps 1000 --image_size 512 --learn_sigma True --noise_schedule linear --num_channels 256 --num_head_channels 64 --num_res_blocks 2 --resblock_updown True --use_fp16 False --use_scale_shift_norm True"
python classifier_sample.py $MODEL_FLAGS --classifier_scale 4.0 --classifier_path models/512x512_classifier.pt --model_path models/512x512_diffusion.pt $SAMPLE_FLAGS

Upsampling

For these runs, we assume you have some base samples in a file 64_samples.npz or 128_samples.npz for the two respective models.

64 -> 256:

MODEL_FLAGS="--attention_resolutions 32,16,8 --class_cond True --diffusion_steps 1000 --large_size 256  --small_size 64 --learn_sigma True --noise_schedule linear --num_channels 192 --num_heads 4 --num_res_blocks 2 --resblock_updown True --use_fp16 True --use_scale_shift_norm True"
python super_res_sample.py $MODEL_FLAGS --model_path models/64_256_upsampler.pt --base_samples 64_samples.npz $SAMPLE_FLAGS

128 -> 512:

MODEL_FLAGS="--attention_resolutions 32,16 --class_cond True --diffusion_steps 1000 --large_size 512 --small_size 128 --learn_sigma True --noise_schedule linear --num_channels 192 --num_head_channels 64 --num_res_blocks 2 --resblock_updown True --use_fp16 True --use_scale_shift_norm True"
python super_res_sample.py $MODEL_FLAGS --model_path models/128_512_upsampler.pt $SAMPLE_FLAGS --base_samples 128_samples.npz

LSUN models

These models are class-unconditional and correspond to a single LSUN class. Here, we show how to sample from lsun_bedroom.pt, but the other two LSUN checkpoints should work as well:

MODEL_FLAGS="--attention_resolutions 32,16,8 --class_cond False --diffusion_steps 1000 --dropout 0.1 --image_size 256 --learn_sigma True --noise_schedule linear --num_channels 256 --num_head_channels 64 --num_res_blocks 2 --resblock_updown True --use_fp16 True --use_scale_shift_norm True"
python image_sample.py $MODEL_FLAGS --model_path models/lsun_bedroom.pt $SAMPLE_FLAGS

You can sample from lsun_horse_nodropout.pt by changing the dropout flag:

MODEL_FLAGS="--attention_resolutions 32,16,8 --class_cond False --diffusion_steps 1000 --dropout 0.0 --image_size 256 --learn_sigma True --noise_schedule linear --num_channels 256 --num_head_channels 64 --num_res_blocks 2 --resblock_updown True --use_fp16 True --use_scale_shift_norm True"
python image_sample.py $MODEL_FLAGS --model_path models/lsun_horse_nodropout.pt $SAMPLE_FLAGS

Note that for these models, the best samples result from using 1000 timesteps:

SAMPLE_FLAGS="--batch_size 4 --num_samples 100 --timestep_respacing 1000"

Results

This table summarizes our ImageNet results for pure guided diffusion models:

Dataset	FID	Precision	Recall
ImageNet 64x64	2.07	0.74	0.63
ImageNet 128x128	2.97	0.78	0.59
ImageNet 256x256	4.59	0.82	0.52
ImageNet 512x512	7.72	0.87	0.42

This table shows the best results for high resolutions when using upsampling and guidance together:

Dataset	FID	Precision	Recall
ImageNet 256x256	3.94	0.83	0.53
ImageNet 512x512	3.85	0.84	0.53

Finally, here are the unguided results on individual LSUN classes:

Dataset	FID	Precision	Recall
LSUN Bedroom	1.90	0.66	0.51
LSUN Cat	5.57	0.63	0.52
LSUN Horse	2.57	0.71	0.55

Training models

Training diffusion models is described in the parent repository. Training a classifier is similar. We assume you have put training hyperparameters into a TRAIN_FLAGS variable, and classifier hyperparameters into a CLASSIFIER_FLAGS variable. Then you can run:

mpiexec -n N python scripts/classifier_train.py --data_dir path/to/imagenet $TRAIN_FLAGS $CLASSIFIER_FLAGS

Make sure to divide the batch size in TRAIN_FLAGS by the number of MPI processes you are using.

Here are flags for training the 128x128 classifier. You can modify these for training classifiers at other resolutions:

TRAIN_FLAGS="--iterations 300000 --anneal_lr True --batch_size 256 --lr 3e-4 --save_interval 10000 --weight_decay 0.05"
CLASSIFIER_FLAGS="--image_size 128 --classifier_attention_resolutions 32,16,8 --classifier_depth 2 --classifier_width 128 --classifier_pool attention --classifier_resblock_updown True --classifier_use_scale_shift_norm True"

For sampling from a 128x128 classifier-guided model, 25 step DDIM:

MODEL_FLAGS="--attention_resolutions 32,16,8 --class_cond True --image_size 128 --learn_sigma True --num_channels 256 --num_heads 4 --num_res_blocks 2 --resblock_updown True --use_fp16 True --use_scale_shift_norm True"
CLASSIFIER_FLAGS="--image_size 128 --classifier_attention_resolutions 32,16,8 --classifier_depth 2 --classifier_width 128 --classifier_pool attention --classifier_resblock_updown True --classifier_use_scale_shift_norm True --classifier_scale 1.0 --classifier_use_fp16 True"
SAMPLE_FLAGS="--batch_size 4 --num_samples 50000 --timestep_respacing ddim25 --use_ddim True"
mpiexec -n N python scripts/classifier_sample.py \
    --model_path /path/to/model.pt \
    --classifier_path path/to/classifier.pt \
    $MODEL_FLAGS $CLASSIFIER_FLAGS $SAMPLE_FLAGS

To sample for 250 timesteps without DDIM, replace --timestep_respacing ddim25 to --timestep_respacing 250, and replace --use_ddim True with --use_ddim False.

Top Related Projects

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot