BSRGAN

Designing a Practical Degradation Model for Deep Blind Image Super-Resolution (ICCV, 2021) (PyTorch) - We released the training code!

1,189

179

1,189

View on GitHub

Top Related Projects

SwinIR

4,331

SwinIR: Image Restoration Using Swin Transformer (official repository)

ESRGAN

5,931

ECCV18 Workshops - Enhanced SRGAN. Champion PIRM Challenge on Perceptual Super-Resolution. The training codes are in BasicSR.

Quick Overview

BSRGAN is a GitHub repository containing a state-of-the-art blind super-resolution model. It aims to enhance low-resolution images without prior knowledge of the degradation process. The project includes pre-trained models, training code, and testing scripts for image super-resolution tasks.

Pros

Achieves high-quality image upscaling without knowing the degradation process
Provides pre-trained models for easy implementation and testing
Includes both training and testing code for researchers and developers
Supports various scaling factors (x2, x3, x4)

Cons

Requires significant computational resources for training
Limited documentation for customization and advanced usage
May struggle with extremely low-quality or heavily compressed input images
Dependency on specific versions of PyTorch and other libraries

Code Examples

Loading a pre-trained BSRGAN model:

import torch
from models.network_bsrgan import BSRNet

model = BSRNet(in_nc=3, out_nc=3, nf=64, nb=23, scale=4)
model.load_state_dict(torch.load('model_zoo/BSRGAN.pth'), strict=True)
model.eval()

Performing super-resolution on an image:

from utils import util_image as util

# Load and prepare input image
img_lq = util.imread_uint('input_low_res.png', n_channels=3)
img_lq = util.uint2tensor4(img_lq)

# Super-resolve the image
with torch.no_grad():
    output = model(img_lq)

# Save the result
util.imsave(util.tensor2uint(output), 'output_high_res.png')

Training the BSRGAN model:

from models.model_plain import ModelPlain

model = ModelPlain(opt)
model.init_train()
model.train()

for epoch in range(opt['train']['epoch']):
    for i, train_data in enumerate(train_loader):
        model.feed_data(train_data)
        model.optimize_parameters()
    
    model.save(epoch)

Getting Started

Clone the repository:

git clone https://github.com/cszn/BSRGAN.git
cd BSRGAN

Install dependencies:
```
pip install -r requirements.txt
```
Download pre-trained models from the repository's release page and place them in the model_zoo folder.

Run inference on an image:

python main_test_bsrgan.py --model_path model_zoo/BSRGAN.pth --folder_lq testsets/set5/LR_bicubic/X4 --folder_gt testsets/set5/HR

This will process images in the specified input folder and save the super-resolved results in the results folder.

Competitor Comparisons

SwinIR

4,331

SwinIR: Image Restoration Using Swin Transformer (official repository)

Pros of SwinIR

Utilizes the Swin Transformer architecture, which can capture long-range dependencies more effectively
Demonstrates superior performance on various image restoration tasks, including super-resolution and denoising
Offers pre-trained models for multiple tasks and scales

Cons of SwinIR

Higher computational complexity and memory requirements due to the transformer-based architecture
May require more training data and longer training times compared to BSRGAN

Code Comparison

SwinIR:

model = define_model(task=args.task, scale=args.scale, training_patch_size=args.training_patch_size)
model.train()
loss = criterion(output, target)

BSRGAN:

model = define_model(scale=args.scale)
model.train()
loss = criterion(output, target)

Both repositories provide similar high-level code structures for model definition and training. However, SwinIR offers more flexibility in task selection and patch size configuration, while BSRGAN focuses primarily on super-resolution tasks.

ESRGAN

5,931

ECCV18 Workshops - Enhanced SRGAN. Champion PIRM Challenge on Perceptual Super-Resolution. The training codes are in BasicSR.

Pros of ESRGAN

More established project with a larger community and wider adoption
Extensive documentation and examples available
Supports multiple architectures and training strategies

Cons of ESRGAN

Older implementation, potentially less optimized for modern hardware
May require more computational resources for training and inference

Code Comparison

ESRGAN:

class RRDBNet(nn.Module):
    def __init__(self, in_nc, out_nc, nf, nb, gc=32):
        super(RRDBNet, self).__init__()
        RRDB_block_f = functools.partial(RRDB, nf=nf, gc=gc)
        self.conv_first = nn.Conv2d(in_nc, nf, 3, 1, 1, bias=True)
        self.RRDB_trunk = make_layer(RRDB_block_f, nb)
        self.trunk_conv = nn.Conv2d(nf, nf, 3, 1, 1, bias=True)

BSRGAN:

class BSRGAN(nn.Module):
    def __init__(self, in_nc=3, out_nc=3, nf=64, nb=23, gc=32, scale=4, pca_matrix=None):
        super(BSRGAN, self).__init__()
        self.conv_first = nn.Conv2d(in_nc, nf, 3, 1, 1, bias=True)
        basic_block = functools.partial(ResidualDenseBlock_5C, nf=nf, gc=gc)
        self.recon_trunk = make_layer(basic_block, nb)

Both implementations use similar architectural concepts, but BSRGAN appears to have a more streamlined approach with fewer parameters in its main network class.

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Designing a Practical Degradation Model for Deep Blind Image Super-Resolution

Kai Zhang, Jingyun Liang, Luc Van Gool, Radu Timofte
Computer Vision Lab, ETH Zurich, Switzerland

[Paper] [Code] [Training Code]

Our new work for real image denoising ---> https://github.com/cszn/SCUNet

Our work is the beginning rather than the end of real image super-resolution.

News (2021-08-31): We upload the training code.
News (2021-08-24): We upload the BSRGAN degradation model.

from utils import utils_blindsr as blindsr
img_lq, img_hq = blindsr.degradation_bsrgan(img, sf=4, lq_patchsize=72)

News (2021-07-23): After rejection by CVPR 2021, our paper is accepted by ICCV 2021. For the sake of fairness, we will not update the trained models in our camera-ready version. However, we may update the trained models in github.
News (2021-05-18): Add trained BSRGAN model for scale factor 2.
News (2021-04): Our degradation model for face image enhancement: https://github.com/vvictoryuki/BSRGAN_implementation

Training

Download KAIR: git clone https://github.com/cszn/KAIR.git
Put your training high-quality images into trainsets/trainH or set "dataroot_H": "trainsets/trainH"

Train BSRNet

Modify train_bsrgan_x4_psnr.json e.g., "gpu_ids": [0], "dataloader_batch_size": 4
Training with DataParallel

python main_train_psnr.py --opt options/train_bsrgan_x4_psnr.json

Training with DistributedDataParallel - 4 GPUs

python -m torch.distributed.launch --nproc_per_node=4 --master_port=1234 main_train_psnr.py --opt options/train_bsrgan_x4_psnr.json  --dist True

Train BSRGAN
1. Put BSRNet model (e.g., '400000_G.pth') into superresolution/bsrgan_x4_gan/models
2. Modify train_bsrgan_x4_gan.json e.g., "gpu_ids": [0], "dataloader_batch_size": 4
3. Training with DataParallel
```
python main_train_gan.py --opt options/train_bsrgan_x4_gan.json
```
1. Training with DistributedDataParallel - 4 GPUs
```
python -m torch.distributed.launch --nproc_per_node=4 --master_port=1234 main_train_gan.py --opt options/train_bsrgan_x4_gan.json  --dist True
```
Test BSRGAN model 'xxxxxx_E.pth' by modified main_test_bsrgan.py
1. 'xxxxxx_E.pth' is more stable than 'xxxxxx_G.pth'

â¨ Some visual examples: oldphoto2; butterfly; comic; oldphoto3; oldphoto6; comic_01; comic_03; comic_04

Testing code
Main idea
Comparison
More visual results on RealSRSet dataset
Visual results on DPED dataset
Citation
Acknowledgments

Testing code

main_test_bsrgan.py
model_zoo (Download the following models from Google drive or è¾è®¯å¾®äº).
- Proposed:
  - BSRGAN.pth [Google drive] [è¾è®¯å¾®äº]ð±
  - BSRNet.pth [Google drive] [è¾è®¯å¾®äº]ð±
- Compared methods:
  - RRDB.pth ---> original link
  - ESRGAN.pth ---> original link
  - FSSR_DPED.pth ---> original link
  - FSSR_DPED.pth ---> original link
  - RealSR_DPED.pth ---> original link
  - RealSR_JPEG.pth ---> original link

Main idea

Design a new degradation model to synthesize LR images for training:

1) Make the blur, downsampling and noise more practical
- Blur: two convolutions with isotropic and anisotropic Gaussian kernels from both the HR space and LR space
- Downsampling: nearest, bilinear, bicubic, down-up-sampling
- Noise: Gaussian noise, JPEG compression noise, processed camera sensor noise
2) Degradation shuffle: instead of using the commonly-used blur/downsampling/noise-addition pipeline, we perform randomly shuffled degradations to synthesize LR images

Some notes on the proposed degradation model:

The degradation model is mainly designed to synthesize degraded LR images. Its most direct application is to train a deep blind super-resolver with paired LR/HR images. In particular, the degradation model can be performed on a large dataset of HR images to produce unlimited perfectly aligned training images, which typically do not suffer from the limited data issue of laboriously collected paired data and the misalignment issue of unpaired training data.
The degradation model tends to be unsuited to model a degraded LR image as it involves too many degradation parameters and also adopts a random shuffle strategy.
The degradation model can produce some degradation cases that rarely happen in real-world scenarios, while this can still be expected to improve the generalization ability of the trained deep blind super-resolver.
A DNN with large capacity has the ability to handle different degradations via a single model. This has been validated multiple times. For example, DnCNN is able to handle SISR with different scale factors, JPEG compression deblocking with different quality factors and denoising for a wide range of noise levels, while still having a performance comparable to VDSR for SISR. It is worth noting that even when the super-resolver reduces the performance for unrealistic bicubic downsampling, it is still a preferred choice for real SISR.
One can conveniently modify the degradation model by changing the degradation parameter settings and adding more reasonable degradation types to improve the practicability for a certain application.

Comparison

These no-reference IQA metrics, i.e., NIQE, NRQM and PI, do not always match perceptual visual quality [1] and the IQA metric should be updated with new SISR methods [2]. We further argue that the IQA metric for SISR should also be updated with new image degradation types, which we leave for future work.

[1] "NTIRE 2020 challenge on real-world image super-resolution: Methods and results." CVPRW, 2020.
[2] "PIPAL: a large-scale image quality assessment dataset for perceptual image restoration." ECCV, 2020.

More visual results on RealSRSet dataset

Left: real images | Right: super-resolved images with scale factor 4

Visual results on DPED dataset

Without using any prior information of DPED dataset for training, our BSRGAN still performs well.

Citation

@inproceedings{zhang2021designing,
    title={Designing a Practical Degradation Model for Deep Blind Image Super-Resolution},
    author={Zhang, Kai and Liang, Jingyun and Van Gool, Luc and Timofte, Radu},
    booktitle={IEEE International Conference on Computer Vision},
    pages={4791--4800},
    year={2021}
}

Acknowledgments

This work was partly supported by the ETH Zurich Fund (OK), a Huawei Technologies Oy (Finland) project, and an Amazon AWS grant.

Top Related Projects

SwinIR

4,331

SwinIR: Image Restoration Using Swin Transformer (official repository)

ESRGAN

5,931

ECCV18 Workshops - Enhanced SRGAN. Champion PIRM Challenge on Perceptual Super-Resolution. The training codes are in BasicSR.

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot