CodeFormer
[NeurIPS 2022] Towards Robust Blind Face Restoration with Codebook Lookup Transformer
Top Related Projects
GFPGAN aims at developing Practical Algorithms for Real-world Face Restoration.
Real-ESRGAN aims at developing Practical Algorithms for General Image/Video Restoration.
Bringing Old Photo Back to Life (CVPR 2020 oral)
Official PyTorch Code and Models of "RePaint: Inpainting using Denoising Diffusion Probabilistic Models", CVPR 2022
Image inpainting tool powered by SOTA AI Model. Remove any unwanted object, defect, people from your pictures or erase and replace(powered by stable diffusion) any thing on your pictures.
Quick Overview
CodeFormer is an open-source project for face restoration, focusing on improving the quality of low-resolution or degraded facial images. It combines both global face priors and local feature corrections to achieve high-quality results in various scenarios, including old photo restoration and AI-generated face enhancement.
Pros
- High-quality face restoration with impressive results
- Versatile application in various scenarios (old photos, AI-generated faces, etc.)
- Combines global and local features for better outcomes
- Provides both a pre-trained model and training code for customization
Cons
- Requires significant computational resources for optimal performance
- May struggle with extreme degradation or unusual facial features
- Limited to face restoration, not applicable to general image enhancement
- Potential privacy concerns when processing personal photos
Code Examples
# Load and prepare an image for face restoration
from basicsr.utils import img2tensor, tensor2img
from torchvision.transforms.functional import normalize
img = cv2.imread('input_image.jpg')
img = img2tensor(img / 255., bgr2rgb=True, float32=True)
normalize(img, (0.5, 0.5, 0.5), (0.5, 0.5, 0.5), inplace=True)
# Perform face restoration using CodeFormer
from basicsr.utils.registry import ARCH_REGISTRY
net = ARCH_REGISTRY.get('CodeFormer')(dim_embd=512, codebook_size=1024, n_head=8, n_layers=9, connect_list=['32', '64', '128', '256'])
restored_img = net(img)
# Save the restored image
from basicsr.utils import tensor2img
restored_img = tensor2img(restored_img, rgb2bgr=True, min_max=(-1, 1))
cv2.imwrite('restored_image.png', restored_img)
Getting Started
-
Clone the repository:
git clone https://github.com/sczhou/CodeFormer.git cd CodeFormer
-
Install dependencies:
pip install -r requirements.txt
-
Download pre-trained models:
python scripts/download_pretrained_models.py facelib python scripts/download_pretrained_models.py CodeFormer
-
Run face restoration:
python inference_codeformer.py -i inputs/whole_imgs -o results -w 0.5 --has_aligned
Competitor Comparisons
GFPGAN aims at developing Practical Algorithms for Real-world Face Restoration.
Pros of GFPGAN
- Faster inference time, especially for batch processing
- Better handling of extreme facial degradations
- More extensive pre-trained models available
Cons of GFPGAN
- Less flexibility in terms of customization options
- May produce over-smoothed results in some cases
- Limited to facial restoration, while CodeFormer has broader applications
Code Comparison
GFPGAN:
from gfpgan import GFPGANer
restorer = GFPGANer(model_path='experiments/pretrained_models/GFPGANv1.3.pth', upscale=2)
restored_img, _ = restorer.enhance(img, has_aligned=False, only_center_face=False, paste_back=True)
CodeFormer:
from basicsr.utils import img2tensor, tensor2img
from torchvision.transforms.functional import normalize
input_img = img2tensor(img / 255., bgr2rgb=True, float32=True)
normalize(input_img, (0.5, 0.5, 0.5), (0.5, 0.5, 0.5), inplace=True)
with torch.no_grad():
output = net(input_img.unsqueeze(0), w=w, adain=True)[0]
restored_img = tensor2img(output, rgb2bgr=True, min_max=(-1, 1))
Both repositories focus on facial restoration and enhancement, but they differ in their approaches and use cases. GFPGAN is generally faster and better suited for extreme degradations, while CodeFormer offers more flexibility and broader applications beyond facial restoration.
Real-ESRGAN aims at developing Practical Algorithms for General Image/Video Restoration.
Pros of Real-ESRGAN
- Focuses on general image restoration and enhancement
- Provides pre-trained models for various use cases
- Offers both CPU and GPU support for inference
Cons of Real-ESRGAN
- Less specialized in face restoration compared to CodeFormer
- May produce artifacts in some cases, especially with faces
- Limited control over specific aspects of the restoration process
Code Comparison
CodeFormer:
from codeformer import CodeFormer
model = CodeFormer()
restored_img = model.restore(input_img, w=0.7)
Real-ESRGAN:
from realesrgan import RealESRGAN
model = RealESRGAN(device='cuda')
restored_img = model.predict(input_img)
Both repositories offer easy-to-use interfaces for image restoration. CodeFormer provides more control over the restoration process with its w
parameter, while Real-ESRGAN focuses on simplicity and general-purpose enhancement. CodeFormer is more specialized for face restoration, whereas Real-ESRGAN is designed for broader image enhancement tasks.
Bringing Old Photo Back to Life (CVPR 2020 oral)
Pros of Bringing-Old-Photos-Back-to-Life
- Specialized in restoring old and damaged photos
- Includes face enhancement and colorization features
- Comprehensive pipeline for handling various photo degradations
Cons of Bringing-Old-Photos-Back-to-Life
- Less versatile for general image restoration tasks
- May require more computational resources due to multiple processing steps
- Limited to specific use case of old photo restoration
Code Comparison
Bringing-Old-Photos-Back-to-Life:
from bringing_old_photos_back_to_life import restore_old_photo
restored_image = restore_old_photo(input_image)
CodeFormer:
from codeformer import restore_image
restored_image = restore_image(input_image, fidelity_weight=0.5)
CodeFormer offers a simpler API with a single function call and adjustable fidelity weight, while Bringing-Old-Photos-Back-to-Life likely involves multiple steps in its restoration pipeline.
CodeFormer is more versatile, handling various image restoration tasks beyond just old photos. It uses a unified model for multiple purposes, potentially offering better performance on a wider range of images. However, Bringing-Old-Photos-Back-to-Life may excel specifically in restoring old and damaged photos due to its specialized focus.
Official PyTorch Code and Models of "RePaint: Inpainting using Denoising Diffusion Probabilistic Models", CVPR 2022
Pros of RePaint
- Focuses specifically on inpainting tasks, potentially offering more specialized results in this area
- Utilizes a diffusion-based approach, which can be effective for generating high-quality, diverse outputs
- Implements a novel "repaint" algorithm for improved inpainting results
Cons of RePaint
- More limited in scope compared to CodeFormer, focusing primarily on inpainting rather than multiple image restoration tasks
- May require more computational resources due to the iterative nature of diffusion models
- Less emphasis on face restoration specifically, which is a key feature of CodeFormer
Code Comparison
CodeFormer:
from basicsr.utils import imwrite, img2tensor, tensor2img
from torchvision.transforms.functional import normalize
img = img2tensor(img, bgr2rgb=True, float32=True)
normalize(img, (0.5, 0.5, 0.5), (0.5, 0.5, 0.5), inplace=True)
RePaint:
from PIL import Image
import numpy as np
image = np.array(Image.open(image_path))
mask = np.array(Image.open(mask_path).convert("L"))
The code snippets show different approaches to image processing and loading. CodeFormer uses PyTorch-based utilities for tensor conversion and normalization, while RePaint uses PIL and NumPy for image loading and manipulation.
Image inpainting tool powered by SOTA AI Model. Remove any unwanted object, defect, people from your pictures or erase and replace(powered by stable diffusion) any thing on your pictures.
Pros of IOPaint
- Offers a user-friendly web interface for inpainting tasks
- Supports multiple inpainting models, including LaMa and SD
- Provides real-time preview and adjustable brush sizes
Cons of IOPaint
- Limited to inpainting functionality, lacking face restoration features
- May require more manual input for complex image editing tasks
- Potentially slower processing times for large images
Code Comparison
IOPaint (Python):
@app.route("/inpaint", methods=["POST"])
def inpaint():
image = request.files["image"]
mask = request.files["mask"]
model = request.form["model"]
result = process_inpainting(image, mask, model)
return jsonify({"result": result})
CodeFormer (Python):
def restore_face(img, codeformer, face_helper, device):
cropped_faces, restored_faces, restored_img = face_helper.enhance(
img, has_aligned=False, only_center_face=False, paste_back=True)
return restored_img
The code snippets demonstrate the different focus areas of each project. IOPaint emphasizes web-based inpainting with multiple model options, while CodeFormer concentrates on face restoration functionality.
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual CopilotREADME
Towards Robust Blind Face Restoration with Codebook Lookup Transformer (NeurIPS 2022)
Paper | Project Page | Video
Shangchen Zhou, Kelvin C.K. Chan, Chongyi Li, Chen Change Loy
S-Lab, Nanyang Technological University
:star: If CodeFormer is helpful to your images or projects, please help star this repo. Thanks! :hugs:
Update
- 2023.07.20: Integrated to :panda_face: OpenXLab. Try out online demo!
- 2023.04.19: :whale: Training codes and config files are public available now.
- 2023.04.09: Add features of inpainting and colorization for cropped and aligned face images.
- 2023.02.10: Include
dlib
as a new face detector option, it produces more accurate face identity. - 2022.10.05: Support video input
--input_path [YOUR_VIDEO.mp4]
. Try it to enhance your videos! :clapper: - 2022.09.14: Integrated to :hugs: Hugging Face. Try out online demo!
- 2022.09.09: Integrated to :rocket: Replicate. Try out online demo!
- More
TODO
- Add training code and config files
- Add checkpoint and script for face inpainting
- Add checkpoint and script for face colorization
-
Add background image enhancement
:panda_face: Try Enhancing Old Photos / Fixing AI-arts
Face Restoration
Face Color Enhancement and Restoration
Face Inpainting
Dependencies and Installation
- Pytorch >= 1.7.1
- CUDA >= 10.1
- Other required packages in
requirements.txt
# git clone this repository
git clone https://github.com/sczhou/CodeFormer
cd CodeFormer
# create new anaconda env
conda create -n codeformer python=3.8 -y
conda activate codeformer
# install python dependencies
pip3 install -r requirements.txt
python basicsr/setup.py develop
conda install -c conda-forge dlib (only for face detection or cropping with dlib)
Quick Inference
Download Pre-trained Models:
Download the facelib and dlib pretrained models from [Releases | Google Drive | OneDrive] to the weights/facelib
folder. You can manually download the pretrained models OR download by running the following command:
python scripts/download_pretrained_models.py facelib
python scripts/download_pretrained_models.py dlib (only for dlib face detector)
Download the CodeFormer pretrained models from [Releases | Google Drive | OneDrive] to the weights/CodeFormer
folder. You can manually download the pretrained models OR download by running the following command:
python scripts/download_pretrained_models.py CodeFormer
Prepare Testing Data:
You can put the testing images in the inputs/TestWhole
folder. If you would like to test on cropped and aligned faces, you can put them in the inputs/cropped_faces
folder. You can get the cropped and aligned faces by running the following command:
# you may need to install dlib via: conda install -c conda-forge dlib
python scripts/crop_align_face.py -i [input folder] -o [output folder]
Testing:
[Note] If you want to compare CodeFormer in your paper, please run the following command indicating --has_aligned
(for cropped and aligned face), as the command for the whole image will involve a process of face-background fusion that may damage hair texture on the boundary, which leads to unfair comparison.
Fidelity weight w lays in [0, 1]. Generally, smaller w tends to produce a higher-quality result, while larger w yields a higher-fidelity result. The results will be saved in the results
folder.
ð§ð» Face Restoration (cropped and aligned face)
# For cropped and aligned faces (512x512)
python inference_codeformer.py -w 0.5 --has_aligned --input_path [image folder]|[image path]
:framed_picture: Whole Image Enhancement
# For whole image
# Add '--bg_upsampler realesrgan' to enhance the background regions with Real-ESRGAN
# Add '--face_upsample' to further upsample restorated face with Real-ESRGAN
python inference_codeformer.py -w 0.7 --input_path [image folder]|[image path]
:clapper: Video Enhancement
# For Windows/Mac users, please install ffmpeg first
conda install -c conda-forge ffmpeg
# For video clips
# Video path should end with '.mp4'|'.mov'|'.avi'
python inference_codeformer.py --bg_upsampler realesrgan --face_upsample -w 1.0 --input_path [video path]
ð Face Colorization (cropped and aligned face)
# For cropped and aligned faces (512x512)
# Colorize black and white or faded photo
python inference_colorization.py --input_path [image folder]|[image path]
ð¨ Face Inpainting (cropped and aligned face)
# For cropped and aligned faces (512x512)
# Inputs could be masked by white brush using an image editing app (e.g., Photoshop)
# (check out the examples in inputs/masked_faces)
python inference_inpainting.py --input_path [image folder]|[image path]
Training:
The training commands can be found in the documents: English | ç®ä½ä¸æ.
Citation
If our work is useful for your research, please consider citing:
@inproceedings{zhou2022codeformer,
author = {Zhou, Shangchen and Chan, Kelvin C.K. and Li, Chongyi and Loy, Chen Change},
title = {Towards Robust Blind Face Restoration with Codebook Lookup TransFormer},
booktitle = {NeurIPS},
year = {2022}
}
License
This project is licensed under NTU S-Lab License 1.0. Redistribution and use should follow this license.
Acknowledgement
This project is based on BasicSR. Some codes are brought from Unleashing Transformers, YOLOv5-face, and FaceXLib. We also adopt Real-ESRGAN to support background image enhancement. Thanks for their awesome works.
Contact
If you have any questions, please feel free to reach me out at shangchenzhou@gmail.com
.
Top Related Projects
GFPGAN aims at developing Practical Algorithms for Real-world Face Restoration.
Real-ESRGAN aims at developing Practical Algorithms for General Image/Video Restoration.
Bringing Old Photo Back to Life (CVPR 2020 oral)
Official PyTorch Code and Models of "RePaint: Inpainting using Denoising Diffusion Probabilistic Models", CVPR 2022
Image inpainting tool powered by SOTA AI Model. Remove any unwanted object, defect, people from your pictures or erase and replace(powered by stable diffusion) any thing on your pictures.
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual Copilot