instant-ngp

Instant neural graphics primitives: lightning fast NeRF and more

15,796

1,899

15,796

481

View on GitHub

Top Related Projects

mitsuba3

2,007

Mitsuba 3: A Retargetable Forward and Inverse Renderer

multinerf

3,604

A Code Release for Mip-NeRF 360, Ref-NeRF, and RawNeRF

pytorch3d

8,643

PyTorch3D is FAIR's library of reusable components for deep learning with 3D data

nerfies

1,627

This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies.

Quick Overview

Instant-ngp is a high-performance implementation of Neural Graphics Primitives, focusing on real-time rendering of 3D scenes using neural networks. It enables rapid training and rendering of complex 3D environments, supporting various applications such as NeRF (Neural Radiance Fields), volume rendering, and image-based 3D reconstruction.

Pros

Extremely fast training and rendering times compared to traditional methods
High-quality results for 3D scene reconstruction and rendering
Supports multiple graphics primitives and applications (NeRF, volume rendering, etc.)
Efficient GPU utilization for real-time performance

Cons

Requires powerful GPU hardware for optimal performance
Limited documentation and examples for beginners
Steep learning curve for those unfamiliar with neural rendering techniques
Dependency on specific CUDA versions and hardware requirements

Code Examples

Loading and rendering a NeRF scene:

import pyngp as ngp

# Load a trained NeRF model
testbed = ngp.Testbed(ngp.TestbedMode.Nerf)
testbed.load_snapshot("path/to/nerf_model.msgpack")

# Render an image
image = testbed.render(width=1920, height=1080, spp=1)

Training a new NeRF model:

import pyngp as ngp

# Create a new NeRF testbed
testbed = ngp.Testbed(ngp.TestbedMode.Nerf)

# Load training data
testbed.load_training_data("path/to/transforms.json")

# Train the model
testbed.train(n_steps=50000)

# Save the trained model
testbed.save_snapshot("path/to/output_model.msgpack")

Performing volume rendering:

import pyngp as ngp

# Create a volume rendering testbed
testbed = ngp.Testbed(ngp.TestbedMode.Volume)

# Load volume data
testbed.load_volume("path/to/volume_data.raw")

# Render the volume
image = testbed.render(width=800, height=600, spp=1)

Getting Started

Clone the repository:

git clone https://github.com/NVlabs/instant-ngp.git
cd instant-ngp

Install dependencies:
```
pip install -r requirements.txt
```

Build the project:

cmake . -B build
cmake --build build --config RelWithDebInfo -j

Run a demo:
```
./build/testbed --scene data/nerf/fox
```

Note: Ensure you have a compatible CUDA-enabled GPU and the required CUDA toolkit installed before building and running the project.

Competitor Comparisons

mitsuba3

2,007

Mitsuba 3: A Retargetable Forward and Inverse Renderer

Pros of Mitsuba3

More comprehensive rendering framework with support for various rendering techniques
Highly extensible and customizable for research purposes
Better documentation and community support

Cons of Mitsuba3

Slower rendering speed compared to Instant-ngp's real-time capabilities
Steeper learning curve for beginners
Requires more computational resources for complex scenes

Code Comparison

Mitsuba3 scene definition:

import mitsuba as mi
mi.set_variant('cuda_ad_rgb')

scene = mi.load_file('scene.xml')
img = mi.render(scene)

Instant-ngp training:

import instant_ngp as ingp

trainer = ingp.Trainer()
trainer.load_dataset('dataset')
trainer.train()

Both projects offer powerful rendering capabilities, but Instant-ngp focuses on real-time neural graphics primitives, while Mitsuba3 provides a more comprehensive physically-based rendering framework. Instant-ngp excels in speed and simplicity for specific use cases, while Mitsuba3 offers greater flexibility and a wider range of rendering techniques at the cost of performance.

multinerf

3,604

A Code Release for Mip-NeRF 360, Ref-NeRF, and RawNeRF

Pros of mip-NeRF

Improved handling of multi-scale scenes and varying camera distances
Better performance on complex, large-scale environments
More robust to varying input image resolutions

Cons of mip-NeRF

Slower training and rendering times compared to Instant NGP
More complex implementation and potentially harder to set up
May require more computational resources for optimal performance

Code Comparison

instant-ngp:

__device__ inline void warp_activation(float& x) {
    x = min(max(x, 0.0f), 1.0f);
    // Smooth polynomial activation
    x = x * x * (3.0f - 2.0f * x);
}

mip-NeRF:

def integrated_pos_enc(x, min_deg, max_deg):
    scales = 2**torch.arange(min_deg, max_deg)
    shape = x.shape[:-1] + (-1,)
    y = torch.reshape(x[..., None, :] * scales[:, None], shape)
    return torch.sin(torch.cat([y, y + 0.5 * torch.pi], dim=-1))

The code snippets highlight the different approaches: Instant NGP uses CUDA for GPU acceleration and focuses on efficient activation functions, while mip-NeRF implements integrated positional encoding in Python, emphasizing its multi-scale capabilities.

pytorch3d

8,643

PyTorch3D is FAIR's library of reusable components for deep learning with 3D data

Pros of pytorch3d

Broader scope: Offers a comprehensive suite of 3D computer vision tools
Integration: Seamlessly works with PyTorch ecosystem
Flexibility: Supports various 3D data representations and operations

Cons of pytorch3d

Performance: Generally slower than instant-ngp for specific tasks like NeRF
Learning curve: More complex API due to its broader feature set
Resource intensity: Can be more memory-intensive for large-scale operations

Code Comparison

instant-ngp (CUDA-based):

__global__ void render_kernel(
    const uint32_t n_elements,
    const uint32_t n_rays,
    BoundingBox aabb,
    const uint32_t max_samples,
    float* __restrict__ rgb,
    float* __restrict__ depth,
    int* __restrict__ n_samples
) {
    // Kernel implementation
}

pytorch3d (Python-based):

def render_points(
    points,
    cameras,
    rasterizer,
    shader,
    device="cpu"
):
    # Rendering implementation

The code snippets highlight the difference in implementation languages and abstraction levels between the two projects. instant-ngp focuses on low-level CUDA optimization, while pytorch3d provides a higher-level Python API for 3D rendering tasks.

nerfies

1,627

This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies.

Pros of nerfies

Focuses on dynamic scenes and deformable objects
Handles non-rigid objects and moving subjects effectively
Provides a more comprehensive framework for capturing and rendering complex, real-world scenes

Cons of nerfies

Generally slower rendering and training times compared to instant-ngp
May require more computational resources for processing complex scenes
Less optimized for real-time applications and interactive rendering

Code Comparison

instant-ngp:

template <typename T>
__global__ void generate_grid_samples_nerf_uniform(
    const uint32_t n_elements,
    default_rng_t rng,
    const BoundingBox aabb,
    T* __restrict__ out_positions,
    float* __restrict__ out_distances
) {
    // ... (implementation details)
}

nerfies:

def render_rays(
    ray_batch,
    model,
    near,
    far,
    num_samples,
    rand=False,
    use_viewdirs=False,
    retraw=False,
    lindisp=False,
    perturb=0.,
    N_importance=0,
    white_bkgd=False,
    raw_noise_std=0.,
    verbose=False,
    pytest=False):
    # ... (implementation details)

The code snippets show different approaches: instant-ngp uses CUDA-accelerated C++ for performance, while nerfies employs Python for flexibility and ease of use in research contexts.

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Instant Neural Graphics Primitives

Ever wanted to train a NeRF model of a fox in under 5 seconds? Or fly around a scene captured from photos of a factory robot? Of course you have!

Here you will find an implementation of four neural graphics primitives, being neural radiance fields (NeRF), signed distance functions (SDFs), neural images, and neural volumes. In each case, we train and render a MLP with multiresolution hash input encoding using the tiny-cuda-nn framework.

Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
Thomas MÃ¼ller, Alex Evans, Christoph Schied, Alexander Keller
ACM Transactions on Graphics (SIGGRAPH), July 2022
Project page / Paper / Video / Presentation / Real-Time Live / BibTeX

For business inquiries, please submit the NVIDIA research licensing form.

Installation

If you have Windows, download one of the following releases corresponding to your graphics card and extract it. Then, start instant-ngp.exe.

Keep reading for a guided tour of the application or, if you are interested in creating your own NeRF, watch the video tutorial or read the written instructions.

If you use Linux, or want the developer Python bindings, or if your GPU is not listed above (e.g. Hopper, Volta, or Maxwell generations), you need to build instant-ngp yourself.

Usage

instant-ngp comes with an interactive GUI that includes many features:

comprehensive controls for interactively exploring neural graphics primitives,
VR mode for viewing neural graphics primitives through a virtual-reality headset,
saving and loading "snapshots" so you can share your graphics primitives on the internet,
a camera path editor to create videos,
NeRF->Mesh and SDF->Mesh conversion,
camera pose and lens optimization,
and many more.

NeRF fox

Simply start instant-ngp and drag the data/nerf/fox folder into the window. Or, alternatively, use the command line:

instant-ngp$ ./instant-ngp data/nerf/fox

You can use any NeRF-compatible dataset, e.g. from original NeRF, the SILVR dataset, or the DroneDeploy dataset. To create your own NeRF, watch the video tutorial or read the written instructions.

SDF armadillo

Drag data/sdf/armadillo.obj into the window or use the command:

instant-ngp$ ./instant-ngp data/sdf/armadillo.obj

Image of Einstein

Drag data/image/albert.exr into the window or use the command:

instant-ngp$ ./instant-ngp data/image/albert.exr

To reproduce the gigapixel results, download, for example, the Tokyo image and convert it to .bin using the scripts/convert_image.py script. This custom format improves compatibility and loading speed when resolution is high. Now you can run:

instant-ngp$ ./instant-ngp data/image/tokyo.bin

Volume renderer

Download the nanovdb volume for the Disney cloud, which is derived from here (CC BY-SA 3.0). Then drag wdas_cloud_quarter.nvdb into the window or use the command:

instant-ngp$ ./instant-ngp wdas_cloud_quarter.nvdb

Keyboard shortcuts and recommended controls

Here are the main keyboard controls for the instant-ngp application.

Key	Meaning
WASD	Forward / pan left / backward / pan right.
Spacebar / C	Move up / down.
= or + / - or _	Increase / decrease camera velocity (first person mode) or zoom in / out (third person mode).
E / Shift+E	Increase / decrease exposure.
Tab	Toggle menu visibility.
T	Toggle training. After around two minutes training tends to settle down, so can be toggled off.
{ }	Go to the first/last training image camera view.
[ ]	Go to the previous/next training image camera view.
R	Reload network from file.
Shift+R	Reset camera.
O	Toggle visualization or accumulated error map.
G	Toggle visualization of the ground truth.
M	Toggle multi-view visualization of layers of the neural model. See the paper's video for a little more explanation.
, / .	Shows the previous / next visualized layer; hit M to escape.
1-8	Switches among various render modes, with 2 being the standard one. You can see the list of render mode names in the control interface.

There are many controls in the instant-ngp GUI. First, note that this GUI can be moved and resized, as can the "Camera path" GUI (which first must be expanded to be used).

Recommended user controls in instant-ngp are:

Snapshot: use "Save" to save the trained NeRF, "Load" to reload.
Rendering -> DLSS: toggling this on and setting "DLSS sharpening" to 1.0 can often improve rendering quality.
Rendering -> Crop size: trim back the surrounding environment to focus on the model. "Crop aabb" lets you move the center of the volume of interest and fine tune. See more about this feature in our NeRF training & dataset tips.

The "Camera path" GUI lets you create a camera path for rendering a video. The button "Add from cam" inserts keyframes from the current perspective. Then, you can render a video .mp4 of your camera path or export the keyframes to a .json file. There is a bit more information about the GUI in this post and in this video guide to creating your own video.

VR controls

To view the neural graphics primitive in VR, first start your VR runtime. This will most likely be either

OculusVR if you have an Oculus Rift or Meta Quest (with link cable) headset, and
SteamVR if you have another headset.
Any OpenXR-compatible runtime will work.

Then, press the Connect to VR/AR headset button in the instant-ngp GUI and put on your headset. Before entering VR, we strongly recommend that you first finish training (press "Stop training") or load a pre-trained snapshot for maximum performance.

In VR, you have the following controls.

Control	Meaning
Left stick / trackpad	Move
Right stick / trackpad	Turn camera
Press stick / trackpad	Erase NeRF around the hand
Grab (one-handed)	Drag neural graphics primitive
Grab (two-handed)	Rotate and zoom (like pinch-to-zoom on a smartphone)

Building instant-ngp (Windows & Linux)

Requirements

An NVIDIA GPU; tensor cores increase performance when available. All shown results come from an RTX 3090.
A C++14 capable compiler. The following choices are recommended and have been tested:
- Windows: Visual Studio 2019 or 2022
- Linux: GCC/G++ 8 or higher
A recent version of CUDA. The following choices are recommended and have been tested:
- Windows: CUDA 11.5 or higher
- Linux: CUDA 10.2 or higher
CMake v3.21 or higher.
(optional) Python 3.7 or higher for interactive bindings. Also, run pip install -r requirements.txt.
(optional) OptiX 7.6 or higher for faster mesh SDF training.
(optional) Vulkan SDK for DLSS support.

If you are using Debian based Linux distribution, install the following packages

sudo apt-get install build-essential git python3-dev python3-pip libopenexr-dev libxi-dev \
                     libglfw3-dev libglew-dev libomp-dev libxinerama-dev libxcursor-dev

Alternatively, if you are using Arch or Arch derivatives, install the following packages

sudo pacman -S cuda base-devel cmake openexr libxi glfw openmp libxinerama libxcursor

We also recommend installing CUDA and OptiX in /usr/local/ and adding the CUDA installation to your PATH.

For example, if you have CUDA 11.4, add the following to your ~/.bashrc

export PATH="/usr/local/cuda-11.4/bin:$PATH"
export LD_LIBRARY_PATH="/usr/local/cuda-11.4/lib64:$LD_LIBRARY_PATH"

Compilation

Begin by cloning this repository and all its submodules using the following command:

$ git clone --recursive https://github.com/nvlabs/instant-ngp
$ cd instant-ngp

Then, use CMake to build the project: (on Windows, this must be in a developer command prompt)

instant-ngp$ cmake . -B build -DCMAKE_BUILD_TYPE=RelWithDebInfo
instant-ngp$ cmake --build build --config RelWithDebInfo -j

If compilation fails inexplicably or takes longer than an hour, you might be running out of memory. Try running the above command without -j in that case. If this does not help, please consult this list of possible fixes before opening an issue.

If the build succeeds, you can now run the code via the ./instant-ngp executable or the scripts/run.py script described below.

If automatic GPU architecture detection fails, (as can happen if you have multiple GPUs installed), set the TCNN_CUDA_ARCHITECTURES environment variable for the GPU you would like to use. The following table lists the values for common GPUs. If your GPU is not listed, consult this exhaustive list.

H100	40X0	30X0	A100	20X0	TITAN V / V100	10X0 / TITAN Xp	9X0	K80
90	89	86	80	75	70	61	52	37

Python bindings

After you have built instant-ngp, you can use its Python bindings to conduct controlled experiments in an automated fashion. All features from the interactive GUI (and more!) have Python bindings that can be easily instrumented. For an example of how the ./instant-ngp application can be implemented and extended from within Python, see ./scripts/run.py, which supports a superset of the command line arguments that ./instant-ngp does.

If you would rather build new models from the hash encoding and fast neural networks, consider tiny-cuda-nn's PyTorch extension.

Happy hacking!

Additional resources

Frequently asked questions (FAQ)

Q: The NeRF reconstruction of my custom dataset looks bad; what can I do?

A: There could be multiple issues:

COLMAP might have been unable to reconstruct camera poses.
There might have been movement or blur during capture. Don't treat capture as an artistic task; treat it as photogrammetry. You want *as little blur as possible* in your dataset (motion, defocus, or otherwise) and all objects must be *static* during the entire capture. Bonus points if you are using a wide-angle lens (iPhone wide angle works well), because it covers more space than narrow lenses.
The dataset parameters (in particular aabb_scale) might have been tuned suboptimally. We recommend starting with aabb_scale=128 and then increasing or decreasing it by factors of two until you get optimal quality.
Carefully read our NeRF training & dataset tips.

Q: How can I save the trained model and load it again later?

A: Two options:

Use the GUI's "Snapshot" section.
Use the Python bindings load_snapshot / save_snapshot (see scripts/run.py for example usage).

Q: Can this codebase use multiple GPUs at the same time?

A: Only for VR rendering, in which case one GPU is used per eye. Otherwise, no. To select a specific GPU to run on, use the CUDA_VISIBLE_DEVICES environment variable. To optimize the compilation for that specific GPU use the TCNN_CUDA_ARCHITECTURES environment variable.

Q: How can I run instant-ngp in headless mode?

A: Use ./instant-ngp --no-gui or python scripts/run.py. You can also compile without GUI via cmake -DNGP_BUILD_WITH_GUI=off ...

Q: Does this codebase run on Google Colab?

A: Yes. See this example inspired on the notebook created by user @myagues. Caveat: this codebase requires large amounts of GPU RAM and might not fit on your assigned GPU. It will also run slower on older GPUs.

Q: Is there a Docker container?

A: Yes. We bundle a Visual Studio Code development container, the .devcontainer/Dockerfile of which you can also use stand-alone.

If you want to run the container without using VSCode:

docker-compose -f .devcontainer/docker-compose.yml build instant-ngp
xhost local:root
docker-compose -f .devcontainer/docker-compose.yml run instant-ngp /bin/bash

Then run the build commands above as normal.

Q: How can I edit and train the underlying hash encoding or neural network on a new task?

A: Use tiny-cuda-nn's PyTorch extension.

Q: What is the coordinate system convention?

A: See this helpful diagram by user @jc211.

Q: Why are background colors randomized during NeRF training?

A: Transparency in the training data indicates a desire for transparency in the learned model. Using a solid background color, the model can minimize its loss by simply predicting that background color, rather than transparency (zero density). By randomizing the background colors, the model is forced to learn zero density to let the randomized colors "shine through".

Q: How to mask away NeRF training pixels (e.g. for dynamic object removal)?

A: For any training image xyz.* with dynamic objects, you can provide a dynamic_mask_xyz.png in the same folder. This file must be in PNG format, where non-zero pixel values indicate masked-away regions.

Troubleshooting compile errors

Before investigating further, make sure all submodules are up-to-date and try compiling again.

instant-ngp$ git submodule sync --recursive
instant-ngp$ git submodule update --init --recursive

If instant-ngp still fails to compile, update CUDA as well as your compiler to the latest versions you can install on your system. It is crucial that you update both, as newer CUDA versions are not always compatible with earlier compilers and vice versa. If your problem persists, consult the following table of known issues.

*After each step, delete the build folder and let CMake regenerate it before trying again.*

Problem	Resolution
CMake error: No CUDA toolset found / CUDA_ARCHITECTURES is empty for target "cmTC_0c70f"	Windows: the Visual Studio CUDA integration was not installed correctly. Follow these instructions to fix the problem without re-installing CUDA. (#18)
	Linux: Environment variables for your CUDA installation are probably incorrectly set. You may work around the issue using `cmake . -B build -DCMAKE_CUDA_COMPILER=/usr/local/cuda-<your cuda version>/bin/nvcc` (#28)
CMake error: No known features for CXX compiler "MSVC"	Reinstall Visual Studio & make sure you run CMake from a developer shell. Make sure you delete the build folder before building again. (#21)
Compile error: A single input file is required for a non-link phase when an outputfile is specified	Ensure there no spaces in the path to instant-ngp. Some build systems seem to have trouble with those. (#39 #198)
Compile error: undefined references to "cudaGraphExecUpdate" / identifier "cublasSetWorkspace" is undefined	Update your CUDA installation (which is likely 11.0) to 11.3 or higher. (#34 #41 #42)
Compile error: too few arguments in function call	Update submodules with the above two `git` commands. (#37 #52)
Python error: No module named 'pyngp'	It is likely that CMake did not detect your Python installation and therefore did not build `pyngp`. Check CMake logs to verify this. If `pyngp` was built in a different folder than `build`, Python will be unable to detect it and you have to supply the full path to the import statement. (#43)

If you cannot find your problem in the table, try searching the discussions board and the issues area for help. If you are still stuck, please open an issue and ask for help.

Thanks

Many thanks to Jonathan Tremblay and Andrew Tao for testing early versions of this codebase and to Arman Toorians and Saurabh Jain for the factory robot dataset. We also thank Andrew Webb for noticing that one of the prime numbers in the spatial hash was not actually prime; this has been fixed since.

This project makes use of a number of awesome open source libraries, including:

tiny-cuda-nn for fast CUDA networks and input encodings
tinyexr for EXR format support
tinyobjloader for OBJ format support
stb_image for PNG and JPEG support
Dear ImGui an excellent immediate mode GUI library
Eigen a C++ template library for linear algebra
pybind11 for seamless C++ / Python interop
and others! See the dependencies folder.

Many thanks to the authors of these brilliant projects!

License and Citation

@article{mueller2022instant,
    author = {Thomas M\"uller and Alex Evans and Christoph Schied and Alexander Keller},
    title = {Instant Neural Graphics Primitives with a Multiresolution Hash Encoding},
    journal = {ACM Trans. Graph.},
    issue_date = {July 2022},
    volume = {41},
    number = {4},
    month = jul,
    year = {2022},
    pages = {102:1--102:15},
    articleno = {102},
    numpages = {15},
    url = {https://doi.org/10.1145/3528223.3530127},
    doi = {10.1145/3528223.3530127},
    publisher = {ACM},
    address = {New York, NY, USA},
}

This work is made available under the Nvidia Source Code License-NC. Click here to view a copy of this license.

Top Related Projects

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot