cloud-hypervisor
A Virtual Machine Monitor for modern Cloud workloads. Features include CPU, memory and device hotplug, support for running Windows and Linux guests, device offload with vhost-user and a minimal compact footprint. Written in Rust with a strong focus on security.
Top Related Projects
Secure and fast microVMs for serverless computing.
Kata Containers is an open source project and community working to build a standard implementation of lightweight Virtual Machines (VMs) that feel and perform like containers, but provide the workload isolation and security advantages of VMs. https://katacontainers.io/
Ignite a Firecracker microVM
Podman: A tool for managing OCI containers and pods.
Podman: A tool for managing OCI containers and pods.
Quick Overview
Cloud Hypervisor is an open-source Virtual Machine Monitor (VMM) designed for running modern, cloud-workload optimized guest operating systems. It's written in Rust, focusing on security, speed, and simplicity while leveraging KVM on Linux systems.
Pros
- High performance due to its lightweight design and Rust implementation
- Strong focus on security, benefiting from Rust's memory safety features
- Optimized for cloud and container workloads
- Supports modern technologies like virtio-fs and vhost-user
Cons
- Limited guest OS support compared to more established hypervisors
- Relatively new project, which may lead to less community support and documentation
- Primarily focused on Linux hosts, with limited support for other platforms
- May lack some advanced features found in more mature hypervisors
Getting Started
To get started with Cloud Hypervisor:
- Install Rust and cargo:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
- Clone the repository:
git clone https://github.com/cloud-hypervisor/cloud-hypervisor.git
cd cloud-hypervisor
- Build Cloud Hypervisor:
cargo build --release
- Run a basic VM (assuming you have a kernel and rootfs):
./target/release/cloud-hypervisor \
--kernel path/to/vmlinux \
--disk path/to/rootfs \
--cpus 4 \
--memory size=512M \
--net "tap=,mac=,ip=,mask="
For more detailed instructions and advanced usage, refer to the project's documentation on GitHub.
Competitor Comparisons
Secure and fast microVMs for serverless computing.
Pros of Firecracker
- Extremely lightweight and fast boot times (as low as 125ms)
- Strong isolation and security features, leveraging KVM
- Optimized for serverless and container workloads
Cons of Firecracker
- Limited device support and customization options
- Primarily focused on Linux guests, less flexible for other OSes
- Steeper learning curve for integration and management
Code Comparison
Firecracker configuration (YAML):
boot-source:
kernel_image_path: "/path/to/kernel"
boot_args: "console=ttyS0 reboot=k panic=1 pci=off"
drives:
- drive_id: "rootfs"
path_on_host: "/path/to/rootfs"
is_root_device: true
is_read_only: false
Cloud-Hypervisor configuration (JSON):
{
"kernel": "/path/to/kernel",
"cmdline": "console=ttyS0 reboot=k panic=1",
"disk": [{"path": "/path/to/rootfs"}],
"cpus": {"boot_vcpus": 1, "max_vcpus": 1},
"memory": {"size": 512}
}
Both projects aim to provide lightweight virtualization solutions, but Firecracker focuses on serverless workloads with minimal overhead, while Cloud-Hypervisor offers more flexibility and device support for general-purpose VMs.
Kata Containers is an open source project and community working to build a standard implementation of lightweight Virtual Machines (VMs) that feel and perform like containers, but provide the workload isolation and security advantages of VMs. https://katacontainers.io/
Pros of Kata Containers
- Provides stronger isolation and security by running containers in lightweight VMs
- Supports multiple hypervisors and architectures, offering greater flexibility
- Offers seamless integration with container ecosystems like Kubernetes
Cons of Kata Containers
- Higher resource overhead compared to Cloud Hypervisor's lightweight approach
- More complex setup and configuration due to additional components
- Potentially slower container startup times
Code Comparison
Kata Containers configuration example:
hypervisor:
path: "/usr/bin/qemu-system-x86_64"
kernel: "/usr/share/kata-containers/vmlinux.container"
image: "/usr/share/kata-containers/kata-containers.img"
Cloud Hypervisor configuration example:
let vm_config = VmConfig {
cpus: CpusConfig { boot_vcpus: 4, max_vcpus: 8 },
memory: MemoryConfig { size: 1024 * 1024 * 1024 },
kernel: Some(KernelConfig { path: String::from("/path/to/kernel") }),
// ...
};
Both projects aim to provide lightweight virtualization solutions, but Kata Containers focuses on secure container isolation using VMs, while Cloud Hypervisor emphasizes a minimalist, high-performance approach to VM management. The choice between them depends on specific use cases and requirements.
Ignite a Firecracker microVM
Pros of Ignite
- Focuses on running containerized applications in VMs, providing better isolation than traditional containers
- Utilizes Firecracker for fast and secure VM provisioning
- Integrates well with Kubernetes ecosystem and container workflows
Cons of Ignite
- Limited to running on Linux hosts only
- Requires specific kernel configurations and may not work on all Linux distributions
- Less flexible than Cloud Hypervisor for general-purpose virtualization tasks
Code Comparison
Ignite (using Docker-like syntax):
ignite run weaveworks/ignite-ubuntu \
--cpus 2 \
--memory 1GB \
--size 6GB \
--ssh
Cloud Hypervisor (using command-line options):
cloud-hypervisor \
--cpus boot=2 \
--memory size=1G \
--disk path=/path/to/disk.img \
--net tap=tap0
Both projects aim to provide lightweight virtualization solutions, but they target different use cases. Ignite focuses on running containerized applications in VMs, while Cloud Hypervisor is a more general-purpose VMM. The code examples show the different approaches to VM creation and configuration, with Ignite using a Docker-like syntax and Cloud Hypervisor using command-line options.
Podman: A tool for managing OCI containers and pods.
Pros of Podman
- Daemonless architecture, improving security and resource usage
- Compatible with Docker commands and images, easing adoption
- Rootless containers support, enhancing security and user isolation
Cons of Podman
- Less mature ecosystem compared to Docker
- Potentially slower performance for some operations due to daemonless design
Code Comparison
Cloud-Hypervisor (Rust):
let vmm = Vmm::new(vm_params)?;
vmm.boot()?;
Podman (Go):
cmd := exec.Command("podman", "run", "-it", "ubuntu")
err := cmd.Run()
Key Differences
- Cloud-Hypervisor is a lightweight VMM for cloud workloads, while Podman is a container engine
- Cloud-Hypervisor focuses on virtualization, Podman on containerization
- Cloud-Hypervisor is written in Rust, Podman in Go
- Cloud-Hypervisor targets cloud and edge computing, Podman aims for general-purpose container management
Use Cases
Cloud-Hypervisor:
- Cloud-native and serverless deployments
- Edge computing with minimal overhead
Podman:
- Local development environments
- CI/CD pipelines
- Production container orchestration (with Kubernetes)
Both projects aim to provide efficient, secure solutions for running isolated workloads, but they approach this goal from different perspectives: virtualization vs. containerization.
Podman: A tool for managing OCI containers and pods.
Pros of Podman
- Daemonless architecture, improving security and resource usage
- Compatible with Docker commands and images, easing adoption
- Rootless containers support, enhancing security and user isolation
Cons of Podman
- Less mature ecosystem compared to Docker
- Potentially slower performance for some operations due to daemonless design
Code Comparison
Cloud-Hypervisor (Rust):
let vmm = Vmm::new(vm_params)?;
vmm.boot()?;
Podman (Go):
cmd := exec.Command("podman", "run", "-it", "ubuntu")
err := cmd.Run()
Key Differences
- Cloud-Hypervisor is a lightweight VMM for cloud workloads, while Podman is a container engine
- Cloud-Hypervisor focuses on virtualization, Podman on containerization
- Cloud-Hypervisor is written in Rust, Podman in Go
- Cloud-Hypervisor targets cloud and edge computing, Podman aims for general-purpose container management
Use Cases
Cloud-Hypervisor:
- Cloud-native and serverless deployments
- Edge computing with minimal overhead
Podman:
- Local development environments
- CI/CD pipelines
- Production container orchestration (with Kubernetes)
Both projects aim to provide efficient, secure solutions for running isolated workloads, but they approach this goal from different perspectives: virtualization vs. containerization.
Convert 
designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual CopilotREADME
- 1. What is Cloud Hypervisor?
- 2. Getting Started
- 3. Status
- 4. Relationship with Rust VMM Project
- 5. Community
1. What is Cloud Hypervisor?
Cloud Hypervisor is an open source Virtual Machine Monitor (VMM) that runs on top of the KVM hypervisor and the Microsoft Hypervisor (MSHV).
The project focuses on running modern, Cloud Workloads, on specific, common, hardware architectures. In this case Cloud Workloads refers to those that are run by customers inside a Cloud Service Provider. This means modern operating systems with most I/O handled by paravirtualised devices (e.g. virtio), no requirement for legacy devices, and 64-bit CPUs.
Cloud Hypervisor is implemented in Rust and is based on the Rust VMM crates.
Objectives
High Level
- Runs on KVM or MSHV
- Minimal emulation
- Low latency
- Low memory footprint
- Low complexity
- High performance
- Small attack surface
- 64-bit support only
- CPU, memory, PCI hotplug
- Machine to machine migration
Architectures
Cloud Hypervisor supports the x86-64 and AArch64 architectures. There are
minor differences in functionality between the two architectures
(see #1125).
Guest OS
Cloud Hypervisor supports 64-bit Linux and Windows 10/Windows Server 2019.
2. Getting Started
The following sections describe how to build and run Cloud Hypervisor.
Prerequisites for AArch64
- AArch64 servers (recommended) or development boards equipped with the GICv3 interrupt controller.
Host OS
For required KVM functionality and adequate performance the recommended host kernel version is 5.13. The majority of the CI currently tests with kernel version 5.15.
Use Pre-built Binaries
The recommended approach to getting started with Cloud Hypervisor is by using a
pre-built binary. Binaries are available for the latest
release.
Use cloud-hypervisor-static for x86-64 or cloud-hypervisor-static-aarch64
for AArch64 platform.
Packages
For convenience, packages are also available targeting some popular Linux distributions. This is thanks to the Open Build Service. The OBS README explains how to enable the repository in a supported Linux distribution and install Cloud Hypervisor and accompanying packages. Please report any packaging issues in the obs-packaging repository.
Building from Source
Please see the instructions for building from source if you do not wish to use the pre-built binaries.
Booting Linux
Cloud Hypervisor supports direct kernel boot (the x86-64 kernel requires the kernel
built with PVH support or a bzImage) or booting via a firmware (either Rust Hypervisor
Firmware or an
edk2 UEFI firmware called CLOUDHV / CLOUDHV_EFI.)
Binary builds of the firmware files are available for the latest release of Rust Hypervisor Firmware and our edk2 repository
The choice of firmware depends on your guest OS choice; some experimentation may be required.
Firmware Booting
Cloud Hypervisor supports booting disk images containing all needed components to run cloud workloads, a.k.a. cloud images.
The following sample commands will download an Ubuntu Cloud image, converting it into a format that Cloud Hypervisor can use and a firmware to boot the image with.
$ wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img
$ qemu-img convert -p -f qcow2 -O raw focal-server-cloudimg-amd64.img focal-server-cloudimg-amd64.raw
$ wget https://github.com/cloud-hypervisor/rust-hypervisor-firmware/releases/download/0.4.2/hypervisor-fw
The Ubuntu cloud images do not ship with a default password so it necessary to
use a cloud-init disk image to customise the image on the first boot. A basic
cloud-init image is generated by this script.
This seeds the image with a default username/password of cloud/cloud123. It
is only necessary to add this disk image on the first boot. Script also assigns
default IP address using test_data/cloud-init/ubuntu/local/network-config details
with --net "mac=12:34:56:78:90:ab,tap=" option. Then the matching mac address
interface will be enabled as per network-config details.
$ sudo setcap cap_net_admin+ep ./cloud-hypervisor
$ ./create-cloud-init.sh
$ ./cloud-hypervisor \
--kernel ./hypervisor-fw \
--disk path=focal-server-cloudimg-amd64.raw path=/tmp/ubuntu-cloudinit.img \
--cpus boot=4 \
--memory size=1024M \
--net "tap=,mac=,ip=,mask="
If access to the firmware messages or interaction with the boot loader (e.g.
GRUB) is required then it necessary to switch to the serial console instead of
virtio-console.
$ ./cloud-hypervisor \
--kernel ./hypervisor-fw \
--disk path=focal-server-cloudimg-amd64.raw path=/tmp/ubuntu-cloudinit.img \
--cpus boot=4 \
--memory size=1024M \
--net "tap=,mac=,ip=,mask=" \
--serial tty \
--console off
Custom Kernel and Disk Image
Building your Kernel
Cloud Hypervisor also supports direct kernel boot. For x86-64, a vmlinux ELF kernel (compiled with PVH support) or a regular bzImage are supported. In order to support development there is a custom branch; however provided the required options are enabled any recent kernel will suffice.
To build the kernel:
# Clone the Cloud Hypervisor Linux branch
$ git clone --depth 1 https://github.com/cloud-hypervisor/linux.git -b ch-6.2 linux-cloud-hypervisor
$ pushd linux-cloud-hypervisor
# Use the x86-64 cloud-hypervisor kernel config to build your kernel for x86-64
$ wget https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/main/resources/linux-config-x86_64
# Use the AArch64 cloud-hypervisor kernel config to build your kernel for AArch64
$ wget https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/main/resources/linux-config-aarch64
$ cp linux-config-x86_64 .config # x86-64
$ cp linux-config-aarch64 .config # AArch64
# Do native build of the x86-64 kernel
$ KCFLAGS="-Wa,-mx86-used-note=no" make bzImage -j `nproc`
# Do native build of the AArch64 kernel
$ make -j `nproc`
$ popd
For x86-64, the vmlinux kernel image will then be located at
linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin.
For AArch64, the Image kernel image will then be located at
linux-cloud-hypervisor/arch/arm64/boot/Image.
Disk image
For the disk image the same Ubuntu image as before can be used. This contains
an ext4 root filesystem.
$ wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img # x86-64
$ wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-arm64.img # AArch64
$ qemu-img convert -p -f qcow2 -O raw focal-server-cloudimg-amd64.img focal-server-cloudimg-amd64.raw # x86-64
$ qemu-img convert -p -f qcow2 -O raw focal-server-cloudimg-arm64.img focal-server-cloudimg-arm64.raw # AArch64
Booting the guest VM
These sample commands boot the disk image using the custom kernel whilst also supplying the desired kernel command line.
- x86-64
$ sudo setcap cap_net_admin+ep ./cloud-hypervisor
$ ./create-cloud-init.sh
$ ./cloud-hypervisor \
--kernel ./linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin \
--disk path=focal-server-cloudimg-amd64.raw path=/tmp/ubuntu-cloudinit.img \
--cmdline "console=hvc0 root=/dev/vda1 rw" \
--cpus boot=4 \
--memory size=1024M \
--net "tap=,mac=,ip=,mask="
- AArch64
$ sudo setcap cap_net_admin+ep ./cloud-hypervisor
$ ./create-cloud-init.sh
$ ./cloud-hypervisor \
--kernel ./linux-cloud-hypervisor/arch/arm64/boot/Image \
--disk path=focal-server-cloudimg-arm64.raw path=/tmp/ubuntu-cloudinit.img \
--cmdline "console=hvc0 root=/dev/vda1 rw" \
--cpus boot=4 \
--memory size=1024M \
--net "tap=,mac=,ip=,mask="
If earlier kernel messages are required the serial console should be used instead of virtio-console.
- x86-64
$ ./cloud-hypervisor \
--kernel ./linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin \
--console off \
--serial tty \
--disk path=focal-server-cloudimg-amd64.raw \
--cmdline "console=ttyS0 root=/dev/vda1 rw" \
--cpus boot=4 \
--memory size=1024M \
--net "tap=,mac=,ip=,mask="
- AArch64
$ ./cloud-hypervisor \
--kernel ./linux-cloud-hypervisor/arch/arm64/boot/Image \
--console off \
--serial tty \
--disk path=focal-server-cloudimg-arm64.raw \
--cmdline "console=ttyAMA0 root=/dev/vda1 rw" \
--cpus boot=4 \
--memory size=1024M \
--net "tap=,mac=,ip=,mask="
3. Status
Cloud Hypervisor is under active development. The following stability guarantees are currently made:
-
The API (including command line options) will not be removed or changed in a breaking way without a minimum of 2 major releases notice. Where possible warnings will be given about the use of deprecated functionality and the deprecations will be documented in the release notes.
-
Point releases will be made between individual releases where there are substantial bug fixes or security issues that need to be fixed. These point releases will only include bug fixes.
Currently the following items are not guaranteed across updates:
- Snapshot/restore is not supported across different versions
- Live migration is not supported across different versions
- The following features are considered experimental and may change substantially between releases: TDX, vfio-user, vDPA.
Further details can be found in the release documentation.
As of 2023-01-03, the following cloud images are supported:
- Ubuntu Focal (focal-server-cloudimg-{amd64,arm64}.img)
- Ubuntu Jammy (jammy-server-cloudimg-{amd64,arm64}.img )
- Fedora 36 (Fedora-Cloud-Base-36-1.5.x86_64.raw.xz / Fedora-Cloud-Base-36-1.5.aarch64.raw.xz)
Direct kernel boot to userspace should work with a rootfs from most distributions although you may need to enable exotic filesystem types in the reference kernel configuration (e.g. XFS or btrfs.)
Hot Plug
Cloud Hypervisor supports hotplug of CPUs, passthrough devices (VFIO),
virtio-{net,block,pmem,fs,vsock} and memory resizing. This
document details how to add devices to a running VM.
Device Model
Details of the device model can be found in this documentation.
Roadmap
The project roadmap is tracked through a GitHub project.
4. Relationship with Rust VMM Project
In order to satisfy the design goal of having a high-performance, security-focused hypervisor the decision was made to use the Rust programming language. The language's strong focus on memory and thread safety makes it an ideal candidate for implementing VMMs.
Instead of implementing the VMM components from scratch, Cloud Hypervisor is importing the Rust VMM crates, and sharing code and architecture together with other VMMs like e.g. Amazon's Firecracker and Google's crosvm.
Cloud Hypervisor embraces the Rust VMM project's goals, which is to be able to share and re-use as many virtualization crates as possible.
Differences with Firecracker and crosvm
A large part of the Cloud Hypervisor code is based on either the Firecracker or the crosvm project's implementations. Both of these are VMMs written in Rust with a focus on safety and security, like Cloud Hypervisor.
The goal of the Cloud Hypervisor project differs from the aforementioned projects in that it aims to be a general purpose VMM for Cloud Workloads and not limited to container/serverless or client workloads.
The Cloud Hypervisor community thanks the communities of both the Firecracker and crosvm projects for their excellent work.
5. Community
The Cloud Hypervisor project follows the governance, and community guidelines described in the Community repository.
Contribute
The project strongly believes in building a global, diverse and collaborative community around the Cloud Hypervisor project. Anyone who is interested in contributing to the project is welcome to participate.
Contributing to a open source project like Cloud Hypervisor covers a lot more than just sending code. Testing, documentation, pull request reviews, bug reports, feature requests, project improvement suggestions, etc, are all equal and welcome means of contribution. See the CONTRIBUTING document for more details.
Slack
Get an invite to our Slack channel, join us on Slack, and participate in our community activities.
Mailing list
Please report bugs using the GitHub issue tracker but for broader community discussions you may use our mailing list.
Security issues
Please contact the maintainers listed in the MAINTAINERS.md file with security issues.
Top Related Projects
Secure and fast microVMs for serverless computing.
Kata Containers is an open source project and community working to build a standard implementation of lightweight Virtual Machines (VMs) that feel and perform like containers, but provide the workload isolation and security advantages of VMs. https://katacontainers.io/
Ignite a Firecracker microVM
Podman: A tool for managing OCI containers and pods.
Podman: A tool for managing OCI containers and pods.
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual Copilot