rdma-core

RDMA core userspace libraries and daemons

1,484

676

1,484

Quick Overview

RDMA-core is a userspace library and drivers for Remote Direct Memory Access (RDMA) technology. It provides a unified, portable, and high-performance RDMA stack for Linux systems, supporting various RDMA-capable hardware and protocols such as InfiniBand, RoCE, and iWARP.

Pros

High performance: Enables low-latency, high-bandwidth data transfer between network nodes
Hardware-agnostic: Supports multiple RDMA-capable hardware and protocols
Actively maintained: Regular updates and contributions from the community and industry partners
Comprehensive documentation: Detailed guides and API references available

Cons

Complexity: RDMA concepts can be challenging for newcomers to understand and implement
Limited use cases: Primarily beneficial for high-performance computing and data center environments
Hardware requirements: Requires specialized RDMA-capable network hardware for optimal performance
Potential security concerns: Direct memory access can introduce security risks if not properly managed

Code Examples

Initializing an RDMA device:

#include <infiniband/verbs.h>

struct ibv_device **dev_list;
struct ibv_context *context;

dev_list = ibv_get_device_list(NULL);
context = ibv_open_device(dev_list[0]);
ibv_free_device_list(dev_list);

Creating a Queue Pair (QP):

struct ibv_qp_init_attr qp_init_attr = {
    .qp_type = IBV_QPT_RC,
    .sq_sig_all = 1,
    .send_cq = send_cq,
    .recv_cq = recv_cq,
    .cap = {
        .max_send_wr = 1,
        .max_recv_wr = 1,
        .max_send_sge = 1,
        .max_recv_sge = 1
    }
};

struct ibv_qp *qp = ibv_create_qp(pd, &qp_init_attr);

Posting a send request:

struct ibv_sge sg = {
    .addr = (uintptr_t)buffer,
    .length = buffer_size,
    .lkey = mr->lkey
};

struct ibv_send_wr wr = {
    .wr_id = 1,
    .sg_list = &sg,
    .num_sge = 1,
    .opcode = IBV_WR_SEND,
    .send_flags = IBV_SEND_SIGNALED
};

struct ibv_send_wr *bad_wr;
ibv_post_send(qp, &wr, &bad_wr);

Getting Started

To get started with rdma-core:

Install the library and development packages:

sudo apt-get install libibverbs-dev librdmacm-dev

Include the necessary headers in your C/C++ program:

#include <infiniband/verbs.h>
#include <rdma/rdma_cma.h>

Compile your program with the appropriate flags:

gcc -o your_program your_program.c -libverbs -lrdmacm

Run your program with proper permissions:
```
sudo ./your_program
```

Note: RDMA programming requires a deep understanding of the RDMA concepts and careful management of resources. It's recommended to refer to the official documentation and examples for more detailed guidance.

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

RDMA Core Userspace Libraries and Daemons

This is the userspace components for the Linux Kernel's drivers/infiniband subsystem. Specifically this contains the userspace libraries for the following device nodes:

/dev/infiniband/uverbsX (libibverbs)
/dev/infiniband/rdma_cm (librdmacm)
/dev/infiniband/umadX (libibumad)

The userspace component of the libibverbs RDMA kernel drivers are included under the providers/ directory. Support for the following Kernel RDMA drivers is included:

bnxt_re.ko
efa.ko
erdma.ko
iw_cxgb4.ko
hfi1.ko
hns-roce.ko
irdma.ko
ib_qib.ko
mana_ib.ko
mlx4_ib.ko
mlx5_ib.ko
ib_mthca.ko
ocrdma.ko
qedr.ko
rdma_rxe.ko
siw.ko
vmw_pvrdma.ko

Additional service daemons are provided for:

srp_daemon (ib_srp.ko)
iwpmd (for iwarp kernel providers)
ibacm (for InfiniBand communication management assistant)

Building

This project uses a cmake based build system. Quick start:

$ bash build.sh

build/bin will contain the sample programs and build/lib will contain the shared libraries. The build is configured to run all the programs 'in-place' and cannot be installed.

Debian Derived

$ apt-get install build-essential cmake gcc libudev-dev libnl-3-dev libnl-route-3-dev ninja-build pkg-config valgrind python3-dev cython3 python3-docutils pandoc

Supported releases:

Debian 9 (stretch) or newer
Ubuntu 16.04 LTS (xenial) or newer

Fedora, CentOS 8

$ dnf builddep redhat/rdma-core.spec

NOTE: Fedora Core uses the name 'ninja-build' for the 'ninja' command.

openSUSE

$ zypper install cmake gcc libnl3-devel libudev-devel ninja pkg-config valgrind-devel python3-devel python3-Cython python3-docutils pandoc

Building on CentOS 7, Amazon Linux 2

Install required packages:

$ yum install cmake gcc libnl3-devel libudev-devel make pkgconfig valgrind-devel

Developers on CentOS 7 or Amazon Linux 2 are suggested to install more modern tooling for the best experience.

CentOS 7:

$ yum install epel-release
$ yum install cmake3 ninja-build pandoc

Amazon Linux 2:

$ amazon-linux-extras install epel
$ yum install cmake3 ninja-build pandoc

NOTE: EPEL uses the name 'ninja-build' for the 'ninja' command, and 'cmake3' for the 'cmake' command.

Usage

To set up software RDMA on an existing interface with either of the available drivers, use the following commands, substituting <DRIVER> with the name of the driver of your choice (rdma_rxe or siw) and <TYPE> with the type corresponding to the driver (rxe or siw).

# modprobe <DRIVER>
# rdma link add <NAME> type <TYPE> netdev <DEVICE>

Please note that you need version of iproute2 recent enough is required for the command above to work.

You can use either ibv_devices or rdma link to verify that the device was successfully added.

Reporting bugs

Bugs should be reported to the linux-rdma@vger.kernel.org mailing list In your bug report, please include:

Information about your system:
- Linux distribution and version
- Linux kernel and version
- InfiniBand hardware and firmware version
- ... any other relevant information
How to reproduce the bug.
If the bug is a crash, the exact output printed out when the crash occurred, including any kernel messages produced.

Submitting patches

See Contributing to rdma-core.

Stable branches

Stable versions are released regularly with backported fixes (see Documentation/stable.md) The current minimum version still maintained is 'v30.X'

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot