foundationdb

FoundationDB - the open source, distributed, transactional key-value store

15,555

1,404

15,555

850

View on GitHub

Top Related Projects

cockroach

31,141

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

etcd

49,955

Distributed reliable key-value store for the most critical data of a distributed system

tidb

38,817

TiDB - the open-source, cloud-native, distributed SQL database designed for modern applications.

🥑 ArangoDB is a native multi-model database with flexible data models for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.

dgraph

21,062

high-performance graph database for real-time use cases

cassandra

9,303

Apache Cassandra®

Quick Overview

FoundationDB is a distributed database designed to handle large volumes of structured data across clusters of commodity servers. It offers a robust, scalable, and transactional key-value store that combines NoSQL's scalability with the ACID guarantees of traditional databases.

Pros

High scalability and performance for distributed systems
Strong consistency and ACID transactions
Multi-model support (key-value, document, graph, etc.)
Language-agnostic with bindings for multiple programming languages

Cons

Steep learning curve for newcomers to distributed systems
Limited built-in support for complex queries compared to traditional SQL databases
Requires careful configuration and tuning for optimal performance
Smaller community compared to some other popular databases

Code Examples

Basic key-value operations:

import fdb
fdb.api_version(630)
db = fdb.open()

@fdb.transactional
def set_value(tr, key, value):
    tr[key] = value

@fdb.transactional
def get_value(tr, key):
    return tr[key]

set_value(db, b'hello', b'world')
result = get_value(db, b'hello')
print(result)  # Output: b'world'

Using atomic operations:

import fdb
fdb.api_version(630)
db = fdb.open()

@fdb.transactional
def increment_counter(tr, key):
    tr.add(key, struct.pack('<q', 1))

@fdb.transactional
def get_counter(tr, key):
    return struct.unpack('<q', tr[key])[0]

increment_counter(db, b'counter')
increment_counter(db, b'counter')
count = get_counter(db, b'counter')
print(count)  # Output: 2

Working with directories:

import fdb
import fdb.directory
fdb.api_version(630)
db = fdb.open()

@fdb.transactional
def create_and_use_directory(tr):
    dir = fdb.directory.create_or_open(tr, ('my_app', 'users'))
    dir['alice'] = b'data for alice'
    return dir['alice']

result = create_and_use_directory(db)
print(result)  # Output: b'data for alice'

Getting Started

Install FoundationDB server and client libraries for your platform.
Install the Python client:
```
pip install foundationdb
```

Create a basic script:

import fdb
fdb.api_version(630)
db = fdb.open()

@fdb.transactional
def hello_world(tr):
    tr[b'hello'] = b'world'
    return tr[b'hello']

result = hello_world(db)
print(result)  # Output: b'world'

Run the script to interact with FoundationDB.

Competitor Comparisons

cockroach

31,141

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

Pros of CockroachDB

Designed for global scale and geographic distribution
SQL-compatible with strong consistency guarantees
Built-in multi-active availability for high resilience

Cons of CockroachDB

Higher resource consumption and operational complexity
Potentially slower performance for certain workloads
Steeper learning curve for optimization and tuning

Code Comparison

CockroachDB (SQL-based):

CREATE TABLE users (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  name STRING,
  created_at TIMESTAMP DEFAULT current_timestamp()
);

FoundationDB (Key-Value based):

@fdb.transactional
def create_user(tr, name):
    user_id = uuid.uuid4()
    tr[f'users/{user_id}/name'] = name
    tr[f'users/{user_id}/created_at'] = time.time()
    return user_id

CockroachDB offers a familiar SQL interface, making it easier for developers with SQL experience. FoundationDB provides a low-level key-value API, offering more flexibility but requiring more custom logic for data modeling.

Both databases focus on scalability and consistency, but CockroachDB is specifically designed for distributed SQL workloads, while FoundationDB serves as a foundation for building various distributed systems.

etcd

49,955

Distributed reliable key-value store for the most critical data of a distributed system

Pros of etcd

Simpler architecture and easier to set up for small to medium-scale distributed systems
Built-in support for distributed consensus using the Raft algorithm
Widely adopted in the Kubernetes ecosystem, making it a natural choice for cloud-native applications

Cons of etcd

Limited scalability compared to FoundationDB, especially for large-scale deployments
Less flexible data model, primarily focused on key-value storage
Lower overall performance in terms of throughput and latency for complex operations

Code Comparison

etcd (Go):

cli, _ := clientv3.New(clientv3.Config{Endpoints: []string{"localhost:2379"}})
defer cli.Close()
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
_, err := cli.Put(ctx, "key", "value")
cancel()

FoundationDB (C++):

Database db = Database::createDatabase("fdb.cluster");
Transaction tr(db);
tr.set(KeyRef("key"), ValueRef("value"));
tr.commit().wait();

Both examples demonstrate basic key-value operations, but FoundationDB's API is more low-level and offers finer control over transactions.

tidb

38,817

TiDB - the open-source, cloud-native, distributed SQL database designed for modern applications.

Pros of TiDB

Designed for distributed HTAP (Hybrid Transactional/Analytical Processing) workloads
SQL-compatible and MySQL protocol support, easier migration for existing MySQL users
Built-in horizontal scalability and high availability features

Cons of TiDB

Higher complexity due to multiple components (TiKV, PD, TiDB)
Potentially higher resource requirements for small-scale deployments
Steeper learning curve for administration and optimization

Code Comparison

TiDB (SQL-based):

CREATE TABLE users (
  id INT PRIMARY KEY,
  name VARCHAR(50),
  age INT
);
INSERT INTO users VALUES (1, 'Alice', 30);
SELECT * FROM users WHERE age > 25;

FoundationDB (Key-Value based):

@fdb.transactional
def create_user(tr, user_id, name, age):
    tr[f'users:{user_id}:name'] = name
    tr[f'users:{user_id}:age'] = str(age)

@fdb.transactional
def get_users_over_25(tr):
    return [k.split(':')[1] for k, v in tr.get_range_startswith('users:')
            if k.endswith(':age') and int(v) > 25]

FoundationDB offers a low-level key-value API, while TiDB provides a SQL interface. FoundationDB requires more custom code for data modeling and querying, but offers fine-grained control over data storage and retrieval. TiDB's SQL interface is more familiar and easier to use for traditional database users, but may have less flexibility for certain use cases.

arangodb

13,878

Pros of ArangoDB

Multi-model database supporting key/value, document, and graph data models
Flexible query language (AQL) for complex data operations
Built-in web interface for administration and data exploration

Cons of ArangoDB

Higher memory usage compared to FoundationDB
Less focus on extreme scalability and distributed systems
Steeper learning curve due to multiple data models and query language

Code Comparison

FoundationDB (C++):

Transaction tr(db);
tr.set(KeyRef("hello"), ValueRef("world"));
tr.commit().wait();

ArangoDB (JavaScript):

db.collection('myCollection').save({ hello: 'world' });

Key Differences

FoundationDB is a distributed key-value store, while ArangoDB is a multi-model database
FoundationDB emphasizes scalability and ACID transactions, ArangoDB focuses on flexibility
FoundationDB uses a lower-level API, ArangoDB provides higher-level abstractions
FoundationDB is better suited for large-scale distributed systems, ArangoDB for diverse data modeling needs

Both databases have their strengths and are suitable for different use cases. FoundationDB excels in highly scalable, transactional environments, while ArangoDB offers more versatility in data modeling and querying.

dgraph

21,062

high-performance graph database for real-time use cases

Pros of Dgraph

Native support for GraphQL, making it easier to work with graph data
Designed for distributed environments, offering better scalability
Includes built-in search capabilities and full-text indexing

Cons of Dgraph

Less mature ecosystem compared to FoundationDB
May have a steeper learning curve for developers unfamiliar with graph databases
Limited support for ACID transactions across multiple shards

Code Comparison

FoundationDB (using Python client):

@fdb.transactional
def add_user(tr, user_id, name):
    tr[f'users/{user_id}/name'] = name

fdb.open()
add_user(db, '123', 'John Doe')

Dgraph (using Go client):

func addUser(c *dgo.Dgraph, userID, name string) error {
    mu := &api.Mutation{
        SetNquads: []byte(`
            _:user <user_id> "` + userID + `" .
            _:user <name> "` + name + `" .
        `),
    }
    _, err := c.NewTxn().Mutate(context.Background(), mu)
    return err
}

Both examples demonstrate adding a user to the database, showcasing the different approaches and query languages used by each system.

cassandra

9,303

Apache Cassandra®

Pros of Cassandra

Highly scalable and distributed architecture, designed for large-scale deployments
Strong support for multi-datacenter replication and high availability
Flexible data model with support for wide-column storage

Cons of Cassandra

Complex setup and maintenance, requiring significant operational expertise
Limited support for ACID transactions compared to FoundationDB
Can be resource-intensive, especially for smaller deployments

Code Comparison

Cassandra (CQL):

CREATE TABLE users (
  id UUID PRIMARY KEY,
  name TEXT,
  email TEXT
);

FoundationDB (Python API):

@fdb.transactional
def create_user(tr, id, name, email):
    tr[f'users/{id}/name'] = name
    tr[f'users/{id}/email'] = email

FoundationDB uses a key-value approach, while Cassandra provides a more structured table-based model. FoundationDB's transactional nature is evident in the code example, showcasing its strong consistency guarantees.

Both databases have their strengths, with Cassandra excelling in large-scale distributed scenarios and FoundationDB offering robust ACID compliance. The choice between them depends on specific use cases, scalability requirements, and consistency needs.

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Build Status

FoundationDB is a distributed database designed to handle large volumes of structured data across clusters of commodity servers. It organizes data as an ordered key-value store and employs ACID transactions for all operations. It is especially well-suited for read/write workloads, but also has excellent performance for write-intensive workloads. Users interact with the database using API language binding.

To learn more about FoundationDB, visit foundationdb.org

Documentation

Documentation can be found online at https://apple.github.io/foundationdb/. The documentation covers details of API usage, background information on design philosophy, and extensive usage examples. Docs are built from the source in this repo.

Forums

The FoundationDB Forums are the home for most of the discussion and communication about the FoundationDB project. We welcome your participation! We want FoundationDB to be a great project to be a part of, and as part of that, we have established a Code of Conduct to define what constitutes permissible modes of interaction.

Contributing

Contributing to FoundationDB can be in contributions to the codebase, sharing your experience and insights in the community on the Forums, or contributing to projects that make use of FoundationDB. Please see the contributing guide for more specifics.

Getting Started

Latest Stable Releases

The latest stable releases are (were) versions that are recommended for production use, which have been extensively validated via simulation and real cluster tests and used in our production environment.

Branch	Latest Production Release	Notes
7.3	7.3.69	Supported
7.2		Experimental
7.1	7.1.57	Bug fixes
7.0		Experimental
6.3	6.3.25	Unsupported

Supported branches are those we actively maintain and will publish new patch releases.
Bug fixes are branches where we still accept bug fixes, but may not publish newer patch releases. The community can build the latest release binaries if needed and is encouraged to upgrade to the Supported branches.
Experimental branches are those used for internal feature testing. They are not recommended for production use.
Unsupported branches are those that will no longer receive any updates.

If you are running on old production releases, we recommend always upgrading to the next major release's latest version, and then continuing to the next major version, e.g., 6.2.X -> 6.3.25 -> 7.1.57 -> 7.3.69. These upgrade paths have been well tested in production (skipping a major release, not marked as Experimental, for an upgrade is only tested in simulation).

Binary Downloads

Developers interested in using FoundationDB can get started by downloading and installing a binary package. Please see the downloads page for a list of available packages.

Compiling from source

Developers on an OS for which there is no binary package, or who would like to start hacking on the code, can get started by compiling from source.

NOTE: FoundationDB has a lot of dependencies. The Docker container listed below tracks them and is what we use internally and is the recommended method of building FDB.

Build Using the Official Docker Image

The official Docker image for building is foundationdb/build, which includes all necessary dependencies. The Docker image definitions used by FoundationDB team members can be found in the dedicated repository.

To build FoundationDB with the clang toolchain,

mkdir /some/build_output_dir
cd /some/build_output_dir
CC=clang CXX=clang++ LD=lld cmake -D USE_LD=LLD -D USE_LIBCXX=1 -G Ninja /some/fdb/source_dir
ninja

To use GCC, a non-default version is necessary. The following modifies environment variables ($PATH, $LD_LIBRARY_PATH, etc) to pick up the right GCC version:

source /opt/rh/gcc-toolset-13/enable
gcc --version  # should say 13
mkdir /some/build_output_dir
cd /some/build_output_dir
cmake -G Ninja /some/fdb/source_dir
ninja

Slightly more elaborate compile commands can be found in the shell aliases defined in /root/.bashrc in the container image.

Build Locally

To build outside of the official Docker image, you'll need at least these dependencies:

CMake version 3.24.2 or higher
Mono
ninja

This list is likely to be incomplete. Refer to the rockylinux9 Dockerfile in the fdb-build-support repo linked above for reference material on specific packages and versions that are likely to be required.

If compiling for local development, please set -DUSE_WERROR=ON in CMake. Our CI compiles with -Werror on, so this way you'll find out about compiler warnings that break the build earlier.

Once you have your dependencies, you can run cmake and then build:

Check out this repository.
Create a build directory (you can place it anywhere you like).
cd <FDB_BUILD_DIR>
cmake -G Ninja <FDB_SOURCE_DIR>
ninja

Building FoundationDB requires at least 8GB of memory. More memory is needed when building in parallel. If the computer freezes or crashes, consider disabling parallelized build using ninja -j1.

FreeBSD

Check out this repo on your server.
Install compile-time dependencies from ports.
(Optional) Use tmpfs & ccache for significantly faster repeat builds
(Optional) Install a JDK for Java Bindings. FoundationDB currently builds with Java 8.
Navigate to the directory where you checked out the FoundationDB repository.

Build from source.

sudo pkg install -r FreeBSD \
    shells/bash devel/cmake devel/ninja devel/ccache  \
    lang/mono lang/python3 \
    devel/boost-libs devel/libeio \
    security/openssl
mkdir .build && cd .build
cmake -G Ninja \
    -DUSE_CCACHE=on \
    -DUSE_DTRACE=off \
    ..
ninja -j 10
# run fast tests
ctest -L fast
# run all tests
ctest --output-on-failure -v

macOS

The build under macOS will work the same way as on Linux. Homebrew can be used to install the boost library and the ninja build tool.

cmake -G Ninja <FDB_SOURCE_DIR>
ninja

To generate an installable package,

<FDB_SOURCE_DIR>/packaging/osx/buildpkg.sh . <FDB_SOURCE_DIR>

Windows

Under Windows, only Visual Studio with ClangCl is supported

Install Visual Studio 2019 (IDE or Build Tools), and enable LLVM support
Install CMake 3.24.2 or higher
Download Boost 1.86.0
Unpack boost to C:\boost, or use -DBOOST_ROOT=<PATH_TO_BOOST> with cmake if unpacked elsewhere
Install Python if it is not already installed by Visual Studio
(Optional) Install OpenJDK 11 to build Java bindings
(Optional) Install OpenSSL 3.x to build with TLS support
(Optional) Install WIX Toolset to build the Windows installer
mkdir build && cd build
cmake -G "Visual Studio 16 2019" -A x64 -T ClangCl <FDB_SOURCE_DIR>
msbuild /p:Configuration=Release foundationdb.sln
To increase build performance, use /p:UseMultiToolTask=true and /p:CL_MPCount=<NUMBER_OF_PARALLEL_JOBS>

Language Bindings

The language bindings that CMake supports will have a corresponding README.md file in the bindings/lang directory corresponding to each language.

Generally, CMake will build all language bindings for which it can find all necessary dependencies. After each successful CMake run, CMake will tell you which language bindings it is going to build.

Generating `compile_commands.json`

CMake can build a compilation database for you. However, the default generated one is not too useful as it operates on the generated files. When running ninja, the build system creates another compile_commands.json file in the source directory. This can then be used for tools such as CCLS and CQuery, among others. This way, you can get code completion and code navigation in flow. It is not yet perfect (it will show a few errors), but we are continually working to improve the development experience.

CMake will not produce a compile_commands.json by default; you must pass -DCMAKE_EXPORT_COMPILE_COMMANDS=ON. This also enables the target processed_compile_commands, which rewrites compile_commands.json to describe the actor compiler source file, not the post-processed output files, and places the output file in the source directory. This file should then be picked up automatically by any tooling.

Note that if the building is done inside the foundationdb/build Docker image, the resulting paths will still be incorrect and require manual fixing. One will wish to re-run cmake with -DCMAKE_EXPORT_COMPILE_COMMANDS=OFF to prevent it from reverting the manual changes.

Using IDEs

CMake provides built-in support for several popular IDEs. However, most FoundationDB files are written in the flow language, which is an extension of the C++ programming language, for coroutine support (Note that when FoundationDB was being developed, C++20 was not available). The flow language will be transpiled into C++ code using actorcompiler, while preventing most IDEs from recognizing flow-specific syntax.

It is possible to generate project files for editing flow with a supported IDE. There is a CMake option called OPEN_FOR_IDE, which creates a project that can be opened in an IDE for editing. This project cannot be built, but you will be able to edit the files and utilize most of the editing and navigation features that your IDE supports.

For example, if you want to use Xcode to make changes to FoundationDB, you can create an Xcode project with the following command:

cmake -G Xcode -DOPEN_FOR_IDE=ON <FDB_SOURCE_DIRECTORY>

A second build directory with the OPEN_FOR_IDE flag off can be created for building and debugging purposes.

Top Related Projects

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

Top Related Projects

Quick Overview

Pros

Cons

Code Examples

Getting Started

Competitor Comparisons

Pros of CockroachDB

Cons of CockroachDB

Code Comparison

Pros of etcd

Cons of etcd

Code Comparison

Pros of TiDB

Cons of TiDB

Code Comparison

Pros of ArangoDB

Cons of ArangoDB

Code Comparison

Key Differences

Pros of Dgraph

Cons of Dgraph

Code Comparison

Pros of Cassandra

Cons of Cassandra

Code Comparison

Convert designs to code with AI

README

Documentation

Forums

Contributing

Getting Started

Latest Stable Releases

Binary Downloads

Compiling from source

Build Using the Official Docker Image

Build Locally

FreeBSD

macOS

Windows

Language Bindings

Generating compile_commands.json

Using IDEs

Top Related Projects

Convert designs to code with AI

Generating `compile_commands.json`