Convert Figma logo to code with AI

microsoft logoFASTER

Fast persistent recoverable log and key-value store + cache, in C# and C++.

6,288
564
6,288
24

Top Related Projects

28,234

A library that provides an embeddable, persistent key-value store for fast storage.

FoundationDB - the open source, distributed, transactional key-value store

4,766

RocksDB/LevelDB inspired key-value database in Go

36,086

LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.

Quick Overview

FASTER is a high-performance, concurrent key-value store and log developed by Microsoft Research. It is designed for large state management in cloud and edge applications, offering both in-memory and larger-than-memory data processing capabilities with low latency and high throughput.

Pros

  • Extremely fast performance, especially for larger-than-memory workloads
  • Supports both in-memory and disk-based operations
  • Provides concurrent access with lock-free operations
  • Offers flexible API for various data types and operations

Cons

  • Steeper learning curve compared to simpler key-value stores
  • Limited documentation for advanced use cases
  • Primarily focused on .NET ecosystem, which may limit adoption in other environments
  • Requires careful tuning for optimal performance in specific scenarios

Code Examples

  1. Creating a FASTER instance:
using FASTER.core;

var log = Devices.CreateLogDevice("hlog.log");
var store = new FasterKV<long, long>(1L << 20, new LogSettings { LogDevice = log });
  1. Performing a read operation:
long key = 42;
long value;
var status = store.Read(ref key, ref value);
if (status == Status.OK)
{
    Console.WriteLine($"Value for key {key}: {value}");
}
  1. Performing an upsert operation:
long key = 42;
long value = 100;
store.Upsert(ref key, ref value);
  1. Using sessions for better performance:
using var session = store.NewSession();
long key = 42;
long value = 100;
session.Upsert(ref key, ref value);

Getting Started

To get started with FASTER in a .NET project:

  1. Install the NuGet package:
dotnet add package Microsoft.FASTER.Core
  1. Create a basic FASTER instance:
using FASTER.core;

var log = Devices.CreateLogDevice("hlog.log");
var store = new FasterKV<long, long>(1L << 20, new LogSettings { LogDevice = log });

using var session = store.NewSession();

// Perform operations using the session
long key = 1;
long value = 100;
session.Upsert(ref key, ref value);

// Read the value back
long readValue;
var status = session.Read(ref key, ref readValue);
if (status == Status.OK)
{
    Console.WriteLine($"Value for key {key}: {readValue}");
}

This example creates a FASTER instance, performs an upsert operation, and then reads the value back.

Competitor Comparisons

28,234

A library that provides an embeddable, persistent key-value store for fast storage.

Pros of RocksDB

  • Mature and widely adopted in production environments
  • Extensive documentation and community support
  • Offers a wide range of features, including column families and transactions

Cons of RocksDB

  • Higher memory usage compared to FASTER
  • Can be slower for certain workloads, especially those involving large key-value pairs
  • More complex configuration and tuning required for optimal performance

Code Comparison

RocksDB:

rocksdb::DB* db;
rocksdb::Options options;
options.create_if_missing = true;
rocksdb::Status status = rocksdb::DB::Open(options, "/path/to/db", &db);
db->Put(rocksdb::WriteOptions(), "key", "value");

FASTER:

var store = new FasterKV<Key, Value>(1L << 20, new LogSettings { ... });
var session = store.NewSession();
session.Upsert(key, value);

Key Differences

  • FASTER is designed for low-latency, high-throughput scenarios, while RocksDB focuses on general-purpose key-value storage
  • FASTER uses a novel hybrid log-structured architecture, whereas RocksDB is based on a Log-Structured Merge-Tree (LSM)
  • RocksDB provides more advanced features like snapshots and column families, while FASTER emphasizes simplicity and performance
  • FASTER is primarily designed for .NET environments, while RocksDB has broader language support

FoundationDB - the open source, distributed, transactional key-value store

Pros of FoundationDB

  • Distributed architecture with strong ACID guarantees
  • Multi-model database supporting key-value, document, and relational models
  • Scalable to handle large datasets across multiple machines

Cons of FoundationDB

  • Steeper learning curve due to its distributed nature
  • Limited language support compared to FASTER
  • May have higher operational complexity for small-scale applications

Code Comparison

FoundationDB (Python):

@fdb.transactional
def add_user(tr, user_id, name):
    tr[f'users:{user_id}:name'] = name

db = fdb.open()
add_user(db, '123', 'John Doe')

FASTER (C#):

using var session = store.NewSession();
session.Upsert(key, value);
session.Refresh();

Key Differences

  • FoundationDB focuses on distributed systems, while FASTER is designed for single-node performance
  • FASTER provides lower-level control over data structures and memory management
  • FoundationDB offers stronger consistency guarantees in a distributed environment
  • FASTER is more suitable for embedded scenarios and high-performance local operations

Both projects have their strengths, with FoundationDB excelling in distributed environments and FASTER optimized for single-node performance and flexibility.

4,766

RocksDB/LevelDB inspired key-value database in Go

Pros of Pebble

  • Designed specifically for key-value storage, optimized for SSDs
  • Better support for concurrent reads and writes
  • More mature and battle-tested in production environments

Cons of Pebble

  • Less flexible than FASTER for different data structures
  • May have higher memory usage in certain scenarios
  • Limited to Go language, while FASTER supports multiple languages

Code Comparison

Pebble (Go):

db, err := pebble.Open("path/to/db", &pebble.Options{})
if err != nil {
    log.Fatal(err)
}
defer db.Close()

FASTER (C#):

var log = Devices.CreateLogDevice("path/to/log");
var store = new FasterKV<Key, Value>(1L << 20, new LogSettings { LogDevice = log });

Both libraries provide simple APIs for opening and interacting with key-value stores, but FASTER offers more flexibility in terms of data types and storage options. Pebble's API is more focused on traditional key-value operations, while FASTER allows for custom data structures and indexing schemes.

Pebble is tailored for use in distributed systems like CockroachDB, offering strong consistency guarantees. FASTER, on the other hand, provides a broader set of features for various use cases, including in-memory operations and hybrid log-based storage.

36,086

LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.

Pros of LevelDB

  • Mature and widely adopted key-value store with a proven track record
  • Simple and lightweight API, easy to integrate into existing projects
  • Supports ordered iteration over keys, useful for range queries

Cons of LevelDB

  • Limited concurrency support, primarily designed for single-threaded access
  • Lacks built-in support for complex data structures or advanced indexing
  • Performance can degrade with large datasets due to compaction overhead

Code Comparison

LevelDB:

leveldb::DB* db;
leveldb::Options options;
options.create_if_missing = true;
leveldb::Status status = leveldb::DB::Open(options, "/tmp/testdb", &db);
db->Put(leveldb::WriteOptions(), "key", "value");

FASTER:

var store = new FasterKV<Key, Value>(1L << 20, new LogSettings { LogDevice = device });
var session = store.NewSession();
session.Upsert(key, value);

FASTER offers a more flexible API with support for concurrent operations and custom key-value types, while LevelDB provides a simpler interface focused on basic key-value operations. FASTER's design allows for better scalability and performance in multi-threaded scenarios, whereas LevelDB excels in simplicity and ease of use for single-threaded applications.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

FASTER logo

NuGet Build Status Gitter

Introduction

Managing large application state easily, resiliently, and with high performance is one of the hardest problems in the cloud today. The FASTER project offers two artifacts to help tackle this problem.

  • FASTER Log is a high-performance concurrent persistent recoverable log, iterator, and random reader library in C#. It supports very frequent commit operations at low latency, and can quickly saturate disk bandwidth. It supports both sync and async interfaces, handles disk errors, and supports checksums.

  • FASTER KV is a concurrent key-value store + cache (available in C# and C++) that is designed for point lookups and heavy updates. FASTER supports data larger than memory, by leveraging fast external storage (local or cloud). It also supports consistent recovery using a fast non-blocking checkpointing technique that lets applications trade-off performance for commit latency.

Both FASTER KV and FASTER Log offer orders-of-magnitude higher performance than comparable solutions, on standard workloads. Start learning about FASTER, its unique capabilities, and how to get started at our official website:

aka.ms/FASTER

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.