Top Related Projects
A library that provides an embeddable, persistent key-value store for fast storage.
LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.
FoundationDB - the open source, distributed, transactional key-value store
Fast key-value DB in Go.
LevelDB key/value database in Go.
An embedded key/value database for Go.
Quick Overview
Pebble is a key-value store written in Go, inspired by RocksDB and designed for performance and configurability. It serves as the storage engine for CockroachDB and can be used as a standalone embedded key-value store in Go applications.
Pros
- High performance and low latency for read and write operations
- Highly configurable with numerous tuning options
- Designed for both in-memory and on-disk storage
- Well-documented and actively maintained by Cockroach Labs
Cons
- Primarily designed for use with CockroachDB, which may limit some use cases
- Steeper learning curve compared to simpler key-value stores
- Limited language support (Go only) for direct integration
Code Examples
- Opening a database:
import "github.com/cockroachdb/pebble"
db, err := pebble.Open("path/to/database", &pebble.Options{})
if err != nil {
log.Fatal(err)
}
defer db.Close()
- Writing a key-value pair:
key := []byte("hello")
value := []byte("world")
err := db.Set(key, value, pebble.Sync)
if err != nil {
log.Fatal(err)
}
- Reading a value:
value, closer, err := db.Get([]byte("hello"))
if err != nil {
log.Fatal(err)
}
defer closer.Close()
fmt.Printf("Value: %s\n", value)
- Iterating over a range of keys:
iter := db.NewIter(&pebble.IterOptions{})
for iter.First(); iter.Valid(); iter.Next() {
fmt.Printf("Key: %s, Value: %s\n", iter.Key(), iter.Value())
}
if err := iter.Close(); err != nil {
log.Fatal(err)
}
Getting Started
To use Pebble in your Go project:
-
Install Pebble:
go get github.com/cockroachdb/pebble
-
Import Pebble in your Go code:
import "github.com/cockroachdb/pebble"
-
Open a database and start using Pebble:
db, err := pebble.Open("path/to/database", &pebble.Options{}) if err != nil { log.Fatal(err) } defer db.Close() // Use db for operations (Set, Get, Delete, etc.)
Refer to the Pebble documentation for more advanced usage and configuration options.
Competitor Comparisons
A library that provides an embeddable, persistent key-value store for fast storage.
Pros of RocksDB
- More mature and widely adopted, with extensive production use
- Broader ecosystem and tooling support
- Offers more advanced features like column families and transactions
Cons of RocksDB
- Higher memory usage and larger binary size
- More complex API and configuration options
- Slower write performance in some scenarios
Code Comparison
RocksDB:
rocksdb::DB* db;
rocksdb::Options options;
options.create_if_missing = true;
rocksdb::Status status = rocksdb::DB::Open(options, "/path/to/db", &db);
Pebble:
db, err := pebble.Open("/path/to/db", &pebble.Options{})
if err != nil {
log.Fatal(err)
}
Key Differences
- Language: RocksDB is primarily written in C++, while Pebble is written in Go
- API Design: Pebble offers a simpler, more streamlined API compared to RocksDB
- Performance: Pebble generally provides better write performance, especially for smaller datasets
- Memory Usage: Pebble typically has lower memory overhead compared to RocksDB
- Feature Set: RocksDB offers more advanced features, while Pebble focuses on core functionality
Both libraries are LSM tree-based key-value stores, but Pebble is designed to be a more lightweight and Go-friendly alternative to RocksDB, particularly suited for use in CockroachDB.
LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.
Pros of LevelDB
- Mature and battle-tested, with a long history of use in production environments
- Lightweight and easy to integrate into existing projects
- Supports multiple platforms, including Windows, Linux, and macOS
Cons of LevelDB
- Limited concurrency support, which can impact performance in multi-threaded applications
- Lacks built-in compression options beyond Snappy
- No native support for range deletions or time-bound operations
Code Comparison
LevelDB:
leveldb::DB* db;
leveldb::Options options;
options.create_if_missing = true;
leveldb::Status status = leveldb::DB::Open(options, "/tmp/testdb", &db);
Pebble:
db, err := pebble.Open("path/to/db", &pebble.Options{})
if err != nil {
log.Fatal(err)
}
defer db.Close()
Pebble, developed by CockroachDB, is designed as a performance-oriented alternative to LevelDB. It offers improved concurrency, additional compression options, and features like range deletions. Pebble is written in Go, making it a natural fit for Go-based projects, while LevelDB is primarily C++ with various language bindings. Both libraries provide key-value storage with similar basic operations, but Pebble includes optimizations and features specifically tailored for high-performance distributed database systems.
FoundationDB - the open source, distributed, transactional key-value store
Pros of FoundationDB
- Distributed architecture with strong consistency and ACID transactions
- Multi-model database supporting key-value, document, and relational models
- Scalable to handle large datasets and high transaction volumes
Cons of FoundationDB
- Steeper learning curve due to its unique architecture and API
- Limited language support compared to Pebble
- Requires more complex setup and configuration for distributed deployments
Code Comparison
FoundationDB (C++):
Transaction tr(db);
tr.set(KeyRef("key"), ValueRef("value"));
tr.commit().wait();
Pebble (Go):
batch := db.NewBatch()
batch.Set([]byte("key"), []byte("value"), nil)
err := batch.Commit(pebble.Sync)
Key Differences
- FoundationDB is designed for distributed systems, while Pebble is primarily an embedded key-value store
- FoundationDB offers a wider range of data models, whereas Pebble focuses on key-value storage
- Pebble provides a simpler API and is easier to integrate into Go applications
- FoundationDB has better support for complex distributed transactions and global consistency
Both databases have their strengths, with FoundationDB excelling in distributed environments and Pebble offering simplicity and performance for embedded use cases.
Fast key-value DB in Go.
Pros of Badger
- Native Go implementation, potentially better performance for Go applications
- Supports ACID transactions out of the box
- Designed for SSDs, optimized for random writes
Cons of Badger
- Less mature and battle-tested compared to Pebble
- Limited ecosystem and tooling support
- May have higher memory usage in certain scenarios
Code Comparison
Badger:
db, err := badger.Open(badger.DefaultOptions("/tmp/badger"))
defer db.Close()
err = db.Update(func(txn *badger.Txn) error {
return txn.Set([]byte("key"), []byte("value"))
})
Pebble:
db, err := pebble.Open("/tmp/pebble", &pebble.Options{})
defer db.Close()
err = db.Set([]byte("key"), []byte("value"), pebble.Sync)
Both Badger and Pebble are key-value stores written in Go, but they have different design goals and trade-offs. Badger is optimized for SSDs and offers built-in ACID transactions, making it suitable for applications requiring strong consistency. Pebble, on the other hand, is designed as a performance-oriented storage engine, originally created for CockroachDB. It may offer better performance in certain scenarios and has been battle-tested in production environments. The choice between the two depends on specific application requirements, performance needs, and ecosystem considerations.
LevelDB key/value database in Go.
Pros of goleveldb
- Simpler implementation, easier to understand and modify
- Longer history and wider adoption in the Go community
- Lighter weight with fewer dependencies
Cons of goleveldb
- Less optimized for performance compared to Pebble
- Fewer advanced features and customization options
- Less active development and maintenance
Code Comparison
goleveldb:
db, err := leveldb.OpenFile("path/to/db", nil)
err = db.Put([]byte("key"), []byte("value"), nil)
value, err := db.Get([]byte("key"), nil)
Pebble:
db, err := pebble.Open("path/to/db", &pebble.Options{})
err = db.Set([]byte("key"), []byte("value"), pebble.Sync)
value, closer, err := db.Get([]byte("key"))
defer closer.Close()
Both libraries provide similar basic key-value storage functionality, but Pebble offers more advanced options and better performance optimizations. Pebble's API is slightly more verbose, requiring explicit closure of resources in some cases.
Pebble is designed for high-performance scenarios and includes features like bloom filters, cache management, and compaction strategies. It's actively maintained by Cockroach Labs and benefits from ongoing improvements.
goleveldb, while simpler and more established, may be sufficient for less demanding use cases or projects prioritizing simplicity and minimal dependencies.
An embedded key/value database for Go.
Pros of bbolt
- Simpler API and easier to use for basic key-value storage needs
- Lighter weight and more suitable for embedded systems or applications with limited resources
- Better suited for smaller datasets and simpler data structures
Cons of bbolt
- Limited feature set compared to Pebble, lacking advanced functionalities like bloom filters and range deletions
- Lower performance for large-scale applications or complex data operations
- Less active development and community support
Code Comparison
bbolt:
db, err := bolt.Open("my.db", 0600, nil)
if err != nil {
log.Fatal(err)
}
defer db.Close()
Pebble:
db, err := pebble.Open("my.db", &pebble.Options{})
if err != nil {
log.Fatal(err)
}
defer db.Close()
Both libraries provide similar basic functionality for opening and closing databases. However, Pebble offers more advanced configuration options through its Options
struct, allowing for fine-tuned performance optimizations and feature customization.
While bbolt is simpler and more straightforward for basic use cases, Pebble provides a more robust and feature-rich solution for complex database requirements, especially in high-performance scenarios or when dealing with large-scale data operations.
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual CopilotREADME
Pebble Coverage
Nightly benchmarks
Pebble is a LevelDB/RocksDB inspired key-value store focused on performance and internal usage by CockroachDB. Pebble inherits the RocksDB file formats and a few extensions such as range deletion tombstones, table-level bloom filters, and updates to the MANIFEST format.
Pebble intentionally does not aspire to include every feature in RocksDB and specifically targets the use case and feature set needed by CockroachDB:
- Block-based tables
- Checkpoints
- Indexed batches
- Iterator options (lower/upper bound, table filter)
- Level-based compaction
- Manual compaction
- Merge operator
- Prefix bloom filters
- Prefix iteration
- Range deletion tombstones
- Reverse iteration
- SSTable ingestion
- Single delete
- Snapshots
- Table-level bloom filters
RocksDB has a large number of features that are not implemented in Pebble:
- Backups
- Column families
- Delete files in range
- FIFO compaction style
- Forward iterator / tailing iterator
- Hash table format
- Memtable bloom filter
- Persistent cache
- Pin iterator key / value
- Plain table format
- SSTable ingest-behind
- Sub-compactions
- Transactions
- Universal compaction style
WARNING: Pebble may silently corrupt data or behave incorrectly if used with a RocksDB database that uses a feature Pebble doesn't support. Caveat emptor!
Production Ready
Pebble was introduced as an alternative storage engine to RocksDB in CockroachDB v20.1 (released May 2020) and was used in production successfully at that time. Pebble was made the default storage engine in CockroachDB v20.2 (released Nov 2020). Pebble is being used in production by users of CockroachDB at scale and is considered stable and production ready.
Advantages
Pebble offers several improvements over RocksDB:
- Faster reverse iteration via backwards links in the memtable's skiplist.
- Faster commit pipeline that achieves better concurrency.
- Seamless merged iteration of indexed batches. The mutations in the batch conceptually occupy another memtable level.
- L0 sublevels and flush splitting for concurrent compactions out of L0 and reduced read-amplification during heavy write load.
- Faster LSM edits in LSMs with large numbers of sstables through use of a copy-on-write B-tree to hold file metadata.
- Delete-only compactions that drop whole sstables that fall within the bounds of a range deletion.
- Block-property collectors and filters that enable iterators to skip tables, index blocks and data blocks that are irrelevant, according to user-defined properties over key-value pairs.
- Range keys API, allowing KV pairs defined over a range of keyspace with user-defined semantics and interleaved during iteration.
- Smaller, more approachable code base.
See the Pebble vs RocksDB: Implementation Differences doc for more details on implementation differences.
RocksDB Compatibility
Pebble strives for forward compatibility with RocksDB 6.2.1 (the latest version
of RocksDB used by CockroachDB). Forward compatibility means that a DB generated
by RocksDB 6.2.1 can be upgraded for use by Pebble. Pebble versions in the v1
series may open DBs generated by RocksDB 6.2.1. Since its introduction, Pebble
has adopted various backwards-incompatible format changes that are gated behind
new 'format major versions'. The Pebble master
branch does not support opening
DBs generated by RocksDB. DBs generated by RocksDB may only be used with recent
versions of Pebble after migrating them through format major version upgrades
using previous versions of Pebble. See the below section of format major
versions.
Even the RocksDB-compatible versions of Pebble only provide compatibility with the subset of functionality and configuration used by CockroachDB. The scope of RocksDB functionality and configuration is too large to adequately test and document all the incompatibilities. The list below contains known incompatibilities.
- Pebble's use of WAL recycling is only compatible with RocksDB's
kTolerateCorruptedTailRecords
WAL recovery mode. Older versions of RocksDB would automatically map incompatible WAL recovery modes tokTolerateCorruptedTailRecords
. New versions of RocksDB will disable WAL recycling. - Column families. Pebble does not support column families, nor does it attempt to detect their usage when opening a DB that may contain them.
- Hash table format. Pebble does not support the hash table sstable format.
- Plain table format. Pebble does not support the plain table sstable format.
- SSTable format version 3 and 4. Pebble does not support version 3
and version 4 format sstables. The sstable format version is
controlled by the
BlockBasedTableOptions::format_version
option. See #97.
Format major versions
Over time Pebble has introduced new physical file formats. Backwards
incompatible changes are made through the introduction of 'format major
versions'. By default, when Pebble opens a database, it defaults to the lowest
supported version. In v1
, this is FormatMostCompatible
, which is
bi-directionally compatible with RocksDB 6.2.1 (with the caveats described
above).
Databases created by RocksDB or Pebble versions v1
and earlier must be upgraded
to a compatible format major version before running newer Pebble versions. Newer
Pebble versions will refuse to open databases in no longer supported formats.
To opt into new formats, a user may set FormatMajorVersion
on the
Options
supplied to
Open
, or
upgrade the format major version at runtime using
DB.RatchetFormatMajorVersion
.
Format major version upgrades are permanent; There is no option to
return to an earlier format.
The table below outlines the history of format major versions, along with what range of Pebble versions support that format.
Name | Value | Migration | Pebble support |
---|---|---|---|
FormatMostCompatible | 1 | No | v1 |
FormatVersioned | 3 | No | v1 |
FormatSetWithDelete | 4 | No | v1 |
FormatBlockPropertyCollector | 5 | No | v1 |
FormatSplitUserKeysMarked | 6 | Background | v1 |
FormatSplitUserKeysMarkedCompacted | 7 | Blocking | v1 |
FormatRangeKeys | 8 | No | v1 |
FormatMinTableFormatPebblev1 | 9 | No | v1 |
FormatPrePebblev1Marked | 10 | Background | v1 |
FormatSSTableValueBlocks | 12 | No | v1 |
FormatFlushableIngest | 13 | No | v1, master |
FormatPrePebblev1MarkedCompacted | 14 | Blocking | v1, master |
FormatDeleteSizedAndObsolete | 15 | No | v1, master |
FormatVirtualSSTables | 16 | No | v1, master |
Upgrading to a format major version with 'Background' in the migration
column may trigger background activity to rewrite physical file
formats, typically through compactions. Upgrading to a format major
version with 'Blocking' in the migration column will block until a
migration is complete. The database may continue to serve reads and
writes if upgrading a live database through
RatchetFormatMajorVersion
, but the method call will not return until
the migration is complete.
For reference, the table below lists the range of supported Pebble format major versions for CockroachDB releases.
CockroachDB release | Earliest supported | Latest supported |
---|---|---|
20.1 through 21.1 | FormatMostCompatible | FormatMostCompatible |
21.2 | FormatMostCompatible | FormatSetWithDelete |
21.2 | FormatMostCompatible | FormatSetWithDelete |
22.1 | FormatMostCompatible | FormatSplitUserKeysMarked |
22.2 | FormatMostCompatible | FormatPrePebblev1Marked |
23.1 | FormatSplitUserKeysMarkedCompacted | FormatFlushableIngest |
23.2 | FormatSplitUserKeysMarkedCompacted | FormatVirtualSSTables |
Pedigree
Pebble is based on the incomplete Go version of LevelDB:
https://github.com/golang/leveldb
The Go version of LevelDB is based on the C++ original:
https://github.com/google/leveldb
Optimizations and inspiration were drawn from RocksDB:
https://github.com/facebook/rocksdb
Getting Started
Example Code
package main
import (
"fmt"
"log"
"github.com/cockroachdb/pebble"
)
func main() {
db, err := pebble.Open("demo", &pebble.Options{})
if err != nil {
log.Fatal(err)
}
key := []byte("hello")
if err := db.Set(key, []byte("world"), pebble.Sync); err != nil {
log.Fatal(err)
}
value, closer, err := db.Get(key)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s %s\n", key, value)
if err := closer.Close(); err != nil {
log.Fatal(err)
}
if err := db.Close(); err != nil {
log.Fatal(err)
}
}
Top Related Projects
A library that provides an embeddable, persistent key-value store for fast storage.
LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.
FoundationDB - the open source, distributed, transactional key-value store
Fast key-value DB in Go.
LevelDB key/value database in Go.
An embedded key/value database for Go.
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual Copilot