Convert Figma logo to code with AI

dgraph-io logobadger

Fast key-value DB in Go.

13,744
1,172
13,744
28
View on GitHub

Top Related Projects

facebook's avatar

rocksdb

28,234

A library that provides an embeddable, persistent key-value store for fast storage.

google's avatar

leveldb

36,086

LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.

syndtr's avatar

goleveldb

6,122

LevelDB key/value database in Go.

cockroachdb's avatar

pebble

4,766

RocksDB/LevelDB inspired key-value database in Go

etcd-io's avatar

bbolt

8,110

An embedded key/value database for Go.

boltdb's avatar

bolt

14,138

An embedded key/value database for Go.

Quick Overview

Badger is a fast, embeddable, persistent key-value store written in pure Go. It's designed to be a performant alternative to non-Go-based key-value stores like RocksDB. Badger is optimized for SSDs and supports ACID transactions, making it suitable for both small-scale and large-scale applications.

Pros

  • High performance, especially on SSDs
  • Pure Go implementation, making it easy to embed and deploy
  • Supports ACID transactions
  • Designed for concurrent access

Cons

  • Larger memory footprint compared to some alternatives
  • Limited query capabilities compared to full-fledged databases
  • Requires manual compaction for optimal performance
  • Less mature ecosystem compared to established databases

Code Examples

  1. Opening a Badger database:
import "github.com/dgraph-io/badger/v3"

db, err := badger.Open(badger.DefaultOptions("/tmp/badger"))
if err != nil {
    log.Fatal(err)
}
defer db.Close()
  1. Writing data to the database:
err = db.Update(func(txn *badger.Txn) error {
    err := txn.Set([]byte("key"), []byte("value"))
    return err
})
  1. Reading data from the database:
err = db.View(func(txn *badger.Txn) error {
    item, err := txn.Get([]byte("key"))
    if err != nil {
        return err
    }
    val, err := item.ValueCopy(nil)
    fmt.Printf("The value is: %s\n", val)
    return nil
})
  1. Iterating over keys with a prefix:
prefix := []byte("user_")
err = db.View(func(txn *badger.Txn) error {
    it := txn.NewIterator(badger.DefaultIteratorOptions)
    defer it.Close()
    for it.Seek(prefix); it.ValidForPrefix(prefix); it.Next() {
        item := it.Item()
        k := item.Key()
        err := item.Value(func(v []byte) error {
            fmt.Printf("key=%s, value=%s\n", k, v)
            return nil
        })
        if err != nil {
            return err
        }
    }
    return nil
})

Getting Started

To use Badger in your Go project:

  1. Install Badger:

    go get github.com/dgraph-io/badger/v3

  2. Import Badger in your code:

    import "github.com/dgraph-io/badger/v3"

  3. Open a database:

    db, err := badger.Open(badger.DefaultOptions("/path/to/db"))
if err != nil {
    log.Fatal(err)
}
defer db.Close()

  4. Use the database for read and write operations as shown in the code examples above.

Competitor Comparisons

facebook logo

rocksdb

28,234

A library that provides an embeddable, persistent key-value store for fast storage.

Pros of RocksDB

  • More mature and battle-tested, with extensive production use at Facebook and other large companies
  • Supports a wider range of storage engines and configurations
  • Offers more advanced features like column families and transactions

Cons of RocksDB

  • Higher memory usage and larger disk footprint
  • More complex API and configuration options, steeper learning curve
  • Written in C++, which may not be ideal for Go-centric projects

Code Comparison

RocksDB (C++):

#include <rocksdb/db.h>

rocksdb::DB* db;
rocksdb::Options options;
options.create_if_missing = true;
rocksdb::Status status = rocksdb::DB::Open(options, "/path/to/db", &db);

Badger (Go):

import "github.com/dgraph-io/badger/v3"

opts := badger.DefaultOptions("/path/to/db")
db, err := badger.Open(opts)

Key Differences

  • Badger is written in Go, making it more suitable for Go projects
  • RocksDB offers more advanced features and configuration options
  • Badger has a simpler API and is easier to get started with
  • RocksDB has broader language support through official and community bindings
  • Badger is optimized for SSDs and uses less memory, while RocksDB is more versatile in terms of storage options
google logo

leveldb

36,086

LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.

Pros of LevelDB

  • Mature and battle-tested, with a long history of use in production environments
  • Excellent performance for read-heavy workloads
  • Smaller memory footprint compared to Badger

Cons of LevelDB

  • Limited concurrency support, which can impact write performance
  • Lacks built-in compression, potentially leading to larger storage requirements
  • No native support for time-based expiration of entries

Code Comparison

LevelDB (C++):

leveldb::DB* db;
leveldb::Options options;
options.create_if_missing = true;
leveldb::Status status = leveldb::DB::Open(options, "/tmp/testdb", &db);

Badger (Go):

opts := badger.DefaultOptions("/tmp/badger")
db, err := badger.Open(opts)
if err != nil {
    log.Fatal(err)
}
defer db.Close()

Both LevelDB and Badger are key-value stores, but they have different design philosophies and target use cases. LevelDB is a more traditional LSM-tree based store, while Badger uses a novel design with separated key and value storage. This allows Badger to offer better write performance and support for larger datasets, but at the cost of increased complexity and potentially higher memory usage. The choice between the two depends on specific application requirements, such as read/write patterns, dataset size, and concurrency needs.

syndtr logo

goleveldb

6,122

LevelDB key/value database in Go.

Pros of goleveldb

  • More mature and widely adopted in the Go ecosystem
  • Smaller memory footprint for certain workloads
  • Simpler API, easier to get started with

Cons of goleveldb

  • Lower write performance compared to Badger
  • Lacks some advanced features like value log and managed mode

Code Comparison

goleveldb:

db, err := leveldb.OpenFile("path/to/db", nil)
err = db.Put([]byte("key"), []byte("value"), nil)
value, err := db.Get([]byte("key"), nil)

Badger:

opts := badger.DefaultOptions("path/to/db")
db, err := badger.Open(opts)
err = db.Update(func(txn *badger.Txn) error {
    return txn.Set([]byte("key"), []byte("value"))
})
err = db.View(func(txn *badger.Txn) error {
    item, err := txn.Get([]byte("key"))
    return item.Value(func(val []byte) error {
        // Use val here
        return nil
    })
})

Both goleveldb and Badger are key-value stores for Go, but they have different design philosophies and performance characteristics. goleveldb is a pure Go implementation of LevelDB, while Badger is a performance-oriented alternative with its own storage format. Badger generally offers better write performance and more advanced features, while goleveldb has a simpler API and may be more suitable for read-heavy workloads or systems with limited memory.

cockroachdb logo

pebble

4,766

RocksDB/LevelDB inspired key-value database in Go

Pros of Pebble

  • Better performance for large datasets and high write throughput scenarios
  • More advanced compaction strategies, leading to improved space efficiency
  • Closer alignment with RocksDB, potentially easier migration for RocksDB users

Cons of Pebble

  • Higher memory usage compared to Badger
  • Less suitable for embedded or resource-constrained environments
  • Steeper learning curve due to more complex configuration options

Code Comparison

Badger:

db, err := badger.Open(badger.DefaultOptions("/tmp/badger"))
defer db.Close()

err = db.Update(func(txn *badger.Txn) error {
    return txn.Set([]byte("key"), []byte("value"))
})

Pebble:

db, err := pebble.Open("/tmp/pebble", &pebble.Options{})
defer db.Close()

err = db.Set([]byte("key"), []byte("value"), pebble.Sync)

Both Badger and Pebble are key-value stores written in Go, but they have different design goals and trade-offs. Badger is optimized for SSDs and aims to provide high performance with low memory usage, making it suitable for embedded systems. Pebble, on the other hand, is designed as a performance-oriented alternative to RocksDB, focusing on large-scale deployments and high write throughput scenarios.

etcd-io logo

bbolt

8,110

An embedded key/value database for Go.

Pros of bbolt

  • Simpler API and easier to use for basic key-value storage
  • Lighter weight and more suitable for embedded applications
  • Better performance for read-heavy workloads

Cons of bbolt

  • Lacks advanced features like transactions with TTL or merge operators
  • Limited to a single writer at a time, which can impact write performance
  • Does not support compression, potentially leading to larger database sizes

Code Comparison

bbolt:

db, _ := bbolt.Open("my.db", 0600, nil)
defer db.Close()

db.Update(func(tx *bbolt.Tx) error {
    b, _ := tx.CreateBucketIfNotExists([]byte("MyBucket"))
    return b.Put([]byte("key"), []byte("value"))
})

Badger:

db, _ := badger.Open(badger.DefaultOptions("").WithInMemory(true))
defer db.Close()

db.Update(func(txn *badger.Txn) error {
    return txn.Set([]byte("key"), []byte("value"))
})

Both bbolt and Badger are embedded key-value stores for Go, but they have different strengths. bbolt is simpler and more lightweight, making it ideal for basic storage needs and read-heavy workloads. Badger offers more advanced features and better write performance, suitable for more complex applications with higher write throughput requirements.

boltdb logo

bolt

14,138

An embedded key/value database for Go.

Pros of Bolt

  • Simpler API and easier to use for basic key-value storage
  • Lighter weight and more suitable for embedded applications
  • Better suited for read-heavy workloads

Cons of Bolt

  • Limited support for concurrent write operations
  • Lacks advanced features like compression and bloom filters
  • May have slower write performance for large datasets

Code Comparison

Bolt:

db, _ := bolt.Open("my.db", 0600, nil)
defer db.Close()

db.Update(func(tx *bolt.Tx) error {
    b, _ := tx.CreateBucketIfNotExists([]byte("MyBucket"))
    return b.Put([]byte("key"), []byte("value"))
})

Badger:

opts := badger.DefaultOptions("").WithInMemory(true)
db, _ := badger.Open(opts)
defer db.Close()

db.Update(func(txn *badger.Txn) error {
    return txn.Set([]byte("key"), []byte("value"))
})

Both Bolt and Badger are key-value stores, but Badger offers more advanced features and better performance for write-heavy workloads. Bolt is simpler and more suitable for smaller, read-heavy applications. Badger uses a Log-Structured Merge (LSM) tree, while Bolt uses a B+tree structure. Badger supports concurrent ACID transactions, while Bolt allows only one write transaction at a time. Choose Bolt for simplicity and embedded use cases, and Badger for high-performance, large-scale applications with frequent writes.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

BadgerDB

Go Reference Go Report Card Sourcegraph ci-badger-tests ci-badger-bank-tests ci-golang-lint

Badger mascot

BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go. It is the underlying database for Dgraph, a fast, distributed graph database. It's meant to be a performant alternative to non-Go-based key-value stores like RocksDB.

Project Status

Badger is stable and is being used to serve data sets worth hundreds of terabytes. Badger supports concurrent ACID transactions with serializable snapshot isolation (SSI) guarantees. A Jepsen-style bank test runs nightly for 8h, with --race flag and ensures the maintenance of transactional guarantees. Badger has also been tested to work with filesystem level anomalies, to ensure persistence and consistency. Badger is being used by a number of projects which includes Dgraph, Jaeger Tracing, UsenetExpress, and many more.

The list of projects using Badger can be found here.

Badger v1.0 was released in Nov 2017, and the latest version that is data-compatible with v1.0 is v1.6.0.

Badger v2.0 was released in Nov 2019 with a new storage format which won't be compatible with all of the v1.x. Badger v2.0 supports compression, encryption and uses a cache to speed up lookup.

Badger v3.0 was released in January 2021. This release improves compaction performance.

Please consult the Changelog for more detailed information on releases.

For more details on our version naming schema please read Choosing a version.

Table of Contents

Getting Started

Installing

To start using Badger, install Go 1.19 or above. Badger v3 and above needs go modules. From your project, run the following command

$ go get github.com/dgraph-io/badger/v4

This will retrieve the library.

Installing Badger Command Line Tool

Badger provides a CLI tool which can perform certain operations like offline backup/restore. To install the Badger CLI, retrieve the repository and checkout the desired version. Then run

$ cd badger
$ go install .

This will install the badger command line utility into your $GOBIN path.

Choosing a version

BadgerDB is a pretty special package from the point of view that the most important change we can make to it is not on its API but rather on how data is stored on disk.

This is why we follow a version naming schema that differs from Semantic Versioning.

  • New major versions are released when the data format on disk changes in an incompatible way.
  • New minor versions are released whenever the API changes but data compatibility is maintained. Note that the changes on the API could be backward-incompatible - unlike Semantic Versioning.
  • New patch versions are released when there's no changes to the data format nor the API.

Following these rules:

  • v1.5.0 and v1.6.0 can be used on top of the same files without any concerns, as their major version is the same, therefore the data format on disk is compatible.
  • v1.6.0 and v2.0.0 are data incompatible as their major version implies, so files created with v1.6.0 will need to be converted into the new format before they can be used by v2.0.0.
  • v2.x.x and v3.x.x are data incompatible as their major version implies, so files created with v2.x.x will need to be converted into the new format before they can be used by v3.0.0.

For a longer explanation on the reasons behind using a new versioning naming schema, you can read VERSIONING.

Badger Documentation

Badger Documentation is available at https://dgraph.io/docs/badger

Resources

Blog Posts

  1. Introducing Badger: A fast key-value store written natively in Go
  2. Make Badger crash resilient with ALICE
  3. Badger vs LMDB vs BoltDB: Benchmarking key-value databases in Go
  4. Concurrent ACID Transactions in Badger

Design

Badger was written with these design goals in mind:

  • Write a key-value database in pure Go.
  • Use latest research to build the fastest KV database for data sets spanning terabytes.
  • Optimize for SSDs.

Badger’s design is based on a paper titled WiscKey: Separating Keys from Values in SSD-conscious Storage.

Comparisons

FeatureBadgerRocksDBBoltDB
DesignLSM tree with value logLSM tree onlyB+ tree
High Read throughputYesNoYes
High Write throughputYesYesNo
Designed for SSDsYes (with latest research 1)Not specifically 2No
EmbeddableYesYesYes
Sorted KV accessYesYesYes
Pure Go (no Cgo)YesNoYes
TransactionsYes, ACID, concurrent with SSI3Yes (but non-ACID)Yes, ACID
SnapshotsYesYesYes
TTL supportYesYesNo
3D access (key-value-version)Yes4NoNo

1 The WISCKEY paper (on which Badger is based) saw big wins with separating values from keys, significantly reducing the write amplification compared to a typical LSM tree.

2 RocksDB is an SSD optimized version of LevelDB, which was designed specifically for rotating disks. As such RocksDB's design isn't aimed at SSDs.

3 SSI: Serializable Snapshot Isolation. For more details, see the blog post Concurrent ACID Transactions in Badger

4 Badger provides direct access to value versions via its Iterator API. Users can also specify how many versions to keep per key via Options.

Benchmarks

We have run comprehensive benchmarks against RocksDB, Bolt and LMDB. The benchmarking code, and the detailed logs for the benchmarks can be found in the badger-bench repo. More explanation, including graphs can be found the blog posts (linked above).

Projects Using Badger

Below is a list of known projects that use Badger:

  • Dgraph - Distributed graph database.
  • Jaeger - Distributed tracing platform.
  • go-ipfs - Go client for the InterPlanetary File System (IPFS), a new hypermedia distribution protocol.
  • Riot - An open-source, distributed search engine.
  • emitter - Scalable, low latency, distributed pub/sub broker with message storage, uses MQTT, gossip and badger.
  • OctoSQL - Query tool that allows you to join, analyse and transform data from multiple databases using SQL.
  • Dkron - Distributed, fault tolerant job scheduling system.
  • smallstep/certificates - Step-ca is an online certificate authority for secure, automated certificate management.
  • Sandglass - distributed, horizontally scalable, persistent, time sorted message queue.
  • TalariaDB - Grab's Distributed, low latency time-series database.
  • Sloop - Salesforce's Kubernetes History Visualization Project.
  • Usenet Express - Serving over 300TB of data with Badger.
  • gorush - A push notification server written in Go.
  • 0-stor - Single device object store.
  • Dispatch Protocol - Blockchain protocol for distributed application data analytics.
  • GarageMQ - AMQP server written in Go.
  • RedixDB - A real-time persistent key-value store with the same redis protocol.
  • BBVA - Raft backend implementation using BadgerDB for Hashicorp raft.
  • Fantom - aBFT Consensus platform for distributed applications.
  • decred - An open, progressive, and self-funding cryptocurrency with a system of community-based governance integrated into its blockchain.
  • OpenNetSys - Create useful dApps in any software language.
  • HoneyTrap - An extensible and opensource system for running, monitoring and managing honeypots.
  • Insolar - Enterprise-ready blockchain platform.
  • IoTeX - The next generation of the decentralized network for IoT powered by scalability- and privacy-centric blockchains.
  • go-sessions - The sessions manager for Go net/http and fasthttp.
  • Babble - BFT Consensus platform for distributed applications.
  • Tormenta - Embedded object-persistence layer / simple JSON database for Go projects.
  • BadgerHold - An embeddable NoSQL store for querying Go types built on Badger
  • Goblero - Pure Go embedded persistent job queue backed by BadgerDB
  • Surfline - Serving global wave and weather forecast data with Badger.
  • Cete - Simple and highly available distributed key-value store built on Badger. Makes it easy bringing up a cluster of Badger with Raft consensus algorithm by hashicorp/raft.
  • Volument - A new take on website analytics backed by Badger.
  • KVdb - Hosted key-value store and serverless platform built on top of Badger.
  • Terminotes - Self hosted notes storage and search server - storage powered by BadgerDB
  • Pyroscope - Open source continuous profiling platform built with BadgerDB
  • Veri - A distributed feature store optimized for Search and Recommendation tasks.
  • bIter - A library and Iterator interface for working with the badger.Iterator, simplifying from-to, and prefix mechanics.
  • ld - (Lean Database) A very simple gRPC-only key-value database, exposing BadgerDB with key-range scanning semantics.
  • Souin - A RFC compliant HTTP cache with lot of other features based on Badger for the storage. Compatible with all existing reverse-proxies.
  • Xuperchain - A highly flexible blockchain architecture with great transaction performance.
  • m2 - A simple http key/value store based on the raft protocol.
  • chaindb - A blockchain storage layer used by Gossamer, a Go client for the Polkadot Network.
  • vxdb - Simple schema-less Key-Value NoSQL database with simplest API interface.
  • Opacity - Backend implementation for the Opacity storage project
  • Vephar - A minimal key/value store using hashicorp-raft for cluster coordination and Badger for data storage.
  • gowarcserver - Open-source server for warc files. Can be used in conjunction with pywb
  • flow-go - A fast, secure, and developer-friendly blockchain built to support the next generation of games, apps and the digital assets that power them.
  • Wrgl - A data version control system that works like Git but specialized to store and diff CSV.
  • Loggie - A lightweight, cloud-native data transfer agent and aggregator.
  • raft-badger - raft-badger implements LogStore and StableStore Interface of hashcorp/raft. it is used to store raft log and metadata of hashcorp/raft.
  • DVID - A dataservice for branched versioning of a variety of data types. Originally created for large-scale brain reconstructions in Connectomics.
  • KVS - A library for making it easy to persist, load and query full structs into BadgerDB, using an ownership hierarchy model.
  • LLS - LLS is an efficient URL Shortener that can be used to shorten links and track link usage. Support for BadgerDB and MongoDB. Improved performance by more than 30% when using BadgerDB
  • lakeFS - lakeFS is an open-source data version control that transforms your object storage to Git-like repositories. lakeFS uses BadgerDB for its underlying local metadata KV store implementation.

If you are using Badger in a project please send a pull request to add it to the list.

Contributing

If you're interested in contributing to Badger see CONTRIBUTING.

Contact