Convert Figma logo to code with AI

Netflix logodynomite

A generic dynamo implementation for different k-v storage engines

4,187
530
4,187
115

Top Related Projects

12,122

A fast, light-weight proxy for memcached and redis

18,411

Vitess is a database clustering system for horizontal scaling of MySQL.

13,421

Zuul is a gateway service that provides dynamic routing, monitoring, resiliency, security, and more.

47,330

Distributed reliable key-value store for the most critical data of a distributed system

12,362

AWS Service registry for resilient mid-tier load balancing and failover.

4,812

HAProxy Load Balancer's development branch (mirror of git.haproxy.org)

Quick Overview

Dynomite is a layer between your client and data store, transforming a non-distributed database into a sharded, multi-datacenter replication system. It supports Redis and Memcached as backends, providing high availability and cross-datacenter replication for these single-server data stores.

Pros

  • Enables horizontal scaling and multi-datacenter replication for Redis and Memcached
  • Provides high availability and fault tolerance
  • Supports multiple protocols (Redis, Memcached)
  • Allows for seamless integration with existing applications

Cons

  • Adds complexity to the infrastructure
  • May introduce additional latency due to the proxy layer
  • Limited to specific backend data stores (Redis and Memcached)
  • Requires careful configuration and tuning for optimal performance

Code Examples

As Dynomite is not a code library but a system-level component, there are no direct code examples for using it. Instead, applications interact with Dynomite as if they were connecting to a regular Redis or Memcached instance.

Getting Started

To get started with Dynomite:

  1. Clone the repository:

    git clone https://github.com/Netflix/dynomite.git
    
  2. Build Dynomite:

    cd dynomite
    autoreconf -fvi
    ./configure --enable-debug=full
    make
    
  3. Configure Dynomite by creating a YAML configuration file (e.g., dynomite.yml) with your desired settings.

  4. Run Dynomite:

    ./src/dynomite -c dynomite.yml -v 11
    
  5. Connect your application to Dynomite using the appropriate Redis or Memcached client, pointing to the Dynomite server address and port instead of the backend data store.

For more detailed instructions and configuration options, refer to the project's documentation on GitHub.

Competitor Comparisons

12,122

A fast, light-weight proxy for memcached and redis

Pros of twemproxy

  • Lightweight and efficient proxy for memcached and Redis
  • Supports multiple hashing modes for consistent distribution
  • Well-established and battle-tested in production environments

Cons of twemproxy

  • Limited support for advanced Redis features
  • No built-in support for sharding or replication
  • Less active development compared to Dynomite

Code Comparison

twemproxy configuration example:

alpha:
  listen: 127.0.0.1:22121
  hash: fnv1a_64
  distribution: ketama
  auto_eject_hosts: true
  redis: true
  server_retry_timeout: 2000
  server_failure_limit: 1
  servers:
   - 127.0.0.1:6379:1

Dynomite configuration example:

dyn_o_mite:
  dyn_listen: 0.0.0.0:8101
  data_store: 0
  listen: 0.0.0.0:8102
  dyn_seed_provider: florida_provider
  servers:
   - 127.0.0.1:6379:1
  tokens: 0

Both projects aim to improve Redis scalability, but Dynomite offers more advanced features like multi-datacenter replication and support for multiple backends. twemproxy is simpler and focuses on efficient proxying, while Dynomite provides a more comprehensive solution for large-scale deployments.

18,411

Vitess is a database clustering system for horizontal scaling of MySQL.

Pros of Vitess

  • Designed for horizontal scaling of MySQL databases, offering better support for relational data models
  • Provides advanced features like query rewriting, connection pooling, and automated failover
  • More mature project with wider adoption and extensive documentation

Cons of Vitess

  • Steeper learning curve due to its complexity and additional components
  • Requires more resources to set up and maintain compared to simpler solutions
  • May introduce additional latency in some scenarios due to its proxy architecture

Code Comparison

Dynomite (Redis-compatible API):

SET key value
GET key
INCR counter

Vitess (MySQL-compatible API):

INSERT INTO table (column) VALUES ('value');
SELECT column FROM table WHERE id = 1;
UPDATE table SET column = column + 1 WHERE id = 1;

Key Differences

  • Dynomite focuses on NoSQL key-value storage, while Vitess is designed for scaling relational databases
  • Vitess offers more advanced features for managing distributed databases, but with added complexity
  • Dynomite provides multi-datacenter replication out of the box, while Vitess requires additional configuration for cross-region setups

Both projects aim to solve scalability issues, but Vitess is better suited for applications with complex relational data models, while Dynomite excels in simpler key-value storage scenarios with multi-datacenter requirements.

13,421

Zuul is a gateway service that provides dynamic routing, monitoring, resiliency, security, and more.

Pros of Zuul

  • Designed as an edge service gateway, providing routing, filtering, and load balancing
  • Highly customizable with filters for authentication, security, and traffic management
  • Supports dynamic routing based on request attributes

Cons of Zuul

  • More complex to set up and configure compared to Dynomite
  • Primarily focused on API gateway functionality, less suitable for distributed caching

Code Comparison

Zuul configuration example:

zuul:
  routes:
    users:
      path: /users/**
      serviceId: user-service

Dynomite configuration example:

dyn_o_mite:
  dyn_listen: 0.0.0.0:8101
  data_store: 0
  servers:
    - 127.0.0.1:22122:1

Key Differences

  • Zuul is primarily an API gateway and edge service, while Dynomite is a distributed cache and database proxy
  • Zuul focuses on request routing and filtering, whereas Dynomite emphasizes data replication and sharding
  • Zuul is typically used in microservices architectures for traffic management, while Dynomite is used for improving database performance and scalability

Use Cases

  • Choose Zuul when building a centralized entry point for microservices or need advanced request routing
  • Opt for Dynomite when requiring a distributed caching layer or database proxy for improved scalability and performance
47,330

Distributed reliable key-value store for the most critical data of a distributed system

Pros of etcd

  • Designed for distributed systems and highly reliable
  • Strong consistency model with support for transactions
  • Built-in support for cluster management and leader election

Cons of etcd

  • Higher complexity for simple use cases
  • Potentially slower write performance compared to Dynomite
  • Requires careful configuration for optimal performance

Code Comparison

etcd (Go):

cli, _ := clientv3.New(clientv3.Config{Endpoints: []string{"localhost:2379"}})
defer cli.Close()
ctx, cancel := context.WithTimeout(context.Background(), timeout)
_, err := cli.Put(ctx, "key", "value")

Dynomite (C):

struct node *node = server_pool_find_node(pool, key, keylen);
status = dnode_peer_forward(ctx, node, msg, rack_data_map);

Key Differences

  • etcd is a distributed key-value store, while Dynomite is a proxy system for Redis
  • etcd focuses on consistency and reliability, Dynomite on scalability and performance
  • etcd uses the Raft consensus algorithm, Dynomite uses a gossip protocol
  • etcd is written in Go, Dynomite in C
  • etcd has built-in clustering, Dynomite relies on external tools for cluster management

Both projects serve different purposes and excel in their respective domains. etcd is better suited for distributed systems requiring strong consistency, while Dynomite shines in high-performance, multi-datacenter Redis deployments.

12,362

AWS Service registry for resilient mid-tier load balancing and failover.

Pros of Eureka

  • Designed specifically for service discovery and registration in microservices architectures
  • Provides a RESTful API for service registration, making it easy to integrate with various applications
  • Supports multiple data centers and regions for high availability

Cons of Eureka

  • Limited to service discovery functionality, not a full-fledged data store
  • May require additional components for complete microservices infrastructure
  • Can be complex to set up and configure for large-scale deployments

Code Comparison

Eureka client registration (Java):

@EnableEurekaClient
@SpringBootApplication
public class Application {
    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }
}

Dynomite configuration (YAML):

dyn_o_mite:
  dyn_listen: 127.0.0.1:8101
  data_store: 0
  servers:
   - 127.0.0.1:22122:1

Key Differences

  • Eureka focuses on service discovery, while Dynomite is a distributed cache and database proxy
  • Eureka is primarily used in Java/Spring ecosystems, whereas Dynomite supports multiple data stores and protocols
  • Dynomite provides data replication and sharding capabilities, which are not part of Eureka's core functionality
4,812

HAProxy Load Balancer's development branch (mirror of git.haproxy.org)

Pros of HAProxy

  • Mature and widely adopted load balancer with extensive features
  • Supports multiple protocols (HTTP, TCP, etc.) and advanced traffic management
  • Highly performant and scalable for large-scale deployments

Cons of HAProxy

  • Primarily focused on load balancing and proxying, not distributed caching
  • Requires additional configuration for high availability setups
  • Less specialized for NoSQL database scaling compared to Dynomite

Code Comparison

HAProxy configuration example:

frontend http_front
   bind *:80
   default_backend http_back

backend http_back
   balance roundrobin
   server server1 192.168.1.1:80 check
   server server2 192.168.1.2:80 check

Dynomite configuration example:

dyn_o_mite:
  dyn_listen: 127.0.0.1:8101
  data_store: 0
  servers:
   - 127.0.0.1:22122:1

Key Differences

  • HAProxy is a general-purpose load balancer and proxy, while Dynomite is specifically designed for distributed caching and NoSQL database scaling
  • Dynomite focuses on providing a Redis-compatible interface for distributed systems, whereas HAProxy excels in traffic management and load distribution
  • HAProxy has a larger community and ecosystem, while Dynomite is more specialized for Netflix's use case but still open-source and adaptable

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

dynomite logo

Dynomite

Build Status Dev chat at https://gitter.im/Netflix/dynomite Apache V2 License

Dynomite, inspired by Dynamo whitepaper, is a thin, distributed dynamo layer for different storage engines and protocols. Currently these include Redis and Memcached. Dynomite supports multi-datacenter replication and is designed for high availability.

The ultimate goal with Dynomite is to be able to implement high availability and cross-datacenter replication on storage engines that do not inherently provide that functionality. The implementation is efficient, not complex (few moving parts), and highly performant.

Workflow

Every branch numbered like v0.5.9, v0.5.8 etc is stable and safe to use in production unless marked as pre-release. The dev branch is the development unstable branch. Over time master branch has fallen behind and is not maintained. We will eventually delete it and may or may not recreate it.

For questions or contributions, please consider reading CONTRIBUTING.md.

Build

To build Dynomite from source with debug logs enabled and assertions disabled:

$ git clone git@github.com:Netflix/dynomite.git
$ cd dynomite
$ autoreconf -fvi
$ ./configure --enable-debug=yes
$ make
$ src/dynomite -h

To build Dynomite in debug mode:

$ git clone git@github.com:Netflix/dynomite.git
$ cd dynomite
$ autoreconf -fvi
$ CFLAGS="-ggdb3 -O0" ./configure --enable-debug=full
$ make
$ sudo make install

Help

Usage: dynomite [-?hVdDt] [-v verbosity level] [-o output file]
                  [-c conf file] [-p pid file] 

Options:
  -h, --help              : this help
  -V, --version           : show version and exit
  -t, --test-conf         : test configuration for syntax errors and exit
  -g, --gossip            : enable gossip (default: disabled)
  -d, --daemonize         : run as a daemon
  -D, --describe-stats    : print stats description and exit
  -v, --verbosity=N       : set logging level (default: 5, min: 0, max: 11)
  -o, --output=S          : set logging file (default: stderr)
  -c, --conf-file=S       : set configuration file (default: conf/dynomite.yml)
  -p, --pid-file=S        : set pid file (default: off)
  -x, --admin-operation=N : set size of admin operation (default: 0)

Configuration

Dynomite can be configured through a YAML 1.1 (YAML 1.1 is not JSON compatible) file specified by the -c or --conf-file command-line argument on process start. The configuration files parses and understands the following keys:

  • env: Specify environment of a node. Currently supports aws and network (for physical datacenter).
  • datacenter: The name of the datacenter. Please refer to architecture document.
  • rack: The name of the rack. Please refer to architecture document.
  • dyn_listen: The port that dynomite nodes use to inter-communicate and gossip.
  • enable_gossip: enable gossip instead of static tokens (default: false). Gossip is experimental.
  • gos_interval: The sleeping time in milliseconds at the end of a gossip round.
  • tokens: The token(s) owned by a node. Currently, we don't support vnode yet so this only works with one token for the time being.
  • dyn_seed_provider: A seed provider implementation to provide a list of seed nodes.
  • dyn_seeds: A list of seed nodes in the format: address:port:rack:dc:tokens (note that vnode is not supported yet)
  • listen: The listening address and port (name:port or ip:port) for this server pool.
  • timeout: The timeout value in msec that we wait for to establish a connection to the server or receive a response from a server. By default, we wait indefinitely.
  • preconnect: A boolean value that controls if dynomite should preconnect to all the servers in this pool on process start. Defaults to false.
  • data_store: An integer value that controls if a server pool speaks redis (0) or memcached (1) or other protocol. Defaults to redis (0).
  • auto_eject_hosts: A boolean value that controls if server should be ejected temporarily when it fails consecutively server_failure_limit times. See liveness recommendations for information. Defaults to false.
  • server_retry_timeout: The timeout value in msec to wait for before retrying on a temporarily ejected server, when auto_eject_host is set to true. Defaults to 30000 msec.
  • server_failure_limit: The number of consecutive failures on a server that would lead to it being temporarily ejected when auto_eject_host is set to true. Defaults to 2.
  • servers: A list of local server address, port and weight (name:port:weight or ip:port:weight) for this server pool. Currently, there is just one.
  • secure_server_option: Encrypted communication. Must be one of 'none', 'rack', 'datacenter', or 'all'. datacenter means all communication between datacenters is encrypted but within a datacenter it is not. rack means all communication between racks and regions is encrypted however communication between nodes within the same rack is not encrypted. all means all communication between all nodes is encrypted. And none means none of the communication is encrypted.
  • stats_listen: The address and port number for the REST endpoint and for accessing statistics.
  • stats_interval: set stats aggregation interval in msec (default: 30000 msec).
  • mbuf_size: size of mbuf chunk in bytes (default: 16384 bytes).
  • max_msgs: max number of messages to allocate (default: 200000).
  • datastore_connections: Maximum number of connections to the local datastore.
  • local_peer_connections: Maximum number of connections to a local DC peer.
  • remote_peer_connections: Maximum number of connections to a remote DC peer.
  • dyn_port: Port used by Dynomite servers to talk to each other.

For example, the configuration file in conf/dynomite.yml

Finally, to make writing syntactically correct configuration files easier, dynomite provides a command-line argument -t or --test-conf that can be used to test the YAML configuration file for any syntax error.

License

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0