snowflake
Snowflake is a network service for generating unique ID numbers at high scale with some simple guarantees.
Top Related Projects
Quick Overview
Snowflake is a network service for generating unique ID numbers at high scale with some simple guarantees. It was originally created by Twitter to provide unique IDs for tweets and other entities in their system. Snowflake is designed to generate roughly sortable unique 64-bit IDs in a distributed environment.
Pros
- High scalability: Can generate up to 4096 unique IDs per millisecond per process
- Roughly time sortable: IDs can be used to determine the order of events
- Distributed: Can be run on multiple machines without coordination
- Customizable: Allows for custom epoch and machine ID configuration
Cons
- Requires synchronized clocks: Relies on system clocks being relatively in sync
- Potential for duplicate IDs: If system clock moves backwards, duplicates can occur
- Limited lifespan: Will eventually run out of IDs (after ~69 years with default settings)
- Not cryptographically secure: IDs are predictable and should not be used for security purposes
Code Examples
- Generating a new Snowflake ID:
IdWorker idWorker = new IdWorker(0, 0);
long id = idWorker.nextId();
System.out.println("Generated ID: " + id);
- Extracting timestamp from a Snowflake ID:
long id = 1234567890123456789L;
long timestamp = (id >> 22) + IdWorker.twepoch;
System.out.println("Timestamp: " + new Date(timestamp));
- Creating a custom IdWorker with a different epoch:
long customEpoch = 1609459200000L; // January 1, 2021
IdWorker customIdWorker = new IdWorker(0, 0, customEpoch);
long customId = customIdWorker.nextId();
System.out.println("Custom ID: " + customId);
Getting Started
To use Snowflake in your Java project:
-
Clone the repository:
git clone https://github.com/twitter-archive/snowflake.git
-
Build the project:
cd snowflake mvn clean install
-
Add the dependency to your project's
pom.xml
:<dependency> <groupId>com.twitter</groupId> <artifactId>snowflake</artifactId> <version>1.0.0</version> </dependency>
-
Use the
IdWorker
class in your code:import com.twitter.snowflake.IdWorker; IdWorker idWorker = new IdWorker(0, 0); long id = idWorker.nextId();
Competitor Comparisons
A distributed unique ID generator inspired by Twitter's Snowflake
Pros of Sonyflake
- Written in Go, offering better performance and concurrency
- Supports custom epoch times, allowing for more flexible timestamp ranges
- Provides a simpler API with fewer dependencies
Cons of Sonyflake
- Less widely adopted compared to Snowflake
- Lacks some advanced features like worker ID assignment
Code Comparison
Snowflake (Ruby):
require 'snowflake-id'
snowflake = Snowflake::Id.new(worker_id: 1, datacenter_id: 1)
id = snowflake.next_id
Sonyflake (Go):
import "github.com/sony/sonyflake"
flake := sonyflake.NewSonyflake(sonyflake.Settings{})
id, err := flake.NextID()
Both Snowflake and Sonyflake are distributed unique ID generators, but they differ in implementation and features. Snowflake, originally developed by Twitter, is widely adopted and has implementations in various languages. Sonyflake, developed by Sony, offers a Go-based alternative with some unique features.
Sonyflake provides better performance due to Go's efficiency and supports custom epoch times. However, it has a smaller community and fewer advanced features compared to Snowflake. The choice between the two depends on specific project requirements, language preferences, and desired features.
Universally Unique Lexicographically Sortable Identifier (ULID) in Go
Pros of ULID
- Language-agnostic implementation with support for multiple programming languages
- Sortable by timestamp, allowing for easy chronological ordering
- Compact representation (26 characters) compared to Snowflake's 64-bit integer
Cons of ULID
- Lacks built-in distributed system support, unlike Snowflake's datacenter-aware design
- Potential for collisions in high-volume scenarios due to limited entropy
Code Comparison
Snowflake (Java):
long id = snowflake.nextId();
ULID (Go):
id := ulid.New()
Key Differences
- Snowflake focuses on distributed systems and high-scale environments
- ULID prioritizes simplicity and cross-language compatibility
- Snowflake uses a 64-bit integer, while ULID uses a 26-character string
- Snowflake includes machine ID in its structure, ULID does not
Use Cases
Snowflake:
- Large-scale distributed systems
- High-throughput environments requiring unique IDs
ULID:
- Cross-platform applications
- Systems prioritizing human-readable and sortable IDs
Both solutions offer unique identifier generation, but their designs cater to different priorities and use cases.
Collision-resistant ids optimized for horizontal scaling and performance.
Pros of cuid
- Simpler implementation, easier to understand and maintain
- More portable, can be used in various environments (browser, server, mobile)
- Generates shorter IDs, which can be beneficial for storage and transmission
Cons of cuid
- Less scalable for extremely high-volume distributed systems
- May have a slightly higher chance of collisions in certain scenarios
- Lacks some advanced features like worker ID and datacenter ID
Code Comparison
Snowflake (Java):
public synchronized long nextId() {
long timestamp = timeGen();
if (timestamp < lastTimestamp) {
throw new RuntimeException("Clock moved backwards.");
}
if (lastTimestamp == timestamp) {
sequence = (sequence + 1) & sequenceMask;
if (sequence == 0) {
timestamp = tilNextMillis(lastTimestamp);
}
} else {
sequence = 0L;
}
lastTimestamp = timestamp;
return ((timestamp - twepoch) << timestampLeftShift) |
(datacenterId << datacenterIdShift) |
(workerId << workerIdShift) |
sequence;
}
cuid (JavaScript):
function cuid() {
return 'c' + (Date.now()).toString(36) +
counter().toString(36).slice(-4) +
fingerprint() +
random();
}
The Snowflake implementation is more complex and includes features for distributed systems, while cuid offers a simpler approach that's easier to implement across different platforms.
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual CopilotREADME
We have retired the initial release of Snowflake and working on open sourcing the next version based on Twitter-server, in a form that can run anywhere without requiring Twitter's own infrastructure services.
The initial version, released in 2010, was based on Apache Thrift and it predated Finagle, our building block for RPC services at Twitter. The Snowflake we're using internally is a full rewrite and heavily relies on existing infrastructure at Twitter to run. We cannot commit to a date but we're doing our best to add necessary features to make Snowflake fit for many environments outside of Twitter.
Source code is still in the repository and is reachable from snowflake-2010 tag.
We won't be accepting pull requests or responding to issues for the retired release.
Top Related Projects
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual Copilot