snappy

A fast compressor/decompressor

6,094

975

6,094

View on GitHub

Top Related Projects

lz4

10,226

Extremely Fast Compression algorithm

zopfli

3,418

Zopfli Compression Algorithm is a compression library programmed in C to perform very good, but slow, deflate or zlib compression.

Quick Overview

Snappy is a fast compression/decompression library developed by Google. It aims to provide high-speed data compression and decompression, focusing on speed rather than maximum compression ratio. Snappy is widely used in various Google projects and is designed for performance-critical systems.

Pros

Extremely fast compression and decompression speeds
Low memory usage during compression/decompression operations
Stable and well-maintained by Google
Cross-platform support (Linux, Windows, macOS)

Cons

Lower compression ratio compared to other compression algorithms
Not suitable for applications requiring maximum data reduction
Limited file format support (primarily designed for in-memory compression)
May not be the best choice for compressing already-compressed data

Code Examples

Compressing a string:

#include <snappy.h>
#include <string>

std::string input = "Hello, Snappy!";
std::string compressed;
snappy::Compress(input.data(), input.size(), &compressed);

Decompressing a string:

#include <snappy.h>
#include <string>

std::string compressed; // Assume this contains compressed data
std::string decompressed;
snappy::Uncompress(compressed.data(), compressed.size(), &decompressed);

Checking if a string is valid Snappy compressed data:

#include <snappy.h>
#include <string>

std::string data; // Assume this contains some data
bool is_valid = snappy::IsValidCompressedBuffer(data.data(), data.size());

Getting Started

To use Snappy in your C++ project:

Install Snappy using your package manager or build from source.
Include the Snappy headers in your project.
Link against the Snappy library.

Here's a simple example of how to use Snappy in a C++ program:

#include <iostream>
#include <string>
#include <snappy.h>

int main() {
    std::string input = "This is a test string for Snappy compression.";
    std::string compressed;
    std::string decompressed;

    // Compress the input
    snappy::Compress(input.data(), input.size(), &compressed);

    // Decompress the compressed data
    snappy::Uncompress(compressed.data(), compressed.size(), &decompressed);

    std::cout << "Original size: " << input.size() << std::endl;
    std::cout << "Compressed size: " << compressed.size() << std::endl;
    std::cout << "Decompressed size: " << decompressed.size() << std::endl;
    std::cout << "Decompressed matches input: " << (input == decompressed ? "Yes" : "No") << std::endl;

    return 0;
}

Compile this program with the Snappy library linked, and you're ready to start using Snappy in your projects.

Competitor Comparisons

lz4

10,226

Extremely Fast Compression algorithm

Pros of lz4

Significantly faster decompression speed
Better compression ratio in most cases
Supports streaming compression and decompression

Cons of lz4

Slightly slower compression speed
Less mature and widespread adoption compared to Snappy

Code Comparison

lz4

char* compressed = malloc(LZ4_compressBound(inputSize));
int compressedSize = LZ4_compress_default(input, compressed, inputSize, LZ4_compressBound(inputSize));

Snappy

std::string output;
snappy::Compress(input_data, input_size, &output);

Both lz4 and Snappy are fast compression libraries designed for high-speed data compression and decompression. While Snappy focuses on very fast compression and reasonable compression ratios, lz4 aims to provide extremely fast decompression with good compression ratios.

lz4 generally offers better compression ratios and faster decompression speeds, making it ideal for scenarios where decompression performance is critical. However, Snappy may have a slight edge in compression speed and has been around longer, potentially leading to wider adoption in some ecosystems.

The code examples demonstrate the simplicity of using both libraries for basic compression tasks. lz4 requires manual memory allocation, while Snappy handles it internally in this example.

brotli

13,454

Brotli compression format

Pros of Brotli

Higher compression ratio, especially for text-based content
Better performance for web content delivery and HTTPS compression
Supports dictionary compression for improved efficiency

Cons of Brotli

Slower compression speed compared to Snappy
Higher memory usage during compression
Less suitable for real-time compression scenarios

Code Comparison

Brotli:

size_t output_size = BrotliEncoderMaxCompressedSize(input_size);
uint8_t* output = malloc(output_size);
BrotliEncoderCompress(quality, BROTLI_DEFAULT_WINDOW, BROTLI_DEFAULT_MODE,
                      input_size, input, &output_size, output);

Snappy:

size_t output_size = snappy_max_compressed_length(input_size);
char* output = malloc(output_size);
snappy_compress(input, input_size, output, &output_size);

Key Differences

Brotli focuses on high compression ratios, while Snappy prioritizes speed
Brotli is better suited for web content and static file compression
Snappy excels in scenarios requiring fast compression and decompression
Brotli offers more compression options and fine-tuning capabilities
Snappy has a simpler API and is easier to integrate into existing projects

Both libraries are maintained by Google and have their specific use cases. Brotli is preferred for web content and static file compression, while Snappy is ideal for real-time compression and scenarios where speed is crucial.

zopfli

3,418

Zopfli Compression Algorithm is a compression library programmed in C to perform very good, but slow, deflate or zlib compression.

Pros of Zopfli

Achieves higher compression ratios than Snappy
Compatible with existing DEFLATE decoders
Suitable for scenarios where compression time is not critical

Cons of Zopfli

Significantly slower compression speed compared to Snappy
Higher computational resource requirements
Not ideal for real-time or low-latency applications

Code Comparison

Snappy (C++):

char* output = new char[maxCompressedLength];
size_t output_length;
RawCompress(input, input_length, output, &output_length);

Zopfli (C):

unsigned char* out = 0;
size_t outsize = 0;
ZopfliCompress(&options, ZopfliFormat, in, insize, &out, &outsize);

Both libraries offer simple APIs for compression, but Zopfli's implementation focuses on achieving better compression ratios at the cost of speed. Snappy, on the other hand, prioritizes fast compression and decompression speeds.

Snappy is better suited for scenarios requiring quick data compression and decompression, such as in-memory data storage or network transmission. Zopfli is more appropriate for offline compression tasks where compression ratio is paramount, such as reducing file sizes for distribution or storage.

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Snappy, a fast compressor/decompressor.

Introduction

Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. (For more information, see "Performance", below.)

Snappy has the following properties:

Fast: Compression speeds at 250 MB/sec and beyond, with no assembler code. See "Performance" below.
Stable: Over the last few years, Snappy has compressed and decompressed petabytes of data in Google's production environment. The Snappy bitstream format is stable and will not change between versions.
Robust: The Snappy decompressor is designed not to crash in the face of corrupted or malicious input.
Free and open source software: Snappy is licensed under a BSD-type license. For more information, see the included COPYING file.

Snappy has previously been called "Zippy" in some Google presentations and the like.

Performance

Snappy is intended to be fast. On a single core of a Core i7 processor in 64-bit mode, it compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more. (These numbers are for the slowest inputs in our benchmark suite; others are much faster.) In our tests, Snappy usually is faster than algorithms in the same class (e.g. LZO, LZF, QuickLZ, etc.) while achieving comparable compression ratios.

Typical compression ratios (based on the benchmark suite) are about 1.5-1.7x for plain text, about 2-4x for HTML, and of course 1.0x for JPEGs, PNGs and other already-compressed data. Similar numbers for zlib in its fastest mode are 2.6-2.8x, 3-7x and 1.0x, respectively. More sophisticated algorithms are capable of achieving yet higher compression rates, although usually at the expense of speed. Of course, compression ratio will vary significantly with the input.

Although Snappy should be fairly portable, it is primarily optimized for 64-bit x86-compatible processors, and may run slower in other environments. In particular:

Snappy uses 64-bit operations in several places to process more data at once than would otherwise be possible.
Snappy assumes unaligned 32 and 64-bit loads and stores are cheap. On some platforms, these must be emulated with single-byte loads and stores, which is much slower.
Snappy assumes little-endian throughout, and needs to byte-swap data in several places if running on a big-endian platform.

Experience has shown that even heavily tuned code can be improved. Performance optimizations, whether for 64-bit x86 or other platforms, are of course most welcome; see "Contact", below.

Building

You need the CMake version specified in CMakeLists.txt or later to build:

git submodule update --init
mkdir build
cd build && cmake ../ && make

Usage

Note that Snappy, both the implementation and the main interface, is written in C++. However, several third-party bindings to other languages are available; see the home page for more information. Also, if you want to use Snappy from C code, you can use the included C bindings in snappy-c.h.

To use Snappy from your own C++ program, include the file "snappy.h" from your calling file, and link against the compiled library.

There are many ways to call Snappy, but the simplest possible is

snappy::Compress(input.data(), input.size(), &output);

and similarly

snappy::Uncompress(input.data(), input.size(), &output);

where "input" and "output" are both instances of std::string.

There are other interfaces that are more flexible in various ways, including support for custom (non-array) input sources. See the header file for more information.

Tests and benchmarks

When you compile Snappy, the following binaries are compiled in addition to the library itself. You do not need them to use the compressor from your own library, but they are useful for Snappy development.

snappy_benchmark contains microbenchmarks used to tune compression and decompression performance.
snappy_unittests contains unit tests, verifying correctness on your machine in various scenarios.
snappy_test_tool can benchmark Snappy against a few other compression libraries (zlib, LZO, LZF, and QuickLZ), if they were detected at configure time. To benchmark using a given file, give the compression algorithm you want to test Snappy against (e.g. --zlib) and then a list of one or more file names on the command line.

If you want to change or optimize Snappy, please run the tests and benchmarks to verify you have not broken anything.

The testdata/ directory contains the files used by the microbenchmarks, which should provide a reasonably balanced starting point for benchmarking. (Note that baddata[1-3].snappy are not intended as benchmarks; they are used to verify correctness in the presence of corrupted data in the unit test.)

Contributing to the Snappy Project

In addition to the aims listed at the top of the README Snappy explicitly supports the following:

C++11
Clang (gcc and MSVC are best-effort).
Low level optimizations (e.g. assembly or equivalent intrinsics) for:
- x86
- x86-64
- ARMv7 (32-bit)
- ARMv8 (AArch64)
Supports only the Snappy compression scheme as described in format_description.txt.
CMake for building

Changes adding features or dependencies outside of the core area of focus listed above might not be accepted. If in doubt post a message to the Snappy discussion mailing list.

We are unlikely to accept contributions to the build configuration files, such as CMakeLists.txt. We are focused on maintaining a build configuration that allows us to test that the project works in a few supported configurations inside Google. We are not currently interested in supporting other requirements, such as different operating systems, compilers, or build systems.

Contact

Snappy is distributed through GitHub. For the latest version and other information, see https://github.com/google/snappy.

Top Related Projects

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot