Convert Figma logo to code with AI

miloyip logonativejson-benchmark

C/C++ JSON parser/generator benchmark

1,979
261
1,979
40
View on GitHub

Top Related Projects

nlohmann's avatar

json

42,154

JSON for Modern C++

simdjson's avatar

simdjson

19,255

Parsing gigabytes of JSON per second : used by Facebook/Meta Velox, the Node.js runtime, ClickHouse, WatermelonDB, Apache Doris, Milvus, StarRocks

Tencent's avatar

rapidjson

14,146

A fast JSON parser/generator for C++ with both SAX/DOM style API

open-source-parsers's avatar

jsoncpp

8,063

A C++ library for interacting with JSON.

Quick Overview

The miloyip/nativejson-benchmark repository is a comprehensive benchmarking suite for JSON libraries in C/C++. It compares the performance of various JSON parsers and generators across different metrics, including parsing speed, stringify speed, and memory usage. This project aims to provide developers with valuable insights into the efficiency of different JSON libraries.

Pros

  • Extensive comparison of multiple JSON libraries in C/C++
  • Detailed performance metrics for parsing, stringifying, and memory usage
  • Regular updates to include new libraries and maintain relevance
  • Provides both numerical results and visual representations of benchmark data

Cons

  • Focused solely on C/C++ libraries, limiting its usefulness for developers working with other languages
  • May not cover all use cases or specific scenarios that could affect performance in real-world applications
  • Benchmark results can become outdated as libraries evolve and improve over time
  • Limited to JSON processing, not covering other data formats or more complex operations

Code Examples

This repository is primarily a benchmarking tool and does not provide a code library for direct use. However, it does include sample code for running benchmarks and integrating various JSON libraries. Here are a few examples of how to use the benchmarking suite:

// Example of running a parsing benchmark
void ParseBenchmark(Benchmark& benchmark, const char* json, size_t length, int repeat) {
    for (int i = 0; i < repeat; i++)
        benchmark.Parse(json, length);
}

// Example of running a stringify benchmark
void StringifyBenchmark(Benchmark& benchmark, const char* json, size_t length, int repeat) {
    Document d;
    d.Parse(json);
    for (int i = 0; i < repeat; i++)
        benchmark.Stringify(d);
}

// Example of measuring memory usage
size_t GetMemoryUsage() {
    // Platform-specific implementation to measure memory usage
    // ...
}

Getting Started

To use this benchmarking suite:

  1. Clone the repository: git clone https://github.com/miloyip/nativejson-benchmark.git
  2. Navigate to the project directory: cd nativejson-benchmark
  3. Build the project using CMake: 
    mkdir build
cd build
cmake ..
make
  4. Run the benchmark executable: ./nativejson-benchmark

Note that you may need to install additional dependencies or JSON libraries depending on which benchmarks you want to run.

Competitor Comparisons

nlohmann logo

json

42,154

JSON for Modern C++

Pros of json

  • Comprehensive and feature-rich JSON library for modern C++
  • Extensive documentation and wide community adoption
  • Header-only implementation, easy to integrate into projects

Cons of json

  • Potentially slower performance for certain operations compared to nativejson-benchmark
  • Larger memory footprint due to its extensive feature set
  • May be overkill for simple JSON parsing needs

Code Comparison

nativejson-benchmark:

rapidjson::Document d;
d.Parse(json);
std::string result = d["key"].GetString();

json:

nlohmann::json j = nlohmann::json::parse(json);
std::string result = j["key"];

Summary

nativejson-benchmark focuses on benchmarking various JSON libraries, while json is a full-featured JSON library for C++. json offers a more user-friendly API and extensive functionality, but may have performance trade-offs compared to some libraries tested in nativejson-benchmark. The choice between them depends on specific project requirements, with json being more suitable for general-purpose JSON handling in C++ applications.

simdjson logo

simdjson

19,255

Parsing gigabytes of JSON per second : used by Facebook/Meta Velox, the Node.js runtime, ClickHouse, WatermelonDB, Apache Doris, Milvus, StarRocks

Pros of simdjson

  • Highly optimized for performance using SIMD instructions
  • Actively maintained with frequent updates and improvements
  • Provides both DOM and streaming parsing capabilities

Cons of simdjson

  • Focused solely on JSON parsing, while nativejson-benchmark covers multiple libraries
  • May have a steeper learning curve due to its low-level optimizations
  • Limited to C++ implementations, whereas nativejson-benchmark includes various languages

Code Comparison

simdjson:

simdjson::dom::parser parser;
simdjson::dom::element doc = parser.load("example.json");
std::string_view title = doc["metadata"]["title"];

nativejson-benchmark (using RapidJSON):

rapidjson::Document d;
d.Parse(json);
const char* title = d["metadata"]["title"].GetString();

Summary

simdjson is a highly optimized JSON parsing library focusing on performance, while nativejson-benchmark is a comprehensive benchmarking suite for various JSON libraries. simdjson offers superior parsing speed but is limited to C++, whereas nativejson-benchmark provides a broader comparison across different implementations and languages. The choice between them depends on whether you need a high-performance parser (simdjson) or a benchmarking tool for multiple JSON libraries (nativejson-benchmark).

Tencent logo

rapidjson

14,146

A fast JSON parser/generator for C++ with both SAX/DOM style API

Pros of RapidJSON

  • Actively maintained and widely used in production environments
  • Comprehensive documentation and extensive feature set
  • Better performance in parsing and serialization

Cons of RapidJSON

  • Larger codebase and more complex API
  • Higher memory usage due to additional features

Code Comparison

RapidJSON:

Document d;
d.Parse(json);
Value& s = d["stars"];
s.SetInt(s.GetInt() + 1);

NativeJSON-benchmark:

// No direct parsing or manipulation code available
// as it's primarily a benchmarking tool

Key Differences

NativeJSON-benchmark is primarily a benchmarking tool for comparing JSON libraries, while RapidJSON is a full-featured JSON library. RapidJSON offers more functionality and is suitable for production use, whereas NativeJSON-benchmark focuses on performance comparisons across different JSON parsers.

RapidJSON provides a complete set of JSON manipulation tools, including parsing, serialization, and DOM-style access. NativeJSON-benchmark, on the other hand, is designed to measure and compare the performance of various JSON libraries, including RapidJSON itself.

While both projects are related to JSON processing, they serve different purposes and are not direct alternatives to each other.

open-source-parsers logo

jsoncpp

8,063

A C++ library for interacting with JSON.

Pros of jsoncpp

  • Full-featured JSON parsing and manipulation library
  • Extensive documentation and community support
  • Widely used in production environments

Cons of jsoncpp

  • Larger codebase and memory footprint
  • Potentially slower performance for simple parsing tasks
  • More complex API compared to lightweight alternatives

Code Comparison

nativejson-benchmark:

Document d;
d.Parse(json);

jsoncpp:

Json::Value root;
Json::Reader reader;
bool parsingSuccessful = reader.parse(json, root);

nativejson-benchmark focuses on benchmarking various JSON parsers, while jsoncpp is a complete JSON manipulation library. nativejson-benchmark provides a simple interface for parsing JSON, whereas jsoncpp offers more extensive functionality for working with JSON data.

jsoncpp is better suited for projects requiring comprehensive JSON handling, while nativejson-benchmark is ideal for performance comparisons and lightweight parsing needs. The choice between them depends on the specific requirements of your project, balancing features against performance and simplicity.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Native JSON Benchmark

Copyright(c) 2014-2016 Milo Yip (miloyip@gmail.com)

Build Status

Introduction

This benchmark evaluates the conformance and performance of 41 open-source C/C++ libraries with JSON parsing/generation capabilities. Performance means speed, memory, and code size.

Performance should be concerned only if the results are correct. This benchmark also test the conformance of library towards the JSON standards (RFC7159, ECMA-404).

Performance of JSON parsing/generation may be critical for server-side applications, mobile/embedded systems, or any application that requires processing of large size or number of JSONs. Native (C/C++) libraries are important because they should provide the best possible performance, while other languages may create bindings of native libraries.

The results show that several performance measurements vary in large scale among libraries. For example, the parsing time can be over 100 times. These differences came from many factors, including design and implementation details. For example, memory allocation strategies, design of variant type for JSON, number-string conversions, etc.

This benchmark may be useful for optimizing existing libraries and developing new, high-performance libraries.

Disclaimer

The original author (Milo Yip) of this benchmark is also the primary author of RapidJSON.

Although the development of benchmark is attempted to be as objective and fair as possible, every benchmarks have their drawbacks, and are limited to particular testing procedures, datasets and platforms. And also, this benchmark does not compare additional features that a library may support, or the user-friendliness of APIs, securities, cross-platform, etc. The author encourage users to benchmarks with their own data sets and platforms.

Benchmarks and Measurements

Conformance

BenchmarkDescription
Parse ValidationUse JSON_checker test suite to test whether the library can identify valid and invalid JSONs. (fail01.json is excluded as it is relaxed in RFC7159. fail18.json is excluded as depth of JSON is not specified.)
Parse Double66 JSONs, each with a decimal value in an array, are parsed. The parsed double values are compared to the correct answer.
Parse String9 JSONs, each with a string value in an array, are parsed. The parsed strings are compared to the correct answer.
Roundtrip27 condensed JSONs are parsed and stringified. The results are compared to the original JSONs.

Performance

BenchmarkDescription
ParseParse in-memory JSON into DOM (tree structure).
StringifySerialize DOM into condensed JSON in memory.
PrettifySerialize DOM into prettified (with indentation and new lines) JSON in memory.
StatisticsTraverse DOM and count the number of JSON types, total length of string, and total numbers of elements/members in array/objects.
Sax Round-tripParse in-memory JSON into events and use events to generate JSON in memory.
Sax StatisticsParse in-memory JSON into events and use events to conduct the statistics.
Code sizeExecutable size in byte. (Currently only support jsonstat program, which calls "Parse" and "Statistics" to print out statistics of a JSON file. )

All benchmarks contain the following measurements:

MeasurementDescription
TimeDuration in millisecond
MemoryMemory consumption in bytes for the result data structure.
MemoryPeakPeak memory consumption in bytes throughout the parsing process.
AllocCountNumber of memory allocation (including malloc, realloc(), new et al.)

Libraries

Currently 43 libraries are successfully benchmarked. They are listed in alphabetic order:

LibraryLanguageVersionNotes
ArduinoJsonC++5.6.6
Boost.JSONC++1.80.0
CAJUNC++2.0.3
C++ REST SDKC++11v2.8.0Need Boost on non-Windows platform. DOM strings must be UTF16 on Windows and UTF8 on non-Windows platform.
ccan/jsonC
cJSONC1.5.0
ConfiguruC++2015-12-18gcc/clang only
dropbox/json11C++11
Facil.ioC0.5.3
FastJsonC++Not parsing number per se, so do it as post-process.
follyC++112016.08.29.00Need installation
gasonC++11
janssonCv2.7
jeayesonC++14
json-cC0.12.1
jsonconsC++110.97.1
json-voorheesC++v1.1.1
json spiritC++4.08Need Boost
Json BoxC++0.6.2
JsonCppC++1.0.0
hjiang/JSON++C++
jsmnCNot parsing number per se, so do it as post-process.
jvarC++v1.0.0gcc/clang only
JzonC++v2-1
nbsdx/SimpleJSONC++11
Nlohmann/jsonC++11v2.0.3
parsonC
picojsonC++1.3.0
pjsonCNo numbers parsing, no DOM interface
POCOC++1.7.5Need installation
qajson4cC1.0.0gcc/clang only
QtC++5.6.1-1Need installation
RapidJSONC++v1.1.0There are four configurations: RapidJSON (default), RapidJSON_AutoUTF (transcoding any UTF JSON), RapidJSON_Insitu (insitu parsing) & RapidJSON_FullPrec (full precision number parsing)
sajsonC++
SimpleJSONC++
sheredom/json.hCNot parsing number per se, so do it as post-process.
udp/jsonC1.1.0Actually 2 libraries: udp/json-parser & udp/json-builder.
taocpp/jsonC++111.0.0-beta.7Uses PEGTL for parsing
tunnuz/JSON++C++
ujsonC++2015-04-12
ujson4cC
V8C++5.1.281.47Need installation
vincenthz/libjsonC0.8
YAJLC2.1.0
ULibC++v1.4.2Need building: (./configure --disable-shared && make)

Libraries with Git repository are included as submodule in thirdparty path. Other libraries are add as files in thirdparty path.

The exact commit of submodule can be navigated at here.

To measure the overheads of the benchmark process, a strdup test is added for comparison. It simply allocate and copy the input string in Parse and Stringify benchmark.

Besides, some libraries was tried to integrated in this benchmark but failed:

LibraryIssue
libjsonUnable to parse UTF-8 string
lastjson
StiX Json
boost property_treenumber, true, false, null types are converted into string.

JSON data

All tested JSON data are in UTF-8.

JSON fileSizeDescription
canada.json source2199KBContour of Canada border in GeoJSON format. Contains a lot of real numbers.
citm_catalog.json source1737KBA big benchmark file with indentation used in several Java JSON parser benchmarks.
twitter.json632KBSearch "一" (character of "one" in Japanese and Chinese) in Twitter public time line for gathering some tweets with CJK characters.

The benchmark program reads data/data.txt which contains file names of JSON to be tested.

Build and Run

  1. Execute git submodule update --init and git -C thirdparty/boost update --init to download all submodules (libraries).
  2. Obtain premake5.
  3. Copy premake5 executable to build/ path (or system path).
  4. Run premake.bat or premake.sh in build/
  5. On Windows, build the solution at build/vs2015/.
  6. On other platforms, run GNU make -f benchmark.make config=release_x32 && make -f nativejson.make config=release_x32 (or release_x64) at build/gmake/
  7. Optional: run build/machine.sh for UNIX or CYGWIN to use CPU info to generate prefix of result filename.
  8. Run the nativejson_release_... executable is generated at bin/
  9. The results in CSV format will be written to result/.
  10. Run GNU make in result/ to generate results in HTML.

For simplicity, on Linux/OSX users can simply run make (or make CONFIG=release_x32) at project root to run 4-10 above.

Some libraries, such as Boost, POCO, V8, etc., need to be installed by user manually.

Sample Results

Update on: 2016-9-9

A collection of benchmarks results can be viewed HERE. Select "Benchmark" from the menu to check available benchmark configurations. The presentation is powered by Google Charts with interactivity.

The followings are some snapshots from the results of MacBook Pro (Retina, 15-inch, Mid 2015, Corei7-4980HQ@2.80GHz) with clang 7.0 64-bit.

Conformance

Conformance

This is the average score of 4 conformance benchmarks. Higher is better. Details.

Parsing Time

Parsing Time

This is the total duration of parsing 3 JSONs to DOM representation, sorted in ascending order. Lower is better. Details

Parsing Memory

Parsing Memory

This is the total memory after parsing 3 JSONs to DOM representation, sorted in ascending order. Lower is better. Details

(Note: The results for Qt is incorrect as the benchmark failed to hook its memory allocations)

Stringify Time

Stringify Time

This is the total duration of stringifying 3 DOMs to JSONs, sorted in ascending order. Lower is better. Details

Prettify Time

Prettify Time

This is the total duration of prettifying 3 DOMs to JSONs, sorted in ascending order. Lower is better. Details

Code Size

Code Size

The is the size of executable program, which parses a JSON from stdin to a DOM and then computes the statistics of the DOM. Lower is better. Details

FAQ

  1. How to add a library?

    Use submodule add https://...xxx.git thirdparty/xxx to add the libary's repository as a submobule. If that is not possible, just copy the files into 'thirdparty/xxx'.

    For C libary, add a xxx_all.c in src/cjsonlibs, which #include all the necessary .c files of the library. And then create a tests/xxxtest.cpp.it

    For C++ library, just need to create a tests/xxxtest.cpp, which #include all the necessary .cpp files of the library.

    You may find a existing library which similar to your case as a start of implementing tests/xxxtest.cpp.

    Please submit a pull request if the integration work properly.

  2. What if the build process is failed, or the benchmark crashes?

    This happens on some platforms as not every libaray is stable for all platforms.

    You can simply delete the local tests/xxxtest.cpp and re-run the build.

    BTW, if you are adding a library, you can remove all tests/xxxtest.cpp except tests/rapidjsontest.cpp and your test. This make the build process fast. RapidJSON's test is needed as a reference to compare statistics results only.

  3. On which platform the benchmark can be run?

    The author tests it on OSX/clang, Winwdows/vs2015, Ubuntu/clang3.7+gcc5.0 (via Travis CI). The benchmark may work in other platforms as well but you will need to generate the build files via premake5, and resolve any potential issues.

  4. How to reduce the build process time?

    You can preserve the tests of libaries that you only need, as in question 2. You can also use make ARGS=--verify-only, make ARGS=--performance-only, make ARGS=--conformance-only to execute the required part.

Other native JSON benchmarks