fastjson

Fast JSON parser and validator for Go. No custom structs, no code generation, no reflection

2,247

135

2,247

View on GitHub

Top Related Projects

simdjson

19,071

Parsing gigabytes of JSON per second : used by Facebook/Meta Velox, the Node.js runtime, ClickHouse, WatermelonDB, Apache Doris, Milvus, StarRocks

jsonparser

5,427

One of the fastest alternative JSON parser for Go that does not require schema

go

13,330

A high-performance 100% compatible drop-in replacement of "encoding/json"

gjson

14,102

Get JSON values quickly - JSON parser for Go

Quick Overview

FastJSON is a fast JSON parser and validator for Go. It aims to provide high performance JSON processing capabilities, particularly for large JSON files or streams, while maintaining a simple API.

Pros

Extremely fast JSON parsing and validation
Low memory usage compared to standard library's encoding/json
Support for streaming large JSON files
Simple and easy-to-use API

Cons

Less feature-rich compared to the standard library's encoding/json
May not support all edge cases or complex JSON structures
Limited documentation and examples
Not as widely adopted as the standard library solution

Code Examples

Parsing JSON:

var p fastjson.Parser
v, err := p.Parse(`{"name":"John","age":30}`)
if err != nil {
    log.Fatal(err)
}
fmt.Println(v.Get("name").String())  // Output: John
fmt.Println(v.Get("age").Int())      // Output: 30

Validating JSON:

err := fastjson.Validate(`{"key": "value"}`)
if err != nil {
    log.Fatal("Invalid JSON")
}
fmt.Println("JSON is valid")

Working with arrays:

var p fastjson.Parser
v, _ := p.Parse(`["apple", "banana", "cherry"]`)
a, _ := v.Array()
for i, item := range a {
    fmt.Printf("Item %d: %s\n", i, item.String())
}

Getting Started

To use FastJSON in your Go project, follow these steps:

Install the package:
```
go get -u github.com/valyala/fastjson
```
Import the package in your Go code:
```
import "github.com/valyala/fastjson"
```

Use the parser to parse JSON:

var p fastjson.Parser
v, err := p.Parse(`{"key": "value"}`)
if err != nil {
    log.Fatal(err)
}
fmt.Println(v.Get("key").String())

That's it! You can now use FastJSON for high-performance JSON parsing and validation in your Go applications.

Competitor Comparisons

simdjson

19,071

Parsing gigabytes of JSON per second : used by Facebook/Meta Velox, the Node.js runtime, ClickHouse, WatermelonDB, Apache Doris, Milvus, StarRocks

Pros of simdjson

Utilizes SIMD instructions for faster parsing
Supports both DOM and streaming APIs
Extensively tested and benchmarked across various architectures

Cons of simdjson

Larger codebase and more complex implementation
Requires C++17 or later, limiting compatibility with older systems

Code Comparison

simdjson:

simdjson::dom::parser parser;
simdjson::dom::element doc = parser.load("example.json");
std::string_view title = doc["metadata"]["title"];

fastjson:

var p fastjson.Parser
v, err := p.Parse(jsonStr)
title := v.GetStringBytes("metadata", "title")

Key Differences

Language: simdjson is written in C++, while fastjson is in Go
API: simdjson offers both DOM and streaming APIs, fastjson focuses on a simple API
Performance: Both are designed for high performance, but simdjson leverages SIMD instructions
Memory usage: fastjson generally uses less memory due to its simpler design
Ecosystem: simdjson has a larger community and more extensive documentation

Use Cases

simdjson: Ideal for C++ projects requiring maximum parsing speed and flexibility
fastjson: Well-suited for Go projects needing fast, lightweight JSON parsing with minimal dependencies

jsonparser

5,427

One of the fastest alternative JSON parser for Go that does not require schema

Pros of jsonparser

More flexible API with support for various data types and nested structures
Better handling of escaped characters and Unicode
Actively maintained with regular updates and improvements

Cons of jsonparser

Slightly slower performance compared to fastjson
Higher memory usage due to more comprehensive parsing

Code Comparison

jsonparser:

data := []byte(`{"name": "John", "age": 30}`)
name, err := jsonparser.GetString(data, "name")
age, err := jsonparser.GetInt(data, "age")

fastjson:

var p fastjson.Parser
v, err := p.Parse(`{"name": "John", "age": 30}`)
name := v.GetStringBytes("name")
age := v.GetInt("age")

Both libraries offer efficient JSON parsing in Go, but they have different strengths. jsonparser provides a more feature-rich API with better support for complex JSON structures, while fastjson focuses on raw performance and lower memory usage. The choice between them depends on the specific requirements of your project, such as parsing complexity, performance needs, and memory constraints.

go

13,330

A high-performance 100% compatible drop-in replacement of "encoding/json"

Pros of json-iterator/go

Offers both ease of use and high performance
Provides a drop-in replacement for the standard library's encoding/json
Supports custom extensions and more flexible configuration options

Cons of json-iterator/go

Slightly larger memory footprint compared to fastjson
May have a steeper learning curve for advanced features

Code Comparison

fastjson:

var p fastjson.Parser
v, err := p.Parse(jsonStr)
if err != nil {
    // handle error
}
name := v.GetStringBytes("name")

json-iterator/go:

var json = jsoniter.ConfigCompatibleWithStandardLibrary
var result map[string]interface{}
err := json.Unmarshal([]byte(jsonStr), &result)
if err != nil {
    // handle error
}
name := result["name"].(string)

Key Differences

fastjson focuses on parsing JSON without unmarshaling, which can be faster for certain use cases
json-iterator/go provides a more familiar API for developers used to the standard library
fastjson has a smaller memory footprint, while json-iterator/go offers more features and flexibility

Both libraries aim to improve JSON parsing performance in Go, but they take different approaches. The choice between them depends on specific project requirements and performance needs.

gjson

14,102

Get JSON values quickly - JSON parser for Go

Pros of gjson

More feature-rich with advanced query capabilities
Supports modification and creation of JSON
Better documentation and examples

Cons of gjson

Slightly slower performance for simple operations
Larger codebase and API surface

Code Comparison

gjson:

value := gjson.Get(json, "name.last")

fastjson:

var p fastjson.Parser
v, _ := p.Parse(json)
value := v.Get("name", "last")

Both libraries offer efficient JSON parsing and value retrieval, but gjson provides a more concise API for simple operations. fastjson requires creating a parser instance and handling potential errors explicitly.

gjson excels in complex queries and manipulations, offering a powerful path syntax:

result := gjson.Get(json, "users.#(name==Alice).age")

fastjson doesn't have built-in support for such advanced queries, requiring manual traversal and filtering.

Overall, gjson is more user-friendly and feature-rich, while fastjson focuses on raw performance for basic operations. The choice between them depends on specific project requirements, with gjson being more suitable for complex JSON handling and fastjson for high-performance, simple parsing tasks.

easyjson

4,442

Fast JSON serializer for golang.

Pros of easyjson

Generates Go code for faster JSON encoding/decoding
Supports custom types and interfaces
Integrates well with existing Go code and structures

Cons of easyjson

Requires code generation step, which can complicate build process
May produce larger binary sizes due to generated code
Less flexible for dynamic JSON structures

Code Comparison

easyjson:

//easyjson:json
type Person struct {
    Name string `json:"name"`
    Age  int    `json:"age"`
}

fastjson:

var p fastjson.Parser
v, err := p.Parse(`{"name":"John","age":30}`)
name := v.GetStringBytes("name")
age := v.GetInt("age")

Key Differences

easyjson focuses on compile-time code generation for performance
fastjson emphasizes runtime parsing and manipulation of JSON
easyjson works well with static Go structures
fastjson is more flexible for dynamic JSON handling

Use Cases

easyjson: Best for projects with well-defined, static JSON structures
fastjson: Ideal for applications dealing with varied or unknown JSON formats

Both libraries aim to provide high-performance JSON handling in Go, but take different approaches to achieve this goal.

oj

3,132

Optimized JSON

Pros of oj

Written in C, offering potentially better performance for Ruby applications
Supports both parsing and generation of JSON
Extensive feature set including custom parsing modes and options

Cons of oj

Limited to Ruby ecosystem, while fastjson is for Go
May require more setup and configuration for optimal performance
Larger codebase, potentially more complex to maintain

Code Comparison

oj (Ruby):

require 'oj'

json_str = '{"name":"John","age":30}'
parsed = Oj.load(json_str)
puts parsed['name']  # Output: John

obj = { name: "Alice", age: 25 }
json = Oj.dump(obj)
puts json  # Output: {"name":"Alice","age":25}

fastjson (Go):

import "github.com/valyala/fastjson"

var p fastjson.Parser
v, _ := p.Parse(`{"name":"John","age":30}`)
fmt.Println(v.Get("name").String())  // Output: John

// Note: fastjson focuses on parsing, not generation

Both libraries aim to provide fast JSON processing, but they target different languages and have distinct feature sets. oj offers a more comprehensive JSON toolkit for Ruby, while fastjson is a lightweight, performance-focused parsing library for Go.

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

fastjson - fast JSON parser and validator for Go

Features

Fast. As usual, up to 15x faster than the standard encoding/json. See benchmarks.
Parses arbitrary JSON without schema, reflection, struct magic and code generation contrary to easyjson.
Provides simple API.
Outperforms jsonparser and gjson when accessing multiple unrelated fields, since fastjson parses the input JSON only once.
Validates the parsed JSON unlike jsonparser and gjson.
May quickly extract a part of the original JSON with Value.Get(...).MarshalTo and modify it with Del and Set functions.
May parse array containing values with distinct types (aka non-homogenous types). For instance, fastjson easily parses the following JSON array [123, "foo", [456], {"k": "v"}, null].
fastjson preserves the original order of object items when calling Object.Visit.

Known limitations

Requies extra care to work with - references to certain objects recursively returned by Parser must be released before the next call to Parse. Otherwise the program may work improperly. The same applies to objects returned by Arena. Adhere recommendations from docs.
Cannot parse JSON from io.Reader. There is Scanner for parsing stream of JSON values from a string.

Usage

One-liner accessing a single field:

	s := []byte(`{"foo": [123, "bar"]}`)
	fmt.Printf("foo.0=%d\n", fastjson.GetInt(s, "foo", "0"))

	// Output:
	// foo.0=123

Accessing multiple fields with error handling:

        var p fastjson.Parser
        v, err := p.Parse(`{
                "str": "bar",
                "int": 123,
                "float": 1.23,
                "bool": true,
                "arr": [1, "foo", {}]
        }`)
        if err != nil {
                log.Fatal(err)
        }
        fmt.Printf("foo=%s\n", v.GetStringBytes("str"))
        fmt.Printf("int=%d\n", v.GetInt("int"))
        fmt.Printf("float=%f\n", v.GetFloat64("float"))
        fmt.Printf("bool=%v\n", v.GetBool("bool"))
        fmt.Printf("arr.1=%s\n", v.GetStringBytes("arr", "1"))

        // Output:
        // foo=bar
        // int=123
        // float=1.230000
        // bool=true
        // arr.1=foo

Security

fastjson shouldn't crash or panic when parsing input strings specially crafted by an attacker. It must return error on invalid input JSON.
fastjson requires up to sizeof(Value) * len(inputJSON) bytes of memory for parsing inputJSON string. Limit the maximum size of the inputJSON before parsing it in order to limit the maximum memory usage.

Performance optimization tips

Re-use Parser and Scanner for parsing many JSONs. This reduces memory allocations overhead. ParserPool may be useful in this case.
Prefer calling Value.Get* on the value returned from Parser instead of calling Get* one-liners when multiple fields must be obtained from JSON, since each Get* one-liner re-parses the input JSON again.
Prefer calling once Value.Get for common prefix paths and then calling Value.Get* on the returned value for distinct suffix paths.
Prefer iterating over array returned from Value.GetArray with a range loop instead of calling Value.Get* for each array item.

Fuzzing

Install go-fuzz & optionally the go-fuzz-corpus.

go get -u github.com/dvyukov/go-fuzz/go-fuzz github.com/dvyukov/go-fuzz/go-fuzz-build

Build using go-fuzz-build and run go-fuzz with an optional corpus.

mkdir -p workdir/corpus
cp $GOPATH/src/github.com/dvyukov/go-fuzz-corpus/json/corpus/* workdir/corpus
go-fuzz-build github.com/valyala/fastjson
go-fuzz -bin=fastjson-fuzz.zip -workdir=workdir

Benchmarks

Go 1.12 has been used for benchmarking.

Legend:

small - parse small.json (190 bytes).
medium - parse medium.json (2.3KB).
large - parse large.json (28KB).
canada - parse canada.json (2.2MB).
citm - parse citm_catalog.json (1.7MB).
twitter - parse twitter.json (617KB).
stdjson-map - parse into a map[string]interface{} using encoding/json.
stdjson-struct - parse into a struct containing a subset of fields of the parsed JSON, using encoding/json.
stdjson-empty-struct - parse into an empty struct using encoding/json. This is the fastest possible solution for encoding/json, may be used for json validation. See also benchmark results for json validation.
fastjson - parse using fastjson without fields access.
fastjson-get - parse using fastjson with fields access similar to stdjson-struct.

$ GOMAXPROCS=1 go test github.com/valyala/fastjson -bench='Parse$'
goos: linux
goarch: amd64
pkg: github.com/valyala/fastjson
BenchmarkParse/small/stdjson-map         	  200000	      7305 ns/op	  26.01 MB/s	     960 B/op	      51 allocs/op
BenchmarkParse/small/stdjson-struct      	  500000	      3431 ns/op	  55.37 MB/s	     224 B/op	       4 allocs/op
BenchmarkParse/small/stdjson-empty-struct         	  500000	      2273 ns/op	  83.58 MB/s	     168 B/op	       2 allocs/op
BenchmarkParse/small/fastjson                     	 5000000	       347 ns/op	 547.53 MB/s	       0 B/op	       0 allocs/op
BenchmarkParse/small/fastjson-get                 	 2000000	       620 ns/op	 306.39 MB/s	       0 B/op	       0 allocs/op
BenchmarkParse/medium/stdjson-map                 	   30000	     40672 ns/op	  57.26 MB/s	   10196 B/op	     208 allocs/op
BenchmarkParse/medium/stdjson-struct              	   30000	     47792 ns/op	  48.73 MB/s	    9174 B/op	     258 allocs/op
BenchmarkParse/medium/stdjson-empty-struct        	  100000	     22096 ns/op	 105.40 MB/s	     280 B/op	       5 allocs/op
BenchmarkParse/medium/fastjson                    	  500000	      3025 ns/op	 769.90 MB/s	       0 B/op	       0 allocs/op
BenchmarkParse/medium/fastjson-get                	  500000	      3211 ns/op	 725.20 MB/s	       0 B/op	       0 allocs/op
BenchmarkParse/large/stdjson-map                  	    2000	    614079 ns/op	  45.79 MB/s	  210734 B/op	    2785 allocs/op
BenchmarkParse/large/stdjson-struct               	    5000	    298554 ns/op	  94.18 MB/s	   15616 B/op	     353 allocs/op
BenchmarkParse/large/stdjson-empty-struct         	    5000	    268577 ns/op	 104.69 MB/s	     280 B/op	       5 allocs/op
BenchmarkParse/large/fastjson                     	   50000	     35210 ns/op	 798.56 MB/s	       5 B/op	       0 allocs/op
BenchmarkParse/large/fastjson-get                 	   50000	     35171 ns/op	 799.46 MB/s	       5 B/op	       0 allocs/op
BenchmarkParse/canada/stdjson-map                 	      20	  68147307 ns/op	  33.03 MB/s	12260502 B/op	  392539 allocs/op
BenchmarkParse/canada/stdjson-struct              	      20	  68044518 ns/op	  33.08 MB/s	12260123 B/op	  392534 allocs/op
BenchmarkParse/canada/stdjson-empty-struct        	     100	  17709250 ns/op	 127.11 MB/s	     280 B/op	       5 allocs/op
BenchmarkParse/canada/fastjson                    	     300	   4182404 ns/op	 538.22 MB/s	  254902 B/op	     381 allocs/op
BenchmarkParse/canada/fastjson-get                	     300	   4274744 ns/op	 526.60 MB/s	  254902 B/op	     381 allocs/op
BenchmarkParse/citm/stdjson-map                   	      50	  27772612 ns/op	  62.19 MB/s	 5214163 B/op	   95402 allocs/op
BenchmarkParse/citm/stdjson-struct                	     100	  14936191 ns/op	 115.64 MB/s	    1989 B/op	      75 allocs/op
BenchmarkParse/citm/stdjson-empty-struct          	     100	  14946034 ns/op	 115.56 MB/s	     280 B/op	       5 allocs/op
BenchmarkParse/citm/fastjson                      	    1000	   1879714 ns/op	 918.87 MB/s	   17628 B/op	      30 allocs/op
BenchmarkParse/citm/fastjson-get                  	    1000	   1881598 ns/op	 917.94 MB/s	   17628 B/op	      30 allocs/op
BenchmarkParse/twitter/stdjson-map                	     100	  11289146 ns/op	  55.94 MB/s	 2187878 B/op	   31266 allocs/op
BenchmarkParse/twitter/stdjson-struct             	     300	   5779442 ns/op	 109.27 MB/s	     408 B/op	       6 allocs/op
BenchmarkParse/twitter/stdjson-empty-struct       	     300	   5738504 ns/op	 110.05 MB/s	     408 B/op	       6 allocs/op
BenchmarkParse/twitter/fastjson                   	    2000	    774042 ns/op	 815.86 MB/s	    2541 B/op	       2 allocs/op
BenchmarkParse/twitter/fastjson-get               	    2000	    777833 ns/op	 811.89 MB/s	    2541 B/op	       2 allocs/op

Benchmark results for json validation:

$ GOMAXPROCS=1 go test github.com/valyala/fastjson -bench='Validate$'
goos: linux
goarch: amd64
pkg: github.com/valyala/fastjson
BenchmarkValidate/small/stdjson 	 2000000	       955 ns/op	 198.83 MB/s	      72 B/op	       2 allocs/op
BenchmarkValidate/small/fastjson         	 5000000	       384 ns/op	 493.60 MB/s	       0 B/op	       0 allocs/op
BenchmarkValidate/medium/stdjson         	  200000	     10799 ns/op	 215.66 MB/s	     184 B/op	       5 allocs/op
BenchmarkValidate/medium/fastjson        	  300000	      3809 ns/op	 611.30 MB/s	       0 B/op	       0 allocs/op
BenchmarkValidate/large/stdjson          	   10000	    133064 ns/op	 211.31 MB/s	     184 B/op	       5 allocs/op
BenchmarkValidate/large/fastjson         	   30000	     45268 ns/op	 621.14 MB/s	       0 B/op	       0 allocs/op
BenchmarkValidate/canada/stdjson         	     200	   8470904 ns/op	 265.74 MB/s	     184 B/op	       5 allocs/op
BenchmarkValidate/canada/fastjson        	     500	   2973377 ns/op	 757.07 MB/s	       0 B/op	       0 allocs/op
BenchmarkValidate/citm/stdjson           	     200	   7273172 ns/op	 237.48 MB/s	     184 B/op	       5 allocs/op
BenchmarkValidate/citm/fastjson          	    1000	   1684430 ns/op	1025.39 MB/s	       0 B/op	       0 allocs/op
BenchmarkValidate/twitter/stdjson        	     500	   2849439 ns/op	 221.63 MB/s	     312 B/op	       6 allocs/op
BenchmarkValidate/twitter/fastjson       	    2000	   1036796 ns/op	 609.10 MB/s	       0 B/op	       0 allocs/op

FAQ

Q: There are a ton of other high-perf packages for JSON parsing in Go. Why creating yet another package? A: Because other packages require either rigid JSON schema via struct magic and code generation or perform poorly when multiple unrelated fields must be obtained from the parsed JSON. Additionally, fastjson provides nicer API.
Q: What is the main purpose for fastjson? A: High-perf JSON parsing for RTB and other JSON-RPC services.
Q: Why fastjson doesn't provide fast marshaling (serialization)? A: Actually it provides some sort of marshaling - see Value.MarshalTo. But I'd recommend using quicktemplate for high-performance JSON marshaling :)
Q: fastjson crashes my program! A: There is high probability of improper use.
- Make sure you don't hold references to objects recursively returned by Parser / Scanner beyond the next Parser.Parse / Scanner.Next call if such restriction is mentioned in docs.
- Make sure you don't access fastjson objects from concurrently running goroutines if such restriction is mentioned in docs.
- Build and run your program with -race flag. Make sure the race detector detects zero races.
- If your program continue crashing after fixing issues mentioned above, file a bug.

Top Related Projects

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot