Convert Figma logo to code with AI

protocolbuffers logoprotobuf-go

Go support for Google's protocol buffers

2,905
408
2,905
7
View on GitHub

Top Related Projects

gogo's avatar

protobuf

5,664

[Deprecated] Protocol Buffers for Go with Gadgets

jhump's avatar

protoreflect

1,332

Reflection (Rich Descriptors) for Go Protocol Buffers

twitchtv's avatar

twirp

7,150

A simple RPC framework with protobuf service definitions

bufbuild's avatar

protoc-gen-validate

3,750

Protocol Buffer Validation - Being replaced by github.com/bufbuild/protovalidate

Quick Overview

protocolbuffers/protobuf-go is the official Go implementation of Protocol Buffers, Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data. This repository provides Go runtime support for Protocol Buffers, including code generation tools and runtime libraries for encoding and decoding protobuf messages.

Pros

  • High performance and efficient serialization/deserialization
  • Strong type safety and backward compatibility
  • Excellent integration with Go's ecosystem and tooling
  • Supports both binary and JSON encoding formats

Cons

  • Steeper learning curve compared to simpler serialization formats like JSON
  • Requires separate compilation step for .proto files
  • Limited support for dynamic message manipulation at runtime
  • Can be overkill for simple data structures or small projects

Code Examples

  1. Defining a protobuf message:
syntax = "proto3";

package example;

message Person {
  string name = 1;
  int32 age = 2;
  repeated string hobbies = 3;
}
  1. Generating Go code from the .proto file:
protoc --go_out=. person.proto
  1. Using the generated code in Go:
package main

import (
    "fmt"
    "log"

    "google.golang.org/protobuf/proto"
    pb "path/to/generated/package"
)

func main() {
    person := &pb.Person{
        Name:    "Alice",
        Age:     30,
        Hobbies: []string{"reading", "hiking"},
    }

    data, err := proto.Marshal(person)
    if err != nil {
        log.Fatal("Marshaling error: ", err)
    }

    newPerson := &pb.Person{}
    if err := proto.Unmarshal(data, newPerson); err != nil {
        log.Fatal("Unmarshaling error: ", err)
    }

    fmt.Printf("Unmarshaled person: %+v\n", newPerson)
}

Getting Started

  1. Install the protoc compiler:

    brew install protobuf  # macOS
apt-get install protobuf-compiler  # Ubuntu

  2. Install the Go plugin:

    go install google.golang.org/protobuf/cmd/protoc-gen-go@latest

  3. Define your .proto file and generate Go code:

    protoc --go_out=. your_file.proto

  4. Use the generated code in your Go project:

    import (
    "google.golang.org/protobuf/proto"
    pb "path/to/generated/package"
)

  5. Start using Protocol Buffers in your Go application!

Competitor Comparisons

gogo logo

protobuf

5,664

[Deprecated] Protocol Buffers for Go with Gadgets

Pros of gogo/protobuf

  • Offers additional code generation options for performance optimization
  • Provides custom types and extensions for Go-specific features
  • Supports backward compatibility with older versions of Protocol Buffers

Cons of gogo/protobuf

  • Less actively maintained compared to the official implementation
  • May have compatibility issues with future Protocol Buffers updates
  • Requires additional setup and configuration for custom features

Code Comparison

protobuf-go:

import "google.golang.org/protobuf/proto"

data, err := proto.Marshal(message)
err = proto.Unmarshal(data, message)

gogo/protobuf:

import "github.com/gogo/protobuf/proto"

data, err := proto.Marshal(message)
err = proto.Unmarshal(data, message)

The basic usage is similar, but gogo/protobuf offers additional options:

import "github.com/gogo/protobuf/gogoproto"

// Custom marshaling options
data, err := proto.MarshalWithOptions(message, proto.MarshalOptions{
    UseProtoNames: true,
})

gogo/protobuf provides performance benefits and additional features but may require more setup and maintenance. protobuf-go is the official implementation, ensuring long-term support and compatibility with the Protocol Buffers specification.

jhump logo

protoreflect

1,332

Reflection (Rich Descriptors) for Go Protocol Buffers

Pros of protoreflect

  • Offers more extensive reflection capabilities for Protocol Buffers
  • Provides additional utilities for working with Protocol Buffer descriptors
  • Includes tools for dynamic message creation and manipulation

Cons of protoreflect

  • May have a steeper learning curve due to its more advanced features
  • Not the official implementation, which could lead to potential compatibility issues
  • Might have less frequent updates compared to the official repository

Code Comparison

protobuf-go:

import "google.golang.org/protobuf/proto"

msg := &MyMessage{Field: "value"}
data, err := proto.Marshal(msg)

protoreflect:

import "github.com/jhump/protoreflect/dynamic"

msg := dynamic.NewMessage(myMessageDescriptor)
msg.SetFieldByName("field", "value")
data, err := msg.Marshal()

The protobuf-go example shows the standard way of marshaling a Protocol Buffer message, while the protoreflect example demonstrates its dynamic message creation and manipulation capabilities.

protoreflect offers more flexibility and advanced features for working with Protocol Buffers, but it may be more complex to use and maintain compared to the official protobuf-go implementation. The choice between the two depends on the specific requirements of your project and the level of control you need over Protocol Buffer operations.

twitchtv logo

twirp

7,150

A simple RPC framework with protobuf service definitions

Pros of Twirp

  • Simpler and more lightweight than protobuf-go, focusing on RPC
  • Built-in HTTP routing and server generation
  • Easier to set up and use for small to medium-sized projects

Cons of Twirp

  • Less feature-rich compared to protobuf-go
  • Limited to HTTP/JSON and Protocol Buffers
  • Smaller community and ecosystem

Code Comparison

Twirp:

service Haberdasher {
  rpc MakeHat(Size) returns (Hat);
}

type Size struct {
  inches int32
}

type Hat struct {
  inches int32
  color string
  name string
}

protobuf-go:

syntax = "proto3";

message Person {
  string name = 1;
  int32 age = 2;
  repeated string phones = 3;
}

message AddressBook {
  repeated Person people = 1;
}

Twirp focuses on defining RPC services with a simpler syntax, while protobuf-go provides a more comprehensive Protocol Buffers implementation with additional features and flexibility. Twirp is better suited for quick RPC setups, while protobuf-go offers more options for complex data serialization needs.

bufbuild logo

protoc-gen-validate

3,750

Protocol Buffer Validation - Being replaced by github.com/bufbuild/protovalidate

Pros of protoc-gen-validate

  • Provides advanced validation rules for Protocol Buffers
  • Supports multiple programming languages (Go, Java, Python, etc.)
  • Offers custom validation options beyond basic Proto3 constraints

Cons of protoc-gen-validate

  • Requires additional setup and integration compared to protobuf-go
  • May introduce overhead due to extra validation checks
  • Limited to validation-specific functionality

Code Comparison

protoc-gen-validate:

message User {
  string email = 1 [(validate.rules).string.email = true];
  int32 age = 2 [(validate.rules).int32 = {gte: 18, lte: 99}];
}

protobuf-go:

type User struct {
    Email string `protobuf:"bytes,1,opt,name=email,proto3" json:"email,omitempty"`
    Age   int32  `protobuf:"varint,2,opt,name=age,proto3" json:"age,omitempty"`
}

protoc-gen-validate focuses on adding validation rules to Protocol Buffer definitions, while protobuf-go provides the core functionality for working with Protocol Buffers in Go. protoc-gen-validate offers more fine-grained control over data validation, but protobuf-go is simpler to use for basic Protocol Buffer operations.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Go support for Protocol Buffers

Go Reference Build Status

This project hosts the Go implementation for protocol buffers, which is a language-neutral, platform-neutral, extensible mechanism for serializing structured data. The protocol buffer language is a language for specifying the schema for structured data. This schema is compiled into language specific bindings. This project provides both a tool to generate Go code for the protocol buffer language, and also the runtime implementation to handle serialization of messages in Go. See the protocol buffer developer guide for more information about protocol buffers themselves.

This project is comprised of two components:

See the developer guide for protocol buffers in Go for a general guide for how to get started using protobufs in Go.

This project is the second major revision of the Go protocol buffer API implemented by the google.golang.org/protobuf module. The first major version is implemented by the github.com/golang/protobuf module.

Package index

Summary of the packages provided by this module:

  • proto: Package proto provides functions operating on protobuf messages such as cloning, merging, and checking equality, as well as binary serialization.
  • encoding/protojson: Package protojson serializes protobuf messages as JSON.
  • encoding/prototext: Package prototext serializes protobuf messages as the text format.
  • encoding/protowire: Package protowire parses and formats the low-level raw wire encoding. Most users should use package proto to serialize messages in the wire format.
  • reflect/protoreflect: Package protoreflect provides interfaces to dynamically manipulate protobuf messages.
  • reflect/protoregistry: Package protoregistry provides data structures to register and lookup protobuf descriptor types.
  • reflect/protodesc: Package protodesc provides functionality for converting descriptorpb.FileDescriptorProto messages to/from the reflective protoreflect.FileDescriptor.
  • reflect/protopath: Package protopath provides a representation of a sequence of protobuf reflection operations on a message.
  • reflect/protorange: Package protorange provides functionality to traverse a protobuf message.
  • testing/protocmp: Package protocmp provides protobuf specific options for the cmp package.
  • testing/protopack: Package protopack aids manual encoding and decoding of the wire format.
  • testing/prototest: Package prototest exercises the protobuf reflection implementation for concrete message types.
  • types/dynamicpb: Package dynamicpb creates protobuf messages at runtime from protobuf descriptors.
  • types/known/anypb: Package anypb is the generated package for google/protobuf/any.proto.
  • types/known/timestamppb: Package timestamppb is the generated package for google/protobuf/timestamp.proto.
  • types/known/durationpb: Package durationpb is the generated package for google/protobuf/duration.proto.
  • types/known/wrapperspb: Package wrapperspb is the generated package for google/protobuf/wrappers.proto.
  • types/known/structpb: Package structpb is the generated package for google/protobuf/struct.proto.
  • types/known/fieldmaskpb: Package fieldmaskpb is the generated package for google/protobuf/field_mask.proto.
  • types/known/apipb: Package apipb is the generated package for google/protobuf/api.proto.
  • types/known/typepb: Package typepb is the generated package for google/protobuf/type.proto.
  • types/known/sourcecontextpb: Package sourcecontextpb is the generated package for google/protobuf/source_context.proto.
  • types/known/emptypb: Package emptypb is the generated package for google/protobuf/empty.proto.
  • types/descriptorpb: Package descriptorpb is the generated package for google/protobuf/descriptor.proto.
  • types/pluginpb: Package pluginpb is the generated package for google/protobuf/compiler/plugin.proto.
  • compiler/protogen: Package protogen provides support for writing protoc plugins.
  • cmd/protoc-gen-go: The protoc-gen-go binary is a protoc plugin to generate a Go protocol buffer package.

Reporting issues

The issue tracker for this project is currently located at golang/protobuf.

Please report any issues there with a sufficient description of the bug or feature request. Bug reports should ideally be accompanied by a minimal reproduction of the issue. Irreproducible bugs are difficult to diagnose and fix (and likely to be closed after some period of time). Bug reports must specify the version of the Go protocol buffer module and also the version of the protocol buffer toolchain being used.

Contributing

This project is open-source and accepts contributions. See the contribution guide for more information.

Compatibility

This module and the generated code are expected to be stable over time. However, we reserve the right to make breaking changes without notice for the following reasons:

  • Security: A security issue in the specification or implementation may come to light whose resolution requires breaking compatibility. We reserve the right to address such issues.
  • Unspecified behavior: There are some aspects of the protocol buffer specification that are undefined. Programs that depend on unspecified behavior may break in future releases.
  • Specification changes: It may become necessary to address an inconsistency, incompleteness, or change in the protocol buffer specification, which may affect the behavior of existing programs. We reserve the right to address such changes.
  • Bugs: If a package has a bug that violates correctness, a program depending on the buggy behavior may break if the bug is fixed. We reserve the right to fix such bugs.
  • Generated additions: We reserve the right to add new declarations to generated Go packages of .proto files. This includes declared constants, variables, functions, types, fields in structs, and methods on types. This may break attempts at injecting additional code on top of what is generated by protoc-gen-go. Such practice is not supported by this project.
  • Internal changes: We reserve the right to add, modify, and remove internal code, which includes all unexported declarations, the protoc-gen-go/internal_gengo package, the runtime/protoimpl package, and all packages under internal.

Any breaking changes outside of these will be announced 6 months in advance to protobuf@googlegroups.com.

Users should use generated code produced by a version of protoc-gen-go that is identical to the runtime version provided by the protobuf module. This project promises that the runtime remains compatible with code produced by a version of the generator that is no older than 1 year from the version of the runtime used, according to the release dates of the minor version. Generated code is expected to use a runtime version that is at least as new as the generator used to produce it. Generated code contains references to protoimpl.EnforceVersion to statically ensure that the generated code and runtime do not drift sufficiently far apart.

Historical legacy

This project is the second major revision (released in 2020) of the Go protocol buffer API implemented by the google.golang.org/protobuf module. The first major version (released publicly in 2010) is implemented by the github.com/golang/protobuf module.

The first version predates the release of Go 1 by several years. It has a long history as one of the first core pieces of infrastructure software ever written in Go. As such, the Go protobuf project was one of many pioneers for determining what the Go language should even look like and what would eventually be considered good design patterns and “idiomatic” Go (by simultaneously being both positive and negative examples of it).

Consider the changing signature of the proto.Unmarshal function as an example of Go language and library evolution throughout the life of this project:

// 2007/09/25 - Conception of Go

// 2008/11/12
export func UnMarshal(r io.Read, pb_e reflect.Empty) *os.Error

// 2008/11/13
export func UnMarshal(buf *[]byte, pb_e reflect.Empty) *os.Error

// 2008/11/24
export func UnMarshal(buf *[]byte, pb_e interface{}) *os.Error

// 2008/12/18
export func UnMarshal(buf []byte, pb_e interface{}) *os.Error

// 2009/01/20
func UnMarshal(buf []byte, pb_e interface{}) *os.Error

// 2009/04/17
func UnMarshal(buf []byte, pb_e interface{}) os.Error

// 2009/05/22
func Unmarshal(buf []byte, pb_e interface{}) os.Error

// 2011/11/03
func Unmarshal(buf []byte, pb_e interface{}) error

// 2012/03/28 - Release of Go 1

// 2012/06/12
func Unmarshal(buf []byte, pb Message) error

These changes demonstrate the difficulty of determining what the right API is for any new technology. It takes time multiplied by many users to determine what is best; even then, “best” is often still somewhere over the horizon.

The change on June 6th, 2012 added a degree of type-safety to Go protobufs by declaring a new interface that all protobuf messages were required to implement:

type Message interface {
   Reset()
   String() string
   ProtoMessage()
}

This interface reduced the set of types that can be passed to proto.Unmarshal from the universal set of all possible Go types to those with a special ProtoMessage marker method. The intention of this change is to limit the protobuf API to only operate on protobuf data types (i.e., protobuf messages). For example, there is no sensible operation if a Go channel were passed to the protobuf API as a channel cannot be serialized. The restricted interface would prevent that.

This interface does not behaviorally describe what a protobuf message is, but acts as a marker with an undocumented expectation that protobuf messages must be a Go struct with a specific layout of fields with formatted tags. This expectation is not statically enforced by the Go language, for it is an implementation detail checked dynamically at runtime using Go reflection. Back in 2012, the only types with this marker were those generated by protoc-gen-go. Since protoc-gen-go would always generate messages with the proper layout of fields, this was deemed an acceptable and dramatic improvement over interface{}.

Over the next 10 years, use of Go would skyrocket and use of protobufs in Go would skyrocket as well. With increased popularity also came more diverse usages and requirements for Go protobufs and an increased number of custom proto.Message implementations that were not generated by protoc-gen-go.

The increasingly diverse ecosystem of Go types implementing the proto.Message interface led to incompatibilities, which often occurred when:

  • Passing custom proto.Message types to the protobuf APIs: A concrete message implementation might work with some top-level functions (e.g., proto.Marshal), but cause others (e.g., proto.Equal) to choke and panic. This occurs because the type only had partial support for being an actual message by only implementing the proto.Marshaler interface or having malformed struct field tags that happened to work with one function, but not another.

  • Using Go reflection on any proto.Message types: A common desire is to write general-purpose code that operates on any protobuf message. For example, a microservice might want to populate a trace_id field if it is present in a message. To accomplish this, one would use Go reflection to introspect the message type, and assume it were a pointer to a Go struct with a field named TraceId (as would be commonly produced by protoc-gen-go). If the concrete message type did not match this expectation, it either failed to work or even resulted in a panic. Such was the case for concrete message types that might be backed by a Go map instead of a Go struct.

Both of these issues are solved by following the idiom that interfaces should describe behavior, not data. This means that the interface itself should provide sufficient functionality through its methods that users can introspect and interact with all aspects of a protobuf message through a principled API. This feature is called protobuf reflection. Just as how Go reflection provides an API for programmatically interacting with any arbitrary Go value, protobuf reflection provides an API for programmatically interacting with any arbitrary protobuf message.

Since an interface cannot be extended in a backwards compatible way, this suggested the need for a new major version that defines a new proto.Message interface:

type Message interface {
    ProtoReflect() protoreflect.Message
}

The new proto.Message interface contains a single ProtoReflect method that returns a protoreflect.Message, which is a reflective view over a protobuf message. In addition to making a breaking change to the proto.Message interface, we took this opportunity to cleanup the supporting functionality that operate on a proto.Message, split up complicated functionality apart into manageable packages, and to hide implementation details away from the public API.

The goal for this major revision is to improve upon all the benefits of, while addressing all the shortcomings of the old API. We hope that it will serve the Go ecosystem well for the next 10 years and beyond.