ProtoBuf

ProtoBuf#

Protocol Buffers (“protobuf”) is Google’s schema and IDL (interface definition language) for typed structured data. The schema (.proto file) is the source of truth; codegen produces matching types in Go, Rust, Python, TypeScript, Java, and many others; the wire format is compact, fast, and forward-/backward-compatible.

The Pieces#

A protobuf project has four moving parts: the .proto schema file, the compiler that emits language code, the wire format that travels between processes, and gRPC as the most common RPC framework built on top. The list below names each and how they fit together.

  • ``.proto`` file, the schema. Defines messages and (optionally) services.

  • ``protoc`` (or buf), the compiler that generates code.

  • Wire format, compact binary; not human-readable; small and fast.

  • gRPC, the RPC framework that uses protobuf as its IDL.

A Minimal Schema#

syntax = "proto3";
package example;

message Person {
  int32  id    = 1;
  string name  = 2;
  string email = 3;
  repeated string tags = 4;
}

message GetPersonRequest {
  int32 id = 1;
}

service People {
  rpc GetPerson (GetPersonRequest) returns (Person);
}

The numbers (= 1, = 2) are field tags, the wire-format identifiers. They never change once assigned.

Scalar Types#

The numeric type choices in a .proto aren’t decorative. int32 is the wrong default for negatives; use sint32 for ZigZag encoding. fixed32 wins when most values are large. The table below is the cheat sheet that prevents quietly wasted bytes on the wire.

Type

Notes

int32 /

int64

variable-length encoding; smaller for small numbers

uint32 /

uint64

unsigned variants

sint32 /

sint64

ZigZag-encoded; better for negatives

fixed32 /

fixed64

fixed 4 / 8 bytes; better when most values are large

float /

double

IEEE-754

bool

string

UTF-8

bytes

raw bytes

Choose int32 vs. sint32 vs. fixed32 based on the value distribution; the default int32 is wrong for negative numbers.

Composite Types#

enum Status {
  STATUS_UNSPECIFIED = 0;
  STATUS_ACTIVE      = 1;
  STATUS_DELETED     = 2;
}

message Address {
  string street  = 1;
  string city    = 2;
  string country = 3;
}

message User {
  int32   id          = 1;
  string  name        = 2;
  Address address     = 3;
  repeated string roles = 4;       // list
  map<string, string> metadata = 5; // map
  Status  status      = 6;
}

oneof#

For mutually exclusive fields (a tagged union).

message Result {
  oneof outcome {
    Success success = 1;
    Failure failure = 2;
  }
}

Schema Evolution#

The protobuf golden rule: don’t change field tags.

Safe changes.

  • Add a new field with a new tag.

  • Remove a field (mark it reserved to prevent reuse).

  • Rename a field (the tag is identity).

  • Make a field optional (optional keyword in proto3, since 3.15).

  • Add a new value to an enum (with care).

Unsafe changes.

  • Reuse a deleted tag for a different field.

  • Change a field’s type incompatibly.

  • Change required to optional in proto2, proto3 doesn’t have required fields by design.

message User {
  reserved 3, 5;             // never reuse these tags
  reserved "old_field_name";

  int32  id   = 1;
  string name = 2;
  string email = 4;          // new
}

proto2 vs. proto3#

  • proto2, explicit required / optional / repeated; field presence is always knowable.

  • proto3, no required; defaults are used for absent fields; optional since 3.15 brings explicit presence back.

Most new code uses proto3, with optional where presence matters.

gRPC#

Protobuf shines in gRPC:

service People {
  rpc GetPerson    (GetPersonRequest)    returns (Person);
  rpc ListPeople   (ListPeopleRequest)   returns (stream Person);
  rpc CreatePerson (CreatePersonRequest) returns (Person);
}

Streaming variants.

  • Server streaming, returns (stream X).

  • Client streaming, rpc Foo (stream X).

  • Bidirectional, both.

Tooling#

The CLIs and frameworks built around protobuf. protoc is the standard compiler with per-language plugins; buf is the modern driver in 2026 with linting, breaking-change detection, and a schema registry; Connect and Twirp give RPC layers atop the same proto schemas.

  • protoc, the original compiler. Per-language plugins (e.g. protoc-gen-go, protoc-gen-ts).

  • buf, modern protobuf tooling: linting, breaking-change detection, dependency management, schema registry. In 2026, buf is the recommended driver for most teams.

  • Connect, gRPC-compatible plus HTTP/JSON; better browser story than gRPC.

  • Twirp, simpler RPC over HTTP/Protobuf.

Wire Format#

Protobuf encodes each field as (tag << 3) | wire_type followed by the value.

  • Varint, variable-length integer encoding.

  • Length-delimited, strings, bytes, sub-messages.

  • 64-bit / 32-bit, fixed types.

The wire format is compact and fast; no field names cross the wire, which is why renaming a field is safe but reusing a tag is fatal.

JSON Mapping#

Protobuf has a standard JSON form for debugging and HTTP gateways.

{
  "id": 1,
  "name": "Ada",
  "email": "operator@example.com",
  "tags": ["math", "computing"]
}

Note: int64 and uint64 are JSON strings, not numbers (precision).

When (Not) to Use Protobuf#

Use when you need.

  • Strict schemas across languages.

  • Compact, fast wire format.

  • Strong forward / backward compatibility.

  • RPC at scale, gRPC builds on protobuf.

Don’t use when.

  • The audience is humans (use JSON / YAML for config).

  • The audience is browsers without a build step (use JSON; Connect / gRPC-Web mitigate).

  • The schema changes weekly with no governance (the rigor is the point).

Alternatives#

The other IDLs and serialization formats an operator may weigh against protobuf. JSON Schema, Avro, Thrift, Cap’n Proto, FlatBuffers, MessagePack, CBOR; each occupies a different niche, with different trade-offs around schema travel, zero-copy, and ecosystem maturity.

  • JSON Schema, describes JSON structure; no codegen for messages.

  • Avro, similar to protobuf; popular in Hadoop / Kafka ecosystems; schemas often live with the data.

  • Thrift, Facebook’s IDL; similar features, different ecosystem.

  • Cap’n Proto, “infinity times faster”; zero-copy.

  • FlatBuffers, zero-copy reads; designed for games / mobile.

  • MessagePack / CBOR, schema-less binary serialization.

Contents