ProtoBuf

ProtoBuf#

Protocol Buffers is Google’s compact binary serialization format. Unlike JSON, YAML, and XML, the wire format is not human-readable; it’s typed bytes meant for machines. Backed by a schema (.proto file), with code generation per language; the wire format is small, fast, and forward / backward compatible.

Protobuf as a data format, the wire format, when to choose it, the ecosystem. For the schema language and the IDL / gRPC angle, see ProtoBuf.

Why Protobuf vs. JSON#

The everyday choice an operator faces when picking a wire format. Protobuf and JSON solve the same problem (carry structured data between processes) but trade off readability against compactness, optional schemas against required ones, and self-description against numeric tags.

Differences that matter at the format level.

Property

Protobuf JSON

Encoding

Binary Text (UTF-8)

Size

3-10x smaller (typical)

Larger; field names repeat

Speed (encode/decode)

Often 5-20x faster

Slower; string parsing

Schema

Required (.proto)

Optional (JSON Schema separate)

Field names on wire

No, only numeric tags

Yes, repeated each message

Forward / backward compat

First-class (tags + reserved)

DIY conventions

Self-describing

No

Yes

Human readable

No (protoc --decode_raw helps)

Yes

Browser support

Indirect (via gRPC-Web / Connect)

Native

In one sentence: protobuf trades human-readability and self-description for speed, size, and contract-driven evolution.

The Wire Format#

The on-the-wire structure is the part most operators never see, and the part that explains every other property of the format. Once the (key, value) framing and the wire-type table are in head, protoc --decode_raw output stops looking like noise and starts reading like a file.

A protobuf message is a sequence of (key, value) pairs. The key combines the field’s tag number with its wire type.

key = (field_tag << 3) | wire_type

Wire types (3 bits).

Code

Type

Used by

0

Varint

int32, int64, uint32, uint64, sint32, sint64, bool, enum

1

I64

fixed64, sfixed64, double

2

Length-delim.

string, bytes, embedded message, packed repeated

5

I32

fixed32, sfixed32, float

(Codes 3 and 4, start-group and end-group, are deprecated and only appear in proto2 group syntax.)

The encoder writes every field this way; the decoder reads keys and matches them to the schema. Field names are never on the wire.

Varints#

Variable-length integer encoding. Each byte uses 7 bits of value plus a 1-bit continuation flag.

  • Small numbers (< 128) take 1 byte.

  • Large numbers grow by 1 byte per 7 bits.

  • Negative int32 / int64 always take 10 bytes, they’re encoded as if 64-bit. Use sint32 / sint64 (ZigZag) for negatives.

ZigZag encoding maps signed integers to unsigned for varint friendliness.

n

ZigZag(n)

0

0

-1

1

1

2

-2

3

2

4

Length-delimited#

Strings, bytes, and embedded messages are length-prefixed (varint length, then the bytes). Embedded messages are written recursively in the same format.

Packed Repeated#

Repeated scalar fields use length-delimited encoding with a single length prefix and the values concatenated. Default in proto3; opt-in in proto2 with [packed=true].

A Tiny Example#

Schema.

syntax = "proto3";

message Person {
  int32  id    = 1;
  string name  = 2;
}

Message Person { id: 150, name: "Ada" } on the wire:

08 96 01 12 03 41 64 61
│  │     │  │  │  │  │
│  │     │  │  │  │  └ 'a'
│  │     │  │  │  └─── 'd'
│  │     │  │  └────── 'A'
│  │     │  └────────── length 3
│  │     └─────────── tag 2 (name), wire-type 2 (length-delim)
│  └─── varint 150
└────── tag 1 (id), wire-type 0 (varint)

Eight bytes; JSON equivalent ({"id":150,"name":"Ada"}) is 23 bytes including delimiters.

Schema Evolution#

The reason protobuf wins in long-lived multi-team systems. Old binaries keep talking to new ones, and vice versa, because the wire format identifies fields by numeric tag and any unknown tag is silently ignored. The rules below codify what is safe and what is not.

The format’s killer feature. Forward / backward compatibility comes from the rule that field tags identify the field, not field names.

Safe changes.

  • Add a new field with a new tag. Old readers ignore unknown fields.

  • Remove a field (mark its tag reserved so nobody reuses it).

  • Rename a field. The wire is unchanged.

Unsafe changes.

  • Reuse a deleted tag for a different field type.

  • Change a field’s type incompatibly (e.g. int32 to string).

  • Promote a singular to repeated without care (varies by version).

message User {
  reserved 3, 5;
  reserved "old_field";

  int32  id    = 1;
  string name  = 2;
  string email = 4;          // new
}

JSON Mapping#

Every protobuf message has a defined JSON projection, used by HTTP gateways, debug tooling, and browser-friendly RPC. The mapping is mostly intuitive, with a handful of quirks around 64-bit integers, bytes, enums, and field presence that catch operators reading or writing the JSON form.

Protobuf has a standard JSON form for debugging / HTTP gateways.

{
  "id": 1,
  "name": "Ada",
  "email": "operator@example.com",
  "tags": ["math", "computing"]
}

Notable quirks.

  • int64 / uint64 are JSON strings (precision: JS Number loses bits past 2^53).

  • bytes are base64-encoded.

  • Enums are strings by default in JSON form.

  • Default-valued fields are omitted in proto3 unless always_print_* options are set.

Tooling#

The tools an operator reaches for when generating code, linting schemas, detecting breaking changes, and inspecting opaque .bin payloads. protoc is the standard compiler underneath, but buf is the modern driver that operators actually invoke from CI and the command line.

  • protoc, the original compiler. Per-language plugins (protoc-gen-go, protoc-gen-ts).

  • buf, modern protobuf tooling: linting, breaking-change detection, dependency management, schema registry, remote codegen. The recommended driver in 2026.

  • Connect, gRPC-compatible plus HTTP/JSON; better browser story.

  • protoc –decode_raw < message.bin`, dump unknown bytes when you don’t have the schema.

Inspect bytes.

$ xxd message.bin
$ protoc --decode_raw < message.bin
$ protoc --decode=Person -I . person.proto < message.bin

Per-Language Bindings#

Every mainstream language has at least one protobuf binding, and most have several. The official Google plugin sets the baseline; community bindings (prost, betterproto, protobuf-es) often emit cleaner code or smaller dependency trees and are widely used in production.

Language

Library

Go

google.golang.org/protobuf (with protoc-gen-go)

TypeScript / JavaScript

protobuf-es, ts-proto, protobufjs

Python

protobuf (official), betterproto

Rust

prost, tonic, protobuf

Java / Kotlin

protobuf-java / protobuf-kotlin

C++

protobuf (the original)

C# / .NET

Google.Protobuf

Ruby

google-protobuf

PHP

google/protobuf

Most languages have multiple bindings; pick the official one unless there’s a specific reason otherwise (smaller dep tree, friendlier generated code, faster encoder).

gRPC#

The flagship use case. gRPC is HTTP/2 + protobuf as the message format, with optional streaming. Schema.

service People {
  rpc GetPerson    (GetPersonRequest)    returns (Person);
  rpc ListPeople   (ListPeopleRequest)   returns (stream Person);
  rpc CreatePerson (CreatePersonRequest) returns (Person);
}

Streaming variants: server, client, bidirectional. See Communication for the broader picture.

When to Pick Protobuf (Format)#

The kind of system where protobuf is the obvious choice. Cross-language RPC, high-volume payloads, multi-team contract work, and constrained mobile or IoT networks all play to the format’s strengths, compactness, speed, and forward / backward compatibility built into the wire.

  • Inter-service RPC at any scale, gRPC builds on it.

  • High-volume event payloads, size and speed matter.

  • Strongly-versioned wire contracts between many teams.

  • Mobile / IoT where bytes on the wire are expensive.

  • Cross-language data interchange with codegen.

When to Skip#

The mirror image. Workloads where the binary, schema-required, opaque-on-the-wire form works against the goal, in public APIs, configuration, throwaway scripts, human-inspected logs. For these, JSON or YAML is the right answer; for browsers specifically, Connect bridges both worlds.

  • Public web APIs for browsers, JSON is friendlier (or use Connect for both).

  • Configuration files, humans can’t read protobuf.

  • One-off scripts, the schema overhead doesn’t pay back.

  • Logs and traces, structured JSON / OTel-protobuf is the norm, but JSON wins for human inspection.

Alternatives#

The other binary serialization formats an operator may meet or evaluate alongside protobuf. Each has a different sweet spot (streaming, zero-copy, schema-less, JSON-shaped), and several remain widely used in their niches even where protobuf has become the default elsewhere.

  • Apache Avro, similar features; schema travels with data; popular in Kafka / Hadoop.

  • Apache Thrift, Facebook’s IDL + RPC; pre-dates gRPC; less common in 2026.

  • Cap’n Proto, “infinity times faster”; zero-copy reads.

  • FlatBuffers, zero-copy; designed for games / mobile.

  • MessagePack, schema-less binary JSON alternative.

  • CBOR, IETF binary; similar role to MessagePack.

  • Smile, BSON (MongoDB), niche binary JSON variants.

  • JSON Schema, not a binary format, but the JSON-shaped equivalent of “have a schema”.

Pitfalls#

The traps that catch teams new to protobuf, especially when crossing language boundaries. Most surface around field presence, integer width, enum evolution, and reserved tag hygiene; all areas where the format leans on convention that the schema author has to actively maintain.

  • Default values in proto3, 0 / empty string / false look identical to “not set” on the wire. Use optional (since 3.15) when presence matters.

  • 64-bit IDs in JS, precision loss; use the JSON string form or BigInt-aware bindings.

  • Reserved tags, track them in the .proto; buf breaking enforces it.

  • Cross-language enum mismatch, adding an enum value can break older readers in some languages. Use UNRECOGNIZED / unknown values where the binding offers them.

  • Performance assumptions, protobuf is fast, but bad schemas (giant nested messages, deeply repeated fields) can erase the win. Profile.

Workflow#

Extract, parse, filter, save. Protobuf needs the schema (.proto) to be useful: with it, protoc decodes binary into text; without it, protoscope walks the wire format for sniffing.

Extract and inspect.

# without the schema, dump wire format
$ protoscope events.bin | head

# with the schema, decode to text proto
$ protoc --decode=EventBatch events.proto < events.bin

Parse and filter (Python, schema-aware).

from events_pb2 import Event, EventBatch

batch = EventBatch()
batch.ParseFromString(open("events.bin", "rb").read())
high  = [e for e in batch.events if e.severity == Event.HIGH]

Save the filtered subset.

out = EventBatch(events=high)
open("high.bin", "wb").write(out.SerializeToString())

Streaming (length-delimited records) when the batch does not fit in memory.

from google.protobuf.internal.decoder import _DecodeVarint
from events_pb2 import Event

def read_stream(path):
    buf = open(path, "rb").read()
    i = 0
    while i < len(buf):
        size, j = _DecodeVarint(buf, i)
        msg = Event(); msg.ParseFromString(buf[j:j+size])
        yield msg
        i = j + size

with open("high.bin", "wb") as out:
    for e in read_stream("events.bin"):
        if e.severity == Event.HIGH:
            data = e.SerializeToString()
            out.write(len(data).to_bytes(4, "little") + data)

See Also#

Contents