Notesencoding🌱 seedling
Protocol Buffers
Also: Protobuf · protobuf · proto3 · proto2 · Protobuf Editions
Protocol Buffers is a schema-driven binary format in which every field has a permanent number. On the wire each field is a tag (field number plus wire type) and a value, so parsers can skip fields they don't know, and the number, not the name, is the field's identity.
Why it exists
Google needed a compact, typed, language-neutral format whose schemas could change without coordinated deploys. It ran internally from the early 2000s and was open-sourced in 2008.
How it works
- Schema:
int64 id = 1; string name = 2;. Names never reach the binary wire. - Each field: tag =
(field_number << 3) | wire_type, then the value. Wire types: VARINT (0), I64 (1), LEN (2), I32 (5). - Integers are base-128 varints: 7 bits per byte, top bit = "more follows", least significant group first. 150 →
96 01; the record {id 150, name "Ada"} is08 96 01 12 03 41 64 61. - Negative
int32/int64always take 10 bytes;sint32/sint64use zig-zag (−1 → 1, 1 → 2). - Unknown field numbers are skipped using the wire type, and preserved on re-serialization (proto3 since 3.5).
- A non-repeated scalar that appears twice: the last value wins. Concatenated messages merge.
Trade-offs
| Gains | Costs |
|---|---|
| compact, fast, typed codegen in many languages | field numbers must be managed forever |
| forward compatible by construction (skippable tags) | needs the .proto to interpret meaning |
| self-delimiting fields, mergeable messages | schema changes need discipline or tooling (buf breaking) |
In practice
- gRPC uses Protobuf as its default message format.
- Editions (first shipped 2024) replace
proto2/proto3syntax with per-feature settings such as field presence; the wire format is unchanged.
Pitfalls
- Reusing a deleted field number: old code decodes the new field as the old one. Always
reservednumbers and names. - Converting to JSON or copying into a domain object drops unknown fields.
Connections
- Part of
- Serialization
- Contrasts with
- Apache Avro
- In real systems
- gRPC
Recall
These cards are also in your review deck, scheduled with spaced repetition.