Encoding · 1 of 2

Encoding & Evolution

Data outlives the code that wrote it. How a record becomes bytes, why JSON, Protobuf and Avro make such different bets, and the rules that let old and new code read each other's data for years.

≈ 25 min

The whole idea

Press ← → to step, or Space to play and pause, whenever a diagram has focus.

Data outlives code

Inside a process, data is objects, structs, lists and hash maps, held together by pointers. None of that survives leaving the process. To write a record to disk or send it to another machine, you translate it into a self-contained sequence of bytes. That translation is encoding (or serialization, or marshalling), and the way back is decoding.

If code never changed, any format would do. But code changes all the time, and it never changes everywhere at once. A server-side deploy is a rolling upgrade: for minutes or hours, old and new versions run side by side. Mobile and desktop clients upgrade whenever their users feel like it. And a database row written five years ago is still there, waiting for today’s code to read it.

So every encoding has to cope with two directions of mismatch:

  • Backward compatibility: newer code can read data written by older code. This one is relatively easy, because when you write the new code you know exactly what the old format looked like.
  • Forward compatibility: older code can read data written by newer code. This is harder: the old code was written before anyone knew what the new format would contain, so it has to ignore additions gracefully and, ideally, not destroy them.

Serialization

Most languages ship a built-in way to turn objects into bytes: Java’s Serializable, Python’s pickle, Ruby’s Marshal. They’re convenient and they’re a trap for anything that leaves the process.

  • They tie you to one language. Another team, or your own future rewrite, can’t read the data.
  • They’re a security hole. To restore arbitrary objects, the decoder must be able to instantiate arbitrary classes. Python’s documentation warns that pickle is not secure and that you should only unpickle data you trust. In 2015, researchers showed that ordinary libraries on a Java classpath, Apache Commons Collections among them, could be chained into remote code execution through deserialization of untrusted bytes.
  • Versioning is an afterthought, and efficiency often is too.

They’re fine for a cache that dies with the process. For anything stored or sent, use a format designed to be read by code you didn’t write, possibly years later.

JSON and binary encodings

The obvious alternative is a textual format everyone can read: JSON, XML or CSV. JSON in particular is the default language of web APIs. It is readable, universal and needs no shared schema. It also has real problems:

  • Numbers are vague. JSON doesn’t distinguish integers from floats or say how precise they are. Many parsers, JavaScript’s included, read every number as a 64-bit float, which holds integers exactly only up to 253. In October 2010, weeks before its new Snowflake tweet IDs were due to pass that, Twitter began sending every ID twice: as a number (id) and as a string (id_str).
  • No binary strings. Bytes must be smuggled in as Base64 text, which is 33% bigger.
  • The schema is optional. JSON Schema exists, but nothing forces a writer to follow one, so every reader must hard-code its own idea of the format.
  • Field names travel with every record.

One record, three encodings

JSON · field names in every record{"id":150,"name":"Ada"}23 BProtocol Buffers · tags instead of names08960112034164618 BAvro · values only, in schema orderac0206416461150lenAda6 B✓ any field can be skipped by its tag! unreadable without the writer’s schema
Same record, same scale: one box per byte. The bytes are the real encodings, not a sketch.

Binary dialects of JSON, like MessagePack and BSON, shave some of this off (the record above is 15 bytes in MessagePack), but they keep the field names, because they keep JSON’s data model: no schema, so every record must describe itself.

The big savings come from schema-driven binary formats: Apache Thrift (built at Facebook, open-sourced in 2007), Protocol Buffers (used inside Google since the early 2000s and open-sourced in 2008) and Apache Avro (started in the Hadoop project in 2009). Writer and reader agree on a schema in advance, so the bytes only need to carry values. Thrift’s compact protocol encodes the record above in 9 bytes, Protobuf in 8 and Avro in 6.Size is rarely the whole argument for a few-byte record. It matters for terabytes of events, and the schema matters everywhere.

Size is the smaller win. A schema is also documentation that can’t drift from the code. It lets statically typed languages generate classes. And it lets a tool check, before you deploy, whether a change is compatible with everything already written. JSON remains the right choice at the edges, for public APIs and browsers. Between your own services and in your own storage, the schema is worth having.

Protocol Buffers

A Protobuf schema gives every field a type, a name and a field number:

message User {
  int64  id   = 1;
  string name = 2;
}

On the wire, names never appear. Each field is written as a tag followed by a value. The tag packs the field number and a three-bit wire type into one integer, (field_number << 3) | wire_type. The wire type says how to find the end of the value: VARINT (0) for integers and booleans, I64 (1) and I32 (5) for fixed-width numbers, and LEN (2), a length prefix followed by that many bytes, for strings, bytes, nested messages and packed lists.

Integers, and the tags themselves, are varints: seven bits of value per byte, with the top bit saying whether another byte follows.

How 150 becomes 96 01

150 =1001011000000010high 7 bits = 1low 7 bits = 22written firstwritten secondmore followslast byte0x960x0100001000last bytefield number 1wire type 0tag= 0x08
Base-128 varints: small numbers cost one byte, and the largest 64-bit values cost ten.

A few consequences are worth remembering:

  • Small numbers are cheap; negative ones aren’t. A negative int32 or int64 is always ten bytes, because its sign bit is the top bit. The sint32 and sint64 types use zig-zag encoding instead (0 → 0, −1 → 1, 1 → 2, −2 → 3…), so small negative numbers stay small.
  • Order and repetition are flexible. If a scalar field appears twice, the last value wins, and concatenating two encoded messages is the same as merging them.
  • A reader can skip anything. Meeting a field number it doesn’t know, a parser reads the wire type, works out how many bytes to skip, and carries on. That single property is what makes Protobuf forward compatible.

Thrift works the same way, with field IDs and type codes in its binary and compact protocols. Protobuf is also the default message format of gRPC, which the next lesson covers.

Schema evolution

Because the field number is the field’s identity on the wire, the rules for changing a Protobuf schema follow directly:

  • Adding a field is safe, as long as it gets a new number. Old code skips it (forward compatible). New code reading old data finds it missing and sees the field’s default value: zero, empty string, or “not set” for fields with explicit presence (backward compatible).
  • Removing a field is safe, but you must reserve its number, and its name too, so nobody can reuse them.
  • Never change a field’s number, and never reuse one. Changing a number is the same as deleting the field and adding a new one.
  • Changing a type is risky. int32, uint32, int64, uint64 and bool are interchangeable on the wire, but values are truncated as if cast in C++. string and bytes are compatible only if the bytes are valid UTF-8. A singular string, bytes or message field can become repeated; numeric fields can’t.
  • Required fields are forever. proto2’s required meant a reader rejected any message missing the field, so it could never be removed. proto3 dropped it.Protobuf Editions, first shipped in 2024, replace syntax = "proto2"/"proto3" with per-feature switches such as field_presence. The wire format doesn’t change.

Forward compatibility needs one more thing from old code: it must not just tolerate unknown fields but keep them. This is the classic way a rolling upgrade loses data:

Old code, new data

The naive version loses data without a single error. The fix happens inside the decoder.

Protobuf’s binary format preserves unknown fields by default. Proto3 originally dropped them, and Google restored preservation in version 3.5 (2017). The bytes are still lost if your code converts the message to JSON, or copies its fields one by one into a domain object of its own. Whatever the format, the rule is the same: read, modify and write back without throwing away what you didn’t understand.

The rule about numbers matters even more. A reused field number doesn’t fail loudly; old code decodes the new field as the old one, with complete confidence.

Reusing a field number

Identical bytes, two meanings. The receiver running old code has no way to know.

Apache Avro

Avro takes the opposite bet. Its schema, written in JSON, has no field numbers at all:

{ "type": "record", "name": "User", "fields": [
    { "name": "id",   "type": "long" },
    { "name": "name", "type": "string" } ] }

The encoding is just the values, concatenated in schema order: integers as zig-zag varints, strings as a length and then UTF-8 bytes, and a union as the index of its branch followed by the value. Nothing in the bytes says which field is which, or even where one field ends unless you know its type. So to read Avro you need the exact schema it was written with, the writer’s schema. The reader also has its own reader’s schema, the shape its code expects, and Avro translates from one to the other.

Writer’s schema meets reader’s schema

writer’s schema · v2idlongnamestringemail[null, string]skippedreader’s schema · v3namestringidlongplanstring = "free"default →bytes · 16 Bac0206416461021061646140782e696fidnameemailname "Ada" · id 150plan "free"
Resolution happens at read time. The two schemas only have to be compatible, not identical.

The resolution rules define what changes are compatible:

  • Fields are matched by name, so they can be reordered.
  • A field only the writer has is ignored.
  • A field only the reader has takes the reader’s default. If it has none, reading fails.
  • So you may add or remove a field only if it has a default. An optional field is a union with null, like ["null", "string"], with null as its default.
  • Renames use aliases in the reader’s schema: a new reader can read old data, but an old reader can’t read the new name.
  • Some type changes are promoted automatically: int to long, float or double, for example.

Where does the reader find the writer’s schema? Avro offers three answers. A container file puts it once in the header (the file starts with Obj and a version byte), ahead of millions of records, which is why Avro is at home in data lakes. A message stream puts a schema ID or fingerprint in front of each record: Avro’s single-object encoding uses a two-byte marker, C3 01, plus an 8-byte schema fingerprint. And two parties on a long-lived connection can agree on a schema when the connection opens.

Leaving out field numbers makes Avro friendly to generated schemas. Dump a database table and each column becomes a field; when the table changes, generate a new schema from it. Nobody has to hand out tag numbers, and resolution by name does the rest. That’s why change-data-capture tools like Debezium commonly emit Avro with a schema registry.

Backward and forward compatibility

Now the two directions from the start, made precise. With two versions of a schema there are four pairings of reader code and stored data, and compatibility is a claim about the two that mix versions.

Who can read what

data written with v1data written with v2v1 code readsv2 code reads✓same version?forward?backward✓same versionupgrade orderv1v2v3BACKWARD checks v3 against v2BACKWARD_TRANSITIVE also checks v3 against v1
Each compatibility mode is a promise about the off-diagonal cells, and that promise decides who upgrades first.

Every change is compatible in one direction, both or neither:

Change Backward (new code reads old data) Forward (old code reads new data)
Add a field with a default ✓ the default fills in ✓ old code skips it
Add a field with no default ✗ old data lacks it ✓
Remove a field that had a default ✓ new code ignores it ✓ old code uses its default
Remove a field old code requires ✓ ✗
Rename (Avro alias in the new reader) ✓ ✗
Reuse a Protobuf field number ✗ silently wrong ✗ silently wrong

Databases need backward compatibility for as long as old rows exist, which usually means forever, because rewriting every row is expensive. So many don’t: since PostgreSQL 11 and MySQL 8.0, adding a column with a constant default changes only metadata, and old rows report the default when read. That is schema resolution, done by the database. A rolling deploy of a service that both reads and writes shared data needs both directions, because old and new nodes read each other’s writes. So does a rollback: once new code has written new data, rolling back means old code must read it.

Schema registries

In a message stream, someone has to keep track of every schema that was ever written, hand out IDs, and refuse changes that break compatibility. That’s a schema registry. For Kafka, Apicurio, AWS Glue Schema Registry and the Buf Schema Registry for Protobuf are the well-known ones.

A registry in the write path

Compatibility stops being a convention: an incompatible schema can’t be registered, so it can’t be produced.

How a registry works:

  • Schemas are versioned per subject, by default one per topic for keys and one for values. Avro, Protobuf and JSON Schema are supported.
  • Each message is a magic byte 0, a 4-byte big-endian schema ID and then the payload. Protobuf adds a short list of message indexes, saying which message type in the file it is. Newer versions can carry a schema GUID in a record header instead.
  • Every new version is checked against a compatibility mode: BACKWARD, FORWARD, FULL, their _TRANSITIVE variants, or NONE. The default is BACKWARD, so that consumers can always rewind to the start of a topic and read everything with the newest schema. The trade: under BACKWARD, consumers must upgrade before producers.
  • For Protobuf, BACKWARD_TRANSITIVE is the common recommendation, because adding a new message type isn’t forward compatible.

Choosing a format

JSON Protocol Buffers Avro
Our record 23 bytes 8 bytes 6 bytes
Schema optional (JSON Schema) required, .proto required, JSON
Field identity on the wire name number (tag) position, resolved by name
Unknown fields kept only by generic parsers skipped, preserved in binary dropped by resolution
Reading needs nothing the .proto for meaning, not for parsing the exact writer’s schema
Typical home public APIs, browsers, config RPC (gRPC), service contracts Kafka, data files, generated schemas
Main risk number precision, drifting formats reused field numbers missing defaults, lost writer schema

Analytics adds a fourth shape: columnar files like Parquet, which store each field of millions of records together. That’s a different trade, covered in the analytics lesson.

In the real world

23 → 8 → 6bytes for one small record in JSON, Protobuf and Avro
$460M+lost by Knight Capital in about 45 minutes after a repurposed flag hit old code (2012)
21 vs 20input fields defined vs supplied in CrowdStrike’s Channel File 291 (2024)
  • Knight Capital, August 1, 2012. New trading code reused a flag that had once activated an obsolete feature called Power Peg. A technician didn’t copy the new code to one of the eight servers, so orders carrying the flag reached a server where it still meant “run Power Peg”. In about 45 minutes, while trying to fill 212 customer orders, the router sent millions of child orders that drew more than 4 million executions, and Knight lost more than $460 million. It wasn’t Protobuf, but it was exactly the reused-field-number failure: identical bits, a new meaning, and one node still on old code.
  • CrowdStrike, July 19, 2024. A content update for the Falcon sensor, Channel File 291, used a template type that defined 21 input fields, while the sensor code that fed it supplied only 20. The new content was the first to actually read the 21st field. The read went out of bounds inside a kernel driver, and about 8.5 million Windows machines (Microsoft’s estimate) crashed. The content validator had checked against 21 fields too. Writer and reader disagreed about the schema, and nothing at runtime checked.
  • Twitter’s id_str, 2010. Twitter’s new Snowflake tweet IDs were scheduled to pass 253 on 26 November 2010, after which JavaScript would silently round them. So from October the API sent every tweet, user and message ID as a string as well as a number. X’s current API sends IDs as strings by default.
  • Google built gRPC’s predecessor, Stubby, on Protobuf and ran it across its data centres for over a decade. Its guidance on field numbers (“never reuse”) and on required (“required is forever”) is earned.
  • Kafka shops lean on a schema registry’s BACKWARD default so consumers can replay a topic from the beginning, and many teams add buf breaking or registry compatibility checks to CI so a bad schema fails the build, not production.

Pitfalls

  • Treating JSON numbers as 64-bit integers. Send large IDs as strings. Any JavaScript client, and many others, will round them.
  • Old code that drops unknown fields. Round-tripping through a domain object, an ORM or JSON loses the bytes. Preserve unknown fields wherever old code writes back data it read.
  • Reusing or renumbering a field. Always reserved deleted numbers and names. The failure is silent.
  • “Wire-compatible” type changes that lose data. int64 to int32 parses fine and truncates.
  • New enum values. Old code will meet values it doesn’t know; give every switch a default branch.
  • Avro fields without defaults. Adding one breaks backward compatibility; removing one breaks forward.
  • Checking compatibility only against the latest version while data from much older versions still exists. Use a transitive mode.
  • Forgetting rollbacks. After new code writes new data, rolling back needs forward compatibility.

Recall

Go deeper

Keep going

The atomic notes for this lesson live in the knowledge graph:

Next on the map: Modes of Dataflow (how encoded data moves between processes: databases, REST and RPC, messages and durable workflows) and Message Brokers & Logs.

Done with this lesson?

Mark it learned and the concepts it teaches will light up in your galaxy.

Next in EncodingModes of Dataflow