Notesencoding🌱 seedling
JSON vs Binary Encoding
Also: JSON · Binary Encoding · Textual Encoding · MessagePack · Base64 Overhead
Textual formats like JSON describe themselves, so they carry field names in every record and have no enforced schema. Schema-driven binary formats carry only values, which makes them several times smaller and lets compatibility be checked before deploy, at the cost of human readability.
Why it exists
JSON is the default for web APIs because anyone can read it with no shared setup. Inside a system, where you control both ends and store terabytes, its costs add up.
How it works
- Field names repeat:
{"id":150,"name":"Ada"}is 23 bytes, 10 of them names and quotes. - Numbers are vague: no int/float distinction or precision. Parsers that use IEEE 754 doubles (JavaScript) are exact only up to 2^53.
- No byte strings: binary data must be Base64-encoded, about 33% larger.
- Binary JSON (MessagePack, BSON) packs values tighter but keeps the names: 15 bytes for the record above in MessagePack.
- Schema-driven binary drops names: the same record is 9 bytes in Thrift's compact protocol, 8 in Protocol Buffers and 6 in Apache Avro.
Trade-offs
| JSON | Schema-driven binary | |
|---|---|---|
| readable without tools | yes | no |
| size | large | small |
| schema | optional (JSON Schema) | required, doubles as documentation |
| compatibility checks | by convention | automatable in CI or a Schema Registry |
In practice
- Twitter (2010): its new Snowflake tweet IDs were about to pass 2^53, which JavaScript would round, so the API added
id_stralongsideidfor every ID. X's current API returns IDs as strings by default. - Public APIs stay JSON for reach; internal RPC and event streams favour binary with schemas.
Pitfalls
- Sending 64-bit IDs as JSON numbers.
- Treating "binary" as the main win. The schema, and the compatibility rules it enables, usually matter more than the bytes saved.
Connections
- Part of
- Serialization
- Mentioned
- Schema Registry
Recall
These cards are also in your review deck, scheduled with spaced repetition.