Foundations · 2 of 7

Scalability

"It scales" means nothing on its own. What grows, how you measure what users feel, why the slowest 1% decides everything, and how to add capacity when the load arrives.

≈ 30 min

The whole idea

Press ← → to step, or Space to play and pause, whenever a diagram has focus.

What scalability means

A system that works today can fail tomorrow without a single line of code changing. It had 10,000 users and now has 100,000. It processed a gigabyte a day and now processes a terabyte. The most common cause of degradation is not a bug, it’s growth.

So people say a system “scales” or “doesn’t scale”, as if it were a property like colour. It isn’t. Scalability is a system’s ability to cope with increased load, and it only means something once you say which load and how much. The useful questions are concrete: if the system grows in this particular way, what happens to performance? And what are our options for adding resources to keep it acceptable?

Answering them takes three steps, which shape the rest of this lesson:

  1. Describe the load with a few numbers that actually drive the work.
  2. Describe the performance users feel, as a distribution rather than an average.
  3. Choose how to add capacity: a bigger machine, more machines, or a different design.

Describing load

Load is described with load parameters: the handful of numbers that determine how much work the system does. Which ones matter depends on the architecture. Requests per second to a web server. The ratio of reads to writes in a database. The number of concurrent users in a chat room. The hit rate of a cache. The size of the dataset.

Two things make choosing them harder than it sounds.

  • Averages hide the peak. Traffic follows the day, spikes at the top of the hour when cron jobs fire, and explodes during events. You size for the peak, or for how fast you can grow toward it.
  • Distributions hide the outlier. One customer with a hundred times the data, one key that everyone reads, one account with millions of followers. Often the load parameter that matters isn’t a total at all; it’s the shape of a distribution.

The classic example is a social network’s home timeline. Twitter shared its numbers in 2012: users posted an average of about 5,000 tweets a second (more than 12,000 during big events), while home timelines were read about 300,000 times a second. Handling 12,000 writes a second is not hard. The hard part was fan-out: each tweet has to appear in the timeline of every follower.

Fan-out on write

The obvious design stores each post once and assembles a timeline when someone asks for it.

Naive: fan-out on read

Correct, simple, and repeated hundreds of thousands of times a second for posts that haven't changed.

That puts the expensive part, the join across everyone you follow, on the hottest path in the system. The alternative flips the work around: do it once, when the post is written, and keep a ready-made timeline for every user. This is fan-out on write.

Fix: fan-out on write

Each new post is pushed into every follower's timeline cache. Reads become a single lookup.

Twitter’s 2012 pipeline did exactly this: each tweet was inserted into the follower timelines held in a large Redis cluster, every timeline replicated three times and capped at 800 entries, adding up to around 30 billion timeline deliveries a day. To save memory, it only kept cached timelines for active users, defined as anyone who had logged in within the last 30 days. Everyone else’s timeline was rebuilt on demand.

Fan-out on write trades cheap reads for expensive, amplified writes. The amplification is proportional to the follower count, and follower counts are wildly skewed.

Breaks again: celebrities

Follower timelines
The same design, one very large account. The fix is to stop treating every account the same.

The same pressure shows up in newer systems from the other direction. Bluesky fans posts out into per-user timelines in ScyllaDB, paging through followers 10,000 at a time. Its hot spots came from accounts that follow hundreds of thousands of others, whose timeline partitions took far more writes and trimming than their neighbours on the same shard. In early 2025 the team made those timelines deliberately lossy: each write to a follower’s timeline is kept with probability min(limit ÷ follows, 1), so everyone gets the same “follows’ worth” of fan-out. The p99 time to fan out a full post dropped by more than 96%, and jobs for the largest accounts went from 5–10 minutes to under 10 seconds.The example limit in the post is 2,000 follows. Someone following 8,000 accounts keeps about a quarter of the writes, and a heavy follower reading that many accounts won’t notice the missing posts.

Latency is a distribution

Once load is described, describe performance. For a batch job that’s usually throughput: records per second, or how long a job of a given size takes. For an online service it’s response time: how long a client waits between sending a request and getting the answer.

The words are often used loosely, so be precise:

  • Response time is what the client sees: the work, plus queueing, plus network round trips.
  • Service time is just the time spent actively processing the request.
  • Latency is the time a request spends latent, waiting to be handled. In everyday use it often means response time; when it matters, say which you mean.

Send the same request a thousand times and you get a thousand different answers. A context switch, a lost packet and a TCP retransmit, a garbage-collection pause, a page fault, a cache miss: each adds noise. So response time is not a number, it’s a distribution.

1,000 requests, one endpoint

1,000requests to one endpoint
10 ms30 ms100 ms300 ms1 s2 sresponse time (log scale)requests

Each bar counts requests in one latency band. Bars past the p99 line turn red once it appears: the tail.

Most requests cluster around 40 ms. The mean lands well to their right, pulled by a tail it doesn't describe.

The mean is a poor summary: it doesn’t tell you how many users experienced what. Use percentiles instead. Sort the response times; the median (p50) is the halfway point, so half your requests were faster and half slower. The higher percentiles, p95, p99 and p99.9, tell you how bad the outliers are. These are the tail latencies.

The tail matters because of who lands in it. Amazon’s 2007 Dynamo paper describes internal SLAs written at the 99.9th percentile, for example a response within 300 ms for 99.9% of requests at a peak of 500 requests per second. An average-based SLA would hide exactly the customers with the longest histories, who need the most processing and so land in the slow tail, and they tend to be among the most engaged. Going past p99.9 was judged too expensive for the benefit.

Slow requests also slow down other requests. A server can only work on a few things at once, so a handful of slow requests at the front of a queue hold up everything behind them: head-of-line blocking. The server sees a fast request; the client saw a slow one. That’s why response time should be measured on the client side.

Measuring it honestly

Load testers have a subtle way of lying. Most send a request, wait for the reply, then send the next one. When the server stalls, the tester stalls with it and quietly stops generating the load you asked for.

Coordinated omission

The tester and the server cooperate to hide the stall. The users it was standing in for don't get that luxury.

Gil Tene named this coordinated omission. His wrk2 load generator fixes it by sending at a constant rate and timing each response from when the request should have been sent. Its README pauses a server for 1.4 seconds: the naive run reports a p99 of 6 ms, the corrected one 1.27 s.

Two more rules keep your percentiles honest:

  • Never average percentiles. The mean of ten servers’ p99s is not the fleet’s p99. Keep a histogram per interval and merge histograms, then read percentiles off the result. HdrHistogram, t-digest and DDSketch are the standard structures; Prometheus histograms do the same with fixed buckets.
  • Look at the full distribution over time, not a single summary. A p99 that’s flat while p99.9 doubles is a story your dashboard should show.

Tail latency amplification

The tail gets much worse once one user request touches many servers. A search query, a page made of dozens of service calls, a scatter-gather over shards: the user waits for the slowest part.

The slowest leaf sets the pace

100 leaf servers (5 shown)
Every leaf is fast 99% of the time. The user is not.

The arithmetic is unforgiving. If each server is slow for 1 request in 100, a request that needs all of 100 servers is slow with probability 1 − 0.99¹⁰⁰, about 63%.

1%of requests slow, when a user request hits one server
63%slow, when it needs 100 such servers in parallel
≈ 1 in 5slow with 2,000 servers, even if each is slow only 1 time in 10,000

Dean and Barroso’s 2013 paper The Tail at Scale measured this in a real Google service. Measured at the root, the p99 for one random leaf to answer was 10 ms. The p99 for all leaves to answer was 140 ms, and for 95% of them it was 70 ms. Waiting for the slowest 5% accounted for half of the total.

You can’t make every server consistently fast; shared machines, background work and garbage collection guarantee some variability. So, just as fault-tolerant systems build a reliable whole out of unreliable parts, tail-tolerant systems build a predictable whole out of unpredictable parts. The simplest technique uses replicas you already have.

Fix: hedged requests

Wait about as long as a normal slow request, then race a second copy.

Hedging only after the p95 has elapsed limits the extra load to about 5%. Tied requests go further: enqueue the request on two replicas, a millisecond or so apart, and the first to start working tells the other to cancel. The same paper describes the rest of the toolbox: splitting data into many small micro-partitions so load can move in small steps, extra replicas of hot partitions, temporarily excluding slow machines from the pool, and returning “good enough” results that skip a few slow leaves when the answer allows it.

Little’s Law and queueing

The hero diagram showed a server whose p99 jumped tenfold when load doubled. Why so sudden? Two pieces of queueing theory explain it.

Little’s Law relates three long-run averages in any stable system:

L = λW: requests in the system = arrival rate × time each request spends in the system.

At 2,000 requests a second and a 50 ms average response time, there are on average 2000 × 0.05 = 100 requests in flight. That’s how many threads, connections or concurrent executions you need. AWS sizes Lambda concurrency with exactly this formula: 5,000 requests a second at 200 ms each needs a concurrency of 1,000, which is also the default regional limit for an account.

The second piece is what happens as a server gets busy. With random arrivals, a request often shows up while the server is still working on others, so it waits. The busier the server, the longer the wait, and the curve is anything but linear.

Why busy servers get slow

queues dominate0 ms50 ms100 ms150 ms200 ms250 ms0%20%40%60%80%100%utilization (fraction of time the server is busy)10 ms of actual work
Utilization ρ—
Arrival rate λ—
Time in system W—
In flight L = λW—

One server, 10 ms of work per request (so at most 100 req/s), random arrivals. Mean time in system is 10 ms ÷ (1 − utilization).

Four utilizations, one server. Response time doubles at 50% busy and is 20× the actual work at 95%. Little's Law turns that into requests piling up in flight.

For the simplest queue (one server, random arrivals), mean time in the system is the service time divided by (1 − utilization). At 50% busy a request takes twice its service time; at 90%, ten times. Real systems are lumpier than this model, which makes things worse, not better.

The feedback loops make it worse. Slow responses cause client timeouts; timeouts cause retries; retries raise the arrival rate, which pushes utilization further up the curve. That spiral is where overload and cascading failures begin.

Scaling up vs scaling out

Now the options for adding capacity. There are two basic moves, and most real systems use both.

Two ways to add capacity

Scale up (vertical)Scale out (horizontal)
Up is simple until it isn't possible. Out has no ceiling, but state makes it hard.

Vertical scaling (scaling up) means a more powerful machine: more CPUs, RAM and faster disks under one operating system. Everything stays in one address space, so there’s no network between components, no partial failure, no distributed transactions. Its limits are a ceiling (the biggest cloud VMs, AWS’s U7i high-memory instances, top out at 1,920 vCPUs and 32 TiB of RAM), cost, and the fact that one machine is one failure domain. Close relatives are shared-disk designs, where several machines share one storage array; they ease the ceiling but contend on locks.

Horizontal scaling (scaling out) means spreading load across many machines, each with its own CPU, memory and disk, coordinating over the network: a shared-nothing architecture. It has no hard ceiling, you can buy commodity machines, and you can spread them across data centres. But it pushes distributed-systems problems into your application.

The key difference is state. Stateless services scale out trivially: put a load balancer in front and add copies. Stateful data systems have to be split into shards, and then you own routing, rebalancing, hot keys and queries that cross shards. That’s why the usual path is to scale databases up as far as is sensible, add read replicas, and shard only when you must.

Some systems are elastic: they add and remove resources automatically as load changes. Others are scaled by hand. Elasticity is great for unpredictable load; manual scaling is simpler and has fewer operational surprises.

Scale up Scale out
Ceiling the largest machine you can buy practically none for stateless work
Programming model one machine, no network between parts partial failure, network, coordination
Failure one machine is one failure domain lose a node, keep serving
Upgrades usually a restart onto new hardware add or replace one node at a time
Stateful data just works needs partitioning and replication
Best for databases and anything hard to split stateless tiers, huge datasets, global reach

In the real world

OpenAI’s ChatGPT database (January 2026). PostgreSQL behind ChatGPT’s 800 million users runs as a single primary on Azure, with a hot standby and nearly 50 read replicas across regions, serving millions of queries per second at a low double-digit millisecond p99. Load grew more than 10× in a year. The scaling moves were the textbook ones: route reads to replicas, pool connections with PgBouncer (average connection time fell from 50 ms to 5 ms in their benchmarks), and move shardable, write-heavy workloads to sharded systems such as Azure Cosmos DB. New tables are no longer allowed on the main cluster. The incidents followed a pattern: something upstream, such as a cache-miss storm or a launch-day write surge, suddenly multiplied the load on the database.

Figma (2020–2024). In 2020 Figma ran a single Postgres database on AWS’s largest physical instance. As its database stack grew almost 100×, it first split groups of tables into about a dozen databases (vertical partitioning), plus caches and replicas. That ran out when single tables reached terabytes and billions of rows and approached RDS’s IOPS limits; the smallest unit of vertical partitioning is one table. Horizontal sharding came next, behind a new query router called DBProxy. Sharding the first table took about nine months and shipped in September 2023 with ten seconds of partial unavailability on the primaries.

Twitter and Bluesky. Both chose fan-out on write for home timelines and both ran into the skewed tail of the follower graph, from opposite ends: very large accounts for Twitter, which in 2012 was working toward merging their posts at read time, and accounts that follow enormous numbers of others for Bluesky, which made their timelines lossy.

Google. Hedged requests cut BigTable’s p99.9 for a 100-server read by 96% for 2% more requests, using replicas that already existed for fault tolerance. Search, whose queries fan out to thousands of leaves, leans on micro-partitions, extra replicas of hot data and “good enough” answers that skip the slowest leaves.

Pitfalls

  • “It scales.” Scales along which load parameter, to what level, at what cost? Without those, it’s a sentiment, not a property.
  • Reporting the mean. It’s dominated by the tail and describes no real user. Use percentiles, and measure on the client.
  • Averaging percentiles across servers or time windows. Merge histograms instead.
  • Trusting a closed-loop benchmark. If the load generator waits for replies, stalls disappear from its results. Check for coordinated omission.
  • Testing one server and extrapolating. Fan-out multiplies tail latency; a 1% problem per server is a majority problem per user.
  • Running hot to save money. Past about 70–80% utilization, queueing dominates response time, and there’s no headroom for a spike or a lost machine.
  • Designing for scale you don’t have. Sharding and fan-out pipelines are expensive to build and run. Scale up and add replicas first; split when a specific load parameter forces it.

Recall

Go deeper

Keep going

The atomic notes for this lesson live in the knowledge graph:

Next on the map: Maintainability (keeping the system easy to run and change for years), then Load Balancing, Caching and Timeouts, Retries & Overload, which pick up where queueing leaves off.

Done with this lesson?

Mark it learned and the concepts it teaches will light up in your galaxy.

Next in FoundationsMaintainability & architecture trade-offs