Foundations · 7 of 7

Doing the Right Thing

Every system that stores data about people makes ethical decisions, whether or not anyone meant to. What to collect, how long to keep it, how to really delete it, and how a model trained on yesterday learns yesterday's mistakes.

≈ 25 min

The whole idea

Deleting a person is harder than deleting a row. Press ← → to step through any diagram.

Why engineers own this

Most of the ethics in a data system isn’t decided in an ethics meeting. It’s decided in a schema migration, a logging statement, a retention setting left at “forever”, a training set nobody questioned. Engineers make those choices daily, usually without noticing they’re choices.

The stakes are concrete. Data about people can be breached, and stays breached forever. It can be repurposed: collected for one thing, used for another the person never imagined. It can be subpoenaed, sold (including in a bankruptcy), or used to make automated decisions about someone’s job, loan or benefits. And a model trained on it can quietly learn, and then scale, the biases in the decisions that produced it.

This lesson treats those as engineering problems with engineering answers. The law gives a useful frame. Under the EU’s GDPR, personal data must be processed lawfully, fairly and transparently; collected for specified purposes; “adequate, relevant and limited to what is necessary”; accurate; kept no longer than necessary; and secured, with the organisation able to demonstrate all of that. Those principles predate the regulation and work as design rules anywhere.The ACM’s Code of Ethics says the same thing in engineering terms: “only the minimum amount of personal information necessary should be collected in a system”.

Privacy by design

The idea, developed in the 1990s by Ann Cavoukian, later Ontario’s privacy commissioner, is that privacy has to be built in from the first design, not bolted on after launch. Her seven principles (2009) read like an architecture checklist: be proactive, not reactive; make privacy the default setting; embed it into the design; aim for full functionality (positive-sum, not a trade against features); protect data across its whole lifecycle; stay visible and transparent; and keep it user-centric.

GDPR Article 25 turned the first two into law as data protection by design and by default. By design means technical measures such as pseudonymisation and minimization, chosen when the system is designed. By default is sharper: “only personal data which are necessary for each specific purpose of the processing are processed”, which covers how much is collected, how far it’s processed, how long it’s stored and who can see it. The default has to be the private option; the user opts into sharing, never out of it.

This isn’t aspirational. In December 2024, Ireland’s Data Protection Commission fined Meta €251 million over a 2018 breach in which attackers abused the “View As” feature to steal access tokens. Of that, €240 million was for violating Article 25: failing to build data protection into the design. The regulator’s statement was blunt about why: failing to build in data protection “throughout the design and development cycle” exposes people to serious risk.Earlier that year, the same regulator fined Meta €91 million for storing some users’ passwords in plaintext.

€251MDPC fine on Meta, December 2024
€240Mof it for failing data protection by design and by default
4%of global annual turnover: GDPR's top fine tier

Data minimization

The strongest privacy control is the field you never stored. It can’t be breached, subpoenaed, repurposed or sold, and it needs no deletion pipeline. Minimization asks, for every piece of data: what specific purpose needs this, and is there a smaller version that serves it?

The same breach, two designs

Collect everythingCollect what the purpose needs
Minimization is the only privacy control that can't be misconfigured later: the data simply isn't there.

The smaller version usually exists:

  • Derive, then discard. Check the passport, keep over_18 = true. Verify the address, keep the postcode region.
  • Coarsen. Birth year instead of birth date; city instead of GPS track; timestamps rounded to the hour.
  • Aggregate early. Count events per day instead of storing every event with a user ID.
  • Separate identifiers. Keep the user ID in one place and pseudonymous tokens everywhere else, so most systems never hold anything directly identifying.
  • Don’t log payloads. Request logs are where personal data goes to live forever, unreviewed.

Signal is the reference case for minimization as architecture. It has published its responses to US grand-jury subpoenas and search warrants. Each time, the only data it could hand over was the date an account was created and the date it last connected. Not because it refused, but because nothing else existed.

Anonymization is harder than it looks

The tempting escape is to “anonymize” data and then use it freely. It fails far more often than people expect, because removing names isn’t the same as removing identity. Every remaining field is a potential join key.

Re-identification by linkage

Neither dataset identifies anyone on its own. The join does.
87%of Americans unique by ZIP + birth date + sex (1990 census data)
99%of Netflix Prize records uniquely identified by 8 ratings with approximate dates
700Mactivities in Strava's 2017 heatmap, which exposed military bases

Latanya Sweeney’s 2000 study found that ZIP code, birth date and sex were likely unique for 87% of the US population (a 2006 re-analysis of 2000 census data put it at 63%, still most people). In 2008, Narayanan and Shmatikov showed that the “anonymized” Netflix Prize dataset of 100 million ratings could be de-anonymized: 8 ratings with dates accurate to within two weeks uniquely identified 99% of records, and cross-referencing public IMDb reviews exposed apparent political views.

Aggregates leak too. Strava’s global heatmap, built from 700 million activities, contained no names at all. In January 2018 an analyst noticed that the glowing jogging loops in otherwise dark regions outlined military bases.

Data retention

Data that served its purpose doesn’t become harmless; it becomes pure risk. Storage limitation says personal data should be kept no longer than its purpose needs. In practice, that means every dataset needs a written retention period and a job that enforces it.

Retention limits are blast-radius limits

The breach is identical. What it can take is decided years earlier, by a retention setting.

The enforcement record is instructive:

  • Deutsche Wohnen (Berlin, 2019). A large German landlord was fined about €14.5 million because its archive system had no way to remove tenants’ data that was no longer needed: payslips, bank statements, employment contracts, tax, social-security and health-insurance data, some of it years old. The regulator called such archives “data graveyards”. The fine went through the EU’s highest court and back; in June 2026 a Berlin court cut it to €900,000 (not yet final), still finding intentional breaches of data minimization and storage limitation.
  • Blackbaud (FTC, 2024). The US Federal Trade Commission found that the software company kept customer data “far longer than was necessary”, which widened the damage of its 2020 breach. The order requires it to delete data it no longer needs and to publish a retention schedule.

Deletion takes time even when done well, and saying so honestly is part of the design. Google, for example, documents that deleting data from its active systems “generally takes around 2 months”, including a recovery period of up to a month in case the deletion was a mistake, and that data can remain in encrypted backups for up to 6 months.

Right to erasure

GDPR Article 17 gives people the right to have their data erased “without undue delay” when, for example, it’s no longer needed or they withdraw consent. The answer is due within a month (extendable by two more for complex cases), and anyone the data was disclosed to must be told, unless that proves impossible or disproportionate. There are real exceptions: legal obligations (tax records), legal claims, public health, research, freedom of expression. But for most product data, the request must be honoured.

For engineers this is a hard distributed-systems problem, as the hero diagram shows. Personal data fans out into caches, search indexes, logs, warehouses, event streams, backups, ML training sets and third parties, usually with no central record of where. Doing it properly looks like this:

Deletion as a checked workflow

Every store registers what it holds; every erasure is a job that isn't done until every store confirms.

Meta’s deletion framework, DELF, is the best public description of this at scale. Developers annotate each data type and each edge in the social graph with how deletion should cascade (“deleting a user deletes the photos they created”). DELF then runs deletions asynchronously, retries failures, keeps a restoration log so mistaken deletions can be reversed for a while, and checks the annotations automatically. Meta reports it handles billions of deletions a day, and that its checks found thousands of missing annotations and dozens of mistakes that would otherwise have left data behind.

Crypto-shredding

Some stores can’t delete one record. An append-only event log, a Kafka topic used for event sourcing, a WORM archive, last month’s backup tapes: rewriting them to remove one person is impractical or impossible. Crypto- shredding deletes by destroying keys instead of data.

Delete the key, not the data

Three users, three keys. Every copy of B's events dies with one deletion.

Encrypt each user’s data with a key unique to that user, held in a separate key store. To erase the user, delete the key. Every copy of their data, in the log, replicas, caches and backups, becomes ciphertext nobody can read, and nothing else changes. NIST’s media-sanitization guidance (SP 800-88, revised in 2025) recognises cryptographic erase as a purge technique for exactly this reason.

The catches are real and worth stating plainly:

  • The key store is now the crown jewels. Lose it and you lose all data; leak it and you lose all privacy. It needs its own backups, and those backups need their own deletion story.
  • It only covers ciphertext. Plaintext that was copied out, into a search index, a warehouse or a log line, before the key was deleted is untouched.
  • Encryption ages. Ciphertext kept for decades has to stay secure for decades.
  • It’s not settled law. No EU regulator has clearly confirmed that key deletion counts as erasure. The UK regulator’s guidance on backups asks for data to be put “beyond use”, which crypto-shredding plausibly achieves, but treat it as defence in depth alongside real deletion where real deletion is possible.

Where the log can forget, use that instead. Kafka’s compacted topics treat a record with a key and a null value (a tombstone) as a delete: compaction removes earlier values for that key, and the tombstone itself is removed after delete.retention.ms (one day by default).

Algorithmic bias

A model learns the patterns in its training data, including the ones you’d never write down as a rule. When the labels are past human decisions, the model learns those decisions’ biases, applies them consistently, at scale, and with an air of objectivity.

How a model inherits and amplifies bias

Removing the sensitive column doesn't help when other features stand in for it.

This is what happened to Amazon’s experimental résumé-ranking tool, reported by Reuters in 2018. Trained on ten years of résumés submitted to the company, mostly by men, it learned to penalise résumés containing the word “women’s” (as in “women’s chess club captain”) and downgraded graduates of two all-women’s colleges. Amazon disbanded the team by early 2017 and said the tool was never used by recruiters to evaluate candidates.

The subtler failure is a proxy label. In 2019, Obermeyer and colleagues studied a widely used US healthcare algorithm that ranked patients for extra care by predicting their future healthcare costs. Because less money is spent on Black patients with the same level of illness, the algorithm systematically underrated how sick they were. Fixing the disparity would raise the share of Black patients flagged for extra help from 17.7% to 46.5%. Nobody coded race into it. The bias was in the choice of what to predict.

17.7% → 46.5%Black patients flagged for extra care, before and after fixing a proxy label
44.9% vs 23.5%COMPAS: Black vs white defendants labelled higher risk who didn't reoffend

Fairness also can’t be one number. ProPublica’s 2016 analysis of the COMPAS recidivism tool found that Black defendants who didn’t reoffend were almost twice as likely as white defendants to have been labelled higher risk. The vendor responded that its scores were equally calibrated: a given score meant the same risk of reoffending regardless of race. Both were true. Kleinberg, Mullainathan and Raghavan (2016) and Chouldechova (2017) proved that when groups have different base rates, a risk score can’t be equally calibrated for both groups and also have equal error rates for both, except in degenerate cases such as perfect prediction. Choosing which fairness property to hold is a values decision, and it should be made explicitly by people accountable for it, not left implicit in a loss function.

The worst outcomes come from automated decisions with no human check. In the Dutch childcare-benefits scandal, the tax authority’s risk-classification model used nationality as a risk indicator, and tens of thousands of families were wrongly pursued as fraudsters. The Dutch data protection authority fined the tax administration €2.75 million in 2021 for discriminatory processing, and the government resigned in January 2021 after a parliamentary inquiry titled Ongekend onrecht, “unprecedented injustice”. By mid-2026, more than 44,000 parents had been recognised as affected.

The law is catching up. GDPR Article 22 already limits decisions based solely on automated processing; in 2026 the Dutch authority fined Uber nearly €825 million for fully automated decisions that deactivated drivers’ accounts over suspected fraud or low ratings, in 2018–2022 (Uber has appealed). The EU AI Act, in force since August 2024, has banned some practices since February 2025, and after a 2026 amendment its main obligations for high-risk systems such as hiring and credit scoring apply from December 2027.

Trade-offs

Practice Protects against Costs Common failure
Data minimization breach, misuse, subpoena, repurposing some analytics and features become impossible “we might need it later”
Privacy by default over-sharing nobody chose lower engagement with optional sharing dark patterns that nudge users out of defaults
Pseudonymisation casual exposure inside the company key management; still personal data treating it as anonymization
Retention limits breach scope, stale-data liability historical analysis is lost no job enforces the policy
Deletion workflow incomplete erasure registration discipline for every store new store added, never registered
Crypto-shredding undeletable logs and backups key store becomes critical infrastructure plaintext copies made elsewhere
Bias audits discriminatory outcomes at scale labelled demographic data, ongoing effort auditing once, at launch

In the real world

  • 23andMe (2023–2025). Attackers used reused passwords to get into about 14,000 accounts, and through the DNA Relatives feature viewed profile data of roughly 6.9 million people. In March 2025 the company filed for bankruptcy, and its genetic database was sold as an asset, to a nonprofit led by its co-founder, for $305 million. The UK regulator fined it £2.31 million. Data outlives the promises made when it was collected.
  • Strava (2018). An aggregate, nameless heatmap revealed the layout of military bases. Strava responded by reviewing features that could be “compromised by people with bad intent” and simplifying privacy controls.
  • Signal. Subpoenaed repeatedly, able to provide two timestamps per account. Minimization made the legal question moot.
  • California’s Delete Act. Since January 2026, Californians can send one deletion request to every registered data broker through a state platform, and since August 2026 brokers must process those requests at least every 45 days. Erasure is becoming infrastructure.

Pitfalls

  • “It’s anonymized.” Usually it’s pseudonymised, or joinable. Assume re-identification is possible until shown otherwise.
  • Deleting the primary row and calling it erasure. The copies are the problem: indexes, logs, warehouses, backups, partners, models.
  • Retention policies on paper only. If no scheduled job deletes the data, the policy is “forever”.
  • Logging request bodies. The most common way personal data ends up everywhere, unreviewed and unretained.
  • Dropping the sensitive column to fix bias. Proxies carry it back in. Measure outcomes by group.
  • Defaults that share. If privacy needs the user to find a setting, it isn’t privacy by default.

Recall

Go deeper

Keep going

The atomic notes for this lesson live in the knowledge graph:

Next on the map: Data Models & Query Languages (where the shape of personal data is first decided) and, further on, Change Data Capture & Event Sourcing (where crypto-shredding earns its keep).

Done with this lesson?

Mark it learned and the concepts it teaches will light up in your galaxy.

Foundations: roadmapBack to the roadmap