How one config change takes down production, and how to contain yours

Four outages, one shape

Between July 2024 and November 2025, four of the largest outages in cloud and security did not start with an attack or a hardware failure. Each started with a normal change, and each reached far more customers than anyone intended, quickly. The postmortems are public and they are worth reading in full.

The details differ. The shape is the same: a change that was safe in the place it was tested and unsafe in the place it landed.

Blast radius is a design decision

If every change goes to every customer at once, your blast radius is 100% before anything has gone wrong. Progressive delivery changes that default. You ship to a small ring first, wait for an automated check, and only then widen.

It works the same whether you use Argo Rollouts or Flagger on Kubernetes, a feature-flag service or deploy stages in a pipeline: an internal ring, a small share of real traffic, a bigger share, then everyone. The gate matters more than the rings. A person watching a dashboard at 3 a.m. is a slow gate, and a gate that compares error rate and latency against the old version is a fast one.

Data and configuration need the same treatment as code. Google's report says the bad policy data was replicated across regions, and its fix list includes incremental replication with time to validate. CrowdStrike's report says template instances should have staged deployment. Both are the code-rollout lesson applied to content.

Treat generated config as untrusted input

Cloudflare's Bot Management system has a limit of 200 machine learning features at runtime, well above the roughly 60 in use, because memory for them is preallocated. A database permission change made the query that builds the feature file return duplicate rows, and the file grew past the limit. The proxy code that loaded it hit an unhandled error and returned HTTP 5xx.

The file was rebuilt every five minutes, and only part of the database cluster had the new permissions, so the network swung between good and bad files. That is why the failure came and went. The figure replays that pattern with two kinds of proxy.

A program that reads generated config should validate it before use, with checks on schema, size and count, keep the last known good version, and degrade or fail open instead of crashing. Cloudflare's own list of fixes starts with hardening how it ingests its generated configuration files, and adds more global kill switches.

let active = loadFromDisk(); // last known good

function onNewConfig(raw) {
  const next = parse(raw);
  if (!schemaOk(next) || next.features.length > LIMIT) {
    metrics.increment("config.rejected");
    alert("bad config, still serving version " + active.version);
    return; // keep serving with the old one
  }
  active = next;
}

Race conditions hide in automation

AWS's DynamoDB outage began in DNS management. A planner produces plans for the service's endpoints, and several enactors apply them. One enactor was slow, another applied a newer plan and started cleaning up old ones, and the slow one then wrote its old plan over the new one. The cleanup deleted that old plan, and with it every IP address for the regional endpoint.

The first enactor's check, is my plan newer than what is live, was true when it ran and false by the time it wrote. A check followed by a write leaves a gap. Closing it means making the write itself conditional on the version you saw, with compare-and-swap or a conditional update, so a stale writer fails instead of winning.

Recovery from the DNS fix was a load event too. In AWS's account, the EC2 lease manager fell into what it calls congestive collapse from restart queues, and engineers throttled requests to get out. Network Load Balancer health checks failed on instances that had not yet received their network state, and automatic failover kept removing capacity until engineers turned it off. EC2 was not fully recovered until 1:50 PM, almost eleven and a half hours after the DNS record was repaired.

Make failures small, then make them visible

Cells are the architectural version of rings. Customers are split across independent copies of the stack, each with its own share of users, and a bad deploy or bad data lands in one cell at a time. AWS describes the pattern in its Well-Architected material on reducing the scope of impact with cell-based architecture.

Once the blast radius is small, speed of detection decides the rest. An alert on raw error counts is either noisy or late. Google's SRE Workbook recommends alerting on how fast the error budget is burning instead: page at 14.4 times the planned burn over one hour, which spends 2% of a 30-day budget, page at 6 times over six hours, and open a ticket at 1 times over three days.

With a 99.9% target, a failure that hits 10% of requests burns the whole month's budget in 7.2 hours, and the first alert fires in under ten minutes. A failure that hits 0.5% never pages anyone, and that is correct: it is a ticket, and it takes six days to use the budget.

A checklist for your own platform

When I review a platform, these are the questions I start with.

1. Can any change reach more than 5% of production without passing an automated health gate?
2. Are config and data deployed through the same pipeline as code, with the same validation?
3. Does every kill switch exist, and has anyone used one this quarter?
4. Is there a last-known-good fallback for every generated file the system reads?
5. Do retries use jitter and a budget?
6. Do alerts fire on burn rate rather than on raw error counts?
7. When did you last roll back on purpose, in production, on a quiet day?