The Amazon S3 Outage of 2017: What a Typo Taught the Internet About Blast Radius

*Part of the Forward Deployment Engineer horror-story series. If you have not read the main guide yet, start with How to Ace the Forward Deployment Engineer Interview at /blog/forward-deployment-engineer-interview-guide-2026.*

Why FDEs read outage postmortems

If you are preparing for a Forward Deployment Engineer loop, "and what happens when X fails?" is one of the most reliable question shapes on the interview. The strong candidate has a specific, honest, mechanical answer. The weak candidate hand-waves about "resilience" and "redundancy."

The fastest way to build a real vocabulary for failure is to read the postmortems that shaped modern infrastructure thinking. The Amazon S3 outage of February 28, 2017 is at the top of that list — not because it was the worst outage ever, but because the postmortem is unusually honest and the lessons still map to almost every AI system being built in 2026.

What happened, hour by hour

At approximately 9:37 AM PST on Tuesday, February 28, 2017, an authorized engineer on the Amazon S3 team was executing a routine debugging playbook for the S3 billing system in the US-East-1 (Northern Virginia) region. The playbook required removing a small number of servers from one of the S3 subsystems.

The engineer typed the command. There was a typo in the arguments. The typo caused a much larger set of servers to be removed than intended — including servers running two S3 subsystems that the operator did not intend to touch: the index subsystem (which manages the metadata and location information for all S3 objects in the region) and the placement subsystem (which manages allocation of new storage when objects are written).

Both subsystems required a full restart to recover, because they had not been restarted for many years. The scale of the S3 metadata store had grown in that time. The restart took significantly longer than expected: index subsystem recovery completed at 12:26 PM PST, placement completed at 1:18 PM PST. That is nearly four hours of degraded or unavailable S3 in the region.

What went down with it

Because S3 is a load-bearing primitive for most of the internet, the blast radius was staggering:

• Slack, Trello, Quora, Medium, Business Insider, and thousands of smaller services became unavailable or degraded
• AWS console for many services (which store their own UI assets in S3) failed
• The AWS Service Health Dashboard — the page you visit to check if AWS is broken — was itself broken, because its status indicators were served from S3 in the affected region
• Users could not tell whether it was their code that was broken, or AWS
• Deploys froze at hundreds of companies because artifact stores lived in S3

The last point is the one FDE interviewers love. AWS could not tell its own customers what was happening because their own operational tools depended on the service that was down. This is called a circular dependency in postmortem language, and identifying them before an outage is one of the hardest and most valuable jobs in any AI system design.

The postmortem AWS actually published

AWS's summary is worth reading in full: https://aws.amazon.com/message/41926/

The five commitments AWS made afterward, and what they mean for any system you design:

1. Modify the tooling that removes capacity to prevent removing capacity below minimum thresholds. Translation: build safety rails into the operator interface so that a typo cannot cause a catastrophic outcome. This is exactly the argument for well-designed approval gates in agentic AI systems — the interface should make the wrong action harder to take, not just possible-to-undo after the fact.

2. Audit other operational tools for similar risks. Translation: one instance of a class of bug means the class exists elsewhere too. This is the "pattern-scan after every fix" rule that any competent SRE or FDE knows to apply.

3. Break S3 into smaller cells. Translation: the blast radius of any single operational mistake should be bounded by design, not by luck. In modern AI system design, this maps directly to per-tenant isolation, sharded queues, and regional cells. If a bad prompt template can only take down one cell, you have a bad hour. If it can take down the whole system, you have a career-ending outage.

4. Improve the recovery time of the index and placement subsystems. Translation: assume everything you built five years ago has drifted past its recovery-time assumptions, because scale grows silently. Regularly force-restart your systems in staging (game-day drills) to prove your MTTR is still what your runbook says it is.

5. Change the way status is reported so it does not depend on any single region. Translation: your operational plane must not depend on the data plane it monitors. AI-system analog: your alerting, dashboards, and rollback tooling should not rely on the same models, agents, or vector stores that they are watching.

What an FDE should take from this

When you are on-site with a customer and they ask about resilience, the S3 story gives you three concrete talking points that carry more weight than any framework.

Every operator tool is a weapon. Someone is going to type the wrong thing on the wrong day. The design question is not "how do we prevent typos" — it is "how do we bound the damage of any single command, and how do we detect it and undo it fast." Ask customers how their operators trigger changes and where the confirmation prompts live. Most enterprises have never mapped it.

Blast radius is a design property. In your agentic-system designs, be able to answer explicitly: if a single tool call goes wrong, what is the maximum damage? If a single model version rollout is bad, how many customers hit it before you notice? "All of them" is a failing answer.

Your monitoring must not share fate with the system it monitors. If your dashboards, alerts, and rollback tools live in the same AWS region as your workload, or worse, are served by the same LLM provider as the workload, the day the workload goes down you have zero visibility. This is a lesson AWS learned publicly in 2017 and that most AI teams have not yet re-learned.

The interview version

If an FDE interviewer describes a customer scenario and asks about failure, walk through the S3 lens: what is the largest possible blast radius, is there a circular dependency between the workload and its operational tools, and what is the MTTR your customer can actually prove today (not what the runbook claims). If you can put that on a whiteboard in under ten minutes, you will separate yourself from most candidates.

Further reading

• AWS official postmortem: https://aws.amazon.com/message/41926/
• The next post in this series: **The GitLab Database Deletion of 2017** at /blog/gitlab-database-deletion-2017-postmortem
• The full FDE interview guide: /blog/forward-deployment-engineer-interview-guide-2026

Related concepts from The AI Security Handbook (https://a.co/d/03l3YNxS): cell architecture, tenant isolation, and operator-tool policy — all of which apply to modern agentic systems, not just object storage.