The GitLab Database Deletion of 2017: When Every Backup Failed at Once

*Part of the Forward Deployment Engineer horror-story series. Start with the main guide at /blog/forward-deployment-engineer-interview-guide-2026 or the previous post: The Amazon S3 Outage of 2017 at /blog/amazon-s3-outage-2017-postmortem-what-fdes-should-learn.*

Why this one is different

Most outages are about one system failing. The GitLab January 2017 incident is famous because SIX independent things failed in sequence — one live database, and five backup mechanisms every one of which had been marked "green" for months. It is the canonical example of what an FDE mentor of mine calls "the belief-vs-reality gap in operational readiness."

If you take one thing from this post to your next interview, take this: untested backups are not backups. They are wishes.

What happened

On the evening of January 31, 2017, GitLab.com was fighting a spam-driven load spike that had produced database replication lag on their secondary Postgres server. A senior engineer working late — call him YP as the postmortem does — was investigating the replication issue and attempting to clear the state on the secondary so it could re-sync from the primary.

The two servers were named similarly. YP was SSH'd into what he believed was the secondary. He ran sudo rm -rf on the Postgres data directory to wipe it before triggering a fresh sync. About two seconds in, he realized the deletion was happening much faster than expected. He canceled the command.

He was on the primary. About 300GB of production data was gone.

The five backup mechanisms that all failed

GitLab had, on paper, five independent ways to recover from exactly this kind of incident. Every one of them failed at the moment it was needed:

1. LVM snapshots. These were being taken but only every 24 hours, and the most recent snapshot was 6 hours old at the time of the incident. Recoverable data would be stale but existed.

2. Regular database dumps via pg_dump. These had been silently failing for months. The reason was subtle: the dumps were being taken by a pg_dump binary that was a slightly different version from the running Postgres, and the version mismatch caused the dumps to produce empty files. Nobody had checked. The alert for "backup succeeded" was firing on the exit code of the script, not on the size of the output.

3. Disk snapshots via Azure. Azure disk snapshots were not enabled for the database server. The team believed they were.

4. S3 backups. The S3 bucket where nightly backups were supposed to go was empty. The upload script had been failing silently for a long time.

5. Replication to a secondary. The secondary that was being repaired at the moment of the incident was also lag-behind and partially clobbered by the SSH-into-wrong-server confusion.

Six backup and recovery paths, five of them dead, one of them (the LVM snapshot) six hours stale.

How they actually recovered

The company was ultimately saved by a completely unofficial copy of the data. Six hours before the incident, an engineer had taken a copy of the production database to their laptop for a development task. That laptop copy was newer than any official backup and became the source of truth for restoring GitLab.com.

The recovery itself took hours and GitLab did something remarkable during it: they livestreamed the entire process on YouTube. Thousands of people watched a global service being restored in real time, with the on-call engineers explaining what they were doing and why. It became an unlikely marketing win because it demonstrated an extraordinary level of technical transparency.

Data lost in the final tally: approximately 6 hours of user activity — issues, merge requests, comments, and projects created between the last-good snapshot and the deletion.

The postmortem GitLab published

GitLab's public writeup is unusually detailed and worth reading: https://about.gitlab.com/blog/2017-02-01-gitlab-dot-com-database-incident/

What the FDE should take from this

Untested backups are lies. Every backup mechanism must be tested end-to-end by restoring it into a working system on a regular schedule. If you have never restored from your backup this quarter, you do not have a backup — you have a probability distribution.

Alerting on "success" is not alerting. If a script that produces an empty file exits with status zero, your monitoring will happily tell you everything is fine. Alerts must check the property the backup is supposed to have (a valid dump file above N megabytes, restorable to a test cluster), not whether the script that produced it exited cleanly.

Cognitive load is a failure mode. YP was tired, fighting an active incident, and using an SSH prompt that did not distinguish clearly between primary and secondary. Interface design that makes destructive operations easy to confuse is a security bug. A hostname that just says the color of the environment (green vs red) or refuses to accept a destructive command without a typed confirmation of the environment name will save someone's career eventually.

The human on the day of the incident is not the one who should design the safety rails. This is one of the deepest lessons in operational engineering. The person under pressure at 2am will not remember your policy document. The interface has to be safe by construction.

Postmortems that are honest earn more trust than perfect uptime. GitLab lost data. They wrote up exactly how, named the mechanisms, took responsibility, and shipped the fix in public. Enterprise customers evaluating them AFTER this incident cited the transparency as a reason to trust them more, not less. If you are ever on an FDE deployment and you are involved in an incident, the way you write it up matters as much as how you fix it.

The AI-system version of this problem

Every point above maps almost 1:1 to modern agentic AI systems, which is why the story stays in every FDE reading list I recommend.

• Agent state, memory, and audit trails have the same "backup vs recovery" gap. A system that writes every trace to a log store you have never restored from is one bad day away from GitLab territory.
• Prompt templates, model versions, and tool contracts have the same "was this deployed correctly" ambiguity as backup scripts. A CI job that reports success on the exit code of a deploy script but does not verify the deployed prompt actually produces the expected eval score is the AI-native version of an empty-file backup.
• Operator interfaces for agentic systems — the "kill this run" and "roll back this agent version" buttons — need the same defense-in-depth as production database CLIs. If your kill switch has no confirmation and your rollback lives in the same dashboard that the runaway agent is DoSing, you have a GitLab-shaped problem waiting.

The interview version

If an interviewer asks "how do you make a system recoverable," the strong FDE answer starts with "we test the recovery on a schedule and prove the RTO/RPO by actually doing it." Not "we take backups." Not "we have retention policies." Real recovery is a property that gets proven, not asserted.

Further reading

• GitLab's public postmortem: https://about.gitlab.com/blog/2017-02-01-gitlab-dot-com-database-incident/
• The YouTube livestream of the recovery (still available): search "GitLab database livestream 2017"
• The next post in the series: **The Cloudflare Regex Outage of 2019** at /blog/cloudflare-regex-outage-2019-postmortem
• The FDE interview guide: /blog/forward-deployment-engineer-interview-guide-2026

For the leadership frame on operational transparency and the "prove recovery" discipline, see Applied AI Leadership: A Practitioner's Guide to Enterprise AI Transformation at https://a.co/d/0ecKO4U0.