Skip to main content
🎓 Claude Code Masterclass Learn AI-assisted development on Udemy — plus the companion book on Leanpub & Amazon. Start Learning
An engineer speaking on stage in front of the RoachFest logo at RoachFest London 2026, recounting a real GCP outage
database

Surviving a Real GCP Outage: A Multi-Cloud CockroachDB Story from RoachFest London 2026

A RoachFest London account of a real GCP outage with zero platform impact thanks to a multi-cloud CockroachDB architecture.

LB
Luca Berton
· 5 min read

The most interesting resilience story I picked up at RoachFest London 2026 did not come from a keynote slide. It came from a short conference-floor clip of an engineer recounting their own commute home. The recording doesn’t carry a name badge for this segment, but the setting, timing, and subject matter line up closely with the same Form3 platform team’s session on multi-cloud resilience and chaos testing covered elsewhere on this site — I’m treating this as very likely the same team’s talk rather than a separate, unrelated speaker, though I can’t confirm that with full certainty from the clip alone. What I do have either way is a specific, concrete account of a distributed platform meeting a real, unscheduled failure in production, not a chaos-engineering drill or a tabletop exercise, and coming out the other side untouched.

A Degraded Stream on the Train Home

The story started somewhere completely mundane. On the train home, the engineer was watching a football YouTube channel they follow regularly — normally well produced, with a live ticker and a tactics board overlaid on the broadcast. That evening, the production suddenly collapsed to almost nothing: no ticker, no tactics board, just one person talking against a plain black background, streaming at noticeably degraded quality. Before they could form a theory, a viewer in the channel’s live chat supplied one: Google Cloud Platform (GCP) was in the middle of what the speaker described as one of its largest outages ever, triggered by a failure in what they called its “iron service” — the kind of foundational, load-bearing internal dependency whose failure ripples out to everything built on top of it, apparently including a football commentary channel’s production pipeline.

Checking Their Own Platform

That is the moment most engineers will recognize instantly: the reflex to stop watching and start checking. The speaker logged into their own company’s platform right there, mid-commute, to assess impact. The result, as they told it: zero. Nothing was affected. No degraded checkout flow, no dropped writes, no failover scramble — the platform kept running exactly as if nothing anywhere had gone wrong, while one of the largest cloud providers on earth was visibly struggling elsewhere.

That gap — a public-facing video production knocked flat, versus a backend that did not notice — is the entire argument for architecting resilience deliberately rather than inheriting whatever your primary cloud provider’s uptime happens to be that day. The speaker’s team had already done the unglamorous work of spreading their platform’s dependencies, including its database layer, across more than one cloud provider, with CockroachDB’s distributed-consensus model keeping writes and reads consistent regardless of which underlying provider was currently degraded. When the failure actually arrived — not scheduled, not simulated — that investment paid out exactly once, in the only currency that matters: nothing broke.

Multi-Cloud Is Hard, But It Is Now a Real Option

A second clip, recorded a minute later and appearing to catch the closing minutes of the same session, picked up the broader architectural argument — again without a name badge visible on screen, but consistent with the same speaker. The framing was refreshingly honest rather than promotional: multi-cloud is genuinely hard, full stop, and nobody serious pretends otherwise. But the tooling and database primitives that make it achievable exist now in a way they simply did not a few years ago. Quorum-based distributed databases, portable container orchestration, and private cross-cloud networking have matured to the point where “one logical system spread across independent providers” is buildable rather than theoretical.

The tradeoffs were named plainly too: multi-cloud adds real cost and real operational complexity compared with staying on a single provider. It is not a silver bullet, and nobody should adopt it because it is fashionable. But it is now a legitimate option in the architectural toolbox for any platform that cannot tolerate its cloud provider’s bad day becoming its own. The closing piece of advice was the most operationally important line in the whole clip: run automated failover and recovery procedures regularly, on a schedule, whether or not anything is currently on fire — because the only way to know a failover path actually works is to have exercised it before you need it for real, not during the ten minutes an actual outage gives you to find out.

Why This Matters Beyond One Commute

Most resilience stories told at conferences are conditional: here is what would happen if a region went down, here is our disaster-recovery runbook, here is our target recovery time on a slide. This one was not conditional. It happened, on an ordinary weekday, discovered secondhand through a degraded YouTube stream rather than a pager alert — arguably the best possible outcome, since the platform’s own monitoring had nothing to page about. That is the real test any resilience architecture eventually faces: not whether it passes a scheduled drill, but whether it survives contact with a failure nobody scheduled.

Read together with Form3’s own multi-cloud rebuild — very likely the same team’s story, told from a different angle within the same session — this is less “two platforms, two paths” and more one platform’s resilience bet paying out twice in the same conference talk: once as a deliberate regulatory response, and once as an unscheduled real-world test. That is a useful data point for anyone still weighing whether multi-cloud complexity is worth carrying before an outage forces the question.

About the Author

I am Luca Berton, AI and Cloud Advisor. I work at the intersection of distributed systems, platform engineering, and enterprise AI deployments. Book a consultation.

Free 30-min AI & Cloud consultation

Book Now