Every few months, our industry becomes fixated on a fresh existential panic. Lately, the conversation around tech leadership, AI safety, and policy has been dominated by a single, dramatic question: What is the probability that artificial intelligence will trigger human extinction over the next decade? I'll leave it as an exercise to the reader but if you scan recent surveys of AI researchers, you'll find median estimates hovering around 5% that AI will kill us all, with mean figures reaching up to 16%! Some prominent voices actively argue for a 10-year "p(doom)" well above 10%.
Having spent a few years working on fault-tolerant distributed systems, transactions and high-availability software architectures, I think these figures miss the point. When you break down the required steps, from raw capability to unconstrained execution, resource acquisition and complete control failure, the realistic baseline for AI-driven extinction over the next decade is closer to ~3%, with a reasonable bounds range between 0.5% and 10%.
To put that in perspective against other existential hazards between 2026 and 2036:
Engineered pandemic / biological catastrophe: ~0.3–1%
Nuclear war causing extinction: ~0.05–0.3%
Abrupt climate change causing extinction: Well below 0.01%
Super volcano eruption: Well below 0.01%
Asteroid or comet impact: Far below 0.001%
Notice where that leaves us: even with conservative discounting, climate change and broader ecological instability pose a far more continuous, physical and systemic threat to civilisational stability than a runaway software script. Yet the headlines remain fixated on Sci-Fi apocalypse scenarios. Why? Setting aside the theories that it might be the AI vendors trying to justify a slowdown to their investors, I think it's because the current AI narrative treats software safety as a mysterious, unsolvable alignment puzzle rather than what it actually is, which is an engineering discipline we have already solved in other mission-critical domains.
The idea that an intelligent system will spontaneously escape, replicate across global infrastructure and wipe out humanity assumes a complete absence of basic systems engineering. In reliable distributed computing, aerospace, nuclear power and financial transaction processing, we do not rely on a system "promising to behave." Rather, we build structural constraints.
If the AI sector actually applied established mission-critical design principles, the extinction risk from autonomous systems would plummet from several percent to well below 1%:
Least Privilege and Restricted Envelopes: A software component, no matter how clever, should never hold production credentials, unrestricted outbound network access, cloud account creation privileges and access to sensitive biological or military data simultaneously.
Defence in Depth & Air-Gapping: Safety-critical systems rely on layered isolation. Disposable execution environments, immutable logging, outbound network filtering and hardware-enforced air gaps mean a failure in one layer does not compromise the host system.
Fail-Safe Defaults & Independent Interlocks: In high-consequence environments, human-in-the-loop approval isn't a UI feature, it's a structural requirement. Critical actions require physically separated authority, formal change control and independent shutdown mechanisms that cannot be overridden by the executing process.
I think that the exaggerated fear of AI extinction isn't coming from a gap in scientific possibility, it comes from a gap in engineering discipline. If you build an autonomous agent, give it shell access, grant it live web credentials, remove human oversight and skip sandboxing, of course you get dangerous failure modes. That isn’t an existential mystery, that’s just bad software architecture.
When we treat safety as a core variable, enforcing audited compute limits, strict runtime isolation, and rigorous fault isolation, AI risk drops by orders of magnitude. It stops being a runaway threat and becomes what it should have been all along: a complex, distributed systems challenge that we have all the tools to handle. Let's focus on putting real safety engineering into practice, instead of pretending the machine is out of our control.