P(doom): The Problem With AI Extinction Numbers

A P(doom) number tells you what somebody believes, not what anybody measured. That is the gap Tim Harford of the Financial Times points at when he asks what these forecasts are for. A prediction nobody can check is a prediction nobody can act on.

The number cannot be wrong

Forecasts earn credibility through track records: say it often, count the hits. Human extinction offers no track record and no second run, so a low estimate and a high estimate are equally safe from correction.

What the percentage does buy is tone. It makes a hunch sound like arithmetic and shifts the burden of proof without carrying any evidence of its own.

The Security Council heard claims, not percentages

According to derStandard, Amodei, Altman and Hugging Face co-founder Clement Delangue spoke on Wednesday before the 15-member body. Amodei said that, badly managed, AI could pose a risk to all of humanity — a statement about possibility, deliberately not a number. Whatever one makes of the warning, it is more honest than a freely chosen probability. The same report notes that President Trump rejects a global control regime.

What testable safety looks like

Researchers at Argonne National Laboratory in Lemont, Illinois built a real-time monitor for salt-cooled high-temperature reactors. Fiber-optic temperature sensors feed a machine learning model that flags local freezing of the molten salt before it blocks the heat exchanger. Across eight candidate methods, XGBoost won: a recall of 0.99 on severe blockages, trained on nearly 5,000 synthetic data sets seeded with artificial sensor noise.

Konstantinos Prantikos and senior electrical engineer Alexander Heifetz are named on the work, which uses Shapley values and partial order analysis to keep the model's decisions inspectable. The contrast with P(doom) is methodological rather than rhetorical. A recall of 0.99 is a claim that the next test case can break.

Why regulators need the second kind of number

Rules need thresholds somebody can audit: error rates, reporting duties, escalation paths. A doom probability supplies none of that and mainly sorts speakers into camps. Both the council warning and the Lemont sensors describe the same risk landscape, but only one of them can be checked.

What we could not verify

The Golem piece on P(doom) sat behind a consent dialog during reporting and its text could not be retrieved, so Harford's framing here rests on the publicly visible teaser rather than the full article. We did not read the Nature Scientific Reports paper itself; the figures and the date come from the t3n report. The derStandard article was cut off by a paywall after the introduction.