Enterprise AI Vendor Risk: Why Anthropic's 25% P(doom) Belongs in Your Procurement Process
Many procurement frameworks were built on an assumption so fundamental that it was never written down. The assumption was that vendors believe in the safety of what they sell. That assumption no longer holds in the category of technology that most firms are now most dependent on. No procurement questionnaire, vendor risk framework, or fiduciary review process in use at the Fortune 500 today has a field for what to do when the CEO of your most critical vendor publicly assigns material probability to their own product causing catastrophic harm to society, the world, potentially leading to an extinction-level event.
This isn’t to say that the worst case scenarios will come to pass with any certainty, but if you were buying a car with a 25% of killing your and your family through no fault of your own, you might ask some questions. I hope.
That is a procurement failure. It is increasingly a fiduciary one.
The reframe
Frontier AI is the first category of enterprise technology sourced from vendors whose leadership has on-the-record doubts about the product's safety. Dario Amodei, CEO of Anthropic, has publicly estimated a 25 percent probability that AI goes "really, really badly." Sam Altman, CEO of OpenAI, signed the 2023 Center for AI Safety statement equating AI extinction risk with pandemics and nuclear war. Sundar Pichai, CEO of Google, has said on record that the underlying risk is "pretty high." These are not activist critics. These are the counterparties.
No previous enterprise technology category has operated under this configuration. Aviation, pharmaceuticals, nuclear power, chemical manufacturing, financial derivatives: in every case, vendor leadership asserted the safety of their product, and the burden was on regulators and buyers to challenge it. In frontier AI, the vendors have done the disclosure for you. What they have not done is translate that disclosure into a procurement posture that the buyer can act on.
What P(doom) actually is
P(doom) is the probability, expressed as a number, that advanced AI causes human extinction or permanent civilizational loss. The term originated inside the rationalist and AI safety research communities in the early 2010s as a shorthand for probability estimates traded privately between researchers. It was technical jargon, not public discourse.
That changed in 2023. After the release of GPT-4, Geoffrey Hinton resigned from Google to speak publicly about AI risk. Yoshua Bengio published his own estimate of roughly 20 percent. In May 2023, the Center for AI Safety released a one-sentence statement equating AI extinction risk with pandemics and nuclear war. It was signed by Sam Altman, Demis Hassabis, Dario Amodei, Bill Gates, and over three hundred other researchers and executives. By late 2023, p(doom) had moved from private research correspondence into congressional testimony, earnings calls, and the regulatory posture of the UK, EU, and US executive branches.
A 2023 survey of AI researchers produced a mean estimate of 14.4 percent and a median of 5 percent over a one-hundred-year horizon. Individual public estimates since then have ranged from Altman's historically cited low single digits to Amodei's 25 percent to Emmett Shear's 5 to 50 percent range during his brief tenure as interim OpenAI CEO. The distribution is wide. It is also entirely non-zero among the people closest to the technology.
Published p(doom) estimates
Probability of existentially catastrophic AI outcomes, by named individual
Source: Wikipedia, P(doom). Figures reflect publicly stated estimates; ranges shown where individuals have given a band rather than a point value.
Why the number being non-zero is what matters
The debate inside the research community is about whether p(doom) is 2 percent, 15 percent, or 50 percent. The debate that matters for procurement, fiduciary review, and capital allocation is whether it is zero. It is not. No credible frontier lab CEO, no major AI researcher in a senior role, and no government AI advisory body currently holds the position that the probability is zero. The floor is a few percentage points. The ceiling is disputed.
For the purposes of corporate governance, the floor is the only number that matters. A few percentage points of catastrophic tail risk, acknowledged by the seller, absorbed silently by the buyer, is the exact fact pattern that every other mature risk discipline is built to price. Nuclear liability frameworks assume low-probability catastrophic outcomes. Pandemic insurance assumes the same. Cyber insurance assumes the same. Enterprise AI procurement is the only category where low-probability catastrophic outcomes are disclosed by the vendor and not priced by the buyer.
Why the term now shows up in executive contexts
Three things converged in 2024 and 2025 to pull p(doom) from technical jargon into the executive vocabulary.
First, the insurance market began pricing it. Specialty insurers started writing AI-specific exclusions into cyber and D&O policies that mapped closely to the failure modes described in p(doom) literature. The insurance market does not use the term. It prices the concept.
Second, the sovereign response began reflecting it. The US compute export control regime, the EU AI Act's systemic risk provisions, the UK AI Safety Institute, and equivalent bodies in Japan, Singapore, and the UAE are all policy responses to some version of the p(doom) thesis taken seriously by governments that will not use the word publicly.
Third, the vendors themselves kept naming numbers. Every quarter produces a new public estimate from a frontier lab CEO, a departing safety researcher, or a regulatory filing. The term is now stable in executive discourse because the disclosures are continuous.
Why this matters for the governance file
The board-level implication is not that p(doom) is correct. The board-level implication is that p(doom) is a term the firm's counterparties, regulators, insurers, and capital providers are all using, in some form, to price the risk of the technology the firm is adopting. A governance process that does not reference the concept is a governance process that is out of phase with the environment the firm is actually operating in.
That is the specific gap this piece addresses. The vendor has named the number. The regulator is building policy around the number. The insurer is writing exclusions based on the number. The buyer's governance process is silent on the number. That is hidden catastrophic AI risk exposure.
Observed versus inferred
What is documented: the three most capable frontier labs have leadership on record with non-trivial probability estimates of catastrophic outcomes from their own products. The numbers range from Altman's historically cited low single digits to Amodei's 25 percent.
What follows: the standard vendor risk framework assumes vendor self-assessment is a ceiling on risk. In this category, vendor self-assessment is a floor. The buyer is operating in a regime where the vendor has publicly pre-disclosed the tail risk, and the buyer has absorbed it silently through procurement by signing a contract that does not reference it.
What does not follow: that the probability estimates are accurate, that they are comparable across labs, or that they will be stable over time. The point is not the number. The point is that the number is non-zero, disclosed by the seller, and absent from the buyer's governance documentation.
Three operating shifts
The vendor risk questionnaire needs a catastrophic-disclosure field. Not a generic AI risk line. A specific field: has the vendor's senior leadership, on record in the past twenty-four months, assigned material probability to catastrophic outcomes from their product category? If yes, has that disclosure been reviewed by the buyer's risk committee? If no, why is the buyer's risk assessment more optimistic than the vendor's?
Fiduciary review must treat vendor safety statements as material. Board-level AI adoption decisions that do not reference the vendor's own published risk posture are operating on less information than is publicly available. When that gap produces a loss event, the plaintiff's bar will not need to prove the board should have known. The board will have chosen not to know what the vendor was saying out loud.
Indemnification language built for conventional software fails in this category. Standard enterprise software contracts cap vendor liability at fees paid, exclude consequential damages, and assume that vendor warranties represent the outer bound of risk. None of those structures contemplate a vendor whose CEO has publicly disclosed a 25 percent catastrophic probability. Either the indemnification terms reflect the vendor's own risk disclosure, or the buyer is underwriting the gap.
Role-by-role translation
CFO. The cost of capital implication is the first-order effect. Lenders and insurers are already adjusting for AI-exposed balance sheets. A firm whose AI vendor relationships are not documented against the vendor's own safety disclosures will be priced as if the disclosure were not public. The information asymmetry cuts against the buyer.
CIO. The vendor risk framework in use today was almost certainly built before the bimodal leadership disclosures reached their current volume. Most major AI vendor relationships signed in the past eighteen months were likely evaluated against a framework that did not ask the question the vendor has already answered publicly. The remediation is a framework update, not a vendor change.
General Counsel. The disclosure gap between what the vendor says in public and what the buyer acknowledges in its own governance documentation is the cleanest fact pattern a plaintiff's attorney could ask for. The fix is procedural. Document that the vendor's public safety posture has been reviewed. Document the buyer's decision to proceed. Document the risk committee sign-off. The documentation is the defense.
Chief Risk Officer. The enterprise risk register most likely contains a line item for AI risk. It almost certainly does not contain a line item for vendor-disclosed catastrophic AI risk. The two are not the same. The second is a counterparty disclosure. The first is a generic category. The register needs both.
Suggested quarterly cadence
Within thirty days: inventory the AI vendor relationships that carry material operational dependency. For each, locate the most recent on-record safety statement from the vendor's senior leadership. File it in the vendor record.
Within sixty days: update the vendor risk questionnaire to include a catastrophic-disclosure field. Run every existing material AI vendor through the updated questionnaire. Document the gaps.
Within ninety days: bring the consolidated disclosure review to the audit and risk committee. The committee's job is not to resolve whether the vendor's probability estimate is correct. The committee's job is to document that the firm has seen it, considered it, and decided to proceed on specific terms.
Quarterly thereafter: the safety posture of frontier AI vendors is not stationary. Leadership statements shift. New advancements in the field happen. The quarterly review is the mechanism that keeps the governance current with a non-stationary counterparty.
The stake
Every other enterprise technology category gives the buyer the option of being more cautious than the vendor. Frontier AI is the first category where the vendor is more cautious than the buyer, and the buyer has not noticed. The only question that matters is whether the governance and procurement process reflects what the vendor has already said out loud.
