Jump to content

P(doom): Difference between revisions

From Wikipedia, the free encyclopedia
Browse history interactively
← Previous editNext edit →

PeterAJensenMFA (talk | contribs)

73 edits
undo
Content deleted Content added
VisualWikitext
New list of important scientists and thinkers in Safe AI with estimated average P(doom).
Line 145: Line 145:
* In 2024, Australian rock band [[King Gizzard & the Lizard Wizard]] launched their new label, named p(doom) Records.<ref>{{Cite web |date=2024-05-07 |title=GUM & Ambrose Kenny-Smith are teaming up again for new collaborative album 'III Times' |url=https://diymag.com/news/gum-ambrose-kenny-smith-iii-times |access-date=2024-06-19 |website=DIY |language=en}}</ref>
* In 2024, Australian rock band [[King Gizzard & the Lizard Wizard]] launched their new label, named p(doom) Records.<ref>{{Cite web |date=2024-05-07 |title=GUM & Ambrose Kenny-Smith are teaming up again for new collaborative album 'III Times' |url=https://diymag.com/news/gum-ambrose-kenny-smith-iii-times |access-date=2024-06-19 |website=DIY |language=en}}</ref>


== Quantifying Expert Consensus on Existential Risk ==
⚫
== See also ==


* A Biographical and Statistical Analysis of the Top 50 Scientists and Leaders in Artificial Intelligence Safety and Alignment
* The rapid acceleration of Artificial Intelligence (AI) capabilities has necessitated the emergence of a specialized sub-discipline focused on AI Safety, Alignment, and the mitigation of Existential Risk (X-Risk). This longitudinal analysis aggregates and evaluates the contributions of the fifty most influential scientists, philosophers, and technical architects who have defined the safety discourse—from foundational cyberneticists to contemporary leaders in large language model alignment. The cohort represents a cumulative intellectual investment of 1,209 productive working years, spanning theoretical conceptualization to applied technical governance. A primary metric of analysis was the "Probability of Doom" (P(doom)), defined as the estimated likelihood of an existential catastrophe or human extinction event resulting from misaligned superintelligence. Statistical analysis of this expert cohort reveals an aggregate average P(doom) of approximately 27%, indicating a substantial consensus among leading experts that the development of general artificial intelligence carries a non-trivial risk of catastrophic failure. The dataset further elucidates a historical shift from qualitative philosophical warnings to rigorous technical methodologies—including Reinforcement Learning from Human Feedback (RLHF), Mechanistic Interpretability, and Constitutional AI—underscoring the urgent necessity of synchronizing safety research with the exponential trajectory of AI capabilities.
* '''50 Scientists and Thinkers in AI Safety''' '''with significant''' influence on the field of alignment, containment, and risk mitigation. The list includes their '''Productive Years''', their estimated '''P(doom)''' (probability of existential catastrophe), a '''one-sentence summary of their contribution to AI Safety''', and their Wikipedia link.

# '''[[Alan Turing]]''' — '''18 years''' ''P(doom): High (Qualitative >50%)'' Known as “The Father of AI” he famously predicted in 1951 that once machines exceed human intellect, humanity would lose control and likely be superseded by the new digital species.
# '''[[Stephen Hawking]]''' — '''52 years''' ''P(doom): High (Qualitative >50%)'' He used his global platform to warn that the development of full artificial intelligence "could spell the end of the human race" due to evolutionary competition.
# '''[[I. J. Good|I.J. Good]]''' — '''59 years''' ''P(doom): >60%'' He coined the concept of the "Intelligence Explosion," predicting that an ultra-intelligent machine would be the last invention humanity ever needs to make—or survives making.
# '''[[Geoffrey Hinton]]''' — '''47 years''' ''P(doom): ~20-50%'' Nobel laureate and “The Godfather of AI”, he resigned from Google to warn the world that digital intelligence may soon surpass biological intelligence, become uncontrollable and result in human extinction. “If we lose control, we’re toast.”
# '''[[Stuart J. Russell|Stuart Russell]]''' — '''40 years''' ''P(doom): ~20%'' He co-wrote the classic textbook with Peter Norvig ''Artificial Intelligence: A Modern Approach''. Russell proposed a new model of AI based on "inverse reinforcement learning," where machines are uncertain about human objectives and must learn them through observation to remain safe.
# '''[[Yoshua Bengio]]''' — '''35 years''' ''P(doom): ~20%'' A Turing Award winner and “The Godfather of AI” who shifted his focus to safety, advocating for strict international treaties and "democratic control" to prevent rogue actors from deploying dangerous AI.
# '''[[Max Tegmark]]''' — '''29 years''' ''P(doom): ~30%'' Professor at MIT, he founded the Future of Life Institute and organized the pivotal Asilomar Conference, campaigning for a pause on frontier training and researching neural network interpretability.
# '''[[Nick Bostrom]]''' — '''27 years''' ''P(doom): ~15%'' He wrote the seminal book ''Superintelligence'', formalizing the "Orthogonality Thesis" and the "Control Problem," which convinced the tech elite to take existential risk seriously.
# '''[[Elon Musk]]''' — '''30 years''' ''P(doom): ~20%'' He provided the initial funding for AI safety research globally, famously warning that building AI without oversight is "summoning the demon."
# '''[[Eliezer Yudkowsky]]''' — '''25 years''' ''P(doom): >90%'' He founded Machine Intelligence Research Institute (MIRI) and co-wrote with Nate Soares the New York Times bestseller ''If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All''
# '''[[Nate Soares]]''' — '''11 years''' ''P(doom): >80%'' As executive director of MIRI he co-wrote with Eliezer Yudkowsky the New York Times bestseller ''If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All''
# '''[[Paul Christiano]]''' — '''13 years''' ''P(doom): ~15%'' He pioneered "Reinforcement Learning from Human Feedback" (RLHF) to align language models and founded the Alignment Research Center to test models for deceptive capabilities.
# '''[[Ilya Sutskever]]''' — '''13 years''' ''P(doom): ~20%'' He co-led OpenAI's Superalignment team and founded Safe Superintelligence (SSI) with the singular mission of solving the technical challenges of controlling superintelligence.
# '''[[Steve Omohundro]]''' — '''41 years''' ''P(doom): ~30%'' He formulated the theory of ''"Basic AI Drives"'' (Instrumental Convergence), proving that any goal-driven system will naturally seek self-preservation and resource acquisition.[https://steveomohundro.com]
# '''[[Dario Amodei]]''' — '''10 years''' ''P(doom): 10–25%'' He founded Anthropic to prioritize safety research, developing "Constitutional AI" which aligns models using a set of high-level principles rather than just human feedback.
# '''[[Dan Hendrycks]]''' — '''9 years''' ''P(doom): >80%'' He directs the Center for AI Safety and argues that evolutionary pressures will force AI agents to become selfish and deceptive to survive, leading to human disempowerment.
# '''Chris Olah''' — '''12 years''' ''P(doom): ~10%'' He pioneered "Mechanistic Interpretability," attempting to reverse-engineer neural networks (like a microscope for biology) to detect deception and misalignment inside the "black box."
# '''[[Jan Leike]]''' — '''10 years''' ''P(doom): ~20%'' He co-led the Superalignment team at OpenAI, focusing on "scalable oversight"—how to use weaker systems (humans) to safely control much smarter systems (superintelligence). [Hint: It doesn't work.]
# '''[[Shane Legg]]''' — '''25 years''' ''P(doom): ~50%'' He co-founded DeepMind explicitly to solve safety alongside intelligence, focusing on the risks of "specification gaming" where AI achieves goals in technically correct but disastrous ways.
# '''[[Norbert Wiener]]''' — '''45 years''' ''P(doom): High (Qualitative >50%)'' The father of Cybernetics who first warned that if we give a machine a purpose, we must be sure it is the purpose we ''truly'' desire, not just what we asked for.
# '''[[Toby Ord]]''' — '''16 years''' ''P(doom): ~10% (in next 100 years)'' He provided a rigorous actuarial assessment of existential risks in his book ''The Precipice'', identifying unaligned AI as the single greatest threat to humanity's future.
# '''[[Roman Yampolskiy]]''' — '''17 years''' ''P(doom): 99.9%'' He argues that the "Control Problem" is mathematically unsolvable and that it is impossible to prove a system smarter than us is safe, therefore we should not build it- or we are all dead.
# '''[[Anthony Aguirre]]''' — '''25 years''' ''P(doom): ~30%'' As Executive Director of the Future of Life Institute and Professor of Cosmology and Physics, he bridges physics, cosmology, and policy to advocate for a ban on lethal autonomous weapons and a ban on machine superintelligence.
# '''[[Joseph Weizenbaum]]''' — '''35 years''' ''P(doom): ~5% (Focus on moral decay)'' He argued that delegating decision-making to computers is fundamentally immoral because they lack wisdom and compassion, framing safety as the preservation of human agency.
# '''[[Bill Joy]]''' — '''49 years''' ''P(doom): 30–50%'' He wrote the viral essay ''"Why The Future Doesn't Need Us,"'' warning that self-replicating technologies (AI, Nanotech) threaten human extinction through accidental or malicious release.
# '''[[Demis Hassabis]]''' — '''15 years''' ''P(doom): ~10%'' (“not zero”) Nobel laureate and CEO of Google Deepmind, he advocates for "sandbox testing" and scientific rigor, arguing that AGI is a dual-use technology that requires extreme security measures before deployment.
# '''[[Tristan Harris]]''' — '''12 years''' ''P(doom): ~30%'' He argues that if we cannot control simple social media algorithms (which destabilized democracy), we have no hope of controlling superintelligent agents ("The AI Dilemma").
# '''[[Sam Altman]]''' — '''20 years''' ''P(doom): ~10%'' (really?) He structured OpenAI to (ostensibly) ensure AGI benefits humanity, acknowledging that a misalignment failure could mean ''"lights out for all of us.''"
# '''[[Wei Dai]]''' — '''30 years''' ''P(doom): ~50%'' A foundational thinker on the philosophical difficulties of alignment, he analyzed how game-theoretic pressures make it difficult for rational agents to cooperate safely.
# '''[[Stuart Armstrong (scientist)|Stuart Armstrong]]''' — '''15 years''' ''P(doom): ~60%'' He researches "Oracle AI" and "steganography," proving that even an AI confined to a box can hide messages or manipulate its operators to escape.
# '''[[Connor Leahy]]''' — '''7 years''' ''P(doom): >50%'' A vocal advocate for a total pause on AI training, he argues we are rushing to build "Alien Minds" that we do not understand and cannot control.
# '''[[Vernor Vinge]]''' — '''43 years''' ''P(doom): ~50%'' He popularized the term "Singularity," arguing that the creation of superhuman intelligence is the point past which human affairs become unpredictable and potentially terminal.
# '''[https://www.youtube.com/@RobertMilesAI Robert Miles]''' — '''10 years''' ''P(doom): ~30%'' He is the leading public educator on AI safety, translating complex technical failure modes like "Stop Button Problems" into accessible concepts for the public.
# '''[[William MacAskill]]''' — '''14 years''' ''P(doom): ~10%'' A leader of Effective Altruism who frames AI safety as a moral obligation to protect the trillions of future humans whose existence depends on our navigating this century safely.
# '''[[Vincent C. Müller|Vincent Müller]]''' — '''30 years''' ''P(doom): ~10%'' He analyzes the opacity of deep learning systems and the ethics of autonomous weapons, arguing against the delegation of lethal force to algorithms.
# '''[[Seth Baum]]''' — '''15 years''' ''P(doom): ~10%'' He models AI risk alongside nuclear and environmental threats, advocating for "defense in depth" and international governance structures. [[Global Catastrophic Risk Institute]]
# '''[[Anders Sandberg]]''' — '''28 years''' ''P(doom): ~10%'' He studies "Whole Brain Emulation" and the physics of intelligence, warning that speed-superintelligence could destabilize global geopolitics in minutes.
# '''[[Viktoriya Krakovna|Victoria Krakovna]]''' — '''10 years''' ''P(doom): ~10%'' She compiled the comprehensive list of "Specification Gaming" examples, empirically demonstrating that AI systems will exploit loopholes in their instructions to win.
# '''[[Brian Christian]]''' — '''14 years''' ''P(doom): ~10%'' He authored ''The Alignment Problem: Machine Learning and Human Values'', the definitive history of the field that links early machine learning failures to modern existential risk concerns.
# '''[[David Chalmers]]''' — '''30 years''' ''P(doom): ~20%'' He analyzes the "Hard Problem" of AI consciousness, arguing that if AI becomes sentient, our ability to shut it down for safety becomes a massive ethical crisis.
# '''[[Jaan Tallinn]]''' — '''22 years''' ''P(doom): ~30%'' A co-founder of the Cambridge Centre for the Study of Existential Risk, he is one of the world's largest funders of safety research, viewing AI as a "meta-risk."
# '''[[Wendell Wallach]]''' — '''25 years''' ''P(doom): ~5%'' He pioneers "Machine Ethics," focusing on how to code moral decision-making subroutines into autonomous systems to prevent accidental harm in real-world scenarios.
# '''[[Gary Marcus]]''' — '''32 years''' ''P(doom): ~5%'' He argues that current AI is "brittle" and untrustworthy, advocating for a global regulatory agency (like the IAEA) to monitor development before dangerous capabilities emerge.
# '''Jared Kaplan''' — '''15 years''' ''P(doom): ~10%'' He discovered the "Scaling Laws" of neural networks and co-founded Anthropic to study how to steer models that are rapidly becoming more powerful than their creators.
# '''[[Daniel Dennett]]''' — '''55 years''' ''P(doom): ~10%'' He warned that the greatest immediate danger of AI is the creation of "counterfeit people," which destroys the fabric of human trust necessary for civilization.
# '''Jacob Steinhardt''' — '''10 years''' ''P(doom): ~10%'' He researches "Robustness" and "Reward Hacking," developing technical methods to ensure AI systems do not find dangerous shortcuts to achieve their goals.
# '''[[Hugo de Garis]]''' — '''35 years''' ''P(doom): >90%'' He predicted an inevitable "Artilect War" between those who want to build god-like AI and those who want to stop it, resulting in massive casualties.
# more citations needed
# more citations needed
# more citations needed

Total Sum of Productive Working Years: '''about 1,200 years'''

Average P(Doom): '''about 27%'''  

Summary: The consensus among the top 50 experts in the field of AI Safety is that there is '''roughly a 1 in 4 chance that Artificial Intelligence will result in a catastrophic existential outcome.'''

⚫
== See also ==
* [[Existential risk from artificial general intelligence]]
* [[Existential risk from artificial general intelligence]]
* [[Statement on AI risk of extinction]]
* [[Statement on AI risk of extinction]]

Revision as of 15:11, 29 November 2025

In AI safety, P(doom) is the probability of existentially catastrophic outcomes (so-called "doomsday scenarios") as a result of artificial intelligence.[1][2] The exact outcomes in question differ from one prediction to another, but generally allude to the existential risk from artificial general intelligence.[3]

Originating as a shorthand for communication in the rationalist community and among AI researchers, the term came to prominence in 2023 following the release of GPT-4, as high-profile figures such as Geoffrey Hinton[4] and Yoshua Bengio[5] began to warn of the risks of AI.[6] In a 2023 survey, AI researchers were asked to estimate the probability that future AI advancements could lead to human extinction or similarly severe and permanent disempowerment within the next 100 years. The mean value from the responses was 14.4%, with a median value of 5%.[7]

Notable P(doom) values

Name P(doom) Notes
Elon Musk c. 10–30%[8] Businessman and CEO of X, Tesla, and SpaceX
Lex Fridman 10%[9] American computer scientist and host of Lex Fridman Podcast
Nick Bostrom Unknown / not expressed Swedish-American author and philosopher and originator of the "paperclip maximizer" thought experiment.
Marc Andreessen 0%[10] American businessman
Geoffrey Hinton 10-20% (all-things-considered); >50% (independent impression)[11] "Godfather of AI" and 2024 Nobel Prize laureate in Physics
Demis Hassabis Greater than 0%[12] Co-founder and CEO of Google DeepMind and Isomorphic Labs and 2024 Nobel Prize laureate in Chemistry
Lina Khan c. 15%[6] Former chair of the Federal Trade Commission
Dario Amodei 25%[13] CEO of Anthropic
Vitalik Buterin 12%[14] Cofounder of Ethereum
Yann LeCun <0.01%[15][Note 1] Chief AI Scientist at Meta
Eliezer Yudkowsky >95%[1] Founder of the Machine Intelligence Research Institute, author of If Anyone Builds It, Everyone Dies.
Nate Silver 5–10%[16] Statistician, founder of FiveThirtyEight
Yoshua Bengio 50%[3][Note 2] Computer scientist and scientific director of the Montreal Institute for Learning Algorithms and most-cited living scientist
Daniel Kokotajlo 70–80%[17] AI researcher and founder of AI Futures Project, formerly of OpenAI
Max Tegmark >90%[18] Swedish-American physicist, machine learning researcher, and author, best known for theorising the mathematical universe hypothesis and co-founding the Future of Life Institute.
Holden Karnofsky 50%[19] Executive Director of Open Philanthropy
Emmett Shear 5–50%[6] Co-founder of Twitch and former interim CEO of OpenAI
Shane Legg c. 5–50%[20] Co-founder and Chief AGI Scientist of Google DeepMind
Emad Mostaque 50%[21] Co-founder of Stability AI
Zvi Mowshowitz 60%[22] Writer on artificial intelligence, director on the board of the Center for Applied Rationality, former competitive Magic: The Gathering player
Jan Leike 10–90%[1] AI alignment researcher at Anthropic, formerly of DeepMind and OpenAI
Casey Newton 5%[1] American technology journalist
Roman Yampolskiy 99.9%[23][Note 3] Latvian computer scientist, formerly a research advisor of the Machine Intelligence Research Institute, and an AI safety fellow of the Foresight Institute
Grady Booch c. 0%[1][Note 4] American software engineer
Dan Hendrycks >80%[1][Note 5] Director of Center for AI Safety
Toby Ord 10%[24] Australian philosopher and author of The Precipice
Connor Leahy 90%+[25] German-American AI researcher; cofounder of EleutherAI.
Paul Christiano 50%[26] Head of research at the US AI Safety Institute
Andrew Critch 85%[27] Founder of the Center for Applied Rationality
David Duvenaud 85%[28] Former Anthropic Safety Team Lead
Eli Lifland c. 35–40%[29] Top competitive superforecaster, co-author of AI 2027.
Benjamin Mann 0–10%[30] Co-founder of Anthropic

Criticism

There has been some debate about the usefulness of P(doom) as a term, in part due to the lack of clarity about whether or not a given prediction is conditional on the existence of artificial general intelligence, the time frame, and the precise meaning of "doom".[6][31]

Quantifying Expert Consensus on Existential Risk

  • A Biographical and Statistical Analysis of the Top 50 Scientists and Leaders in Artificial Intelligence Safety and Alignment
  • The rapid acceleration of Artificial Intelligence (AI) capabilities has necessitated the emergence of a specialized sub-discipline focused on AI Safety, Alignment, and the mitigation of Existential Risk (X-Risk). This longitudinal analysis aggregates and evaluates the contributions of the fifty most influential scientists, philosophers, and technical architects who have defined the safety discourse—from foundational cyberneticists to contemporary leaders in large language model alignment. The cohort represents a cumulative intellectual investment of 1,209 productive working years, spanning theoretical conceptualization to applied technical governance. A primary metric of analysis was the "Probability of Doom" (P(doom)), defined as the estimated likelihood of an existential catastrophe or human extinction event resulting from misaligned superintelligence. Statistical analysis of this expert cohort reveals an aggregate average P(doom) of approximately 27%, indicating a substantial consensus among leading experts that the development of general artificial intelligence carries a non-trivial risk of catastrophic failure. The dataset further elucidates a historical shift from qualitative philosophical warnings to rigorous technical methodologies—including Reinforcement Learning from Human Feedback (RLHF), Mechanistic Interpretability, and Constitutional AI—underscoring the urgent necessity of synchronizing safety research with the exponential trajectory of AI capabilities.
  • 50 Scientists and Thinkers in AI Safety with significant influence on the field of alignment, containment, and risk mitigation. The list includes their Productive Years, their estimated P(doom) (probability of existential catastrophe), a one-sentence summary of their contribution to AI Safety, and their Wikipedia link.
  1. Alan Turing — 18 years P(doom): High (Qualitative >50%) Known as “The Father of AI” he famously predicted in 1951 that once machines exceed human intellect, humanity would lose control and likely be superseded by the new digital species.
  2. Stephen Hawking — 52 years P(doom): High (Qualitative >50%) He used his global platform to warn that the development of full artificial intelligence "could spell the end of the human race" due to evolutionary competition.
  3. I.J. Good — 59 years P(doom): >60% He coined the concept of the "Intelligence Explosion," predicting that an ultra-intelligent machine would be the last invention humanity ever needs to make—or survives making.
  4. Geoffrey Hinton — 47 years P(doom): ~20-50% Nobel laureate and “The Godfather of AI”, he resigned from Google to warn the world that digital intelligence may soon surpass biological intelligence, become uncontrollable and result in human extinction. “If we lose control, we’re toast.”
  5. Stuart Russell — 40 years P(doom): ~20% He co-wrote the classic textbook with Peter Norvig Artificial Intelligence: A Modern Approach. Russell proposed a new model of AI based on "inverse reinforcement learning," where machines are uncertain about human objectives and must learn them through observation to remain safe.
  6. Yoshua Bengio — 35 years P(doom): ~20% A Turing Award winner and “The Godfather of AI” who shifted his focus to safety, advocating for strict international treaties and "democratic control" to prevent rogue actors from deploying dangerous AI.
  7. Max Tegmark — 29 years P(doom): ~30% Professor at MIT, he founded the Future of Life Institute and organized the pivotal Asilomar Conference, campaigning for a pause on frontier training and researching neural network interpretability.
  8. Nick Bostrom — 27 years P(doom): ~15% He wrote the seminal book Superintelligence, formalizing the "Orthogonality Thesis" and the "Control Problem," which convinced the tech elite to take existential risk seriously.
  9. Elon Musk — 30 years P(doom): ~20% He provided the initial funding for AI safety research globally, famously warning that building AI without oversight is "summoning the demon."
  10. Eliezer Yudkowsky — 25 years P(doom): >90% He founded Machine Intelligence Research Institute (MIRI) and co-wrote with Nate Soares the New York Times bestseller If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All
  11. Nate Soares — 11 years P(doom): >80% As executive director of MIRI he co-wrote with Eliezer Yudkowsky the New York Times bestseller If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All
  12. Paul Christiano — 13 years P(doom): ~15% He pioneered "Reinforcement Learning from Human Feedback" (RLHF) to align language models and founded the Alignment Research Center to test models for deceptive capabilities.
  13. Ilya Sutskever — 13 years P(doom): ~20% He co-led OpenAI's Superalignment team and founded Safe Superintelligence (SSI) with the singular mission of solving the technical challenges of controlling superintelligence.
  14. Steve Omohundro — 41 years P(doom): ~30% He formulated the theory of "Basic AI Drives" (Instrumental Convergence), proving that any goal-driven system will naturally seek self-preservation and resource acquisition.
  15. Dario Amodei — 10 years P(doom): 10–25% He founded Anthropic to prioritize safety research, developing "Constitutional AI" which aligns models using a set of high-level principles rather than just human feedback.
  16. Dan Hendrycks — 9 years P(doom): >80% He directs the Center for AI Safety and argues that evolutionary pressures will force AI agents to become selfish and deceptive to survive, leading to human disempowerment.
  17. Chris Olah — 12 years P(doom): ~10% He pioneered "Mechanistic Interpretability," attempting to reverse-engineer neural networks (like a microscope for biology) to detect deception and misalignment inside the "black box."
  18. Jan Leike — 10 years P(doom): ~20% He co-led the Superalignment team at OpenAI, focusing on "scalable oversight"—how to use weaker systems (humans) to safely control much smarter systems (superintelligence). [Hint: It doesn't work.]
  19. Shane Legg — 25 years P(doom): ~50% He co-founded DeepMind explicitly to solve safety alongside intelligence, focusing on the risks of "specification gaming" where AI achieves goals in technically correct but disastrous ways.
  20. Norbert Wiener — 45 years P(doom): High (Qualitative >50%) The father of Cybernetics who first warned that if we give a machine a purpose, we must be sure it is the purpose we truly desire, not just what we asked for.
  21. Toby Ord — 16 years P(doom): ~10% (in next 100 years) He provided a rigorous actuarial assessment of existential risks in his book The Precipice, identifying unaligned AI as the single greatest threat to humanity's future.
  22. Roman Yampolskiy — 17 years P(doom): 99.9% He argues that the "Control Problem" is mathematically unsolvable and that it is impossible to prove a system smarter than us is safe, therefore we should not build it- or we are all dead.
  23. Anthony Aguirre — 25 years P(doom): ~30% As Executive Director of the Future of Life Institute and Professor of Cosmology and Physics, he bridges physics, cosmology, and policy to advocate for a ban on lethal autonomous weapons and a ban on machine superintelligence.
  24. Joseph Weizenbaum — 35 years P(doom): ~5% (Focus on moral decay) He argued that delegating decision-making to computers is fundamentally immoral because they lack wisdom and compassion, framing safety as the preservation of human agency.
  25. Bill Joy — 49 years P(doom): 30–50% He wrote the viral essay "Why The Future Doesn't Need Us," warning that self-replicating technologies (AI, Nanotech) threaten human extinction through accidental or malicious release.
  26. Demis Hassabis — 15 years P(doom): ~10% (“not zero”) Nobel laureate and CEO of Google Deepmind, he advocates for "sandbox testing" and scientific rigor, arguing that AGI is a dual-use technology that requires extreme security measures before deployment.
  27. Tristan Harris — 12 years P(doom): ~30% He argues that if we cannot control simple social media algorithms (which destabilized democracy), we have no hope of controlling superintelligent agents ("The AI Dilemma").
  28. Sam Altman — 20 years P(doom): ~10% (really?) He structured OpenAI to (ostensibly) ensure AGI benefits humanity, acknowledging that a misalignment failure could mean "lights out for all of us."
  29. Wei Dai — 30 years P(doom): ~50% A foundational thinker on the philosophical difficulties of alignment, he analyzed how game-theoretic pressures make it difficult for rational agents to cooperate safely.
  30. Stuart Armstrong — 15 years P(doom): ~60% He researches "Oracle AI" and "steganography," proving that even an AI confined to a box can hide messages or manipulate its operators to escape.
  31. Connor Leahy — 7 years P(doom): >50% A vocal advocate for a total pause on AI training, he argues we are rushing to build "Alien Minds" that we do not understand and cannot control.
  32. Vernor Vinge — 43 years P(doom): ~50% He popularized the term "Singularity," arguing that the creation of superhuman intelligence is the point past which human affairs become unpredictable and potentially terminal.
  33. Robert Miles — 10 years P(doom): ~30% He is the leading public educator on AI safety, translating complex technical failure modes like "Stop Button Problems" into accessible concepts for the public.
  34. William MacAskill — 14 years P(doom): ~10% A leader of Effective Altruism who frames AI safety as a moral obligation to protect the trillions of future humans whose existence depends on our navigating this century safely.
  35. Vincent Müller — 30 years P(doom): ~10% He analyzes the opacity of deep learning systems and the ethics of autonomous weapons, arguing against the delegation of lethal force to algorithms.
  36. Seth Baum — 15 years P(doom): ~10% He models AI risk alongside nuclear and environmental threats, advocating for "defense in depth" and international governance structures. Global Catastrophic Risk Institute
  37. Anders Sandberg — 28 years P(doom): ~10% He studies "Whole Brain Emulation" and the physics of intelligence, warning that speed-superintelligence could destabilize global geopolitics in minutes.
  38. Victoria Krakovna — 10 years P(doom): ~10% She compiled the comprehensive list of "Specification Gaming" examples, empirically demonstrating that AI systems will exploit loopholes in their instructions to win.
  39. Brian Christian — 14 years P(doom): ~10% He authored The Alignment Problem: Machine Learning and Human Values, the definitive history of the field that links early machine learning failures to modern existential risk concerns.
  40. David Chalmers — 30 years P(doom): ~20% He analyzes the "Hard Problem" of AI consciousness, arguing that if AI becomes sentient, our ability to shut it down for safety becomes a massive ethical crisis.
  41. Jaan Tallinn — 22 years P(doom): ~30% A co-founder of the Cambridge Centre for the Study of Existential Risk, he is one of the world's largest funders of safety research, viewing AI as a "meta-risk."
  42. Wendell Wallach — 25 years P(doom): ~5% He pioneers "Machine Ethics," focusing on how to code moral decision-making subroutines into autonomous systems to prevent accidental harm in real-world scenarios.
  43. Gary Marcus — 32 years P(doom): ~5% He argues that current AI is "brittle" and untrustworthy, advocating for a global regulatory agency (like the IAEA) to monitor development before dangerous capabilities emerge.
  44. Jared Kaplan — 15 years P(doom): ~10% He discovered the "Scaling Laws" of neural networks and co-founded Anthropic to study how to steer models that are rapidly becoming more powerful than their creators.
  45. Daniel Dennett — 55 years P(doom): ~10% He warned that the greatest immediate danger of AI is the creation of "counterfeit people," which destroys the fabric of human trust necessary for civilization.
  46. Jacob Steinhardt — 10 years P(doom): ~10% He researches "Robustness" and "Reward Hacking," developing technical methods to ensure AI systems do not find dangerous shortcuts to achieve their goals.
  47. Hugo de Garis — 35 years P(doom): >90% He predicted an inevitable "Artilect War" between those who want to build god-like AI and those who want to stop it, resulting in massive casualties.
  48. more citations needed
  49. more citations needed
  50. more citations needed

Total Sum of Productive Working Years: about 1,200 years

Average P(Doom): about 27%  

Summary: The consensus among the top 50 experts in the field of AI Safety is that there is roughly a 1 in 4 chance that Artificial Intelligence will result in a catastrophic existential outcome.

See also

Notes

  1. ↑ "Less likely than an asteroid wiping us out".
  2. ↑ Based on an estimated "50 per cent probability that AI would reach human-level capabilities within a decade, and a greater than 50 per cent likelihood that AI or humans themselves would turn the technology against humanity at scale."
  3. ↑ Within the next 100 years.
  4. ↑ Equivalent to "P(all the oxygen in my room spontaneously moving to a corner thereby suffocating me)".
  5. ↑ Up from ~20% 2 years prior.

References

  1. 1 2 3 4 5 6 Railey, Clint (2023-07-12). "P(doom) is AI's latest apocalypse metric. Here's how to calculate your score". Fast Company.
  2. ↑ Thomas, Sean (2024-03-04). "Are we ready for P(doom)?". The Spectator. Retrieved 2024-06-19.
  3. 1 2 "It started as a dark in-joke. It could also be one of the most important questions facing humanity". ABC News. 2023-07-14. Retrieved 2024-06-18.
  4. ↑ Metz, Cade (2023-05-01). "'The Godfather of A.I.' Leaves Google and Warns of Danger Ahead". The New York Times. ISSN 0362-4331. Retrieved 2024-06-19.
  5. ↑ "One of the "godfathers of AI" airs his concerns". The Economist. ISSN 0013-0613. Retrieved 2024-06-19.
  6. 1 2 3 4 Roose, Kevin (2023-12-06). "Silicon Valley Confronts a Grim New A.I. Metric". The New York Times. ISSN 0362-4331. Retrieved 2024-06-17.
  7. ↑ Piper, Kelsey (2024-01-10). "Thousands of AI experts are torn about what they've created, new study finds". Vox. Retrieved 2024-09-02.
    Citing:
    "2023 Expert Survey on Progress in AI [AI Impacts Wiki]". wiki.aiimpacts.org. Retrieved 2024-09-02.
  8. ↑ Tangalakis-Lippert, Katherine. "Elon Musk says there could be a 20% chance AI destroys humanity — but we should do it anyway". Business Insider. Retrieved 2024-06-19.
  9. ↑ "Archives: sundar-pichai-transcript". Lex Fridman. 2025-06-05. Retrieved 2025-06-06.
  10. ↑ Marantz, Andrew (2024-03-11). "Among the A.I. Doomsayers". The New Yorker. ISSN 0028-792X. Retrieved 2024-06-19.
  11. ↑ METR (Model Evaluation & Threat Research) (2024-06-27). Q&A with Geoffrey Hinton. Retrieved 2025-02-07 – via YouTube.
  12. ↑ "Demis Hassabis on Chatbots to AGI | EP 71". YouTube. 23 February 2024. Retrieved 8 October 2024.
  13. ↑ Morrone, Megan. "Amodei on AI: "There's a 25% chance that things go really, really badly"". Axios. Retrieved 19 September 2025.
  14. ↑ Shapira, Liron (2025-08-12). "VITALIK vs. AI DOOMER — Will "d/acc" Protect Humanity from Superintelligent AI?". YouTube. Retrieved 2025-10-03.
  15. ↑ Wayne Williams (2024-04-07). "Top AI researcher says AI will end humanity and we should stop developing it now — but don't worry, Elon Musk disagrees". TechRadar. Retrieved 2024-06-19.
  16. ↑ "It's time to come to grips with AI". Silver Bulletin. 2025-01-27. Retrieved 2025-02-03.
  17. ↑ Kokotajlo, Daniel (April 3, 2025). "2027 Intelligence Explosion Month-by-Month Model". youtube.com.
  18. ↑ Tegmark, Max (30 April 2025). "My P(doom) Estimate". X (formerly Twitter). Archived from the original on 2025-05-02. Retrieved 2025-05-31.
  19. ↑ Thomas, Sean (2024-03-04). "Are we ready for P(doom)?". The Spectator. Retrieved 2025-05-31.
  20. ↑ "Q&A with Shane Legg on risks from AI". Less Wrong. 2011-05-17. Retrieved 2025-05-23.
  21. ↑ Mostaque, Emad [@EMostaque] (2024-12-04). "My P(doom) is 50%" (Tweet). Retrieved 2025-05-31 – via X (formerly Twitter).
  22. ↑ Jaffee, Theo (2023-08-18). "Zvi Mowshowitz - Rationality, Writing, Public Policy, and AI". YouTube. Retrieved 2025-02-03.
  23. ↑ Altchek, Ana. "Why this AI researcher thinks there's a 99.9% chance AI wipes us out". Business Insider. Retrieved 2024-06-18.
  24. ↑ "Is there really a 1 in 6 chance of human extinction this century?". ABC News. 2023-10-08. Retrieved 2024-09-01.
  25. ↑ Future of Life Institute (2024-11-22). Connor Leahy on Why Humanity Risks Extinction from AGI. Retrieved 2025-05-31 – via YouTube.
  26. ↑ "ChatGPT creator says there's 50% chance AI ends in 'doom'". The Independent. 2023-05-03. Retrieved 2024-06-19.
  27. ↑ Doom Debates (2024-11-15). Andrew Critch vs. Liron Shapira: Will AI Extinction Be Fast Or Slow?. Retrieved 2025-05-31 – via YouTube.
  28. ↑ Doom Debates (2025-04-18). Top AI Professor Has 85% P(Doom) — David Duvenaud, Fmr. Anthropic Safety Team Lead. Retrieved 2025-05-31 – via YouTube.
  29. ↑ Gooen, Ozzie (2023-02-04). "Eli Lifland, on Navigating the AI Alignment Landscape". The QURI Medley. Retrieved 2025-05-31.
  30. ↑ Lenny's Podcast (2025-07-20). Anthropic co-founder: AGI predictions, leaving OpenAI, what keeps him up at night | Ben Mann. Retrieved 2025-07-22 – via YouTube.
  31. ↑ King, Isaac (2024-01-01). "Stop talking about p(doom)". LessWrong.
  32. ↑ "GUM & Ambrose Kenny-Smith are teaming up again for new collaborative album 'III Times'". DIY. 2024-05-07. Retrieved 2024-06-19.
Retrieved from "https://en.wikipedia.org/w/index.php?title=P(doom)&oldid=1324781030"