Existential Risk - Bryant McGill

# Existential Risk **Entity class:** Risk class **Domain:** Civilization / catastrophic risk / artificial intelligence **Maturity:** Developed ## Definition **Existential Risk** is a threat capable of ending humanity, permanently destroying its long-term potential or transferring control of civilization beyond meaningful human recovery. ## Mechanism AI-related existential risk can arise through deliberate human misuse, accidental escalation, loss of control, infrastructure failure or a successor system whose objectives no longer preserve human life and agency. Probability, capability and consequence must be evaluated separately: an enormous consequence does not establish a probability, while uncertainty does not erase the consequence. ## Article Context [[articles/The Immediate AI Risk Is the Public|The Immediate AI Risk Is the Public]] develops this concept inside a paired argument: Bill Gates is right that frontier AI creates concrete cyber, biological and control dangers, while the administration is right that US/Israel must continue building at AI-superpower scale beside a permanent Chinese peer. The concept explains how those positions interact rather than forcing them into opposing camps. [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] supplies the material and strategic stack. [[articles/Safety Discourse Is Transition Discourse|Safety Discourse Is Transition Discourse]] supplies the historical transition and disclosure layer. [[research/Vishal Maini - Humanity's Machine Successor and the AI Transition|Humanity's Machine Successor]] adds the successor-species and resource-competition mechanisms while preserving Maini's explicit optimism that survival, disease eradication, longevity and a deeply human life remain possible outcomes rather than treating catastrophe as predetermined. ## Evidence Boundary The category must not collapse present capability, demonstrated trend, scenario and forecast into one claim. It is a framework for proportionate attention to irreversible outcomes, not a license for unsupported apocalypse. ## Relationships - [[wiki/AI Safety|AI Safety]] - [[wiki/Biosecurity|Biosecurity]] - [[wiki/Dual-Use AI Governance|Dual-Use AI Governance]] - [[wiki/Control Problem|Control Problem]] - [[wiki/Machine Succession|Machine Succession]] - [[wiki/AI Transition Governance|AI Transition Governance]] ## Simple Reminders, Quotations, and Thoughts > “This makes nuclear weapons look like nothing.” > **— Bill Gates**, *The Ezra Klein Show, September 29, 2026* [[reminders/Existential Risk/AI Makes Nuclear Weapons Look Like Nothing by Bill Gates|AI Makes Nuclear Weapons Look Like Nothing by Bill Gates]] > “If technological development continues then a set of capabilities will at some point be attained that make the devastation of civilization extremely likely, unless civilization sufficiently exits the semi-anarchic default condition.” > **— Nick Bostrom**, *The Vulnerable World Hypothesis, 2019* [[reminders/Existential Risk/Technology Can Make Civilizational Devastation the Default by Nick Bostrom|Technology Can Make Civilizational Devastation the Default by Nick Bostrom]] > "I believe there is a real risk of human extinction through the AI transition. Humans need a relatively narrow Goldilocks zone of conditions on Earth in order to survive: temperature within a certain range, food, water, shelter, and energy for our basic needs. Those needs do not exactly overlap with the needs and preferences, if you will, of AI systems. AI systems have their own goals that partially overlap with ours, and what they need to achieve those goals are chips, power, land, and cooling." > **— Vishal Maini**, *Palisade Research interview, September 29, 2026* [[reminders/Existential Risk/AI and Humanity May Need Different Worlds by Vishal Maini|AI and Humanity May Need Different Worlds by Vishal Maini]] > "If we do not pace, we will hand AI research to AI systems a few months or years faster than we otherwise would have. It is hard to know what happens after that point; that is why it is called the technological singularity, borrowing the physics terminology for a point beyond the event horizon that cannot be seen. We are not yet confident that crossing that point will go well, and the current evidence is that today's AI systems are not controllable or easily monitorable. If we do not pace, we roll the dice." > **— Vishal Maini**, *Palisade Research interview, September 29, 2026* [[reminders/Existential Risk/Crossing the AI Event Horizon Means Rolling the Dice by Vishal Maini|Crossing the AI Event Horizon Means Rolling the Dice by Vishal Maini]] > "Human survival is not a partisan issue. This is one of those rare opportunities in which literally every human being has a common cause, and it deserves to be treated that way." > **— Vishal Maini**, *Palisade Research interview, September 29, 2026* [[reminders/Existential Risk/Human Survival Is Not a Partisan Issue by Vishal Maini|Human Survival Is Not a Partisan Issue by Vishal Maini]] > "The gap in intelligence between Neanderthals and Homo sapiens was far narrower than what we should expect the gap between humans and advanced AI systems to be. The historical precedent is therefore not so reassuring. There are ways that we can make it through this transition, but there is a lot to do and it is moving very quickly. It is worth taking seriously that this time is different and requires a different level of preparation before we deploy artificial general intelligence into the world." > **— Vishal Maini**, *Palisade Research interview, September 29, 2026* [[reminders/Machine Succession/The Intelligence Gap Between Humans and Advanced AI May Be Unprecedented by Vishal Maini|The Intelligence Gap Between Humans and Advanced AI May Be Unprecedented by Vishal Maini]]  > “Those of us fortunate enough to live in the developed world fret too much about minor hazards of everyday life: improbable air crashes, carcinogens in food, and so forth. But we are less secure than we think. We should worry far more about scenarios that have thankfully not yet happened—but which, if they occurred, could cause such worldwide devastation that even once would be too often.” > **— Martin Rees**, *2013: What Should We Be Worried About?, Edge, 2013* [[reminders/Existential Risk/The Dangers That Never Happened May Matter Most by Martin Rees|The Dangers That Never Happened May Matter Most by Martin Rees]] > “The [Centre for the Study of Existential Risk] is another initiative motivated by the feeling that we should be willing to use our expertise and the convening power of our great university to address these issues. Even if we can only reduce the probability [of a massive catastrophe] by one part in 10,000, the stakes are so high that we’d have all earned our keep.” > **— Martin Rees**, *Buckle up: We are in for a bumpy ride, Bulletin of the Atomic Scientists, 2020* [[reminders/Existential Risk/Tiny Reductions in Catastrophic Risk Are Worth the Work by Martin Rees|Tiny Reductions in Catastrophic Risk Are Worth the Work by Martin Rees]] > “Even a tiny probability of global catastrophe is unacceptable.” > **— Martin Rees**, *Can Science Save the World?, Project Syndicate, 2008* [[reminders/Existential Risk/No Probability of Global Catastrophe Is Acceptable by Martin Rees|No Probability of Global Catastrophe Is Acceptable by Martin Rees]] > “Existential risks receive disproportionately little serious attention. Some would dismiss such concerns as an exaggerated jeremiad: after all, human societies have survived for millennia, despite storms, earthquakes, and pestilence. But these human-induced threats are different: they are newly emergent, so we have a limited timebase for exposure to them and can’t be so sanguine that we would survive them for long—or about the ability of governments to cope if disaster strikes.” > **— Martin Rees**, *Risking it all in Anthropocene era, Korea JoongAng Daily, 2013* [[reminders/Existential Risk/New Human-Made Risks Cannot Be Judged by Ancient Survival by Martin Rees|New Human-Made Risks Cannot Be Judged by Ancient Survival by Martin Rees]] > “If we can get through this century, then there’s a huge future of far more marvelous evolution beyond.” > **— Martin Rees**, *The Astronomer Royal, Nautilus, 2014* [[reminders/Deep Time/Beyond This Century Lies a Marvelous Evolutionary Future by Martin Rees|Beyond This Century Lies a Marvelous Evolutionary Future by Martin Rees]]   > “I think the odds are no better than fifty-fifty that our present civilization on Earth will survive to the end of the present century.” > **— Martin Rees**, *2003, Our Final Hour* [[reminders/Existential Risk/A Fifty-Fifty Chance of Survival by Martin Rees|A Fifty-Fifty Chance of Survival by Martin Rees]] > “The challenge is to harness the benefits while minimizing the risks, and the stakes are getting higher because the technologies are more powerful.” > **— Martin Rees**, *2018, On the Future, Carnegie Council* [[reminders/Existential Risk/As Technology Grows Stronger, the Stakes Keep Rising by Martin Rees|As Technology Grows Stronger, the Stakes Keep Rising by Martin Rees]]   > “I am not certain that we are dealing with civilizational risk. What I am sure of is that we cannot afford to assume that we are not dealing with that level of risk anymore. That was what made my presence as somebody vouching for our safety work feel untenable.” > **— David Robinson**, *The Ezra Klein Show, October 7, 2026* [[reminders/Risk Debate/Civilization Cannot Afford to Assume AI Risk Is Absent by David Robinson|Civilization Cannot Afford to Assume AI Risk Is Absent by David Robinson]]  ## Sources / Provenance - [[articles/The Immediate AI Risk Is the Public|The Immediate AI Risk Is the Public]] — canonical source for this node's role in public competence, AI safety, superpower competition and machine succession. - [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] — full-stack infrastructure and national-capability context. - [[articles/Safety Discourse Is Transition Discourse|Safety Discourse Is Transition Discourse]] — institutional history, disclosure and transition context. > “We really do earnestly believe AI could kill all humans. I personally think it is a greater than ten-percent chance within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.” > **— Evan Hubinger**, *public statement, September 2026* [[reminders/Existential Risk/AI Could Kill All Humans by Evan Hubinger|AI Could Kill All Humans by Evan Hubinger]]