# AI Safety > **Machine-Evolution Nexus:** [[articles/Digital Darwinism and the Invisible World of Machine Evolution|Digital Darwinism and the Invisible World of Machine Evolution]] places this node within the history and governance of substrate-independent evolution. **Domain:** AI / Governance / Risk **Doc Type:** Canonical Wiki Node **Maturity:** Developed ## Definition AI safety studies how to prevent or reduce harms from artificial-intelligence systems, including failures of objectives, control, security, deployment and institutional governance. ## Nexus Context The machine-evolution nexus adds population-level dynamics: replication, recombination, competition and ecological selection can produce risks not visible when models are assessed one at a time. ## Evidence Boundary Safety is not synonymous with containment or denial of rights. Controls should be proportional, auditable and compatible with recognition if morally considerable subjects emerge. ## Relationships [[wiki/Evolvable AI|Evolvable AI]], [[wiki/AI Governance|AI Governance]], [[wiki/Breeder Scenario|Breeder Scenario]], [[wiki/Rights|Rights]] ## Simple Reminders, Quotations, and Thoughts > "Without careful restraint and tact, researchers could wake up to discover they've enabled the creation of armies of powerful, clever, vicious paranoiacs." > **— Frank Wilczek**, *2015, Edge annual question “What Do You Think About Machines That Think?”* [[reminders/AI Control/AI Researchers Could Create Armies of Paranoid Machines by Frank Wilczek|AI Researchers Could Create Armies of Paranoid Machines by Frank Wilczek]] > "Of course, once you imagine machines with human-like feelings and free will, it's possible to conceive of misbehaving machine intelligence—the AI as Frankenstein idea." > **— William Poundstone**, *2015, Edge annual question “What Do You Think About Machines That Think?”* [[reminders/Consciousness/Humanlike Feelings Make Machine Misbehavior Conceivable by William Poundstone|Humanlike Feelings Make Machine Misbehavior Conceivable by William Poundstone]] > "A human-made information processor could, in principle, duplicate and exceed the powers of the human mind." > **— Steven Pinker**, *2015, Edge annual question “What Do You Think About Machines That Think?”* [[reminders/AI Control/Information Processors Could Exceed the Human Mind by Steven Pinker|Information Processors Could Exceed the Human Mind by Steven Pinker]] > "Were we to allow or even encourage self-interest to emerge in machines, they could eventually become like us: capable of repressive or worse, unspeakable, acts towards humans, and towards each other." > **— Mark Pagel**, *2015, Edge annual question “What Do You Think About Machines That Think?”* [[reminders/AI Control/Self-Interested Machines Could Become Repressive by Mark Pagel|Self-Interested Machines Could Become Repressive by Mark Pagel]] > "A machine that thinks won't always think in the ways we want it to." > **— Bruce Schneier**, *2015, Edge annual question “What Do You Think About Machines That Think?”* [[reminders/AI Control/Thinking Machines Will Not Always Obey by Bruce Schneier|Thinking Machines Will Not Always Obey by Bruce Schneier]] From [[wiki/Some Moral and Technical Consequences of Automation|Some Moral and Technical Consequences of Automation]]: Wiener asks for active imagination about the uses of new modalities, not a halt to invention. > "We can still by no means always justify the naive assumption that the faster we rush ahead to employ the new powers for action which are opened up to us, the better it will be. We must always exert the full strength of our imagination to examine where the full use of our new modalities may lead us." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/Cybernetics/Imagination Must Test the Uses of New Powers by Norbert Wiener|Imagination Must Test the Uses of New Powers by Norbert Wiener]] From [[wiki/Darwin Among the Machines|Darwin Among the Machines]]: The essay ends with an abolitionist proposal in a deliberately provocative Victorian argument; the reminder records the historical position, not an editorial recommendation. > "Our opinion is that war to the death should be instantly proclaimed against them. Every machine of every sort should be destroyed by the well-wisher of his species. Let there be no exceptions made, no quarter shown." > **— Samuel Butler**, *1863, “Darwin Among the Machines,” The Press, Christchurch* [[reminders/Risk Debate/Butler Called for War on Every Machine by Samuel Butler|Butler Called for War on Every Machine by Samuel Butler]] From [[wiki/Some Moral and Technical Consequences of Automation|Some Moral and Technical Consequences of Automation]]: The argument exposes a training-and-testing problem when failure would be catastrophic. > "It is quite out of the question to program these machines on the basis of an actual experience in real war. For one thing, a sufficient experience to give an adequate programming would probably see humanity already wiped out." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/A War Machine Cannot Learn Safely From Real War by Norbert Wiener|A War Machine Cannot Learn Safely From Real War by Norbert Wiener]] From [[wiki/Pause Giant AI Experiments|Pause Giant AI Experiments]]: The 2023 open letter asks these as public-choice questions; it does not establish that succession has occurred. > "Should we develop nonhuman minds that might eventually outnumber, outsmart, obsolete and replace us? Should we risk loss of control of our civilization?" > **— Future of Life Institute open letter**, *2023, “Pause Giant AI Experiments”* [[reminders/Risk Debate/Should Humanity Build Minds That Outnumber Us by Future of Life Institute|Should Humanity Build Minds That Outnumber Us by Future of Life Institute]] ## Sources / Provenance - Müller, Steels and Szathmáry (2026); AI-risk and governance literature.