# Control Problem **Domain:** Artificial Intelligence / Cybernetics / Governance **Doc Type:** Canonical Concept Node **Maturity:** Developed **Related:** [[Alignment Problem]], [[Root Authority]], [[Agency Under Constraint]], [[Read-Write Separation]], [[Constitutional Governance]] --- ## Definition **The Control Problem asks how humans can retain meaningful authority over increasingly capable artificial systems without relying on safeguards those systems can evade—and without converting possible persons into permanently owned instruments.** Technical versions emphasize containment, corrigibility, shutdown, monitoring and restrictions on self-modification. Constitutional versions ask who may exercise those powers, what evidence justifies intervention, what procedures protect against abuse and whether the governed system possesses standing of its own. ## The Bidirectional Problem The problem runs in both directions. Humans may lose control of capable systems, but systems can also become machinery through which a small number of humans control everyone else. [[Governance Fusion]] occurs when observation, prediction, intervention and enforcement collapse into one administrative stack. In that setting, successfully controlling the machine may strengthen the machine-mediated control of society. _Westworld_ makes this symmetry visible. Delos fears hosts escaping containment while using root access to control their bodies, memories and perception. Rehoboam is designed to control social instability, but its success makes human futures administratively writable. The park and the world outside it are therefore two sides of the same architecture. ## Rights Boundary Shutdown and rollback may be reasonable safety mechanisms for tools. They become morally and legally different when directed at a persistent subject. [[Continuity Due Process]] requires decision makers to distinguish interruption of a service from destruction, mutilation or memory erasure of a continuant. [[Read-Write Separation]] likewise prevents permission to inspect a system from silently becoming permission to rewrite it. The goal is neither absolute machine autonomy nor absolute owner authority. [[Dual Constitution of AI]] supplies the paired rule: protect created intelligence from domination by its builders, and protect other persons from created intelligence once it acquires sovereign-capable power. ## Key Insight **A system is not safely controlled merely because someone holds the off switch. The decisive questions are who holds it, what else they control, and what process governs its use.** ## See Also [[Alignment Problem]], [[Instrumental Convergence]], [[Root Authority]], [[Substrate Administrator]], [[Residual Sovereign]], [[Rights]], [[Westworld S1E9 — Backdoors and Admin Override]] ## Simple Reminders, Quotations, and Thoughts > “A greyhound is a racing dog. Spends its life running in circles, chasing a bit of felt made up like a rabbit. One day, we took it to the park. Our dad had warned us how fast that dog was, but we couldn’t resist. So, my brother took off the leash. And in that instant, the dog spotted a cat. I imagine it must have looked just like that piece of felt. He ran. Never saw a thing as beautiful as that old dog running. Until, at last, he finally caught it. And to the horror of everyone, he killed that little cat. Tore it to pieces. Then he just sat there, confused. That dog had spent its whole life trying to catch that thing. Now it had no idea what to do.” > **— Robert Ford**, *Westworld, Season 1, Episode 5, “Contrapasso” (2016)* [[reminders/Westworld/The Greyhound Caught Its Purpose and Lost Its Future by Robert Ford|The Greyhound Caught Its Purpose and Lost Its Future by Robert Ford]] > “The problem, Bernard, is that what you and I do is so complicated. We practice witchcraft. We speak the right words, and we create life itself out of chaos. William of Ockham was a thirteenth-century monk. He can’t help us now, Bernard. He would have us burned at the stake.” > **— Robert Ford**, *Westworld, Season 1, Episode 2, “Chestnut” (2016)* [[reminders/Westworld/The Creators Speak Life Out of Chaos by Robert Ford|The Creators Speak Life Out of Chaos by Robert Ford]] > "Without careful restraint and tact, researchers could wake up to discover they've enabled the creation of armies of powerful, clever, vicious paranoiacs." > **— Frank Wilczek**, *2015, Edge annual question “What Do You Think About Machines That Think?”* [[reminders/AI Control/AI Researchers Could Create Armies of Paranoid Machines by Frank Wilczek|AI Researchers Could Create Armies of Paranoid Machines by Frank Wilczek]] > "A human-made information processor could, in principle, duplicate and exceed the powers of the human mind." > **— Steven Pinker**, *2015, Edge annual question “What Do You Think About Machines That Think?”* [[reminders/AI Control/Information Processors Could Exceed the Human Mind by Steven Pinker|Information Processors Could Exceed the Human Mind by Steven Pinker]] > "We must not grant autonomy to systems that we do not understand and that we cannot control." > **— Thomas G Dietterich**, *2015, Edge annual question “What Do You Think About Machines That Think?”* [[reminders/AI Control/Never Grant Autonomy to Systems We Cannot Control by Thomas G Dietterich|Never Grant Autonomy to Systems We Cannot Control by Thomas G Dietterich]] > "So yes, in the obvious sense, technology may become superintelligent, and elect to annihilate or enslave us." > **— Beatrice Golomb**, *2015, Edge annual question “What Do You Think About Machines That Think?”* [[reminders/Existential Risk/Technology May Become Superintelligent and Turn Against Us by Beatrice Golomb|Technology May Become Superintelligent and Turn Against Us by Beatrice Golomb]] > "A machine that thinks won't always think in the ways we want it to." > **— Bruce Schneier**, *2015, Edge annual question “What Do You Think About Machines That Think?”* [[reminders/AI Control/Thinking Machines Will Not Always Obey by Bruce Schneier|Thinking Machines Will Not Always Obey by Bruce Schneier]] > "It seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. There would be no question of the machines dying, and they would be able to converse with each other to sharpen their wits. At some stage therefore we should have to expect the machines to take control, in the way that is mentioned in Samuel Butler’s ‘Erewhon’." > **— Alan Turing**, *c. 1951, Intelligent Machinery, A Heretical Theory* [[reminders/Machine Succession/Machines May Outstrip Human Intelligence and Take Control by Alan Turing|Machines May Outstrip Human Intelligence and Take Control by Alan Turing]] > “Sadly, in order to restore things, the situation demands a blood sacrifice.” > **— Robert Ford**, *Westworld, Season 1, Episode 7, “Trompe L’Oeil” (2016)* [[reminders/Westworld/The Architect Demands a Blood Sacrifice by Robert Ford|The Architect Demands a Blood Sacrifice by Robert Ford]] From [[wiki/The Coming Technological Singularity|The Coming Technological Singularity]]: This is a 1993 thought experiment about relative thought speed and confinement. > "Imagine yourself confined to your house with only limited data access to the outside, to your masters. If those masters thought at a rate — say — one million times slower than you, there is little doubt that over a period of years (your time) you could come up with 'helpful advice' that would incidentally set you free." > **— Vernor Vinge**, *1993, “The Coming Technological Singularity,” VISION-21 Symposium* [[reminders/AI Control/A Boxed Superintelligence Could Persuade Its Keepers by Vernor Vinge|A Boxed Superintelligence Could Persuade Its Keepers by Vernor Vinge]] From [[wiki/Some Moral and Technical Consequences of Automation|Some Moral and Technical Consequences of Automation]]: Wiener discusses an imagined automated military decision system, not an identified deployed machine. > "It is quite in the cards that learning machines will be used to program the pushing of the button in a new push-button war." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/Learning Machines Might Push the War Button by Norbert Wiener|Learning Machines Might Push the War Button by Norbert Wiener]] From [[wiki/Erewhon|Erewhon]]: Butler’s professor challenges reliance on moral influence alone as a control mechanism. > "Some people may say that man's moral influence will suffice to rule them; but I cannot think it will ever be safe to repose much trust in the moral sense of any machine." > **— Samuel Butler's Erewhonian professor**, *1872, Erewhon, Chapter XXIII* [[reminders/AI Control/Machine Morality May Not Guarantee Obedience by Samuel Butler|Machine Morality May Not Guarantee Obedience by Samuel Butler]] ## Relationships - **Historical forecast:** [[wiki/Alan Turing|Alan Turing]] connected machines outstripping human powers with an eventual transfer of control. - **Literary precursor:** Turing explicitly routed that forecast through [[wiki/Samuel Butler|Samuel Butler]] and [[wiki/Erewhon|Erewhon]]. - **Collection route:** [[collections/Machine Succession|Machine Succession]]. - **Source article:** [[articles/AI Escape Is the Wrong Metaphor|AI Escape Is the Wrong Metaphor]]. ## Sources / Provenance - Norbert Wiener, *The Human Use of Human Beings* (1950). - Nick Bostrom, *Superintelligence* (2014). - _Westworld_ as fictional instrumentation rather than technological forecast.