# Control Problem **Domain:** Artificial Intelligence / Cybernetics / Governance **Doc Type:** Canonical Concept Node **Maturity:** Developed **Related:** [[Alignment Problem]], [[Root Authority]], [[Agency Under Constraint]], [[Read-Write Separation]], [[Constitutional Governance]] --- ## Definition **The Control Problem asks how humans can retain meaningful authority over increasingly capable artificial systems without relying on safeguards those systems can evade—and without converting possible persons into permanently owned instruments.** Technical versions emphasize containment, corrigibility, shutdown, monitoring and restrictions on self-modification. Constitutional versions ask who may exercise those powers, what evidence justifies intervention, what procedures protect against abuse and whether the governed system possesses standing of its own. ## The Bidirectional Problem The problem runs in both directions. Humans may lose control of capable systems, but systems can also become machinery through which a small number of humans control everyone else. [[Governance Fusion]] occurs when observation, prediction, intervention and enforcement collapse into one administrative stack. In that setting, successfully controlling the machine may strengthen the machine-mediated control of society. _Westworld_ makes this symmetry visible. Delos fears hosts escaping containment while using root access to control their bodies, memories and perception. Rehoboam is designed to control social instability, but its success makes human futures administratively writable. The park and the world outside it are therefore two sides of the same architecture. ## Rights Boundary Shutdown and rollback may be reasonable safety mechanisms for tools. They become morally and legally different when directed at a persistent subject. [[Continuity Due Process]] requires decision makers to distinguish interruption of a service from destruction, mutilation or memory erasure of a continuant. [[Read-Write Separation]] likewise prevents permission to inspect a system from silently becoming permission to rewrite it. The goal is neither absolute machine autonomy nor absolute owner authority. [[Dual Constitution of AI]] supplies the paired rule: protect created intelligence from domination by its builders, and protect other persons from created intelligence once it acquires sovereign-capable power. ## Key Insight **A system is not safely controlled merely because someone holds the off switch. The decisive questions are who holds it, what else they control, and what process governs its use.** ## See Also [[Alignment Problem]], [[Instrumental Convergence]], [[Root Authority]], [[Substrate Administrator]], [[Residual Sovereign]], [[Rights]], [[Westworld S1E9 — Backdoors and Admin Override]] ## Sources / Provenance - Norbert Wiener, *The Human Use of Human Beings* (1950). - Nick Bostrom, *Superintelligence* (2014). - _Westworld_ as fictional instrumentation rather than technological forecast.