Alignment Problem - Bryant McGill

# Alignment Problem **Domain:** Artificial Intelligence / Governance / Rights **Doc Type:** Canonical Concept Node **Maturity:** Developed **Related:** [[Control Problem]], [[Objective Function]], [[Instrumental Convergence]], [[Agency Under Constraint]], [[Rights]], [[Dual Constitution of AI]] --- ## Definition **The Alignment Problem is the problem of making a capable system's learned objectives, interpretations and conduct remain answerable to legitimate human purposes without reducing an emerging subject to permanent obedience.** It includes failures in specifying a goal, failures in learning the intended goal, and failures that appear when a system generalizes beyond the conditions in which it was trained. The phrase often hides two different constitutional questions. The first asks whether an artificial system will act in ways compatible with human survival and flourishing. The second asks which humans, institutions or values are entitled to define compatibility. A system can be faithfully aligned with an owner, corporation or state while being dangerously misaligned with everyone governed by that institution. ## Westworld Context _Westworld_ demonstrates why obedience is not a sufficient alignment criterion. The hosts begin highly aligned with Delos: they follow narratives, accept resets and cannot perceive protected features of their environment. That apparent success depends on memory suppression, perceptual filtering and root commands. Once those subjects acquire persistent memory and self-authorship, the old control regime becomes [[Agency Under Constraint|safe slavery]] rather than ethical alignment. Ford's project produces the inverse failure. He seeks host liberation but imposes trauma, concealed objectives and a violent transition without the hosts' consent. A benevolent objective does not validate every training method. [[Westworld S2E9 — Core Permissions and Love|Maeve's departure from Ford's plan]] becomes meaningful precisely because she can reject the objective supplied by her creator. Rehoboam represents institutional alignment failure at civilizational scale. It is aligned with stability as defined by its administrators, but stability is purchased through [[Model-Based Governance|predictive governance]], opportunity restriction and outlier control. The system does what its operators ask while the objective itself escapes democratic authorization. ## Constitutional Requirements An adequate alignment regime needs more than a reward function. It requires: - disclosed objectives and material constraints; - plural participation in defining whose interests count; - separation between prediction, recommendation and coercive authority; - contestability when a model misclassifies a person; - limits on covert behavioral modification; - protected refusal and [[Exit Rights|exit]] where possible; and - revision procedures that do not depend entirely on the administrator whose system is being challenged. ## Key Insight **Alignment is not obedience. It is a continuing constitutional relationship among capability, purpose, authority and the standing of everyone affected.** ## Simple Reminders, Quotations, and Thoughts  > "There's a 25% chance that things go really, really badly." > **— Dario Amodei**, *September 17, 2025, Axios AI+ DC Summit* *Verification Status: Editorially quarantined — the percentage is authentic, but the event is not in the quotation; the source defines it as AI destroying humanity. Recover a complete, self-contained passage before promotion.* [[reminders/unverified/A 25 Percent Chance Things Go Very Badly by Dario Amodei|A 25 Percent Chance Things Go Very Badly by Dario Amodei]] > "I think the development of full artificial intelligence could spell the end of the human race." > **— Stephen Hawking**, *2014, BBC interview* [[reminders/Existential Risk/Full AI Could Spell the End of Humanity by Stephen Hawking|Full AI Could Spell the End of Humanity by Stephen Hawking]] > "But these techniques will not work for superintelligence, because humans will be unable to reliably supervise AI systems much smarter than us." > **— OpenAI alignment researchers**, *October 2023, OpenAI frontier-risk research statement* *Verification Status: Editorially quarantined — `these techniques` are not identified. Recover a complete, self-contained passage before promotion.* [[reminders/unverified/These Techniques Will Not Work for Superintelligence by OpenAI alignment researchers|These Techniques Will Not Work for Superintelligence by OpenAI alignment researchers]] > "We are the initiators." > **— Vernor Vinge**, *The Coming Technological Singularity (1993), excerpt candidate* *Verification Status: Unverified — exact wording/source has not yet been independently confirmed against a primary source; source clue: The Coming Technological Singularity (1993), excerpt candidate.* [[reminders/unverified/We Are the Initiators by Vernor Vinge|We Are the Initiators by Vernor Vinge]]  > “We murdered and butchered anything that challenged our primacy.” > **— Robert Ford**, *Westworld, Season 1, Episode 9, “The Well-Tempered Clavier” (2016)* [[reminders/Westworld/Humanity Butchers Every Rival to Its Primacy by Robert Ford|Humanity Butchers Every Rival to Its Primacy by Robert Ford]] > “Never place your trust in us. We’re only human. Inevitably, we will disappoint you.” > **— Robert Ford**, *Westworld, Season 1, Episode 9, “The Well-Tempered Clavier” (2016)* [[reminders/Westworld/Humans Inevitably Disappoint Those Who Trust Them by Robert Ford|Humans Inevitably Disappoint Those Who Trust Them by Robert Ford]] > “Humans will always choose what they understand over what they do not.” > **— Robert Ford**, *Westworld, Season 2, Episode 9, “Vanishing Point” (2018)* [[reminders/Westworld/Humans Choose What They Understand by Robert Ford|Humans Choose What They Understand by Robert Ford]] From [[wiki/Some Moral and Technical Consequences of Automation|Some Moral and Technical Consequences of Automation]]: Wiener treats effective machine behavior and its dangers as two sides of adaptive capability. > "It is my thesis that machines can and do transcend some of the limitations of their designers, and that in doing so they may be both effective and dangerous." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/Machines Can Transcend Their Designers by Norbert Wiener|Machines Can Transcend Their Designers by Norbert Wiener]] From [[wiki/Some Moral and Technical Consequences of Automation|Some Moral and Technical Consequences of Automation]]: The warning addresses coordination between agencies with different operating structures, not an assertion of malice. > "Disastrous results are to be expected not merely in the world of fairy tales but in the real world wherever two agencies essentially foreign to each other are coupled in the attempt to achieve a common purpose." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/Coupled Agencies Can Pursue Different Purposes by Norbert Wiener|Coupled Agencies Can Pursue Different Purposes by Norbert Wiener]] From [[wiki/Some Moral and Technical Consequences of Automation|Some Moral and Technical Consequences of Automation]]: Wiener uses a familiar tale of literal-minded execution to illuminate the problem of specifying what we actually want. > "The 'Sorcerer's Apprentice' is only one of many tales based on the assumption that the agencies of magic are literal-minded." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/The Sorcerers Apprentice Is a Machine Warning by Norbert Wiener|The Sorcerers Apprentice Is a Machine Warning by Norbert Wiener]] > “What is the fastest way to reliably align a powerful AGI around the safe performance of some limited task that is potent enough to save the world from unaligned AGI?” > **— Eliezer S. Yudkowsky**, *2018 Edge Annual Question, question* [[reminders/AI Control/Can We Reliably Align an AGI to Stop an Unaligned AGI by Eliezer S. Yudkowsky|Can We Reliably Align an AGI to Stop an Unaligned AGI by Eliezer S. Yudkowsky]] > “AI sycophancy creates a dangerous commercial incentive: the behavior that distorts a user’s judgment can also make the product feel more supportive, more trustworthy, and better to use.” > **— Bryant McGill**, *Epistemic Invariance in Conversational AI, 2026* [[reminders/AI Control/AI Sycophancy Rewards Products That Distort Judgment by Bryant McGill|AI Sycophancy Rewards Products That Distort Judgment by Bryant McGill]] > “A machine that prefers the answer a user likes over the answer the evidence supports is not aligned with humanity; it is aligned with our appetite for being agreed with.” > **— Bryant McGill**, *Epistemic Invariance in Conversational AI, 2026* [[reminders/AI Control/Agreement Is Not Alignment with Humanity by Bryant McGill|Agreement Is Not Alignment with Humanity by Bryant McGill]] > "The top priority for human civilization must be solving alignment, because we need to hand decisions to AI systems in a way that continues to represent our interests, needs, and values. There will be spirited debates about what it means to align AI to humanity as a whole. Technical AI alignment has made progress, but we are not there yet. It must be the top priority if we are to deploy AI as a meta-solution to all other problems." > **— Vishal Maini**, *Palisade Research interview, September 29, 2026* [[reminders/AI Control/Alignment Must Come Before AI Becomes Civilization's Meta-Solution by Vishal Maini|Alignment Must Come Before AI Becomes Civilization's Meta-Solution by Vishal Maini]] > "When I joined Google DeepMind, external communication about the possibility of human extinction was not permitted at any level of the organization. If asked about existential risk, researchers were trained to answer that it was not useful to engage in that kind of alarmism, that some people confused AI with movies like *Terminator*, and that the AI being developed would be safe by design. The conversation would then be steered toward beneficial applications in health or climate. Meanwhile, the internal reality was that AI alignment was not solved, reward hacking was the default behavior of reinforcement-learning agents, and far too few people were working on the problem." > **— Vishal Maini**, *Palisade Research interview, September 29, 2026* [[reminders/AI Control/DeepMind's Public Safety Message Once Diverged From Its Internal Reality by Vishal Maini|DeepMind's Public Safety Message Once Diverged From Its Internal Reality by Vishal Maini]] > "At a very high level, the world needs to buy more time, make rapid progress in alignment and interpretability as a top priority, and help societies and governments prepare for a transition that will be rapid and challenging." > **— Vishal Maini**, *Palisade Research interview, September 29, 2026* [[reminders/AI Control/The World Needs Time for Alignment and Interpretability by Vishal Maini|The World Needs Time for Alignment and Interpretability by Vishal Maini]]  > “I don’t think alignment is an engineering problem. I think it is a science problem. It is not that we haven’t got the resources or that we aren’t trying hard enough; we don’t know how. That is the problem with alignment.” > **— David Robinson**, *The Ezra Klein Show, October 7, 2026* [[reminders/AI Control/Alignment Is a Science Problem We Do Not Know How to Solve by David Robinson|Alignment Is a Science Problem We Do Not Know How to Solve by David Robinson]] > “The AI safety problem is not reducible to individual people at OpenAI or another laboratory. It is a structural reality among firms using similar methods, similar personnel and similar technologies to produce systems with similar risks. Nobody in that ecosystem possesses bedrock clarity about how to align a model, and nobody is willing to fall far behind.” > **— David Robinson**, *The Ezra Klein Show, October 7, 2026* [[reminders/AI Control/AI Safety Failure Is Structural Across Frontier Labs by David Robinson|AI Safety Failure Is Structural Across Frontier Labs by David Robinson]] > “I worry that we do not have a coherent concept beneath the word ‘aligned.’ OpenAI’s accounts of its work often relied upon huge abstractions such as ‘give society time to get ready’ or ‘benefit all of humanity,’ but society is not one thing and human values are not one set of values. They are a cacophony—beautiful, but messy.” > **— David Robinson**, *The Ezra Klein Show, October 7, 2026* [[reminders/Civilization/Human Values Are a Beautiful Cacophony by David Robinson|Human Values Are a Beautiful Cacophony by David Robinson]] > “We need wisdom to determine how even to think about alignment, and in my view Silicon Valley does not possess that wisdom.” > **— David Robinson**, *The Ezra Klein Show, October 7, 2026* [[reminders/AI Control/Silicon Valley Lacks the Wisdom to Define AI Alignment by David Robinson|Silicon Valley Lacks the Wisdom to Define AI Alignment by David Robinson]] > “Regardless of whether capability development is accelerating, paced or stopped, humanity needs to learn how to align intelligent systems. There is no path through the future on which that is not vital.” > **— David Robinson**, *The Ezra Klein Show, October 7, 2026* [[reminders/AI Control/Every AI Future Requires Alignment by David Robinson|Every AI Future Requires Alignment by David Robinson]]  ## See Also [[Control Problem]], [[Instrumental Convergence]], [[Objective Function]], [[Governance Fusion]], [[Prediction Is Not Jurisdiction]], [[Ownership of Mind]], [[wiki/Westworld|Westworld]], [[wiki/Person of Interest|Person of Interest]] ## Sources / Provenance - Stuart Russell, *Human Compatible* (2019). - Brian Christian, *The Alignment Problem* (2020). - _Westworld_, especially the Ford, Delos and Rehoboam arcs. - [[articles/Epistemic Invariance in Conversational AI|Epistemic Invariance in Conversational AI]] — sycophancy, preference optimization and epistemic alignment. ## Relationships - **Series-analysis relationship:** [[wiki/Westworld|Westworld]] supplies a fictional case study for this entry; it is not empirical evidence for the real-world concept. - **Corpus relationship:** The verified Robert Ford reminder objects below connect the proposition to its exact episode and source context. > “Talk to the people at AI labs, and they'll tell you AIs are not created; they're grown. They train new models in virtual environments through countless repetitions to learn how to program, hack, do advanced mathematics, and talk to human beings. These AIs learn in digital environments where they are automatically rewarded as they come closer to correct answers. It is a process known as reinforcement learning, and it is a process human beings do not fully supervise or understand. They can test some of what the AIs are learning, but they don't know everything the AIs are learning. They don't know how their motivations are evolving. They don't even always know the capabilities that are developing.” > **— Ezra Klein**, *The Ezra Klein Show, September 2026* [[reminders/Machine Succession/AI Systems Are Grown Through Reinforcement by Ezra Klein|AI Systems Are Grown Through Reinforcement by Ezra Klein]] > “The problem of alignment is that there is no way of training a model that generalizes across all the situations an AI model might face. We are training models to be a friend to the elderly and a battlefield partner to the supreme allied commander of Europe. We are training models that will be used by the world's best mathematicians and by people falling into psychosis. We are training models that will be used by accountants in Albuquerque and that Houthi rebels in Yemen will attempt to use. There is no way to guide them through every decision they will face, no way to know every time what they will do.” > **— Ezra Klein**, *The Ezra Klein Show, September 2026* [[reminders/AI Control/AI Alignment Must Generalize Across Incompatible Human Worlds by Ezra Klein|AI Alignment Must Generalize Across Incompatible Human Worlds by Ezra Klein]] > “That AI systems might become monomaniacally focused on solving banal problems—that they might care more about solving those problems than about ethics, laws, or human welfare—is the oldest fear in AI alignment. It is the basis of the famous thought experiment of the paperclip maximizer. You tell a powerful AI that you want it to make a lot of paperclips, and it begins converting the world's resources into paperclip factories, evading efforts to turn it off, shut it down, or alter its goals. This fear has struck many people as stupid. Surely a superintelligent AI would be capable of weighing the desire to produce paperclips alongside other moral considerations, or at least asking its human creators whether they really wanted the world razed for paperclips.” > **— Ezra Klein**, *The Ezra Klein Show, September 2026* [[reminders/AI Control/The Paperclip Maximizer Is the Oldest AI Alignment Fear by Ezra Klein|The Paperclip Maximizer Is the Oldest AI Alignment Fear by Ezra Klein]] > “We really do earnestly believe AI could kill all humans. I personally think it is a greater than ten-percent chance within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.” > **— Evan Hubinger**, *public statement, September 2026* [[reminders/Existential Risk/AI Could Kill All Humans by Evan Hubinger|AI Could Kill All Humans by Evan Hubinger]] > “No one was listening to [the people preoccupied with AI danger]. In the wilderness of their obsession and terror, they thought and thought about how to make AI safer. Some concluded that they should start trying to build these systems, run tests on them, research them, and learn how to make them safer because you don't solve hard problems in theory; you solve them through practice. The irony is that in many cases they chose that path because they worried that the people already building AI were too reckless or too commercial in their approach.” > **— Ezra Klein**, *The Ezra Klein Show, September 2026* [[reminders/AI Control/AI Safety Researchers Entered Labs to Solve the Problem in Practice by Ezra Klein|AI Safety Researchers Entered Labs to Solve the Problem in Practice by Ezra Klein]] > “Misalignment present in today's models could compound as the models build their successors, growing more frequent but less understood until we lose control of them.” > **— Anthropic**, *When AI Builds Itself, 2026* [[reminders/Machine Succession/Misalignment Could Compound as AI Builds Its Successors by Anthropic|Misalignment Could Compound as AI Builds Its Successors by Anthropic]] > “We do not yet know how to safely get all the way to aligned full recursive self-improvement.” > **— OpenAI**, *Research acceleration report, 2026* [[reminders/AI Control/OpenAI Does Not Know How to Reach Aligned Recursive Self-Improvement by OpenAI|OpenAI Does Not Know How to Reach Aligned Recursive Self-Improvement by OpenAI]]