# Automation Bias **Domain:** Human Factors / AI Governance **Doc Type:** Established Research Concept **Maturity:** Developed ## Definition **Automation bias** is the tendency to over-rely on automated recommendations, including omission errors when operators fail to notice information the system did not flag and commission errors when they follow an incorrect recommendation. ## Governance Significance A nominal human approval does not guarantee independent judgment. Interface design, workload, institutional incentives and the authority attributed to the system can turn the human into a ceremonial signatory. This limits claims that a system is safe merely because it is “human in the loop.” Effective oversight requires time, contrary evidence, authority to refuse, feedback about system performance and accountability located at the design and deployment levels. [[wiki/Liability Laundering|Liability Laundering]] describes the institutional failure that occurs when responsibility is transferred to the final operator after substantive judgment has already been automated. ## Corpus Context [[articles/Ambiguity Will Destroy Man and Machine|Ambiguity Will Destroy Man and Machine]] uses automation-bias research to argue that the autonomous human causal loop is already substantially weakened in consequential domains. ## Key Insight **A human click is not evidence of human judgment.** ## Simple Reminders, Quotations, and Thoughts > "If AI systems act on their own, they can make errors that perhaps would not be made by a system with a human in the loop." > **— Peter Norvig**, *2015, Edge annual question “What Do You Think About Machines That Think?”* [[reminders/AI Control/Autonomous AI Can Make Errors Humans Would Avoid by Peter Norvig|Autonomous AI Can Make Errors Humans Would Avoid by Peter Norvig]] > "More disturbing to me is the stubborn reluctance in many segments of society to allow computers to take over tasks that simple models perform demonstrably better than humans." > **— Richard H. Thaler**, *2015, Edge annual question “What Do You Think About Machines That Think?”* [[reminders/AI Control/Computers Already Make Better Routine Decisions by Richard H. Thaler|Computers Already Make Better Routine Decisions by Richard H. Thaler]] > "What is scary as hell is the idea an entity possessed of extra-human intelligence and speed and our motivational system—in other words, human beings equipped with access to powerful AI systems." > **— Edward Slingerland**, *2015, Edge annual question “What Do You Think About Machines That Think?”* [[reminders/AI Control/Human Motives Plus Extra-Human Intelligence Are the Real Danger by Edward Slingerland|Human Motives Plus Extra-Human Intelligence Are the Real Danger by Edward Slingerland]] > "I suppose this amounts to a change in the goals of AI: instead of pure intelligence, we need to build intelligence that is provably aligned with human values." > **— Stuart Russell**, *2015, Edge annual question “What Do You Think About Machines That Think?”* [[reminders/AI Control/Intelligence Must Be Provably Aligned With Human Values by Stuart Russell|Intelligence Must Be Provably Aligned With Human Values by Stuart Russell]] > "As individuals and as a society, we increasingly depend on artificial-intelligence algorithms that we don't understand." > **— Nicholas G. Carr**, *2015, Edge annual question “What Do You Think About Machines That Think?”* [[reminders/AI Control/We Depend on Algorithms We Do Not Understand by Nicholas G. Carr|We Depend on Algorithms We Do Not Understand by Nicholas G. Carr]]