# Recursive Opacity Loop **Domain:** Algorithmic Governance / Feedback Failure **Doc Type:** Canonical Failure Mode **Maturity:** Developed ## Definition A **recursive opacity loop** occurs when a system intervenes invisibly, interprets behavior produced by that intervention as independent evidence, and uses the resulting classification to justify deeper invisible intervention. ## Sequence 1. The system changes a person's environment or opportunities without acknowledgment. 2. The person responds to an unexplained perturbation. 3. The response is classified without accounting for the system's causal role. 4. The classification authorizes additional opacity or restriction. 5. The resulting instability becomes new training data. This is the governance analogue of measurement contamination. It connects [[wiki/Institutional Retrocausality|Institutional Retrocausality]], [[wiki/Forecast-Induced Dissipation|Forecast-Induced Dissipation]] and [[wiki/Continuous Adjudication|Continuous Adjudication]]. ## Constitutional Response [[wiki/Epistemic Status Separation|Epistemic Status Separation]], intervention provenance, [[wiki/Semantic Feedback Channel|Semantic Feedback Channels]] and independent [[wiki/Model Contestability|Model Contestability]] are necessary because the operating system cannot audit an effect while omitting itself from the causal graph. ## Key Insight **Opacity produces instability; instability is classified as danger; danger justifies deeper opacity.** ## Relationships - **Historical comparison:** [[wiki/Some Moral and Technical Consequences of Automation|Wiener's 1960 automation paper]] describes loss of operator understanding inside an adaptive process. It is a precursor to the opacity concern, not itself evidence of this entry's full recursive governance loop. - **Corpus route:** [[wiki/Model Contestability|Model Contestability]], [[articles/AI Escape Is the Wrong Metaphor|AI Escape Is the Wrong Metaphor]], and [[collections/Machine Succession|Machine Succession]] provide adjacent control and accountability contexts. ## Simple Reminders, Quotations, and Thoughts > "Trouble arrives as soon as any of the machine's customers, managers, or assistants start asking a few simple questions." > **— Jon Kleinberg and Sendhil Mullainathan**, *2015, Edge annual question “What Do You Think About Machines That Think?”* [[reminders/Machine Succession/We Built Machines We Cannot Explain by Jon Kleinberg and Sendhil Mullainathan|We Built Machines We Cannot Explain by Jon Kleinberg and Sendhil Mullainathan]] From [[wiki/Some Moral and Technical Consequences of Automation|Some Moral and Technical Consequences of Automation]]: Wiener distinguishes eventual explanation from explanation fast enough to govern the action. > "It may well be that in principle we cannot make any machine the elements of whose behavior we cannot comprehend sooner or later. This does not mean in any way that we shall be able to comprehend these elements in substantially less time than the time required for operation of the machine, or even within any given number of years or generations." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/Machines May Be Comprehensible Only After They Act by Norbert Wiener|Machines May Be Comprehensible Only After They Act by Norbert Wiener]] From [[wiki/Some Moral and Technical Consequences of Automation|Some Moral and Technical Consequences of Automation]]: The passage concerns effective understanding of intermediate stages and tactical intentions. > "It may be seen that the result of a programming technique of automatization is to remove from the mind of the designer and operator an effective understanding of many of the stages by which the machine comes to its conclusions and of what the real tactical intentions of many of its operations may be." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/Automation Can Hide Its Own Decision Path by Norbert Wiener|Automation Can Hide Its Own Decision Path by Norbert Wiener]] From [[wiki/Some Moral and Technical Consequences of Automation|Some Moral and Technical Consequences of Automation]]: The standfirst of Wiener’s 1960 paper summarizes a possible speed-and-understanding mismatch. > "As machines learn they may develop unforeseen strategies at rates that baffle their programmers." > **— Norbert Wiener**, *1960, “Some Moral and Technical Consequences of Automation,” Science 131(3410)* [[reminders/AI Control/Learning Machines May Baffle Their Programmers by Norbert Wiener|Learning Machines May Baffle Their Programmers by Norbert Wiener]]