# Model Contestability **Domain:** AI Governance / Due Process / Administrative Systems **Doc Type:** Canonical Governance Principle **Maturity:** Developed ## Definition **Model contestability** is the practical ability of an affected person or community to discover that a model influenced an outcome, understand the relevant basis, challenge data and assumptions, introduce contrary evidence, obtain human or independent review and receive an effective remedy. Transparency alone is insufficient. A perfectly documented model remains unaccountable if no one can change its consequential output. ## Requirements - notice that an automated or model-mediated determination occurred; - provenance for the material data and classifications; - an intelligible statement of decisive factors; - domain-specific [[wiki/Score Separability|Score Separability]]; - a review authority independent of the original model operator; - timely correction and restoration of wrongly withheld benefits or rights; and - collective standing where a model harms a community rather than one isolated individual. - a [[wiki/Semantic Feedback Channel|Semantic Feedback Channel]] through which context and alternative causal models can enter the operative system; and - [[wiki/Counterfactual Opportunity|Counterfactual Opportunity]] sufficient to test whether changed conditions defeat the forecast. ## Climate and Administrative Context Climate-risk models influence insurance, credit, infrastructure, relocation and sovereign finance. [[articles/Climate Justice and Global Reparative Systems|Climate Justice and Global Reparative Systems]] treats appealability as part of the administrative stack. [[wiki/Prediction Is Not Jurisdiction|Prediction Is Not Jurisdiction]] supplies the corresponding constitutional limit. [[articles/Ambiguity Will Destroy Man and Machine|Ambiguity Will Destroy Man and Machine]] distinguishes being modeled from being heard. Contestability fails if the subject can submit words but the model continues treating behavior as the only authoritative signal. [[articles/Peak Person and the Predicaments of Prediction|Peak Person and the Predicaments of Prediction]] adds that a challenge cannot succeed if the institution has already removed every opportunity capable of falsifying its forecast. ## Key Insight **An explanation describes power; contestability gives the governed a means to resist and correct it.** ## See Also [[wiki/Appeals Mechanisms|Appeals Mechanisms]], [[wiki/Algorithmic Determinations|Algorithmic Determinations]], [[wiki/Algorithmic Constitutionalism|Algorithmic Constitutionalism]], [[wiki/Fiduciary Governance|Fiduciary Governance]], [[wiki/Construction Transparency|Construction Transparency]] ## Educational Cybernetics Context [[Russia and Prussia Kybernetiks]] makes contestability necessary wherever grades, risk models, engagement measures or proctoring systems become error signals that alter a learner’s future. ## Simple Reminders, Quotations, and Thoughts > "The construction of an artificial mind then probably has to wait until we understand better, in physical terms, what a mind is." > **— Lee Smolin**, *2015, Edge annual question “What Do You Think About Machines That Think?”* [[reminders/Risk Debate/Artificial Minds Must Wait for a Physical Theory of Mind by Lee Smolin|Artificial Minds Must Wait for a Physical Theory of Mind by Lee Smolin]] > "The effort to build machines that can think is certain to make us aware of aspects of thought that are not yet fully understood." > **— Mary Catherine Bateson**, *2015, Edge annual question “What Do You Think About Machines That Think?”* [[reminders/Machine Succession/Building Thinking Machines Reveals Thought Itself by Mary Catherine Bateson|Building Thinking Machines Reveals Thought Itself by Mary Catherine Bateson]] > "More disturbing to me is the stubborn reluctance in many segments of society to allow computers to take over tasks that simple models perform demonstrably better than humans." > **— Richard H. Thaler**, *2015, Edge annual question “What Do You Think About Machines That Think?”* [[reminders/AI Control/Computers Already Make Better Routine Decisions by Richard H. Thaler|Computers Already Make Better Routine Decisions by Richard H. Thaler]] > "Thought experiments about these matters are the source of practical insights into human and machine behavior and suggest how to build different and better kinds of machines." > **— Robert Provine**, *2015, Edge annual question “What Do You Think About Machines That Think?”* [[reminders/Machine Succession/Thought Experiments Can Build Better Machines by Robert Provine|Thought Experiments Can Build Better Machines by Robert Provine]] > "Trouble arrives as soon as any of the machine's customers, managers, or assistants start asking a few simple questions." > **— Jon Kleinberg and Sendhil Mullainathan**, *2015, Edge annual question “What Do You Think About Machines That Think?”* [[reminders/Machine Succession/We Built Machines We Cannot Explain by Jon Kleinberg and Sendhil Mullainathan|We Built Machines We Cannot Explain by Jon Kleinberg and Sendhil Mullainathan]] > "Building models is very different from proclaiming truths. It's a never-ending process of discovery and refinement, not a war to win or destination to reach. Uncertainty is intrinsic to the process of finding out what you don't know, not a weakness to avoid. Bugs are features — violations of expectations are opportunities to refine them. And decisions are made by evaluating what works better, not by invoking received wisdom." > **— Neil Gershenfeld**, *2011, Edge Annual Question, “What Scientific Concept Would Improve Everybody's Cognitive Toolkit?”* [[reminders/Information/Uncertainty Is a Feature of Discovery by Neil Gershenfeld|Uncertainty Is a Feature of Discovery by Neil Gershenfeld]] ## Relationships - **Edge Annual Question source relationship:** [[collections/Edge|Edge]] preserves the annual-question source corpus from which a proposition-specific quotation is connected to this entry.