# Runaway Predictive Feedback Loop **Entity class:** Formal failure mode **Domains:** Predictive policing / counterterrorism analytics / machine learning ## Definition A **runaway predictive feedback loop** occurs when a deployed model directs observation toward the places or people it scores as high-risk, the resulting discoveries become new training data, and the model's confidence grows beyond what the underlying distribution supports. Ensign, Friedler, Neville, Scheidegger, and Venkatasubramanian demonstrated the mechanism formally for [[wiki/Predictive Policing|predictive policing]]: police sent to a location discover more incidents there because they are looking there, and the model becomes progressively more certain the location is dangerous. Lum and Isaac described the same dynamic as algorithms predicting future policing rather than future crime when historical enforcement contaminates the labels. ## Security consequences The loop concentrates attention, converts enforcement density into apparent risk, and pairs with **resource diversion**: analyst capacity consumed by self-reinforcing signals is withheld from real threats. GAO's findings on the SPOT behavior-detection program, including valid supporting evidence for only 8 of 36 revised indicators in 2017, show the resource cost of an unvalidated instrument operating at scale. ## Relationships - **general principle:** [[wiki/Observation-Induced Ground Truth|Observation-Induced Ground Truth]]. - **companion mechanism:** [[wiki/Selective Labels|Selective Labels]]. - **applied setting:** [[wiki/Predictive Policing|Predictive Policing]], [[wiki/Hotspot Analysis|Hotspot Analysis]], and [[wiki/Behavior Detection|Behavior Detection]]. - **loop family:** [[wiki/Feedback Loops|Feedback Loops]] and [[wiki/Graph-to-Collection Feedback Loop|Graph-to-Collection Feedback Loop]]. - **accounting frame:** [[wiki/The Second Error Function|The Second Error Function]]. - **analysis:** [[research/Harms Incurred While Bringing Preventive Systems Online|Harms Incurred While Bringing Preventive Systems Online]]. - **collection:** [[collections/Terrorism, Counterterrorism, and the Intelligence Environment|Terrorism, Counterterrorism, and the Intelligence Environment]]. ## Sources / Provenance - [Ensign et al. — Runaway Feedback Loops in Predictive Policing, PMLR 81](https://proceedings.mlr.press/v81/ensign18a.html) (2018). - [Lum and Isaac — To Predict and Serve?, Significance](https://rss.onlinelibrary.wiley.com/doi/abs/10.1111/j.1740-9713.2016.00960.x) (2016). - [GAO-17-608R — TSA Does Not Have Valid Evidence Supporting Most of the Revised Behavioral Indicators](https://www.gao.gov/products/gao-17-608r) (2017). **As of:** 2026-09-23