# Selective Labels **Entity class:** Statistical and machine-learning problem **Domains:** Algorithmic decision-making / risk assessment / security analytics ## Definition The **selective labels problem**, formalized by Lakkaraju, Kleinberg, Leskovec, Ludwig, and Mullainathan (KDD 2017), arises when outcomes are observed only for cases on which a decision-maker acted. Counterfactual outcomes for cases left alone stay unobserved, so evaluation and retraining rest on a sample shaped by prior decisions and by unobservables that influenced them. ## Security relevance In predictive security, the rare target event and the decision-dependent visibility of outcomes combine: a person never investigated produces no observed outcome, while a person investigated produces labels shaped by the investigation. Selective labels are the statistical core of [[wiki/Observation-Induced Ground Truth|observation-induced ground truth]] and a principal reason [[wiki/Base Rate|base-rate]]-constrained systems validate largely through proxy labels. ## Relationships - **general principle:** [[wiki/Observation-Induced Ground Truth|Observation-Induced Ground Truth]]. - **companion failure mode:** [[wiki/Runaway Predictive Feedback Loop|Runaway Predictive Feedback Loop]]. - **causal context:** [[wiki/Causal Inference|Causal Inference]] and [[wiki/Counterfactual|Counterfactual]]. - **accounting frame:** [[wiki/The Second Error Function|The Second Error Function]]. - **analysis:** [[research/Harms Incurred While Bringing Preventive Systems Online|Harms Incurred While Bringing Preventive Systems Online]]. - **collection:** [[collections/Terrorism, Counterterrorism, and the Intelligence Environment|Terrorism, Counterterrorism, and the Intelligence Environment]]. ## Sources / Provenance - [Lakkaraju et al. — The Selective Labels Problem: Evaluating Algorithmic Predictions in the Presence of Unobservables, KDD](https://www.cs.cornell.edu/home/kleinber/kdd17-selective.pdf) (2017). **As of:** 2026-09-23