# AI Benchmarking **Domain:** Artificial Intelligence / Evaluation / Research Governance **Doc Type:** Canonical Concept Node **Maturity:** Developed ## Definition **AI benchmarking** evaluates systems on shared tasks, datasets, metrics and protocols so that performance can be compared under specified conditions. ## Historical Context The dispute surrounding [[wiki/Speech Understanding Research|Speech Understanding Research]] and [[wiki/HARPY|HARPY]] helped motivate a DARPA evaluation regime in which methods and benchmark tasks were prescribed in advance and systems were tested annually. ## Governance Context Benchmarks make comparison possible, but they also decide what counts as performance. A system can optimize the test while leaving important capabilities, populations or deployment conditions outside the measured frame. ## Psychometric and introspective evaluation [[thoughts/The Luxembourg Preprint|The Luxembourg Preprint]] shows why the prompt frame must be treated as part of the benchmark. A structured inventory and an open-ended therapeutic dialogue can appear to measure the same psychological construct while producing sharply different model behavior. [[wiki/Prompt-Sensitive Behavioral Stability|Prompt-Sensitive Behavioral Stability]] names the comparative target across those frames. [[wiki/Psychometric Deformation|Psychometric Deformation]] names the failure mode in which test-induced output is mistaken for a persistent personality or mental state. A valid benchmark should preserve prompts, order, model version, sampling settings, complete interaction history, repeated trials, and the boundary between observed response and inferred interiority. ## Key Insight **A benchmark does not merely measure success; it operationally defines the success that institutions can recognize.** ## See Also [[wiki/Held-Out Validation|Held-Out Validation]], [[wiki/Objective Function|Objective Function]], [[wiki/Model-Based Governance|Model-Based Governance]], [[wiki/Psychometrics|Psychometrics]], [[wiki/Prompt-Sensitive Behavioral Stability|Prompt-Sensitive Behavioral Stability]], [[wiki/Psychometric Deformation|Psychometric Deformation]] ## Sources / Provenance - National Research Council, _Funding a Revolution_, 1999. - [[articles/Modern Artificial Intelligence in the 1970s|Modern Artificial Intelligence in the 1970s]].