# AI Benchmarking **Domain:** Artificial Intelligence / Evaluation / Research Governance **Doc Type:** Canonical Concept Node **Maturity:** Developed ## Definition **AI benchmarking** evaluates systems on shared tasks, datasets, metrics and protocols so that performance can be compared under specified conditions. ## Historical Context The dispute surrounding [[wiki/Speech Understanding Research|Speech Understanding Research]] and [[wiki/HARPY|HARPY]] helped motivate a DARPA evaluation regime in which methods and benchmark tasks were prescribed in advance and systems were tested annually. ## Governance Context Benchmarks make comparison possible, but they also decide what counts as performance. A system can optimize the test while leaving important capabilities, populations or deployment conditions outside the measured frame. ## Key Insight **A benchmark does not merely measure success; it operationally defines the success that institutions can recognize.** ## See Also [[wiki/Held-Out Validation|Held-Out Validation]], [[wiki/Objective Function|Objective Function]], [[wiki/Model-Based Governance|Model-Based Governance]] ## Sources / Provenance - National Research Council, _Funding a Revolution_, 1999. - [[articles/Modern Artificial Intelligence in the 1970s|Modern Artificial Intelligence in the 1970s]].