# Corpus Engineering **Domain:** Computational Linguistics / Data Engineering **Doc Type:** Canonical Infrastructure Node **Maturity:** Developed **Related:** [[wiki/Computational Linguistics|Computational Linguistics]], [[wiki/Information Retrieval|Information Retrieval]], [[wiki/Distributional Semantics|Distributional Semantics]], [[wiki/Trigger-Based Language Modeling|Trigger-Based Language Modeling]], [[wiki/Provenance|Provenance]] --- ## Definition **Corpus engineering is the construction of reliable, queryable linguistic datasets from heterogeneous source material.** It includes acquisition, normalization, filtering, tokenization, deduplication, metadata preservation, transformation and index generation. Corpus engineering differs from merely collecting text. Every transformation encodes a policy about which material survives and which relationships remain recoverable. ## Symbolic Language Engine Context The project described in [[projects/Ten Years Building a Symbolic Language Engine|Ten Years Building a Symbolic Language Engine]] combined approximately 6.35 million source records and filtered them into roughly 900,000 usable phrase-level rhyme candidates. The important fact is not only scale but declared selection policy: retain multiword material, reject unwanted patterns, preserve function words when they carry useful sound structure and build multiple derived products from one canonical core. The pipeline produced rhyme, compound-rhyme, Soundex, synonym, trigger and anagram structures, along with indexes for software, print and web renderings. This is corpus engineering as representation manufacture. ## Evidence Boundary Corpus-derived claims require provenance. Source, license, filtering rules, version, timestamp and transformation history affect what any later result can mean. ## Key Insight **A corpus is not raw language. It is language after a sequence of engineering judgments, and those judgments become part of every result.** ## See Also [[wiki/Information Retrieval|Information Retrieval]], [[wiki/Distributional Semantics|Distributional Semantics]], [[wiki/Switchboard Corpus|Switchboard Corpus]], [[wiki/Broadcast News Corpus|Broadcast News Corpus]], [[wiki/CMU Pronouncing Dictionary|CMU Pronouncing Dictionary]], [[wiki/Provenance|Provenance]] ## Sources / Provenance - Tony McEnery and Andrew Hardie, *Corpus Linguistics* (2012). - Primary project pipeline described in the linked project account.