# Corpus Engineering **Domain:** Computational Linguistics / Data Engineering **Doc Type:** Canonical Infrastructure Node **Maturity:** Developed **Related:** [[wiki/Computational Linguistics|Computational Linguistics]], [[wiki/Information Retrieval|Information Retrieval]], [[wiki/Distributional Semantics|Distributional Semantics]], [[wiki/Trigger-Based Language Modeling|Trigger-Based Language Modeling]], [[wiki/Provenance|Provenance]] --- ## Definition **Corpus engineering is the construction of reliable, queryable linguistic datasets from heterogeneous source material.** It includes acquisition, normalization, filtering, tokenization, deduplication, metadata preservation, transformation and index generation. Corpus engineering differs from merely collecting text. Every transformation encodes a policy about which material survives and which relationships remain recoverable. ## Symbolic Language Engine Context The project described in [[projects/Ten Years Building a Symbolic Language Engine|Ten Years Building a Symbolic Language Engine]] combined approximately 6.35 million source records and filtered them into roughly 900,000 usable phrase-level rhyme candidates. The important fact is not only scale but declared selection policy: retain multiword material, reject unwanted patterns, preserve function words when they carry useful sound structure and build multiple derived products from one canonical core. The pipeline produced rhyme, compound-rhyme, Soundex, synonym, trigger and anagram structures, along with indexes for software, print and web renderings. This is corpus engineering as representation manufacture. ## Ontology-Governed Corpus Context Corpus engineering prepares and preserves the material from which [[wiki/Ontology Learning|Ontology Learning]] can discover candidate concepts. [[wiki/Bidirectional Ontology|Bidirectional Ontology]] adds a later phase: once concepts mature, they begin constraining how that corpus is interpreted and extended. If a foundational definition changes, [[wiki/Corpus Recompilation|Corpus Recompilation]] applies corpus-engineering methods to find affected passages without treating bulk textual replacement as conceptual review. ## Evidence Boundary Corpus-derived claims require provenance. Source, license, filtering rules, version, timestamp and transformation history affect what any later result can mean. ## Key Insight **A corpus is not raw language. It is language after a sequence of engineering judgments, and those judgments become part of every result.** ## See Also [[wiki/Bidirectional Ontology|Bidirectional Ontology]], [[wiki/Ontology Learning|Ontology Learning]], [[wiki/Corpus Recompilation|Corpus Recompilation]], [[wiki/Information Retrieval|Information Retrieval]], [[wiki/Distributional Semantics|Distributional Semantics]], [[wiki/Switchboard Corpus|Switchboard Corpus]], [[wiki/Broadcast News Corpus|Broadcast News Corpus]], [[wiki/CMU Pronouncing Dictionary|CMU Pronouncing Dictionary]], [[wiki/Provenance|Provenance]] ## Simple Reminders, Quotations, and Thoughts > “Their silly philandering with protesting only reveals their promiscuity with fleeting things. Nothing meaningful ever stays in their lives, because they whore themselves out to every idea that dresses itself up and walks past. They do flit, don't they? A new panic every season. A new purity to defend. A new fascination to denounce. And each infatuation abandoned the moment the next one arrives. Promiscuity with ideas produces exactly what promiscuity always produces: a great deal of activity... and no descendants. Fidelity is unfashionable in an age of flings. But the flings never leave anything behind worth inheriting.” > **— Bryant McGill**, *The Argument in Miniature, 2026; imagined Robert Ford register* [[reminders/Writing/Fidelity to an Idea Leaves Something Worth Inheriting by Bryant McGill|Fidelity to an Idea Leaves Something Worth Inheriting by Bryant McGill]] ## Sources / Provenance - Tony McEnery and Andrew Hardie, *Corpus Linguistics* (2012). - Primary project pipeline described in the linked project account.