# Lexical Segmentation **Domain:** Computational Linguistics / Speech and Text Processing **Doc Type:** Canonical Mechanism Node **Maturity:** Developed **Related:** [[wiki/Phonological Representation|Phonological Representation]], [[wiki/Approximate Phonological Matching|Approximate Phonological Matching]], [[wiki/Corpus Engineering|Tokenization]], [[wiki/Natural Language Processing|Natural Language Processing]] --- ## Definition **Lexical segmentation is the assignment of boundaries that divide a linguistic sequence into words or other lexical units.** Written spaces provide one segmentation, but sound, morphology and meaning may require different boundaries. ## Rhyme-Engine Context The [[wiki/Symbolic Language Engine|Symbolic Language Engine]] treats word boundaries as properties of candidates rather than fixed constraints on sound matching. A target word can rhyme with a phrase whose equivalent terminal sequence is distributed across several tokens. The phonological sequence remains the unit of comparison even when lexical segmentation differs. This distinction turns multiword rhyme into an alignment problem rather than a dictionary lookup. It also expands generative freedom: a required sound can be realized through a phrase when no single word preserves both meaning and form. ## Broader NLP Context Segmentation is foundational in tokenization, speech recognition and languages whose writing systems do not consistently mark word boundaries. No segmentation is purely mechanical; the selected units influence what later models can count, align and learn. ## Key Insight **Spaces belong to writing. Linguistic structure may cross them, ignore them or place its important boundaries elsewhere.** ## See Also [[wiki/Corpus Engineering|Tokenization]], [[wiki/Approximate Phonological Matching|Approximate Phonological Matching]], [[wiki/Phonological Representation|Phonological Representation]], [[wiki/Semantic-First Constrained Generation|Semantic-First Constrained Generation]], [[wiki/Corpus Engineering|Corpus Engineering]] ## Sources / Provenance - Primary project account: [[projects/Ten Years Building a Symbolic Language Engine|Ten Years Building a Symbolic Language Engine]]. - Daniel Jurafsky and James H. Martin, *Speech and Language Processing*. - Phrase-level rhyme architecture documented in the project account.