# Deduplication **Entity class:** Identity-Resolution Layer concept **Deduplication** is an identity-resolution concept used to decide whether heterogeneous records refer to the same real-world entity. Deduplication belongs in the ontology because AI can compute over this structure while analysts retain responsibility for interpretation and authorized action. ## Counterterrorism predictive-graph role A relationship or identity match is an evidence-bearing hypothesis with source, time, and confidence. It is not guilt by association. The analytic state should distinguish ground truth, observed evidence, inferred state, and predicted state. ## Relationships - **article-linked mechanism or framework:** [[wiki/Intelligence Cleaning|Intelligence Cleaning]]. - **article-linked mechanism or framework:** [[wiki/Entity Resolution|Entity Resolution]]. - **collection route:** [[collections/The Civilian Intelligence Marketplace|The Civilian Intelligence Marketplace]]. - **counterterrorism architecture:** [[collections/Terrorism, Counterterrorism, and the Intelligence Environment|Terrorism, Counterterrorism, and the Intelligence Environment]]. - **source article:** [[articles/The Coming Market for Civilian Spycraft|The Coming Market for Civilian Spycraft]]. - **domain router:** [[wiki/Identity Resolution|Identity Resolution]]. - **ontology neighbors:** [[wiki/Entity Matching|Entity Matching]] and [[wiki/Disambiguation|Disambiguation]]. - **synthesis:** [[wiki/Counterterrorism Predictive Graph|Counterterrorism Predictive Graph]] and [[wiki/Predictive Intelligence Loop|Predictive Intelligence Loop]]. ## Sources - [NCTC — Guidelines for Access, Retention, Use, and Dissemination of United States Person Information (accessed September 23, 2026)](https://www.odni.gov/files/NCTC/documents/RelatedContent_documents/NCTCGuidelines.pdf) - [IARPA — AGILE technical overview (accessed September 23, 2026)](https://www.iarpa.gov/images/research-programs/AGILE/AGILE_ModSim_-_Final.pdf) **As of:** 2026-09-23 ## Civilian intelligence marketplace Duplicate observations are identified and removed or linked so repeated records do not masquerade as independent support. Cleaning reduces false weight while preserving relevant source history. Collection, cleaning, enrichment, modeling, and productization make a sequence from raw observation to usable intelligence. Each stage performs a distinct kind of work and applies criteria that affect the final product. Bryant’s pipeline thesis connects those operational gates with the contractors, analysts, engineers, and agent builders who staff them. ### Marketplace sources - [[articles/The Coming Market for Civilian Spycraft#XIII. The Pipeline Is the Sovereign|The Coming Market for Civilian Spycraft — XIII. The Pipeline Is the Sovereign]] — October 2026 revision; immediate source for this entry’s place in the marketplace. - [Çağdaş İnal, *İstihbarat Teorisi 2026: Bilişsel Harp, Yapay Zekâ, OSINT, Bayesçi Muhakeme ve Defansif Hibrit İstihbarat*, Chapter 4, "OSINT, Veri Mühendisliği Boru Hatları ve LLM Ajanları" (EVENTICA Yayınları, June 2026; Internet Archive).](https://archive.org/download/istihbarat-teorisi-2026-17.06.2026/Istihbarat_Teorisi_2026%2017.06.2026.pdf) - [Lauren Armbruster, Catherine O'Saben, and Nicholas Tran, MITRE, Scaling the Use of Publicly Available Information Across the U.S. Government (April 2025).](https://www.mitre.org/sites/default/files/2025-04/PR-25-1065-Scaling-the-Use-of-Publicly-Available-Information-Across-the-US-Government.pdf) ## Connected marketplace roles The source article connects this entry with [[wiki/Intelligence Cleaning|Intelligence Cleaning]]. These routes locate the associated people, implementations, methods, and operational contexts in [[articles/The Coming Market for Civilian Spycraft|The Coming Market for Civilian Spycraft]]. **Collections:** [[collections/The Civilian Intelligence Marketplace|The Civilian Intelligence Marketplace]] · [[collections/Terrorism, Counterterrorism, and the Intelligence Environment|Terrorism, Counterterrorism, and the Intelligence Environment]].