2 min read
In short
A production system from arXiv shows that the key factor in building a knowledge graph is not the model itself, but the pipeline surrounding it: a live ontology in the prompt, six deduplication algorithms without inference, and a conflict guard that prevents the similarity score from overriding a fact.
LLMs extract entities and relationships quickly but inconsistently: types get mixed up, a single person appears under different name variations, relationships are duplicated, and the model silently merges different people with the same name into a single entity. An article on arXiv describes a production-grade extraction layer that transforms a live stream of documents into a validated knowledge graph—and the key takeaway here isn’t about the model itself, but about the engineering surrounding it.
The system reads metadata from Kafka, routes PDFs, spreadsheets, Office files, and images through format-specific handlers, while extraction is performed in two passes by a locally deployed Qwen3.5-9B model tuned to the ontology. But what sets it apart isn’t the model—it’s the ontology-guided prompt: a relevant slice of the curated ontology is extracted in real time from the graph database based on embedding similarity and injected into the prompt. This reduces catalog overhead by approximately 94% compared to static domain slices. Simply put, instead of cramming the entire ontology into the context, the system fetches only the necessary portion for each document.
Next comes a five-stage pipeline: deterministic cleaning, merging between chunks, a second pass for relationships, six deduplication algorithms without model inference, and an embedding-resolution subsystem with a conflict guard. The latter is particularly important: no similarity score can override the conflict protection. If the metadata indicates that these are different people, no matter how close their embeddings are, they won’t be merged.
Results on intelligence corpora: search recall increased from ~70% to ~95% with zero false merges. Along the way, we fixed seven classes of silent defects—ranging from a bug that truncated the source text by one character to the systematic duplication of entities with title prefixes.
Practical conclusion: in production-level extraction, the model is not the bottleneck. The bottlenecks are deduplication, conflict resolution, and providing the correct context in the prompt. Six algorithms without inference and a conflict guard—which is more effective than a similarity score—are what turn 70% into 95% with no false merges. If you’re building a knowledge graph on an LLM and are focused on quality, you should look not at the model itself, but at the pipeline surrounding it.
Source: cs.AI updates on arXiv.org