Why LLMs Need Structured Knowledge

Large Language Models and Factual Information

Large language models contain a surprising amount of factual information baked into their parameters during pre-training. Ask GPT-4 who discovered Radium and it will answer correctly. Ask it what the capital of France is and it will not hesitate. This parametric knowledge is impressive, but it comes with three fundamental limitations that make it insufficient for production systems.

Limitations of LLMs

1. Knowledge Becomes Outdated: A model trained on data up to a certain cutoff date cannot know what happened after that date.
2. Hallucinations Are Hard to Control: When a model does not know something, it tends to confabulate plausible-sounding answers rather than admitting ignorance.
3. Multi-Fact Reasoning Is Unreliable: The model may know related facts but fail to connect them reliably when answering related questions.

Because of these limitations, most modern LLM systems rely on external knowledge sources that can be updated, verified, and queried with precision. There are two architectural patterns in this space: knowledge bases and knowledge graphs.

Knowledge Bases

A knowledge base (KB) is a structured repository of facts typically implemented as a relational database or document store. Information is stored as discrete records without explicit representation of relationships between records.

Example of a Knowledge Base

Knowledge bases are commonly used as retrieval sources in Retrieval-Augmented Generation (RAG) pipelines but do not naturally represent relationships between entities.

Knowledge Graphs

A knowledge graph (KG) is a graph-structured representation of knowledge where entities are nodes and relationships are directed edges. Information is stored as triples of the form (head, relation, tail).

Example of Knowledge Graph Triples

(Marie Curie, discovered, Radium)
(Marie Curie, spouse, Pierre Curie)
(Radium, used_for, Cancer treatment)
(Radium, element_of, Periodic table)

In a knowledge graph, relationships are first-class facts. This allows for efficient, relational queries and multi-hop reasoning.

LLM-driven Knowledge Graph Construction

Building a knowledge graph was historically labor-intensive, but advances in LLMs have allowed for automated extraction of structured knowledge from unstructured text. The process operates in three stages:

1. Entity Extraction: Identifies named entities in text and assigns them to types.
2. Relation Extraction: Identifies relationships between extracted entities and expresses them as triples.
3. Knowledge Fusion: Merges duplicate entities and resolves conflicts.

Why Graphs Matter for LLM Reasoning

Knowledge graphs excel at multi-hop reasoning—queries that require traversing a chain of relationships to find answers. The explicit structure of a graph makes this traversal efficient whereas a knowledge base requires multiple lookups.

Properties Enabled by Knowledge Graphs

• Multi-hop Retrieval: The system can find answers that span multiple facts.
• Structured Reasoning: Provides an explicit reasoning trace for queries.
• Explainable Answers: Paths followed in retrieval can be shown to users.

Knowledge Retrieval for LLMs

Modern LLM systems use three retrieval strategies suited for different types of queries:

1. Vector Retrieval: Similarity search in embedding space.
2. Knowledge Base Retrieval: Deterministic, schema-bound retrieval.
3. Knowledge Graph Retrieval: Enables path traversal across a network of relationships.

Why Vector RAG Fails for Multi-Hop Reasoning

Vector retrieval often fails in relational reasoning tasks due to chunk fragmentation and semantic proximity collapse. Knowledge graphs avoid these failures by storing explicit relationships.

GraphRAG: The Emerging Standard

GraphRAG combines LLMs with knowledge graphs for efficient retrieval-augmented generation. It includes processes for extracting entities and relationships, constructing knowledge graphs, and summarizing themes across communities of knowledge.

Key Takeaways

• Knowledge bases are fast for fact lookups but cannot represent relationships between entities.
• Knowledge graphs enable explicit reasoning and multi-hop queries, making them powerful for complex queries.
• LLMs can now build knowledge graphs from unstructured text, paving the way for increased adoption of graph technology.
• Hybrid systems that combine retrieval strategies are becoming the dominant architecture in LLM applications.