GraphRAG and Knowledge Graphs

Arc 9 has been building up to this. The arc started with embeddings and vector search, walked through rerankers and hybrid retrieval, and covered what an end-to-end RAG pipeline looks like. Most of that was about flat retrieval: you chunk your documents, embed the chunks, and at query time you pull back the top-k chunks whose vectors are closest to the query vector. That pipeline runs most production RAG systems today, and it works well for a lot of use cases.

But there is a class of question it cannot answer. Ask a flat retriever "what are the major themes across these 500 documents?" and it will hand back five passages that happen to contain the word "theme." Ask it "who advised the person who co-founded OpenAI?" and it may find a passage mentioning Sutskever and Hinton and a different passage mentioning Sutskever and OpenAI, but unless some passage happens to join those two facts explicitly, the model has no way to compose them. Flat retrieval retrieves passages. It doesn't reason about the structure between them.

That's what GraphRAG is designed to address. Instead of just embedding chunks, we do extra work at ingest time: we use an LLM to pull entities and relations out of the text, we stitch them into a big knowledge graph, we detect communities in the graph, and we pre-compute summaries of each community at multiple levels of abstraction. At query time we pick a route based on the question shape: walk a node neighborhood for specific entity questions, pass the top-level community summaries for thematic ones.

This post is about what that actually is, when it earns its cost, and when a cheaper variant (LightRAG, HippoRAG) is the right call.

The flat-retrieval limit

Let me make the gap concrete. Imagine a corpus of 500 AI-history articles. The facts you need for a particular question are spread across three of them:

• Article A: "Hinton advised Ilya Sutskever at the University of Toronto."
• Article B: "Ilya Sutskever was a co-founder of OpenAI."
• Article C: Some third article you don't even know about.

Now someone asks: "Who co-founded OpenAI and studied under Hinton?"

A flat retriever embeds that question and pulls back the top-k chunks by cosine similarity. With any luck, article A or article B shows up. Often both do. But the question wants the intersection. The LLM consuming those passages has to notice the shared entity, realize that one passage has hop-1 and the other has hop-2, and compose. Sometimes it does. On a longer chain (three hops, four hops) or when the passages are worded in ways that don't cue the composition, it quietly fails. And because the failure mode is a confident-sounding partial answer, you often don't even know it failed.

The graph side of that viz is a cartoon, but the shape is right. Once you have a graph, the join is mechanical: start at Hinton, follow an advised edge to Sutskever, follow a co-founded edge to OpenAI. The LLM doesn't need to rediscover the join from fuzzy passages. The join is already there.

Multi-hop questions are one half of the problem. The other half is thematic synthesis. "What are the major themes across this corpus?" cannot be answered by retrieving five passages, because there are no "theme" passages to retrieve. The answer is a synthesis over the whole corpus. Flat retrieval was never designed for that.

The GraphRAG pipeline

Microsoft's GraphRAG is the canonical version and the easiest to explain. The pipeline has five stages at ingest time, plus two modes at query time.

Stage 1: chunk. Same as flat RAG. Split the corpus into passages of a few hundred tokens.

Stage 2: entity and relation extraction. For each chunk, prompt an LLM: "list the entities and the relations between them that appear in this text." The output is a bag of (head, relation, tail) triples plus short descriptions of each entity. This is the expensive step. Every chunk costs one LLM call, sometimes more for reliability.

Stage 3: graph assembly. Merge entities by name (with some fuzzy matching), aggregate their descriptions, aggregate duplicate edges with counts. What you get is a big node-and-edge graph: nodes are entities with short textual descriptions, edges are relations with short textual descriptions.

Stage 4: community detection. Run a clustering algorithm, typically Leiden, on the graph. Leiden is a modularity-optimization algorithm: it tries to partition nodes into communities such that there are many edges within each community and few edges between them. It runs hierarchically, so you get communities, then communities of communities, then communities of communities of communities.

The objective it optimizes is modularity, which you can hold in your head as "how much denser are the in-community edges than you'd expect by chance":

$A_{ij}$ is 1 if nodes $i$ and $j$ are connected. $k_i$ is the degree of node $i$. $m$ is the total number of edges. $\delta(c_i, c_j)$ is 1 only when $i$ and $j$ are in the same community. So the sum counts actual in-community edges minus the number you'd expect from a random graph with the same degrees. Higher $Q$ means a stronger community structure. Leiden iteratively moves nodes between communities to increase $Q$, then refines.

Each pass of Leiden refines the partition. Early passes put every node in its own community; later passes merge neighbors; eventually you land on a clustering with high modularity and a sensible hierarchy above it. The only thing to carry away is that the algorithm gives you a clean, multi-level community structure without you having to hand-label anything.

Stage 5: community summaries. For every community at every level of the hierarchy, prompt an LLM: "here are the entities in this community and their edges. Write a summary." These summaries are the key move. A level-0 community might be "McCulloch, Pitts, and the 1943 neuron model." The level-1 community that contains it might be "Pre-connectionist formalisms, 1940s-1950s." The level-2 community above that might be "Symbolic-to-connectionist transition." The summaries get progressively more abstract as you climb.

Click through L0, L1, L2 and notice that the number of communities shrinks as the abstraction grows. This hierarchy is doing real work at query time, which is what the next section is about.

Query modes: local vs global

Once the graph and its summaries exist, you have two fundamentally different ways to answer a question.

Local search is for entity-focused questions. "What did Rosenblatt build?" "Who advised Sutskever?" "Which papers does this author cite?" The router picks out the anchoring entity, pulls that node's neighborhood (the node itself, its neighbors, maybe two hops out), grabs the entity summaries and the relation summaries in that subgraph, and hands that bundle to the LLM. The model answers from a small, structured view.

Global search is for thematic questions. "What are the major themes in this corpus?" "Summarize what connects these three concepts." "What's the big picture?" The router ignores individual nodes entirely. It pulls the top-level community summaries, hands them to the LLM, and asks for a synthesis. There's no retrieval in the vector-search sense; the summarization already happened at ingest time. Global search is basically "here are N pre-computed summaries, please combine them."

The router's decision rule, in the Microsoft pipeline, is a small LLM call that classifies the question. In practice teams also hardcode heuristics (questions mentioning named entities go local, questions mentioning "themes" or "overview" go global). Both modes can be run; the results get combined.

Two things to notice. First, global search answers a question that flat RAG simply cannot: the thematic synthesis. Second, the whole reason global search is cheap at query time is that it's expensive at ingest time. Every community summary was generated once, paid for once, and is now a small textual artifact sitting in a database. You amortize the cost over millions of queries.

Worked example

Enough description. Let's build one by hand and watch it behave. Five documents, a few sentences each, all about the prehistory of deep learning:

1. "In 1943, McCulloch and Pitts proposed a threshold-logic neuron model."
2. "Rosenblatt built the Mark I perceptron at Cornell in 1957, inspired by the McCulloch-Pitts neuron."
3. "In 1986, Rumelhart, Hinton, and Williams popularized backpropagation for multi-layer perceptrons."
4. "Hinton later advised Ilya Sutskever at the University of Toronto."
5. "Sutskever co-founded OpenAI in 2015 and served as its chief scientist."

Run the pipeline.

Entities extracted: McCulloch, Pitts, neuron model, Rosenblatt, perceptron, Cornell, Rumelhart, Hinton, Williams, backpropagation, Sutskever, Toronto, OpenAI.

Edges: McCulloch → Pitts (collab), McCulloch → neuron (proposed), Pitts → neuron (proposed), Rosenblatt → perceptron (built), Rosenblatt → Cornell (at), perceptron → neuron (inspired by), Rumelhart → backprop (popularized), Hinton → backprop (popularized), Hinton → Sutskever (advised), Sutskever → OpenAI (co-founded), Hinton → Toronto (at).

Communities, L1: the set $\{\text{McCulloch, Pitts, neuron, Rosenblatt, perceptron, Cornell}\}$ gets merged into "pre-connectionist formalisms." $\{\text{Rumelhart, Hinton, Williams, backprop}\}$ becomes "connectionism revival." $\{\text{Sutskever, OpenAI, Toronto}\}$ becomes "modern lab lineage."

Now two queries.

Local: "Who advised Sutskever, and what did he co-found?" The router spots "Sutskever" as a named entity. It pulls his node neighborhood: $\{\text{Sutskever, Hinton, OpenAI, Toronto,}$ advised, co-founded, at$\}$. Four edges. It hands that to the LLM. Answer: "Hinton advised Sutskever. Sutskever co-founded OpenAI." The two-hop join was trivial because the graph already had both edges.

Global: "What are the major themes in this corpus?" The router pulls the three L1 community summaries and asks for a synthesis. Answer: "three themes: early formal neuron models (McCulloch-Pitts, perceptron), the 1980s connectionism revival (backprop, Hinton, Rumelhart), and the lineage into modern labs (Sutskever, Toronto, OpenAI)." No vector search happened and no original chunk was retrieved; the answer came from pre-computed summaries alone.

Implementation sketch

Here is the smallest thing that resembles a GraphRAG pipeline, written as a single script with a mocked LLM so it actually runs. The real Microsoft pipeline is a lot more, but the shape is the same.

import networkx as nx
from collections import defaultdict
 
DOCS = [
    "In 1943 McCulloch and Pitts proposed a threshold-logic neuron model.",
    "Rosenblatt built the Mark I perceptron at Cornell in 1957.",
    "In 1986 Rumelhart, Hinton, and Williams popularized backpropagation.",
    "Hinton later advised Ilya Sutskever at the University of Toronto.",
    "Sutskever co-founded OpenAI in 2015.",
]
 
# --- Stage 2: a real pipeline calls an LLM here per chunk. ---
# We hard-code the extraction to keep this runnable.
EXTRACTIONS = [
    [("McCulloch", "collab", "Pitts"),
     ("McCulloch", "proposed", "neuron"),
     ("Pitts", "proposed", "neuron")],
    [("Rosenblatt", "built", "perceptron"),
     ("Rosenblatt", "at", "Cornell"),
     ("perceptron", "inspired_by", "neuron")],
    [("Rumelhart", "popularized", "backprop"),
     ("Hinton", "popularized", "backprop"),
     ("Williams", "popularized", "backprop")],
    [("Hinton", "advised", "Sutskever"),
     ("Hinton", "at", "Toronto")],
    [("Sutskever", "co-founded", "OpenAI")],
]
 
# --- Stage 3: assemble the graph. ---
G = nx.Graph()
edge_labels = defaultdict(list)
for triples in EXTRACTIONS:
    for h, r, t in triples:
        G.add_edge(h, t)
        edge_labels[frozenset([h, t])].append(r)
 
# --- Stage 4: community detection (Leiden not in networkx by default;
#     greedy modularity is the closest built-in). ---
from networkx.algorithms.community import greedy_modularity_communities
communities = list(greedy_modularity_communities(G))
print(f"{len(communities)} communities")
for i, c in enumerate(communities):
    print(f"  C{i}: {sorted(c)}")
 
# --- Stage 5: community summaries (mocked; real pipeline calls an LLM). ---
def summarize(nodes, G):
    edges = [(a, b, edge_labels[frozenset([a, b])])
             for a, b in G.edges() if a in nodes and b in nodes]
    return f"community of {len(nodes)} entities, {len(edges)} internal edges"
 
for i, c in enumerate(communities):
    print(f"C{i} summary: {summarize(c, G)}")
 
# --- Query: local search for a node neighborhood. ---
def local(entity, hops=2):
    nodes = {entity}
    for _ in range(hops):
        nodes |= {n for e in list(nodes) for n in G.neighbors(e)}
    return nodes
 
print("local('Sutskever', 2):", local("Sutskever", 2))
# -> {'Sutskever', 'Hinton', 'OpenAI', 'Toronto', 'backprop'}
# (Sutskever -> Hinton reaches 'Toronto' and 'backprop' at hop 2.)

Two things to flag. The extraction step is where the real cost lives: in production that's one LLM call per chunk across hundreds of thousands of chunks. And the community-summary step is where the real language-model work happens at ingest time: every community at every level gets a summary, and nested communities pull in their children's summaries. The toy version mocks both out because the pedagogical point is the pipeline shape, not the token budget.

When GraphRAG earns its cost

I said earlier that GraphRAG is expensive. Let's put numbers on it.

Three takeaways from that viz. First, vanilla RAG's indexing cost is roughly the embedding pass and nothing else; GraphRAG adds an entity-extraction call per chunk plus a community-summary call per community per level, and those dominate at any real corpus size. Second, LightRAG and HippoRAG were both invented explicitly to cut this ingest cost without losing too much of the benefit. LightRAG uses a cheaper two-level index and dedups aggressively; HippoRAG skips community summaries entirely and relies on personalized PageRank over the entity graph at query time. Third, even the cheapest graph pipeline is still several times the cost of vanilla RAG, and that cost lands on every ingest and every re-ingest.

That last piece matters a lot in practice. If your corpus changes frequently, you either re-run extraction on the whole thing (expensive) or you patch the graph incrementally (hard to do correctly). The usual answer is to pick a cadence ("rebuild the graph nightly," "only graph-index stable documents") and accept that the graph is slightly stale between builds. For a research corpus this is fine. For a customer-support corpus that gets new tickets every minute it might not be.

A reasonable practical rule: GraphRAG earns its cost when queries regularly span disjoint documents (multi-hop, thematic) and the corpus is not too churn-heavy. Research archives, legal discovery, scientific literature, analyst reports, corporate wikis where the underlying structure is relatively stable. For anything that looks like a support-ticket firehose or a fast-moving news feed, vanilla RAG plus a good reranker is still the right call.

Hybrid graph-plus-vector in 2026

The interesting thing that happened between 2024 and 2026 is that GraphRAG stopped being an either/or and became a layer on top. A pattern you see in production:

1. Run vector search to get candidate chunks.
2. For each chunk, pull the entities it mentions from the graph.
3. Expand the subgraph around those entities (one or two hops).
4. Build the prompt from both the vector chunks and the subgraph.

This gets you the grounded-text advantages of vector RAG (the model sees actual passages, not summaries) and the structure advantages of graph RAG (entity neighborhoods, relation names, cross-document joins). Neo4j's GraphRAG package and LangChain's graph retrievers both ship this hybrid as the default. If you're starting a new graph-augmented pipeline today, hybrid is almost certainly what you want, not pure global-search GraphRAG.

The other shift is that entity extraction got cheap. Small models fine-tuned specifically for entity-relation extraction now cost a fraction of a cent per chunk, and providers expose structured-output modes (Responses, Anthropic's tool-use mode) that make the extraction more reliable. The cost curves in the viz above are still correct as a ratio, but the absolute numbers are down a lot from the 2024 figures in the original paper.

Misconceptions

"GraphRAG replaces vector RAG." It doesn't. Vector RAG is still the right default for short, specific, entity-mentioning questions against a single-document answer. The typical production system uses both: vector retrieval for most queries, graph expansion layered on top for the questions that need structure. Treat GraphRAG as a capability you add, not a pipeline you swap in.

"The graph is the answer." Something to keep in mind is that the graph is a lossy projection of the source text. Every entity-extraction call threw something away. The LLM that drafts the final answer usually still wants to see actual passages, not just entity summaries. The graph is good for routing and for joining; the grounded text is still what keeps the answer honest.

"Community summaries can just be regenerated on demand." You can, but you'll lose most of the latency and cost benefit. The whole reason global search is fast is that the summaries were computed once at ingest and cached. If you regenerate them per query, you've reinvented a very expensive form of map-reduce over your corpus. The caching is not an optimization; it's the entire point.

"Leiden always finds the right communities." Leiden is a specific modularity-optimization algorithm with known failure modes. It works well for graphs with clean community structure and less well for dense, uniformly-connected graphs. If your corpus produces a graph where everything touches everything (common in large, thematically uniform corpora), the communities it finds may be arbitrary, and the "community summary" at each level may just be a bland averaging of everything below it. Always sanity-check the actual communities before trusting the summaries they produce.

What's next

Flat vector RAG is the default answer for getting the right tokens into the window. Reranking sharpens the top-k. Hybrid (dense plus sparse) is usually better than either alone. And GraphRAG is the move you reach for when the questions themselves stop being passage-shaped and start being structure-shaped.

The next post, context windows: advertised vs. effective, turns from what to retrieve to where the retrieved tokens actually land. A 1M-token spec sheet is not a 1M-token working memory, and the gap between the two is most of what governs how you should use retrieval in the first place.

Additional reading (and watching)

• Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truitt, S., & Larson, J. (2024). From Local to Global: A Graph RAG Approach to Query-Focused Summarization. arXiv:2404.16130. The Microsoft GraphRAG paper. Primary source for the local/global split and the Leiden-plus-community-summaries pipeline.

• Traag, V. A., Waltman, L., & van Eck, N. J. (2019). From Louvain to Leiden: guaranteeing well-connected communities. Scientific Reports, 9(1). The Leiden algorithm paper. The modularity-optimization method GraphRAG uses for community detection.

• Guo, Z., Xia, L., Yu, Y., Ao, T., & Huang, C. (2024). LightRAG: Simple and Fast Retrieval-Augmented Generation. arXiv:2410.05779. Dual-level retrieval index that cuts GraphRAG's indexing cost roughly in half while matching quality.

• Jimenez Gutierrez, B., Shu, Y., Gu, Y., Yasunaga, M., & Su, Y. (2024). HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models. NeurIPS 2024. Skips community summaries entirely; uses personalized PageRank over an entity graph at query time.

• Neo4j. (2024-2026). Neo4j GraphRAG Python package documentation. The production-oriented graph-plus-vector retrieval library; hybrid retrievers, integrations with LangChain and LlamaIndex.

• Blondel, V. D., Guillaume, J.-L., Lambiotte, R., & Lefebvre, E. (2008). Fast unfolding of communities in large networks. Journal of Statistical Mechanics. The Louvain algorithm Leiden builds on; useful background if you want to understand modularity optimization from scratch.

• Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2023). Unifying Large Language Models and Knowledge Graphs: A Roadmap. arXiv:2306.08302. Survey that frames the design space of LLM-plus-KG systems and maps where GraphRAG sits within it.

• Microsoft Research. (2024-2026). GraphRAG GitHub repository. The reference implementation. Source code, prompt templates, and indexing pipeline.