RAG Architectures End to End

This is the integrative post for Arc 9. The previous five covered the pieces: embeddings, dense vs sparse, ANN indexes, chunking, and rerankers. The question now is how those pieces snap together into a system that actually answers questions, and what goes wrong when you try it the obvious way.

"RAG" has come to cover a huge range of things. The minimal version is "embed the docs, stuff the top-k chunks into the prompt." That's one end of a much larger design space, and it's where I want to start, because its failure modes point directly at why the more elaborate patterns exist.

The problem this layer solves

Language models memorize a lot of the world during pretraining, but they don't memorize your world. The model has never read your company's handbook, your codebase, your customer tickets, or anything that was written after its cutoff. If you want the model to answer questions using that kind of information, you have to get that information into the prompt at query time.

The model stays frozen. The knowledge lives outside. At query time, we fish the relevant pieces out of the knowledge store and paste them into the context window as reference material.

The appealing part: it decouples knowledge from the weights. You can update your company handbook without retraining anything. The hard part: "fish the relevant pieces out" is doing a lot of work in that sentence, and how well you do it almost entirely determines whether the system gives good or bad answers.

A mental frame I keep coming back to is that RAG is really two systems glued together. There's a search system, which has to be good at finding relevant passages, and there's a language model, which has to be good at using those passages to answer. A lot of the quality issues in production come from treating these as one problem when they're two, and then not knowing where the quality loss is coming from. We'll come back to this when we talk about evaluation, because once you start measuring the two halves separately, the debugging loop collapses from "vibes" to something closer to regular engineering.

What this post assumes

I'm going to lean on everything from the previous Arc 9 posts without re-explaining it. If any of the following phrases feel blurry, the linked post is the right detour: embeddings and vector search (how a chunk becomes a vector), dense-sparse-hybrid (why you usually want both), vector search and ANN (HNSW, IVF, and friends), chunking strategies (how text becomes chunks), and rerankers (cross-encoders on top-k).

I also assume familiarity with the tool-calling idea from Arc 10 — not the protocol details, just the mental model that the model can emit a structured function call and have the runtime execute it. We'll use that pattern in the agentic section.

Naive RAG, and why it breaks

The minimal version has three moves. You embed the user's query, you pull the top-k most similar chunks from a vector store, and you paste those chunks into the prompt with some instructions. Generate. Done.

A naive RAG demo can work surprisingly well on easy questions and fall over fast on anything subtler.

A few failure modes worth naming, because they come back later:

Retrieval-miss is the most common failure. Top-k cosine similarity is sensitive to surface words and blind to semantics the query doesn't spell out. A query about "Q3 2025" will happily pull chunks about "Q3 2024" because Q3 is the salient token and the year is a minor perturbation in the embedding. The retriever returns something that looks relevant. The model has no way to know it isn't.

Contradiction-in-context is the second. Your corpus almost certainly contains superseded versions of documents, legacy pages, and edge cases. When retrieval pulls three chunks that disagree, the model usually picks whichever one appears first or happens to match its priors, and you get a confident answer that's partially wrong.

The third one, ungrounded hallucination, is the one that wrecks trust. The retrieved chunks genuinely don't contain the answer, the model fills the gap from its parametric knowledge, and because the output reads fluently you don't notice. If you're not citing chunks back to the reader, you have no way to audit this.

None of these are bugs in the model. They are symptoms of asking too much of a single-shot cosine lookup.

These failure modes are almost all fixable, but the fixes require adding stages to the pipeline rather than swapping the embedder for a bigger one. The shape of the pipeline is what changes the quality curve. The embedder is one knob, and usually not the most important one.

The full pipeline

The antidote to naive RAG is to stop thinking of it as "embed and paste" and start thinking of it as two pipelines, one offline and one online, each with several stages where things can be measured and improved independently.

The ingestion side runs when your data changes: parse the raw documents, chunk them into passages, embed the passages, write everything into the index along with metadata. This is the expensive, batched side. You do it once per document and you reuse the result across every query.

The query side runs per request: embed the query, retrieve top-k from the index, optionally rerank, assemble a prompt with the chosen context, and generate. This is the latency-critical side. Every millisecond counts.

I find the most useful thing you can do when reasoning about a RAG system is to step a real question through each stage and look at the data at each boundary. So here is one question walked through an eight-stage pipeline, with the actual payloads at each step.

Input: handbook_2025.pdf (184 pages)

Extracted text (first 600 chars):
"Section 7.3 Parental Leave. Full-time employees
are eligible for up to 16 weeks of paid parental
leave following the birth or adoption of a child.
Eligibility begins after 12 months of continuous
employment. Part-time employees receive a
pro-rated benefit. For details on combining
leave with short-term disability, see Section
7.4 ..."

A few things to notice as this plays through.

Chunking is load-bearing and boring. The shape of your chunks (size, overlap, whether you split on paragraphs vs sentences vs fixed windows) directly controls what the retriever can retrieve. Too-small chunks lose context. Too-large chunks dilute the embedding. This is covered properly in the chunking post earlier in the arc; here I'll just note that if your RAG system is bad, chunking is probably part of why.

Retrieval is usually hybrid in production. Dense similarity handles paraphrase and semantics. BM25 handles rare words, names, identifiers, exact phrases. Reciprocal-rank-fusion (RRF) stitches them together with no tuning. If you're using a modern managed store, hybrid is often the default.

Reranking is where cross-encoders earn their cost. A bi-encoder at retrieval time gives you roughly-right candidates. A cross-encoder looking at the (query, chunk) pair together can sharply reorder them. The reranker post covers why the cross-encoder is strictly more expressive but slow enough that you only run it on the top-50 or top-100.

Assembly is where a lot of quiet quality loss happens. How you format the chunks, whether you include chunk IDs for citation, what instructions you give the model, how you order the chunks (some models attend more to the start or end of context, the "lost in the middle" effect): all of this matters and almost none of it gets benchmarked.

The middle-of-context dip is the most counter-intuitive part of this. The chunk is there, the model reads over it during attention, and the answer still comes out wrong. It's a strong argument for aggressively small context: three well-placed chunks beat fifty evenly-padded ones.

One useful heuristic: write the prompt assembly as its own function, log the exact string that goes to the model, and read it the way a reader would. If you can't tell which claims came from which chunk, your citations are cosmetic. If the instructions contradict the chunks (common when the system prompt says "be concise" but the chunks are long), the model is getting mixed signals, and that shows up in the outputs.

One small change that pays off: give each chunk a stable ID that survives reindexing, and require the model to cite those IDs in its answer. When the IDs match, you can auto-check faithfulness by looking up each cited chunk against the claim. When they don't match or go missing, you have an automatic hallucination flag. That one piece of discipline buys more observability than most eval frameworks do.

Advanced patterns

Once you have the pipeline, you start noticing that the single "embed the query, retrieve, paste" loop is too rigid for a surprising number of real questions. The advanced patterns mostly fall out of relaxing one of those three steps.

HyDE: embed the answer, not the question

Here's a move that sounds strange until you see why it works. Instead of embedding the user's query, ask the model to hallucinate what a good answer would look like, embed that, and retrieve against the embedded answer.

The intuition is that an answer lives in the same embedding neighborhood as the documents that contain it, but a question doesn't. A question is structurally different from an answer: it's shorter, it uses interrogative phrasing, it references the unknown. An embedding of "do part-time employees get paid parental leave?" lands somewhere near "part-time employees" but not especially near the actual policy passage. An embedding of a plausible-sounding answer to that question lands much closer to the real policy passage because they are the same kind of text.

And yes, HyDE feeds on its own hallucinations. That's fine. We throw the hallucinated answer away after embedding it. The retrieved chunks are real. The final answer is grounded in those real chunks. HyDE is a retrieval trick, not a generation trick.

HyDE costs an extra generation call per query, so latency goes up. It also sometimes underperforms on queries where the model's guess is very wrong, which happens more on niche corpora where the model's priors don't reflect your domain. A common compromise is to run HyDE only when the first-pass retrieval looks weak (low top-1 similarity, or low reranker agreement), which keeps the cost low and recovers most of the benefit.

Query rewriting

Related, cheaper, and less magical: have the model rewrite the query before it goes into the retriever. Expand abbreviations, resolve pronouns, split compound questions, add synonyms. Useful when your users type things like "what about the other one" and expect the system to have some memory of the conversation.

Multi-hop RAG

Some questions can't be answered in a single retrieval pass because no single chunk contains the full answer. The canonical example is a compositional question: "which author of the Transformer paper later co-founded Anthropic?" You need to retrieve the authors, retrieve the founders, and intersect them. One-shot retrieval will almost certainly fail.

The pattern is that generate a sub-query, retrieve, inspect what came back, generate the next sub-query conditioned on the partial answer, retrieve again, iterate. Two or three hops is typical. The trickiest part is knowing when to stop, which is usually a "do you have enough to answer?" check the model runs on itself.

If you squint, this is just the agent loop from Arc 10 starting to show up early. Multi-hop RAG is a special case of a tool-using agent whose one tool happens to be a retriever. That framing matters, because most of what makes multi-hop work well (good stopping criteria, clear sub-queries, good assembly of the partial state) are generic agent-loop skills, not retrieval-specific ones.

Agentic RAG: the model decides

Naive RAG retrieves on every turn. That's fine when every user message is a question about your corpus. It's wasteful, and sometimes harmful, when the message is "write me a haiku" or "what is 17 times 23." Retrieval that fires against a haiku request will return three random chunks and push the model off course.

Agentic RAG flips the control flow. Retrieval becomes a tool the model chooses to call, the same way it would choose to call a calculator or a code interpreter. The loop looks a lot like the tool loop that Arc 10 is about to cover in full: model decides, runtime executes, result returns, model continues. This is the hinge between retrieval-as-infrastructure and retrieval-as-agent-capability.

The gating decision can be explicit or implicit. The explicit version is Self-RAG: the model is trained to emit special tokens, [Retrieve] or [No-Retrieve], and the runtime reads those tokens and fires a retrieval call only when instructed. The implicit version is standard tool-calling: the model emits a search(query="...") function call when it wants to retrieve. Behaviorally they look the same from outside.

Self-RAG goes further than just the retrieve gate. After retrieving, the model emits critique tokens that score whether each retrieved passage is relevant ([Relevant] / [Irrelevant]), whether its own candidate answer is supported by the passage ([Fully Supported] / [Partially Supported] / [No Support]), and whether the answer is useful ([Utility:5] through [Utility:1]). The decoder can use those scores to pick among multiple candidate generations. It's retrieval, generation, and self-evaluation rolled into one decoding pass.

The mental handle I carry around: RAG is becoming one more tool the model knows how to use, not a fixed preprocessing step. Once you have that handle, a lot of the 2026 architecture decisions make more sense. Claude's Projects retrieves from uploaded documents as a tool. ChatGPT search fires web search as a tool. Perplexity's whole product is a model that uses search aggressively. The distinction between "RAG app" and "agent that happens to search" is collapsing.

The agentic framing also fixes a failure mode of always-on retrieval. If the user just said "thanks, that makes sense," firing a retrieval call against the vector store with that query returns noise that pushes the model around. Gated retrieval means the model doesn't call the tool there, and the round-trip is saved. It sounds trivial, but it changes the felt quality of the system a lot.

Minimal math: what gets retrieved

The retrieval step itself has exactly one equation worth holding in your head. Given a query embedding $q \in \mathbb{R}^d$ and a set of chunk embeddings $\{c_i\}$, we want the top-k by some similarity:

The similarity is almost always cosine, which is just the dot product after normalization:

That's retrieval. Everything else in this post (hybrid, rerank, HyDE, multi-hop, agentic) is a question of what you feed into that argmax and how you post-process its output. The argmax itself is the same line of code it was in 2020.

For hybrid with BM25, the score you rank by is a fused score. A common choice is reciprocal rank fusion, which avoids having to calibrate the scales of cosine and BM25 against each other:

The constant $k$ is typically 60. The intuition is that a chunk that makes the top-10 under both scoring schemes outscores a chunk that is only strong under one. No calibration required, which is why RRF tends to just work.

Worked example

Here's the pipeline from the stepper, compressed into runnable code. No framework, just embeddings + a cross-encoder + an LLM call, so the moving parts are visible. In a real system you'd swap the inline list for a proper vector store (FAISS, pgvector, Qdrant) and the cosine loop for its ANN index.

import numpy as np
from openai import OpenAI
# pip install sentence-transformers
from sentence_transformers import CrossEncoder
 
client = OpenAI()
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")
 
def embed(texts):
    out = client.embeddings.create(model="text-embedding-3-large", input=texts)
    return np.array([d.embedding for d in out.data])
 
# --- 1. INGEST (offline) ---
chunks = [
    "Full-time employees: up to 16 weeks paid parental leave.",   # c_027
    "Short-term disability may be combined with parental leave.", # c_028
    "Leave requests must be filed 30 days in advance.",           # c_104
    "Bereavement leave covers up to 5 days of paid time.",        # c_211
    "Jury duty leave does not affect PTO accrual.",               # c_062
]
chunk_vecs = embed(chunks)  # shape (N, 3072), L2-normalize and store.
 
# --- 2. QUERY (online) ---
query = "how long is paid parental leave?"
q = embed([query])[0]
 
# retrieve: cosine top-k
sims = chunk_vecs @ q / (np.linalg.norm(chunk_vecs, axis=1) * np.linalg.norm(q))
topk = np.argsort(-sims)[:5]
 
# rerank with cross-encoder
pairs = [(query, chunks[i]) for i in topk]
rerank_scores = reranker.predict(pairs)
order = topk[np.argsort(-rerank_scores)][:3]
 
# assemble prompt
context = "\n".join(f"[c{ i :03d}] { chunks[i] }" for i in order)
prompt = f"Answer using ONLY the context. Cite chunk ids.\n\n{context}\n\nQ: {query}"
 
# generate
resp = client.chat.completions.create(model="gpt-4o",
    messages=[{"role": "user", "content": prompt}])
print(resp.choices[0].message.content)

Everything else you'll see in a real codebase (hybrid search, HyDE, multi-hop, agentic retrieval) is a variation on which text you embed, which scores you combine, and how many times you loop. The skeleton above is the thing they all decorate.

When debugging an unfamiliar RAG system, it can help to mentally map whatever framework you're looking at back onto these three knobs. LangChain chains, LlamaIndex query engines, Haystack pipelines, the Anthropic retrieval tool, the OpenAI file_search tool — under the abstraction, they are all choosing some embedding target, some score combination, and some loop count. Once you see it that way, the framework stops mattering and the design decisions become visible.

Evaluation: the triad

A RAG system has at least three places where it can be wrong, and they need to be measured separately. This is the big lesson of the last two years of RAG practice. Treating "did the answer look right" as the metric hides which stage is actually failing.

The Ragas framework names the three and gives you automated estimators for each.

Context-relevance. Of the chunks retrieved, what fraction are actually relevant to the question? This is a pure retriever-quality metric. If it's low, your problem is upstream of the model. Fix chunking, fix the embedder, add a reranker, add hybrid.

Faithfulness. Given the retrieved context, does every claim in the answer follow from the context? Faithfulness measures whether the model stayed in the box we gave it or wandered off into its parametric knowledge. A faithful answer that cites chunk IDs is auditable; an unfaithful one is a hallucination waiting to be caught.

Answer-quality. Given the ground truth, is the answer actually right? This is the one humans care about end-to-end, but it's also the one that can be high for the wrong reasons (the model's priors happened to be correct) or low even when retrieval and faithfulness are fine (the context was right but the model gave a thin answer).

The point of separating the three is that a single aggregate score hides the diagnosis. A system with 90% answer-quality and 30% faithfulness is one lucky prior away from embarrassing you in production. A system with 95% faithfulness and 20% context-relevance is honestly saying "I don't know" on most queries. Both systems score identically on a naive "did the user like the answer" eval. You can't fix what you can't see.

There's a messier fourth dimension: answer coverage, which asks whether the retrieved context contains the information needed to answer at all. If coverage is zero, nothing downstream can save you. This is where evals that check your corpus against your expected question distribution matter more than any specific generation benchmark.

A workflow that holds up reasonably well: build a small golden set of maybe 50 real questions with gold answers and gold citations. Run the full pipeline and log the retrieved chunks, the prompt, and the answer for each. Then score each case on all three metrics, and look at the ones where faithfulness and context-relevance disagree. Those disagreements are the ones that teach you the most: a high-relevance, low-faithfulness case usually means your prompt assembly or your model choice is off. A low-relevance, high-faithfulness case usually means the model is correctly refusing and your retriever is the problem. A low-low case means you might not have the document in your corpus at all.

Misconceptions

"RAG is a preprocessing step." It used to be. In 2026 it's a tool-call inside a loop. Retrieval now interleaves with generation, can happen multiple times per turn, and is governed by the same mechanism that governs any tool the model uses. If you're designing a new system, design for "generate, possibly retrieve, generate more, possibly retrieve again," not "retrieve, then generate."

"If the model hallucinated, the model is the problem." Usually it isn't. Most RAG hallucinations come from retrieval giving the model the wrong context, not from the model refusing to use good context. The cure is measuring context-relevance and faithfulness separately, as above. If context-relevance is fine and faithfulness is bad, then the model is the problem, but in practice that's rarely where the leak is.

"More retrieved chunks is always better." There's a real trade-off. More chunks give the model more chances to find the answer, and also more chances to drown in distractors, contradict itself, or lose the plot in a too-long context. The sweet spot is almost always smaller than people's first instinct. Three to ten strongly-reranked chunks is often better than fifty raw ones.

"Embeddings are the whole game." They are the most visible piece and the piece with the most published benchmarks, so they get the most attention. They matter, but in a full pipeline they're one of maybe eight stages where quality leaks, and they are almost never the biggest one. Chunking, reranking, prompt assembly, and the retrieve/no-retrieve decision all tend to matter at least as much on real workloads.

"A longer context window makes RAG obsolete." This argument comes up a lot as context windows creep past a million tokens, and I don't think it holds up. Long contexts change the trade-offs, they don't erase them. Putting a million tokens of corpus into every single query is expensive, slow, and (per the "lost in the middle" results) not always as effective as putting a thousand well-chosen tokens. Retrieval doesn't go away when context is cheap; it just competes with "stuff everything in" for different workloads. The next post is about exactly this tension.

What's next

Flat retrieval answers a lot of the queries a real system gets, but not all of them. Thematic questions ("what are the big ideas across this corpus?") and multi-hop joins ("who advised the person who co-founded OpenAI?") don't live in any single passage, and embedding-plus-rerank can't rescue that.

The next post covers GraphRAG and knowledge graphs, where we do extra work at ingest time (extract entities, build a graph, detect communities, pre-summarize) so that structure-shaped questions have somewhere to land at query time.

Additional reading (and watching)

• Chen, H., et al. (2024). Dense X Retrieval: What Retrieval Granularity Should We Use?. EMNLP 2024. Argues for proposition-level chunking and shows how granularity shifts retrieval quality meaningfully.

• Nogueira, R. & Cho, K. (2019). Passage Re-ranking with BERT. arXiv:1901.04085. The foundational paper for BERT-based cross-encoder reranking in IR.

• Liu, N. F., et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. TACL 2024. Documents the U-shaped attention-to-context curve and why middle-of-context chunks underperform.

• Gao, L., et al. (2022). Precise Zero-Shot Dense Retrieval without Relevance Labels. arXiv:2212.10496. The HyDE paper: embed a hallucinated answer, retrieve against it.

• Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020. The original RAG paper, introducing the retrieve-then-generate architecture end-to-end with a trainable retriever.

• Asai, A., et al. (2023). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. ICLR 2024. Introduces retrieve/critique tokens so the model controls its own retrieval and evaluates its own outputs.

• Es, S., et al. (2023). Ragas: Automated Evaluation of Retrieval Augmented Generation. EACL 2024. Defines the context-relevance / faithfulness / answer-quality triad and estimators for each.

• Anthropic. (2024). Introducing Contextual Retrieval. Engineering blog post (September 2024). Augments chunks with model-generated context before embedding, reducing retrieval errors by ~35%.

• Guu, K., et al. (2020). REALM: Retrieval-Augmented Language Model Pre-Training. ICML 2020. Shows how a retriever can be trained jointly with the LM via masked-LM signal; predecessor to RAG.

• Borgeaud, S., et al. (2022). Improving Language Models by Retrieving from Trillions of Tokens. ICML 2022. DeepMind's RETRO; architectural retrieval via cross-attention to neighbors from a massive text database.

• Trivedi, H., et al. (2023). Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions. ACL 2023. The "IRCoT" paper on multi-hop retrieval interleaved with reasoning steps.