Chunking Strategies

In a retrieval system, how you slice your documents matters more than which embedding model you use. Swap a mediocre embedding model for a great one and you might pick up two or three points of recall. Chunk a document badly and you can drop twenty.

It's tempting to spend the first round of RAG tuning on the wrong knob: benchmarking embedding models against each other, A/B-ing distance metrics, tweaking top-k. The thing that actually moves the needle is almost always upstream of all of it. A good embedding model cannot see what isn't in the chunk it was handed.

This post is about the chunking strategies that matter in practice, why the chunk-size sweet spot is task-dependent, and a small worked example of the same document carved three ways, with a query that sits near a boundary.

I'm assuming you've seen the embeddings material earlier in the arc. If "a chunk's embedding is a vector, and retrieval is nearest-neighbor in that space" is already a working sentence for you, you're set.

The problem chunking solves

Embedding models have a maximum context window. Even the generous ones top out somewhere around 8k tokens, and most production embedding models are tuned for far less. So you cannot just embed "the document." You have to split it into pieces, embed each piece, and store those vectors in your index.

That's the mechanical reason. The interesting reason is different. Even if you had an embedding model with infinite context, you would still want to chunk. An embedding is a single vector that averages the semantic content of whatever you handed it. Hand it a 50-page PDF and you get a vector that is the average meaning of the entire thing, which is useful for "what's this document broadly about?" and useless for "which sentence answers my question?"

The word "averages" is doing real work there, so let me make it concrete. Most embedding models produce one vector per token, then mean-pool them: $\mathbf{e}_{\text{chunk}} = \frac{1}{n}\sum_{i=1}^{n} \mathbf{e}_{t_i}$. A chunk with one on-topic sentence surrounded by nineteen off-topic ones has an embedding whose signal is diluted roughly 1-in-20. The query vector has to beat that dilution to win the nearest-neighbor lookup.

So chunking is how you trade between two things. Too small, and a chunk loses the context it needs to be interpretable. Too big, and its embedding smears over so many ideas that no specific query matches it well. The right chunk size is the one where the unit of text is the unit of meaning for the kind of query you expect.

This is the frame I want you to carry through the rest of the post. Every strategy we're about to look at is a different answer to the same question: how do I draw the boundaries so the piece I embed is the right semantic unit?

Fixed-size with overlap: the baseline

The simplest thing that works. Decide on a chunk size (say, 500 tokens), decide on an overlap (say, 50 tokens), and slide a window across the document. Each chunk is 500 tokens long. Consecutive chunks overlap by 50.

The overlap is the important part. Without it, a fact that lands on a boundary is permanently split across two chunks, and neither chunk contains enough of the surrounding context to match a query cleanly. Overlap is the cheapest way to buy robustness against bad boundary luck. You pay in storage (you're indexing some text twice) and in retrieval (a query might hit two overlapping chunks and you need to dedupe), but it almost always pays for itself.

The thing to watch out for is that fixed-size chunkers are usually byte or token based, not sentence based. They don't care that they just split "nineteen miles per hour" into "nineteen" and "miles per hour." A token boundary is not a semantic boundary. Which leads to the next strategy.

Recursive and semantic chunking

A recursive splitter tries to respect natural boundaries in the text. The classic version, the one LangChain's RecursiveCharacterTextSplitter is built around, works like this: try to split on "\n\n" first (paragraph breaks). If any resulting chunk is still too big, split those by "\n". If still too big, split by ". ". If still too big, fall back to splitting by spaces. Characters are the absolute last resort.

The intuition is that stronger separators ("\n\n") correlate with bigger semantic transitions. A paragraph break is usually a topic shift. A sentence break is a thought-unit break. A space is just... a word. The recursive splitter tries to preserve the strongest semantic boundary it can get away with.

"Semantic chunking" is a fancier relative of this. Instead of using character patterns as proxies for meaning, it embeds sentences one at a time, compares consecutive sentence embeddings, and splits wherever the cosine similarity drops below a threshold. The logic is: "when the meaning shifts, cut here." In practice it's slower (you have to embed everything twice, once for boundary detection and again for indexing), it's finicky to tune, and recursive splitting is almost always good enough. I'd reach for semantic chunking last, not first.

There's a subtler flavor I want to flag here: document-structure-aware chunking. Unstructured.io and similar tools parse the structural elements of a document (headings, tables, lists, code blocks) before splitting text. The result is chunks that never cross a section boundary and never mix a table row with surrounding prose. If your corpus has structure (HTML, markdown, PDFs with a table of contents), this is the strategy that respects it.

Parent-document: small for matching, big for context

You can chunk the same document at two different granularities and use them for different jobs.

Index the small chunks. Retrieve on the small chunks. But when you hand results to the LLM, return the parent chunk, the bigger block that the small chunk came from.

Why does this work? Because "what should I embed?" and "what should I show the LLM?" are two different questions with two different right answers.

The best thing to embed is a small, specific unit of meaning. One sentence, one code block, one bullet point. Small chunks have tight, high-signal embeddings. They match queries cleanly.

But when the LLM needs to answer the query, it usually wants more context than that single sentence. It wants the paragraph around it, maybe the whole section, so it can disambiguate pronouns, catch conditions, and quote precisely. Handing the LLM only the 40-token child chunk is starving it of the context the small chunk's embedding was never supposed to carry.

Parent-document retrieval separates those two jobs. LangChain and LlamaIndex both ship first-class implementations. LlamaIndex's SentenceWindowNodeParser is the cleanest version: each node is one sentence, but each node carries a window of ±3 sentences as metadata, and at retrieval time the window is what gets returned.

Of the chunking moves available, parent-document tends to be the highest-leverage one. Compared to a default "split everything into 512-token chunks and return them directly" pipeline, switching to a parent-document scheme will usually help more than anything else you could do to the retriever.

The chunk-size sweet spot is task-dependent

So. You've picked a strategy. You still have to pick a chunk size. What's the right size?

The answer is: it depends on what your queries look like.

Factoid queries ("when did the engine stall happen?", "what's the price of the enterprise plan?") want small chunks. The answer is usually a sentence or two, and the embedding of a small chunk will be dominated by the sentence that actually contains the fact. Big chunks dilute that signal with surrounding text that the query doesn't care about.

Summarization-ish queries ("what's the overall argument of this paper?", "what are the main concerns in this thread?") want bigger chunks. The answer isn't located in one span. The topic of a chunk has to emerge from averaging many sentences, and you need enough sentences for that average to be stable.

Real systems usually face both kinds of queries. The two common responses are: (1) pick a size that's a decent compromise (300–500 tokens is a reasonable default across most workloads), or (2) run a multi-granularity index where the same corpus is chunked at multiple sizes, and the retriever searches all of them. The second option costs more storage but buys robustness across query types. Most production systems end up doing some version of it.

One more note. When people quote a "best chunk size" number from a blog post or a benchmark, they are implicitly quoting it for their queries on their corpus. Those numbers don't transfer. The only reliable way to pick a chunk size is to evaluate on your own queries against your own corpus. We'll come back to retrieval evaluation in a later post.

Metadata enrichment

Here's a move that costs almost nothing and often helps a lot. Before you embed a chunk, prepend the stuff that locates it: the document title, the section heading, the breadcrumb path, the date.

Why does this help? Because most chunks are ambiguous in isolation. A chunk that reads "The engine shuts off after 30 seconds." could be about a car, an IoT device, a video game, or a dishwasher. The embedding of that sentence alone is a blurry vector near many unrelated things. Prepend "Tesla Model Y > Owner Manual > Start-Stop System" and the embedding moves decisively into the automotive region. The query "why does the car turn off at lights?" now lands next to it.

There's a 2024 variant I want to flag, Anthropic's Contextual Retrieval, where instead of using static metadata the system asks an LLM to generate a short contextual sentence for each chunk before embedding it. Something like: "This chunk discusses the Start-Stop System behavior described in the Model Y Owner's Manual, Section 4." Anthropic reports meaningful recall improvements across a variety of corpora, at the cost of an LLM call per chunk at indexing time. For a one-off static corpus, that's a real trade. For a corpus that changes daily, probably not.

Either way, the lesson is the same: the embedding sees only what you give it. If context you have at indexing time is cheap to attach, attach it.

A teaser: late chunking

Everything we've looked at so far embeds each chunk in isolation. The embedding model reads the chunk, produces a vector, and never sees any text outside the chunk boundary.

In 2024, Jina released a technique called late chunking that flips this. You pass the entire document (or as much of it as the model can take) through the embedding model's encoder, get contextualized token embeddings, and only then average-pool them into chunk-level vectors. The chunks are the same. The chunk-embeddings are different, because each one was computed with awareness of the surrounding text.

The effect is subtle but real. A chunk that contains "she" is no longer stuck with the ambiguous token embedding for "she"; it carries the contextual embedding that the encoder built while also attending to the sentence two paragraphs earlier that introduced "Dr. Chen." The chunk gets the benefit of the wider context without being physically enlarged.

I think of late chunking as the logical endpoint of the "embeddings see only what you give them" idea. If the model can see more, let it see more, and only chunk at the end. It's still fairly new, it requires an embedding model that exposes per-token outputs, and the tooling is catching up, but the direction is promising.

Worked example

Let me stitch the pieces together with a concrete example. Here's the document, the query, and what each strategy ends up returning as its top-1 hit.

## Hippo facts

Hippos are the deadliest large mammal in Africa. Adults weigh up to three tons.

Despite their bulk, hippos can run at nineteen miles per hour on land. They can outrun most humans over short distances.

They spend most of the day submerged to stay cool.

The interesting part isn't which one "wins" (recursive and parent-document both return a usable chunk here). It's how they fail when they fail. The fixed-size chunker with no overlap doesn't return a wrong passage; it returns a truncated one. The embedding still ranks it near the top because "hippos can run at nineteen" is close enough to the query. But the LLM now has to answer "how fast?" with a chunk that literally ends mid-number.

This is the pattern you'll see over and over. Most chunking failures aren't failures to retrieve a relevant chunk. They're failures to retrieve enough of the relevant chunk for the downstream model to work with it.

A minimal implementation

Here's a small self-contained comparison in Python. It uses LangChain's RecursiveCharacterTextSplitter (the most-used chunker in production RAG), computes character-length-based chunks for a document, and then does a naive bag-of-words retrieval against the query so you can see which chunk wins without any embedding dependency.

# pip install langchain-text-splitters
from langchain_text_splitters import RecursiveCharacterTextSplitter
 
doc = """## Hippo facts
 
Hippos are the deadliest large mammal in Africa.
Adults weigh up to three tons.
 
Despite their bulk, hippos can run at nineteen miles per hour on land.
They can outrun most humans over short distances.
 
They spend most of the day submerged to stay cool."""
 
query = "how fast can a hippo run?"
 
# 1) fixed-size, no overlap (naive)
def fixed(text, size=80, overlap=0):
    step = size - overlap
    return [text[i:i+size] for i in range(0, len(text), step)]
 
# 2) recursive, paragraph-aware
recursive = RecursiveCharacterTextSplitter(
    chunk_size=200,
    chunk_overlap=20,
    separators=["\n\n", "\n", ". ", " ", ""],
)
 
# 3) parent-document: small children (80 chars), parent = whole section
def children_of(parent, size=80):
    return [parent[i:i+size] for i in range(0, len(parent), size)]
 
parents = doc.split("\n\n")                    # section-sized parents
parent_children = [(p, c) for p in parents for c in children_of(p)]
 
# Naive scoring: count query-word overlaps with the chunk
def score(chunk, q):
    qs = set(q.lower().split())
    cs = set(chunk.lower().split())
    return len(qs & cs)
 
def topk(chunks, q, k=1):
    return sorted(chunks, key=lambda c: score(c, q), reverse=True)[:k]
 
print("fixed-size top-1:")
print(topk(fixed(doc, 80), query)[0])
 
print("\nrecursive top-1:")
print(topk(recursive.split_text(doc), query)[0])
 
# For parent-doc: match on the child, return the parent
winner = max(parent_children, key=lambda pc: score(pc[1], query))
print("\nparent-document top-1 (child matched, parent returned):")
print(winner[0])

Run it and you'll see exactly the pattern the viz above showed: fixed-size returns a cut-off fragment, recursive returns the sentence-aligned match, parent-document returns the whole section. The retrieval scoring here is toy-level, but the shapes the different chunkers produce are representative, which is the part the post is about.

Misconceptions

"Bigger chunks are safer because they preserve more context." They preserve more context for the LLM to read, sure. But a bigger chunk's embedding averages over more content, which blurs the signal the retriever uses to find it. For factoid queries, bigger chunks frequently hurt retrieval even though they would help answering. Parent-document strategies exist precisely to break this trade-off: let the retriever match on small chunks, let the LLM read the big one.

"Semantic chunking is strictly better because it respects meaning." Semantic chunking is slower, more expensive to run, and in practice gives small improvements over a well-tuned recursive splitter on most corpora. The name makes it sound like the obvious choice, but the gap over recursive splitting is usually small once you actually measure. If you're starting from scratch, get recursive working and measure, then consider semantic chunking if the gap is worth the complexity.

"Overlap is wasteful; I'm just duplicating storage." Overlap is an insurance premium against bad boundary luck. A target fact that sits on a chunk boundary with zero overlap is effectively half-hidden from the retriever. With some overlap, either chunk can match. Storage is cheap; missed retrievals are expensive. A small amount of overlap (5–15%) usually pays for itself immediately.

"The right chunk size is a universal constant I can copy from a blog post." A "best chunk size" pulled from someone else's benchmark rarely lands in the same place on a different corpus. Chunk size interacts with document type, query distribution, embedding model, and retrieval top-k in ways that don't transfer. You have to evaluate on your own data. The good news is you don't have to sweep finely; usually a small grid search across 128 / 256 / 512 / 1024 tokens will tell you most of what you need to know.

What's next

This post covered why chunking matters as much as it does, the handful of strategies that do real work in production (fixed-size with overlap, recursive, parent-document, sentence-window), the task-dependent chunk-size sweet spot, metadata enrichment, and a teaser for late chunking.

Once your chunks are good, the next lever is what you do after the initial retrieve. The top-k hits from a vector search are rarely in the right order, and a second model can reorder them dramatically better. The next post covers rerankers: cross-encoder scoring, why a 100x slower model applied to 50 candidates beats a fast model applied to a million, and where rerankers sit in the two-stage retrieval pipeline most production systems run.

Additional reading (and watching)

• Unstructured.io. Chunking strategies documentation. Element-aware chunking for PDFs, HTML, Word, and spreadsheets; worth reading for the "chunk by title" and "chunk by page" primitives.

• LlamaIndex. SentenceWindowNodeParser documentation. First-class implementation of the sentence-window / parent-document pattern described here.

• Anthropic. (2024, September 19). Introducing Contextual Retrieval. Prepend LLM-generated context to each chunk before embedding and BM25 indexing; reports 35–67% reduction in retrieval failure on their internal benchmarks.

• Günther, M., et al. (2024). Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models. Jina AI. Run the full document through the encoder first, then pool token embeddings into chunks; preserves cross-chunk context at embed time.

• Chen, J., et al. (2024). Dense X Retrieval: What Retrieval Granularity Should We Use?. Empirical study of chunk-granularity trade-offs across several retrieval benchmarks; supports the "task-dependent sweet spot" claim.

• Chunkviz. chunkviz.up.railway.app. Greg Kamradt's interactive visualization of how different LangChain splitters carve up a document; useful for building intuition about what each splitter actually does.

• Gao, Y., et al. (2024). Retrieval-Augmented Generation for Large Language Models: A Survey. Broad survey that contextualizes chunking choices within the larger RAG pipeline, including advanced patterns like hierarchical and graph-based chunking.