Chunking Strategies

A retrieved chunk can contain the right keyword and still omit the sentence needed to answer. Chunking decides what the search system can return as a unit, so inspect the boundaries before blaming the encoder.

For a failed query, locate its supporting passage, and see whether the indexed chunks preserve that passage and its subject. Changing the embedding model cannot restore text discarded during parsing or splitting.

We will use the earlier embedding model to represent each indexed chunk and rank it against a query.

The problem chunking solves

An embedding model has a finite input limit. Longer inputs may be rejected or truncated, depending on the library. Chunking keeps each indexed input within that limit; the appropriate limit comes from the model’s documentation.

There is also a retrieval reason to split. A single vector represents a whole input, while a query may concern one small part of it. Smaller retrieval units can isolate that part, at the cost of losing surrounding context.

A small chunk may lose the subject or condition needed to interpret it. A large chunk may mix unrelated topics and make a specific passage harder to retrieve. Test boundaries against the evidence your queries require.

Fixed-size with overlap: the baseline

Choose a chunk size (say, 500 tokens), decide on an overlap (say, 50 tokens), and slide a window across the document. Each full chunk is 500 tokens long; the final chunk may be shorter. Consecutive chunks overlap by 50.

Overlap can put a boundary-spanning sentence into at least one complete chunk, if the sentence and overlap sizes permit it. It also duplicates indexed content. Adjacent-chunk expansion is another way to recover surrounding evidence.

Fixed-size splitters may count characters or model tokens. Those units are not interchangeable, and neither automatically respects sentence boundaries.

Recursive and semantic chunking

A recursive splitter tries stronger boundaries before weaker ones, splitting oversized pieces further. LangChain’s documented default separator order is \n\n, \n, space, then the empty string. You can configure sentence separators too; the tree below illustrates a custom hierarchy.

“Semantic chunking” instead uses embedding similarity to choose boundaries. Instead of using character patterns as proxies for meaning, it embeds sentences one at a time, compares consecutive sentence embeddings, and splits wherever the cosine similarity drops below a threshold. The logic is: "when the meaning shifts, cut here." Boundary detection adds embedding work, and its threshold needs evaluation. Compare it with a simpler recursive splitter before paying that cost across the corpus.

Document-structure-aware chunking starts from parsed elements such as headings, paragraphs and tables. This can preserve useful boundaries, but parser errors and size limits still matter; it does not guarantee that every table or section remains intact.

Parent-document: small for matching, big for context

You can chunk the same document at two different granularities and use them for different jobs.

Index the small chunks. Retrieve on the small chunks. But when you hand results to the LLM, return the parent chunk, the bigger block that the small chunk came from.

A small unit can give retrieval a more specific target. It may also be too ambiguous to match without its heading or neighboring text. Compare retrieval at several granularities rather than assuming one sentence is always optimal.

If an answer depends on a condition or pronoun reference outside the retrieved sentence, returning that sentence alone omits necessary evidence. Parent expansion can restore it, while also adding text the answer may not need.

Parent-document retrieval separates the indexed unit from the returned unit. A sentence-window variant retrieves a sentence and expands to nearby sentences. Store the parent or neighbor identifiers explicitly and deduplicate overlapping expansions.

The chunk-size sweet spot is task-dependent

A factoid query may need only one sentence plus the heading that identifies its subject. A small retrieval unit can work well if it preserves both.

A summary query needs broader coverage. Larger returned sections, multiple retrieval queries or hierarchical summaries can supply it; a single large chunk is not automatically sufficient.

For mixed workloads, compare a fixed-size baseline with parent expansion or multiple retrieval granularities. Count duplicated tokens and evaluate answer support alongside retrieval recall.

A reported best chunk size belongs to a particular corpus, query set, tokenizer and retrieval pipeline. Treat it as a starting hypothesis and compare sizes on your own held-out queries.

Metadata enrichment

Another option is to make the chunk’s source context explicit. Before you embed a chunk, prepend the stuff that locates it: the document title, the section heading, the breadcrumb path, the date.

“Replace it every thirty days” is ambiguous alone. “Example pump manual › Maintenance › Inlet filter” supplies a subject the chunk lost. The title must belong to that passage; incorrect metadata can create a confident wrong match.

Anthropic’s Contextual Retrieval generates a short document-specific context for each chunk before embedding and lexical indexing. Their report evaluates particular datasets and retrieval configurations. Check that each generated description accurately represents its source passage.

Include useful context deliberately, keeping authoritative source metadata separate from generated descriptions.

A teaser: late chunking

Everything we've looked at so far embeds each chunk in isolation. The embedding model reads the chunk, produces a vector, and never sees any text outside the chunk boundary.

Late chunking runs a compatible long-context embedding model over a longer text, then pools token states within the desired chunk boundaries. The token representations can incorporate context outside each final chunk. The document still has to fit the encoder’s supported input strategy.

A pronoun’s token state may then carry information from an earlier sentence. Whether this helps retrieval depends on the model, pooling and corpus; the method adds longer-sequence encoding work.

This moves chunk pooling after contextualization. It is different from generating a textual summary or prepending a breadcrumb before separate chunk encoding.

Worked example

The worked example uses a fictional pump manual. We inspect actual split strings and whether they retain the complete answer sentence.

The fixed boundary cuts the answer sentence in this example. The paragraph and parent preserve it.

Keep evidence preservation and retrieval ranking separate when debugging. First ask whether any indexed/expanded unit can supply the answer. Then ask whether the retriever finds it.

A minimal implementation

The Python sketch uses the same fictional source. Its overlap scorer is deliberately lexical; it is useful for inspecting strings, not a replacement for an embedding evaluation.

# pip install langchain-text-splitters
from langchain_text_splitters import RecursiveCharacterTextSplitter
 
doc = """## Example pump manual
 
The pump accepts clean water only.
 
Disconnect power before opening the casing. Replace the inlet filter every thirty days. Record the replacement date in the service log."""
 
query = "replace inlet filter"
 
# 1) Fixed character windows.
def fixed(text, size=80, overlap=0):
    if size <= 0 or not 0 <= overlap < size:
        raise ValueError("require size > overlap >= 0")
    chunks = []
    for start in range(0, len(text), size-overlap):
        chunks.append(text[start:start+size])
        if start+size >= len(text):
            break
    return chunks
 
# 2) Paragraph-aware recursive splitting, measured in characters.
recursive = RecursiveCharacterTextSplitter(
    chunk_size=200, chunk_overlap=0,
    separators=["\n\n", "\n", ". ", " ", ""],
)
 
# 3) Retrieve a child, return the containing section.
def children_of(parent, size=80):
    return fixed(parent, size)
 
parents = [doc]  # this short manual is one parent section
parent_children = [(p, c) for p in parents for c in children_of(p)]
 
# Toy lexical overlap, with punctuation removed.
import re
def score(chunk, q):
    qs = set(re.findall(r"\w+", q.lower()))
    cs = set(re.findall(r"\w+", chunk.lower()))
    return len(qs & cs)
 
def topk(chunks, q, k=1):
    return sorted(chunks, key=lambda c: score(c, q), reverse=True)[:k]
 
print("fixed-size top-1:")
print(topk(fixed(doc, 80), query)[0])
 
print("\nrecursive top-1:")
print(topk(recursive.split_text(doc), query)[0])
 
# For parent-doc: match on the child, return the parent
winner = max(parent_children, key=lambda pc: score(pc[1], query))
print("\nparent-document top-1 (child matched, parent returned):")
print(winner[0])

Inspect the output and the full split lists. The fixed character fragment may match all three query words while losing the interval; the larger returned units preserve it.

Misconceptions

"Bigger chunks are always safer." They preserve more source context but also return more material. Measure whether the relevant evidence is retrieved and used, rather than treating size as a quality score.

"Semantic chunking is strictly better." It adds a model-dependent boundary decision and encoding cost. Compare it with simpler splitters on the same documents and questions.

"Overlap is only wasted storage." It can preserve boundary-spanning evidence, but it duplicates work and can fill a top-k list with near-duplicates. Size it deliberately and deduplicate retrieved spans.

"The right chunk size is a universal constant I can copy from a blog post." A "best chunk size" pulled from someone else's benchmark rarely lands in the same place on a different corpus. Chunk size interacts with document type, query distribution, embedding model, and retrieval top-k in ways that don't transfer. You have to evaluate on your own data. A coarse sweep such as 128 / 256 / 512 / 1024 tokens can provide a starting comparison; refine it if the results justify doing so.

What's next

The next post adds a second scoring stage to the retrieved candidate set, where query and passage can be read together.

Additional reading (and watching)

• Unstructured.io. Chunking strategies documentation. Element-aware chunking for PDFs, HTML, Word, and spreadsheets, including title and page boundaries.

• LlamaIndex. SentenceWindowNodeParser documentation. First-class implementation of the sentence-window / parent-document pattern described here.

• Anthropic. (2024, September 19). Introducing Contextual Retrieval. Prepend LLM-generated context to each chunk before embedding and BM25 indexing; reports 35–67% reduction in retrieval failure on their internal benchmarks.

• Günther, M., et al. (2024). Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models. Jina AI. Run the full document through the encoder first, then pool token embeddings into chunks; preserves cross-chunk context at embed time.

• Chen, T., et al. (2024). Dense X Retrieval: What Retrieval Granularity Should We Use?. Compares retrieval granularities including propositions.

• Chunkviz. chunkviz.up.railway.app. Greg Kamradt's interactive visualization of how different LangChain splitters carve up a document; useful for building intuition about what each splitter actually does.

• Gao, Y., et al. (2024). Retrieval-Augmented Generation for Large Language Models: A Survey. Broad survey that contextualizes chunking choices within the larger RAG pipeline, including advanced patterns like hierarchical and graph-based chunking.