Chunking Strategies That Make or Break RAG Quality

Most RAG systems underperform not because of the LLM or the vector database, but because of how documents were split before indexing. A chunk that cuts a sentence mid-thought, or lumps three unrelated paragraphs together, means the retriever will either miss the right passage or surface one that dilutes the signal. Teams blame the model when the real problem happened hours earlier in a preprocessing script nobody touched since launch.

Chunk boundaries determine what context the model ever gets to see. If a chunk spans a topic boundary or omits the sentence containing the actual answer, no amount of retriever tuning will fix it. Chunking is a retrieval-quality decision, not a one-time setup step.

At Laxaar, we've rebuilt the chunking layer in several RAG systems after teams hit recall walls they couldn't explain. The pattern is consistent: the first chunking approach was chosen for convenience (fixed size, easy to implement) and it was good enough to pass early tests but fell apart on real-world queries. This guide covers the strategies worth knowing, when each one fits, and the trade-offs you'll accept with each choice.

What you'll learn

• Why chunk size is a retrieval variable, not a config detail
• Fixed-size chunking: when it's fine and when it breaks
• Semantic and sentence-boundary chunking
• Recursive and structure-aware chunking
• Parent-child and hierarchical chunk strategies
• Late chunking and contextual retrieval
• Chunking strategy comparison
• How to test whether your chunking is working
• Frequently Asked Questions

Why chunk size is a retrieval variable, not a config detail

A chunk is the atomic unit your retriever scores against a query embedding. Too small and each chunk lacks enough context to be meaningful: the embedding becomes a floating fact. Too large and the embedding averages over multiple topics, reducing its similarity to any specific query.

The right size depends on your document type, query distribution, and embedding model. A 512-token chunk that works for dense technical documentation is probably wrong for legal contracts, where a single clause determines everything.

Overlap is an often-ignored second variable. A 10-20% token overlap lets adjacent chunks share boundary sentences, reducing the chance that a critical answer falls exactly on a split point.

Treat chunk size and overlap as hyperparameters you tune against a retrieval eval dataset, not values you set once from a blog post.

Fixed-size chunking: when it's fine and when it breaks

Fixed-size chunking splits documents into chunks of N tokens (or characters), optionally with overlap. It's the default in most RAG frameworks because it's simple to implement and fast to run.

from langchain_text_splitters import RecursiveCharacterTextSplitter

# Fixed-size with overlap — the most common starting point
splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=64,
    length_function=len,
)

chunks = splitter.split_text(document_text)

It works well when documents are fairly uniform in structure and density (think: short product descriptions, customer support tickets, news articles). When document structure is irrelevant to queries and text density is consistent, fixed-size chunking produces embeddings that are representative and comparable.

It breaks when documents have variable-density sections. A legal agreement might have a two-line clause that determines everything and a three-page recital that determines nothing. Fixed chunks treat both equally. A 512-token chunk that spans two clauses embeds a meaning somewhere between them, close enough to neither query to rank highly.

Fixed-size chunking also struggles with structured documents (tables, lists, code blocks), where a hard split mid-structure produces an incoherent fragment. For prototyping or uniform corpora it's reasonable. For production systems over mixed document types, treat it as your baseline to beat, not your final answer.

Semantic and sentence-boundary chunking

Semantic chunking groups sentences together until the embedding similarity between consecutive sentences drops below a threshold, treating that drop as a topic boundary.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

# Semantic chunking using embedding similarity between adjacent sentences
splitter = SemanticChunker(
    OpenAIEmbeddings(),
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=90,  # split at the top 10% of similarity drops
)

chunks = splitter.create_documents([document_text])

The appeal is that chunk boundaries align with topic shifts rather than arbitrary character counts. Each chunk tends to cover a single coherent idea. Embeddings are tighter and more discriminative.

The cost: semantic chunking is slower and more expensive. Every document requires embedding individual sentences to find split points. For a corpus of millions of documents, this can mean meaningful extra preprocessing time and embedding API spend.

Sentence-boundary chunking is a simpler middle ground: split on sentence endings and then group sentences into chunks up to a target size. No embedding step required, just a sentence tokenizer. It's significantly better than fixed-size for prose documents and adds almost no complexity.

import nltk
nltk.download("punkt_tab")
from nltk.tokenize import sent_tokenize

def sentence_aware_chunks(text: str, max_tokens: int = 400) -> list[str]:
    sentences = sent_tokenize(text)
    chunks, current, current_len = [], [], 0
    for sentence in sentences:
        token_est = len(sentence.split())
        if current_len + token_est > max_tokens and current:
            chunks.append(" ".join(current))
            current, current_len = [], 0
        current.append(sentence)
        current_len += token_est
    if current:
        chunks.append(" ".join(current))
    return chunks

For most prose-heavy corpora, sentence-boundary chunking at 300-500 tokens is the right starting point before investing in full semantic chunking.

Recursive and structure-aware chunking

Recursive chunking tries a hierarchy of separators in order (paragraphs, then sentences, then words), falling back to the next separator only when a section exceeds the target chunk size. LangChain's RecursiveCharacterTextSplitter follows this approach.

Structure-aware chunking goes further: it parses the document's actual structure (Markdown headings, HTML tags, PDF section breaks, code blocks) and uses those structural boundaries as primary split points.

from langchain_text_splitters import MarkdownHeaderTextSplitter

headers_to_split_on = [
    ("#", "h1"),
    ("##", "h2"),
    ("###", "h3"),
]

splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=headers_to_split_on,
    strip_headers=False,
)

# Each chunk preserves its header context as metadata
chunks = splitter.split_text(markdown_document)
# chunks[0].metadata == {"h1": "Section Title", "h2": "Subsection"}

The metadata attached to each chunk is particularly valuable. When the retriever returns a chunk, you know which section it came from. You can then include that section path in the context sent to the LLM, improving answer grounding without enlarging the chunk itself.

Structure-aware chunking suits technical documentation and knowledge bases where the document's own hierarchy is meaningful. For PDF-heavy corpora, it requires a capable parser (unstructured.io, pdfplumber) before structural splits are possible.

Parent-child and hierarchical chunk strategies

Parent-child chunking indexes two levels of granularity simultaneously. Small child chunks (128-256 tokens) go into the vector index: specific enough to match narrow queries closely. Large parent chunks (512-1024 tokens or full sections) are what the LLM actually reads, with enough surrounding context for the model to answer well.

from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryStore
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma

# Child splitter: small chunks for retrieval precision
child_splitter = RecursiveCharacterTextSplitter(chunk_size=200)
# Parent splitter: larger chunks for generation context
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=800)

store = InMemoryStore()
vectorstore = Chroma(embedding_function=embeddings)

retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=store,
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)
retriever.add_documents(documents)

This approach separates retrieval precision from generation context. That's arguably the most important architectural insight in RAG chunking. The retriever finds the right needle; the LLM reads the surrounding haystack.

The trade-off is storage cost and index complexity: two copies of every document at different granularities, plus a docstore to manage the parent-child relationship. For large corpora, that overhead is real.

The Laxaar team uses parent-child chunking as the default for AI engineering projects where document density varies significantly: legal docs, research papers, technical specifications. For uniform corpora like FAQs or short-form content, the added complexity doesn't pay off.

Late chunking and contextual retrieval

Late chunking (from the JinaAI team) embeds the full document first, then splits the resulting token embeddings into chunk-level pooled representations. Each chunk embedding inherits contextual signal from the whole document. A paragraph that opens with "As described above..." doesn't embed as a meaningless fragment.

Contextual retrieval (introduced by Anthropic) achieves a similar goal differently: before indexing, prepend each chunk with a short LLM-generated summary of where it sits within the document.

import anthropic

client = anthropic.Anthropic()

def add_chunk_context(document: str, chunk: str) -> str:
    response = client.messages.create(
        model="claude-3-haiku-20240307",
        max_tokens=120,
        messages=[{
            "role": "user",
            "content": (
                f"<document>{document}</document>\n"
                f"<chunk>{chunk}</chunk>\n"
                "In 1-2 sentences, describe where this chunk fits in the document "
                "and what broader topic it belongs to. Be concise."
            ),
        }],
    )
    context = response.content[0].text
    return f"{context}\n\n{chunk}"

# Prepend context before embedding
contextual_chunks = [add_chunk_context(doc_text, c) for c in raw_chunks]

The cost is LLM calls per chunk. For corpora that change infrequently, it's a one-time preprocessing expense worth paying. For high-throughput corpora, weigh it against the recall improvement your eval actually shows.

Chunking strategy comparison

Strategy	Best for	Retrieval quality	Implementation cost	Indexing speed
Fixed-size	Uniform corpora, prototypes	Baseline	Low	Fast
Sentence-boundary	Prose documents	Good	Low	Fast
Semantic	Topic-varied documents	High	Medium	Slow (embedding step)
Structure-aware	Docs with rich hierarchy	High	Medium-High	Medium
Parent-child	Variable-density corpora	Very high	High	Medium
Contextual retrieval	High-stakes recall	Very high	High (LLM cost)	Slow

No single strategy wins across all use cases. The right choice depends on your document type, query distribution, and tolerance for indexing complexity. Start with sentence-boundary chunking, establish a retrieval eval dataset, and upgrade only when the eval shows a meaningful gap.

How to test whether your chunking is working

Chunking without a retrieval eval is guesswork. The minimum viable eval is a set of question-answer pairs derived from your actual documents, where you know which chunk contains the ground-truth answer.

# Minimal retrieval eval: does the right chunk appear in top-k results?
def recall_at_k(retriever, eval_pairs: list[dict], k: int = 5) -> float:
    hits = 0
    for pair in eval_pairs:
        results = retriever.invoke(pair["question"])
        retrieved_texts = [r.page_content for r in results[:k]]
        # Check if any retrieved chunk contains the ground-truth answer span
        if any(pair["answer_span"] in text for text in retrieved_texts):
            hits += 1
    return hits / len(eval_pairs)

# Run against two chunking strategies, compare recall@5
print(f"Fixed-size recall@5: {recall_at_k(fixed_retriever, eval_pairs):.2%}")
print(f"Sentence-boundary recall@5: {recall_at_k(sentence_retriever, eval_pairs):.2%}")

Recall@5 is the signal that matters most. Below 0.7, chunking is almost certainly part of the problem. Above 0.85, look at reranking before revisiting chunk strategy.

Build a quick eval dataset with an LLM: sample 50-100 chunks, generate a question answerable only from that chunk, store the chunk as ground truth. An afternoon's work that saves weeks of debugging.

The custom software development analogy holds: don't ship a chunking strategy without a retrieval eval. On every Laxaar RAG project, a recall eval dataset is a non-negotiable deliverable before we consider the retrieval layer production-ready.

Frequently Asked Questions

What chunk size should you start with?

For prose documents, 300-500 tokens with a 10-15% overlap is a reasonable starting point. For technical or structured documents with rich hierarchy, use structure-aware splitting and let the document's own section breaks define chunk boundaries rather than a fixed token count. Always validate against a retrieval eval rather than treating any default as correct.

Does overlapping chunks cause duplicate retrieval?

It can when the query closely matches the overlapping region. Most production systems add a deduplication step after retrieval, returning the top-k unique parent chunks rather than raw results. With parent-child chunking, deduplication happens naturally: multiple child chunks pointing to the same parent collapse into a single context.

How does chunking interact with reranking?

Reranking re-scores retrieved chunks using a cross-encoder that directly compares query and chunk text. A strong reranker can push the right chunk from position 8 to position 1, but it can't fix a chunk that contains only half the answer. Good chunking and good reranking are complementary, not substitutes.

Should chunk metadata be stored in the vector store?

Yes. At minimum, store the source document ID, section path (if structure-aware chunking), and character offsets within the original document. This metadata serves two purposes: filtering (limit retrieval to a specific document or section), and citation (show the user exactly where the retrieved content came from). Most production RAG systems rely on metadata filtering to scope retrieval by user, tenant, or document set.

When is contextual retrieval worth the LLM cost?

When retrieval recall is the bottleneck and your corpus changes infrequently. A fixed knowledge base, a legal library, a set of internal policies: the LLM cost is a one-time preprocessing expense. For high-write corpora, measure the recall gain from your eval before committing to it. For high-value use cases (customer-facing answers, compliance queries), it usually pays off.

---

If you're building a RAG pipeline and hitting recall walls you can't explain, the chunking layer is the first place we'd look. The Laxaar team has rebuilt chunking strategies on several production AI systems. Reach out and we can help diagnose whether your retrieval quality problem is a chunking problem, a reranking problem, or something else entirely. See our generative AI development work for examples of what we've shipped.