Module 3: Chunking Strategies · RAG

You feed a 150-page technical manual into your vector store and retrieval falls apart: answers are vague, citations are wrong, and the system sometimes returns the index page when asked about a specific procedure. The problem is rarely the embedding model or the vector store. It is almost always chunking. By the end of this module you will understand four chunking approaches, their tradeoffs, and how to choose the right one for your document type.

Why Chunking Matters More Than Most People Think

An embedding model converts a chunk of text into a single vector. That vector represents the average meaning of the entire chunk. If a chunk contains three different topics, the vector is a mixture of all three, and no individual topic is represented well. Queries about any one topic will get a low similarity score, even though the relevant information is technically in the document.

The opposite problem is also real. Chunks that are too small lose context. A sentence like "This does not apply to remote employees" is meaningless without knowing what "this" refers to. Without surrounding context, the embedding places this sentence in a confused region of the semantic space, and retrieval fails.

Good chunking is the art of finding units of text that are semantically coherent, self-contained enough to be meaningful in isolation, and consistent enough in size to work with your context window budget. According to the survey of RAG techniques by Gao et al. (arXiv:2312.10997), chunking strategy is among the most impactful decisions in the indexing phase, with effects that compound at every subsequent stage of retrieval.

The Four Main Chunking Approaches

Approach 1

Fixed-Size

Split on token count (e.g., 512 tokens) with an overlap (e.g., 10%). Fast to implement. Works poorly when sentences or logical ideas cross chunk boundaries.

Approach 2

Sentence Boundary

Split at sentence endings detected by a sentence splitter. Ensures each chunk is grammatically complete. May produce very small chunks for technical documents with short sentences.

Approach 3

Recursive / Paragraph

Split at natural boundaries in priority order: paragraphs first, then sentences, then words. Keeps related ideas together while respecting a maximum size. The most commonly used approach in practice.

Approach 4

Document-Aware

Respect the document's own structure: sections, headings, tables, code blocks. Requires parsing the document format (Markdown, HTML, PDF). Produces the highest-quality chunks for structured documents.

The chunk size tradeoff in plain English Small chunks (64-256 tokens): precise retrieval, loses context. Large chunks (512-1024 tokens): keeps context, pollutes embeddings with multiple topics. The sweet spot depends on your document type. Narrative text: 300-512 tokens. Technical specs: 128-256 tokens. Legal documents: paragraph-aligned, variable size.

The Role of Overlap

Overlap means that adjacent chunks share some tokens. If chunk 1 is tokens 1-512 and overlap is 10%, chunk 2 starts at token 461, so tokens 461-512 appear in both chunks.

Overlap helps when key information sits near a chunk boundary. Without overlap, a sentence that crosses a boundary would be split in half and neither chunk would represent it fully. With 10% overlap, the boundary region appears in both neighboring chunks, increasing the chance that at least one of them retrieves correctly.

The cost of overlap is storage and computation: more chunks means a larger index and more embeddings to compute. A 10-20% overlap is the typical starting point. Beyond 30%, the redundancy usually outweighs the benefit.

Fig 3 · Four Chunking Approaches on the Same Document

Document-Aware Chunking

Standard splitters treat a document as a flat sequence of characters. Document-aware chunkers understand the document's structure. A Markdown-aware splitter, for example, splits at heading boundaries first, then at paragraph breaks within each section. This ensures that the heading context stays with the content it introduces.

For structured documents (product manuals, legal agreements, research papers), document-aware chunking can dramatically improve retrieval quality. A table in a financial report, for example, should probably be kept as a single chunk rather than split through its rows. A code block should never be split in the middle of a function.

Python · Recursive and Semantic Chunking with LangChain

from langchain.text_splitter import (
    RecursiveCharacterTextSplitter,
    MarkdownHeaderTextSplitter,
)

# --- Approach 1: Recursive character splitting ---
splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,          # target size in characters
    chunk_overlap=50,        # overlap between adjacent chunks
    separators=["\n\n", "\n", ". ", " ", ""],  # priority order
)
chunks = splitter.split_text(raw_text)
print(f"Produced {len(chunks)} chunks, avg {sum(len(c) for c in chunks)//len(chunks)} chars")

# --- Approach 2: Document-aware (Markdown) ---
md_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[
        ("#", "h1"),
        ("##", "h2"),
        ("###", "h3"),
    ]
)
header_splits = md_splitter.split_text(markdown_text)
# Each split retains its heading metadata for source attribution

# --- Approach 3: Sentence splitting ---
from langchain.text_splitter import SentenceTransformersTokenTextSplitter
sent_splitter = SentenceTransformersTokenTextSplitter(
    chunk_overlap=0,
    model_name="all-MiniLM-L6-v2",
    tokens_per_chunk=128,
)
sent_chunks = sent_splitter.split_text(raw_text)

Try this

Take a document you use at work (a policy, a manual, a proposal) and apply three different chunking strategies to the first three pages. Compare the results: which approach produces the cleanest chunks? Which splits a logical idea across two chunks? For each bad split you find, think about which splitter setting would fix it. This is the fastest way to develop chunking intuition for your specific document type.