RAG chunking strategies, chunk size and overlap explained

The main RAG chunking strategies are fixed size, recursive, structure aware (split at headings, keep tables whole), semantic chunking, and parent document retrieval, where small chunks are searched and larger ones are returned. For most business documents, start with a structure aware or recursive splitter producing chunks of a few hundred tokens, then test against real questions. Published evaluations find that semantic chunking rarely pays for its extra cost, and that default chunk sizes and overlaps are often not the best choice.

This is a Labs note on what the library docs and published tests say. We quote no benchmark of our own here.

Blue and red hanging file folders in a drawer — Most business documents already have sections; structure aware chunking splits along them instead of at an arbitrary character count.

Why chunking decides what the model sees

In retrieval-augmented generation, each chunk becomes one embedding stored in a vector database, and only the top few chunks reach the model. Chunk too large and one vector blurs several topics, so a page about leave, travel and expenses matches all three weakly. Chunk too small and the fact loses its context: the value without its table heading, the clause without its number. There is also a hard ceiling: Microsoft's docs give 8,191 tokens as the input limit for the text-embedding-3-small model, and longer text is cut off.

The five strategies

Strategy	How it splits	Examples in the libraries
Fixed size	Every N characters or tokens, with optional overlap	LangChain CharacterTextSplitter, TokenTextSplitter
Recursive	Tries paragraphs, then lines, then words, until pieces fit	LangChain RecursiveCharacterTextSplitter
Structure aware	At headings, sections, HTML tags or code blocks; tables kept apart	MarkdownHeaderTextSplitter, LlamaIndex MarkdownNodeParser, Unstructured by_title
Semantic	Where the meaning of consecutive sentences shifts	LlamaIndex SemanticSplitterNodeParser
Parent document	Searches small chunks, returns their larger parent	LangChain ParentDocumentRetriever, LlamaIndex AutoMergingRetriever

LangChain's docs advise starting with the recursive splitter, whose default separators are a blank line, a line break, a space, then nothing. Unstructured's by_title strategy never puts text from two sections into one chunk, and it always isolates a table as its own element, splitting it only when it is too large. LlamaIndex's semantic splitter embeds each sentence and starts a new chunk where the difference between neighbours passes the 95th percentile, which is its default. Its hierarchical parser builds three levels by default, 2,048, 512 and 128 tokens, so the auto merging retriever can swap a cluster of small hits for their parent.

What chunk size and overlap should you start with?

The defaults disagree, and some count characters while others count tokens:

LangChain's base splitter: 4,000 characters with 200 overlap.
OpenAI's file search: 800 tokens with 400 overlap.
Azure AI Search guidance: start at 512 tokens with 25% overlap.
Unstructured: at most 500 characters, no overlap.

Chroma's July 2024 technical report compared splitters on about 470 generated questions over five corpora. The recursive splitter at 200 tokens with no overlap performed well across metrics, while OpenAI's 800 token, 400 overlap default, tested as a plain token splitter, gave slightly below-average recall and the lowest scores on the other measures. Less overlap also improved scores that penalise repeated text in the results. The authors note their dataset is small and the questions machine-written. Anthropic's 2024 retrieval write-up likewise says chunks are usually a few hundred tokens or less.

A reasonable start: 200 to 500 tokens, little or no overlap when you split at natural boundaries, and 10% to 25% overlap only when fixed size cuts fall mid-sentence.

Is semantic chunking worth the cost?

The evidence is mixed. A 2024 paper by Qu, Tu and Bao, "Is Semantic Chunking Worth the Computational Cost?", concluded that the extra computation is not justified by consistent gains, with fixed size chunking as good or better on ordinary documents. In Chroma's report, a clustering based semantic chunker did well, but Greg Kamradt's semantic chunker at its default settings scored slightly below average. Try semantic chunking only where documents have no structure to follow: call transcripts, chat logs, long email threads.

How to pick by document type

Document type	Start with
Policies, handbooks, SOPs	Split at headings; recursive inside long sections; heading path in metadata
Contracts	Split at clause numbers; contract name and clause number on every chunk; parent retrieval returns the whole clause
Manuals with tables	Tables whole, with their title and column headings repeated; model number and revision in metadata
Scanned PDFs	OCR first, then by heading or page; keep the page number
FAQs and help articles	One question and its answer per chunk
Transcripts and chat logs	Recursive by speaker turn, or semantic
Spreadsheets and database tables	Do not chunk as text; give the assistant a query tool

An open book lying on a wooden table in warm light — Tables in manuals are where fixed size chunking does most damage, cutting values away from their column headings.

Give each chunk its context back

A chunk that says "the limit is 30 days" is useless if retrieval cannot tell which policy it came from. The cheap fix is to prepend the document title and heading path to every chunk before embedding; Azure's guidance suggests adding the title to chunks from the middle of long documents. Anthropic's contextual retrieval goes further, with a model writing a short explanation for each chunk. On their datasets, top-20 retrieval failures fell by 35% with contextual embeddings, by 49% adding contextual keyword search, and by 67% adding a reranker.

Test, change one thing, test again

Write 50 or more real questions with the passage that answers each, measure how often that passage lands in the top 5, then change one setting at a time. Every change of chunking means re-embedding the whole collection, so settle it before you load years of files.

Our RAG chatbot builds parse document structure first and score chunking changes against a question set written with your team.

If your documents are a mix of scanned PDFs, Word policies and spreadsheets and you are unsure where to cut them, send us a sample and we will suggest a starting point. Book a consultation.