How to Evaluate RAG Chunking in 2026 - Test Chunk Size and Strategy

Chunking is the least glamorous knob in a RAG pipeline and one of the most consequential. How you split documents - the size, the overlap, the strategy - decides what your retriever can possibly find. Get it wrong and no embedding model, reranker or prompt will save you, because the answer was never in the chunks you could retrieve.

And yet most teams set chunk size to whatever the tutorial used, ship it, and never test it. This guide fixes that. It shows how to measure chunking impact with retrieval metrics, run a proper parameter sweep, and choose settings on evidence. If you want the broader RAG picture first, our guide on how to evaluate RAG covers the whole pipeline; this one zooms into the chunking layer.

Why chunking is a retrieval problem, not an answer problem

The first mental shift. Chunking failures show up in retrieval metrics long before they show up in bad answers, so if you only grade the final response you will misdiagnose them.

Here is the causal chain. Your documents get split into chunks. Chunks get embedded and stored. A query retrieves the top-k nearest chunks. Those chunks become the context the model answers from. If the chunking is wrong, the right context never enters the pipeline - and a model asked to answer without the facts either hallucinates or hedges. You will spend a week tuning the prompt when the real problem was that the answer got split across two chunks and only half of it was retrieved.

So the whole discipline of chunking evaluation is: measure the retrieval stage in isolation, and read the two failure signatures.

The two failure signatures

Chunking goes wrong in exactly two directions, and each has a metric that catches it.

Chunks too small or fragmented - low context recall. When you split too aggressively, the passage that answers a question gets scattered across chunks, and a top-k retrieval grabs some but not all of it. The missing piece is what context recall measures: did you retrieve everything needed to answer. Low recall with small chunks means you are fragmenting answers.

Chunks too large - low context precision. When you split too coarsely, each chunk is a big block where a sentence of signal floats in paragraphs of filler. The retriever pulls relevant chunks, but they are mostly noise, and the model has to find the answer inside the clutter. Context precision measures whether the useful content ranked near the top rather than being buried. Low precision with large chunks means you are drowning the signal.

For the full definitions and how they differ, see our breakdown of context precision versus recall and the context precision glossary entry. The trade-off between them is the whole game - smaller chunks push precision up and recall down, larger chunks do the reverse, and your job is to find the balance point on your data.

Step 1: Build a fixed eval set with source labels

You need something to measure against. Assemble 30 to 50 real questions, and for each, label the source passage that actually answers it - not just the correct answer, but which part of which document it should come from. That source label is what makes context recall computable, because recall asks whether you retrieved the passage you know is correct.

This is more work than an answer-only set, and it is worth it. Without source labels you can score precision-ish signals but not recall, and recall is exactly the metric that catches undersized chunks. The RAG evaluation metrics guide explains what each score needs as input.

Step 2: Sweep chunk parameters, holding everything else fixed

Now the experiment. Vary only the chunking, keep everything else constant - same embedding model, same top-k, same generation prompt - so any change in retrieval metrics is attributable to the chunking alone.

Sweep across a grid like this:

Variable	Values to try
Chunk size	128, 256, 512, 1024 tokens
Overlap	0%, 10%, 20%
Strategy	fixed-size, sentence, recursive, semantic

For each combination, re-index your corpus, run the same eval set, and record context recall and context precision. Do not change top-k or the embedding model mid-sweep - if you move two knobs at once you cannot tell which one moved the metrics, and the experiment is wasted.

Step 3: Score each configuration and read the curve

Run the eval and plot recall and precision against chunk size. You will typically see recall rise then plateau as chunks get bigger (more context per chunk means fewer fragmented answers), while precision falls (bigger chunks carry more noise). The winning configuration is the one at the knee of the curve - the largest recall you can get before precision falls off a cliff.

Two tools make this loop fast:

• Arize Phoenix is my default here. It ships 50+ pre-built eval metrics with the best RAG evaluation in the category, so context recall and precision are ready to run against your traces without writing judge prompts. It is OpenTelemetry-native and runs locally in under a minute, so re-scoring each chunk configuration is quick. Just note the server is Elastic License 2.0, source-available, if you ever plan to resell it.
• DeepEval fits teams that want the sweep in a Python test suite - you parameterize the chunk config, assert on the retrieval metrics, and run deepeval test run. The OSS framework is Apache-2.0 and free. Cost-model it, since nearly all metrics are LLM-as-judge and a full sweep multiplies the judge calls.

Step 4: Lock it in and re-test on drift

Once the sweep names a winner, set those chunk parameters and add the eval to your regression suite. Corpora grow and change, and a chunk size that was optimal for last quarter’s documents can degrade as the content shifts - new document types, longer pages, different structure. Re-run the chunking eval whenever you materially change the corpus, and keep the retrieval metrics on your CI dashboard so a re-indexing change that quietly hurts recall gets caught.

Bottom line

Stop guessing chunk size. Chunking is a retrieval problem, so measure it with retrieval metrics - low context recall means chunks too small, low context precision means chunks too large - and find the balance with a proper sweep that varies only the chunking. Build a source-labeled eval set, grid over size, overlap and strategy, score each with Phoenix or DeepEval, and lock in the configuration at the knee of the precision-recall curve. If you want to see how the full field of tools compares for this kind of work, the best RAG evaluation tools roundup ranks them. The knob is small; the payoff is the difference between a retriever that finds the answer and one that never could.