How to Generate Synthetic Data for LLM Evaluation in 2026

The single most common reason teams never start evaluating their LLM app is not that they do not care. It is that they have no labeled data to evaluate against, and building one by hand feels like a project they will get to later. Later never comes.

Synthetic data breaks that stalemate. Instead of waiting for user traffic or paying people to write test cases, you generate a golden dataset from material you already have - your own documents, your own schema, a few seed examples. This guide covers how to generate eval data that is actually useful, the quality traps that make synthetic sets worthless, and the tools that ship a synthesizer so you are not building the generator from scratch. If you want the concepts first, the eval dataset and ground truth glossary entries define what you are building.

When synthetic data is the right move

Be clear about what problem this solves. Synthetic data is a cold-start tool - it gets you a first golden set when you have zero labeled examples, which is exactly when most evaluation efforts stall.

It shines in three situations:

• Bootstrapping - you have a new RAG app and no user traffic yet, but you have the document corpus, so you can generate questions the app should answer.
• Coverage gaps - production traffic clusters around common cases, and you need edge cases and rare scenarios that users have not sent yet.
• Regression sets - you want a fixed, repeatable battery to run on every change, and generating one is faster than curating from logs.

What synthetic data does not do is replace real user data long term - real queries are messier and surface failures you would never think to invent. The mature pattern seeds with synthetic and continuously mixes in curated real traces. Our guide on how to build an LLM eval pipeline covers where the golden set fits in the wider loop.

How generation actually works

The core technique for RAG is document-grounded generation, and it is worth understanding the mechanics. You feed a source document to a generator model and ask it to produce a realistic question that the document answers, plus the grounded answer - so you get an input, an expected output, and the context all at once.

A good synthesizer does more than one-shot this:

1. Chunk and sample the source corpus so questions span the whole document set, not just page one.
2. Generate diverse question types - factual lookups, multi-hop questions that need two passages, comparison questions, and “not answerable from this document” cases that test whether the app abstains.
3. Evolve difficulty - take a simple generated question and rewrite it to be harder, more specific, or more conversational, so your set is not all softballs.
4. Attach the ground truth - the source chunk and the expected answer, so the set supports reference-based metrics like context recall and answer correctness.

The goal is not volume, it is diversity - a hundred varied, grounded cases catch more regressions than a thousand rephrasings of the same easy question.

The quality traps to avoid

Synthetic data earns its bad reputation when teams generate and never look. Four traps, and the fix for each:

• Repetition - generators fall into a groove and produce near-duplicate questions. Fix by sampling widely across the corpus and forcing question-type diversity.
• Too easy - one-shot generation tends to produce trivial factual questions your app aces, which tells you nothing. Fix with the difficulty-evolution step.
• Subtle wrongness - the generated “correct” answer is occasionally wrong, and an eval set with wrong ground truth is worse than none. Fix with human review.
• Distribution mismatch - synthetic questions read like a textbook, real users write like they are in a hurry. Fix by seeding generation with real examples so it mimics your actual traffic style.

The through-line is human-in-the-loop. Generate to draft, then have a person filter and lightly edit - reviewing a generated case takes seconds, writing one from scratch takes minutes, so you keep most of the speed and remove the garbage.

The tools that ship a synthesizer

Two tools make this practical rather than a bespoke prompting project.

DeepEval has the strongest built-in synthesizer of the pair. Its Synthesizer generates golden test cases directly from your documents, with the diversity and difficulty-evolution steps built in, and the output drops straight into its pytest-style eval suite - you generate a dataset, then assert your metrics against it and run deepeval test run. The OSS framework is Apache-2.0 and free, so you can generate and score locally with no account. As always with DeepEval, the metric scoring is LLM-as-judge, so cost-model the eval runs once the set gets large.

Braintrust is the pick if you want generation wired into a managed dataset and experiment system. Its dataset management sits in the same platform as its evals and CI/CD quality gates, so a generated golden set feeds directly into experiments and into merge-blocking regression checks. You write custom scorers as plain functions and its autoevals library gives you working ones out of the box. Watch the processed-data billing meter, since large generated datasets with verbose contexts are what burn the GB allowance.

Here is the shape of DeepEval’s synthesizer (illustrative):

# illustrative - DeepEval Synthesizer
from deepeval.synthesizer import Synthesizer

synthesizer = Synthesizer()
goldens = synthesizer.generate_goldens_from_docs(
    document_paths=["./docs/handbook.pdf", "./docs/faq.md"],
    max_goldens_per_context=3,   # diversity per chunk
)
# review, filter, then evaluate your app against `goldens`

Step by step

The end-to-end process:

1. Gather source material - the documents, schema, or a handful of real seed examples your app works over.
2. Generate with a synthesizer, forcing question-type diversity and difficulty evolution.
3. Review and filter - a human pass to cut duplicates and fix wrong ground truth. This step is non-negotiable.
4. Score your app against the set with your chosen metrics, and add it to CI as a regression battery. Our guide on how to run LLM regression tests covers wiring it in.
5. Grow it from production - as real traffic arrives, curate the interesting failures into the set so it drifts toward reality.

Bottom line

No labeled data is a reason to start evaluating, not a reason to wait. Generate a first golden set from your own documents with a synthesizer, force diversity and difficulty instead of chasing volume, and put a human review pass between generation and use so wrong ground truth never poisons your metrics. Reach for DeepEval for the strongest built-in synthesizer in a free Python framework, and Braintrust if you want the generated set wired to managed experiments and CI gates. Seed with synthetic to break the cold-start, then keep feeding the set real traffic - the bootstrap gets you evaluating this week instead of never.