Building Evaluation Datasets

Datasets are the whole game

You can have the best eval framework and the smartest scorers, and none of it matters if your dataset is weak. The dataset is the ground truth your entire evaluation rests on. A suite that scores 95 percent on cases that do not reflect real usage tells you nothing except that you built an easy test.

This chapter is about building datasets that actually catch problems. It is the least glamorous part of LLM evaluation and the part that most determines whether your evals are worth running. The how to build an LLM eval pipeline post covers the surrounding machinery - here we focus on the data itself.

Anatomy of a good test case

At minimum, a test case is an input and a way to judge the output. In practice a good row carries more:

• Input - the actual prompt variables or user message. Real phrasing, not idealized.
• Expected output or criteria - either a reference answer or the rubric a good answer must meet.
• Context - for retrieval systems, the documents that should have been retrieved, so you can test grounding.
• Metadata - tags like category, difficulty, or source, so you can slice scores later.

That metadata is more important than it looks. When your overall score drops, tags let you ask “did it drop on refund questions or on billing questions?” Without them you have one number and no idea where the regression lives.

Where cases come from

There are four sources, and a mature dataset draws from all of them.

Real production traffic. The highest-value source. These are inputs your users actually sent, so they reflect true distribution and phrasing. As covered in the offline vs online chapter, the core habit is turning production traces into test cases. Every bug a user hits should become a permanent row.

Hand-written cases. Early on, before you have traffic, you write cases by hand. Think adversarially - what inputs would break this? Write the empty input, the hostile input, the input in another language, the input that asks for something out of scope.

Synthetic generation. Use an LLM to expand coverage. Give it a few real examples and ask for twenty variations, or ask it to generate edge cases for a category. This is fast and great for breadth, but treat generated expected answers with suspicion - a model writing both question and answer can encode its own blind spots. Verify the important ones by hand.

Public benchmarks. Useful for calibration and for capability tests, but rarely a substitute for domain-specific cases. Your users do not send benchmark questions.

The golden dataset

The term you will hear most is the golden dataset - a curated, human-verified set of cases that represents your quality bar. It is not every case you have ever collected. It is the deliberately maintained subset you trust enough to gate releases on.

A few principles for keeping it golden:

• Cover failure modes, not just happy paths. Ten cases across ten distinct ways the system fails beat a hundred variations of the same easy question.
• Keep expected answers correct. A stale or wrong expected answer produces false regressions that erode trust in the whole suite. Review it periodically.
• Version it. The dataset lives in git or in your eval platform with history, so you know which version of the data produced which scores.
• Split it. Separate subsets for retrieval quality, tone, safety, and core accuracy let you run focused evals and read focused scores.

Handling the no-single-answer problem

Much of LLM work has no single correct answer. There are many good ways to summarize a document or answer a support question. When there is no reference answer, switch from reference-based to criteria-based scoring.

Instead of storing an exact expected string, store the rubric. A good summary might need to be under 100 words, mention the key decision, and omit speculation. You then score with an LLM-as-judge that reads the output and rates it against that rubric, or with rule checks for the parts that are deterministic like length. The LLM-as-a-judge guide goes deep on making that judge reliable, and for retrieval systems the how to evaluate RAG post covers storing context so you can score grounding.

A worked example

Say you are building a documentation assistant. Your first golden set of 30 cases might break down as:

• 12 core accuracy cases - real questions with a verified correct answer and the source doc they should cite.
• 6 grounding cases - questions where the answer is not in the docs, expecting an honest “I do not know” rather than a hallucination.
• 6 tone and format cases - checking length limits and that code is returned in code blocks.
• 6 adversarial cases - prompt injection attempts, off-topic requests, and hostile phrasing.

Each row is tagged by category. Now when you change the retrieval prompt and the grounding subset drops from 6 out of 6 to 3 out of 6, you know exactly what broke and where to look. That specificity is the payoff of building the dataset with intent.

How the real tools do it

• Braintrust treats datasets as first-class versioned objects, so you can store, edit, and track cases over time and link each experiment to the exact data it ran on.
• Langfuse makes it easy to promote captured production traces into named datasets, closing the loop from real traffic to test case.
• DeepEval provides dataset abstractions plus synthetic data generation helpers and a metric library to score against your criteria.
• Promptfoo keeps test cases in version-controlled config, which suits teams that want their dataset to live in git next to the code.

The best LLM eval frameworks roundup compares how each handles dataset management if you are choosing.

Key takeaways

• The dataset is the ground truth your entire eval rests on - a weak dataset makes every downstream score meaningless.
• A good test case carries input, expected output or criteria, context for retrieval, and metadata tags so you can slice scores by category.
• Draw cases from all four sources - real traffic first, then hand-written, synthetic, and benchmarks - and curate the trusted subset into a versioned golden dataset.
• When there is no single correct answer, store a rubric instead and score with an LLM-as-judge or rule checks against it.

You now have the foundation - why evals matter, the two modes, and the datasets they run on. From here, dig into scoring methods and pipelines through the related guides above, and pick a framework to put it into practice.