Regression Testing

What it means

A regression is when something that used to work stops working. Regression testing for LLM applications means keeping a stable dataset of inputs with known-good outcomes and running the full scorer suite against it whenever you change a prompt, swap a model, bump a provider version or edit surrounding code. If a case that passed before now fails, you have a regression to investigate before shipping.

Why it matters

LLM behavior is sensitive to small changes. Tweaking one instruction to fix a bad case can quietly break three others, and a provider silently updating a model can shift outputs overnight. Because these failures are non-obvious and non-deterministic, you cannot catch them by eyeballing a few examples. A regression suite makes the invisible visible and lets you upgrade models with confidence rather than fear.

In practice

Teams wire regression suites into CI so a pull request cannot merge if scores drop below a threshold. promptfoo and DeepEval run assertion-based suites in a pipeline and report per case pass or fail, while Braintrust tracks experiment scores over time so you can see exactly which cases moved. Watch out for flaky scorers: a noisy judge produces false positive regressions that erode trust in the suite. See how to run LLM regression tests for a practical CI setup.