OpenAI Evals Review (2026)

Two products, one name, one of them dying

Nearly all the confusion about OpenAI Evals comes from the fact that the name covers two different things.

The hosted Evals platform - the dashboard and Evals API on OpenAI’s developer platform - is being shut down.

The openai/evals GitHub repository - the open-source framework and benchmark registry - is a separate artefact, not archived and not formally deprecated, but only lightly maintained.

If someone tells you OpenAI Evals is dead, or that it is fine, they may both be right. Ask which one they mean.

The shutdown timeline

Date	What happens
3 June 2026	Deprecation announced to developers
31 October 2026	Evals become read-only for existing users
30 November 2026	Platform shuts down

Your real deadline is 31 October, not 30 November.

That month matters. Read-only means you lose the ability to create or modify evals four weeks before the service actually disappears. Any migration plan built around the November date is wrong by a month, and it is the kind of error you discover on 1 November when you try to make a change.

Existing content remains available during the transition window, so export should still work.

It is not just evals

This is the part teams underestimate.

On the same day, 3 June 2026, OpenAI announced three platform-level deprecations sharing the 30 November shutdown date: the Evals platform, Agent Builder, and Reusable Prompts (the v1/prompts API).

Teams that built on OpenAI’s platform layer frequently used more than one. If you were storing prompts in the Prompts API and evaluating them through the Evals platform, you are replacing two systems and the integration between them, not swapping an eval tool.

Audit all three dependencies rather than solving for evals in isolation.

OpenAI’s migration recommendation is promptfoo - which OpenAI owns

Credit where it is due - OpenAI published a migration path rather than leaving people stranded. It points to promptfoo, with guidance explicitly titled around moving from OpenAI Evals to promptfoo. For teams new to evaluation or wanting a more iterative environment, it suggests Datasets instead.

The context nobody states alongside that recommendation: OpenAI acquired promptfoo on 9 March 2026, roughly three months before announcing the Evals deprecation.

That reframes what is happening. This is not a vendor gracefully bowing out and pointing customers to the best independent alternative. It is a vendor retiring its homegrown eval platform in favour of the one it just bought, and consolidating users onto it. The Evals platform and promptfoo were overlapping products under one roof, and one of them was always going to lose.

None of this makes the recommendation wrong. promptfoo is genuinely good - open source under MIT, provider-agnostic, roughly 23.5k GitHub stars, and built around the same declarative, CI-first shape OpenAI Evals used, so the mental model transfers rather than needing relearning. OpenAI and promptfoo both stated on the record that it stays open source under its current license and that existing customers continue to be supported.

But you should evaluate it as a recommendation with a commercial interest attached, not as disinterested advice. And if part of your reason for leaving is discomfort with OpenAI platform dependencies, note that the recommended destination is also an OpenAI company now. DeepEval, whose commercial layer Confident AI is an independent startup, is the obvious alternative if vendor independence is what you are actually buying.

The migration does at least fix the structural problem OpenAI Evals always had.

The limitation it always carried

Even before the deprecation, OpenAI Evals was a constrained choice for one reason: it was tightly coupled to OpenAI models.

That was reasonable when OpenAI was the only serious option. It aged badly. Most teams now evaluate across Anthropic, Google and open-weight models, and comparing models is the main reason to run evaluations at all. A framework that assumes one provider makes the primary use case awkward.

Anyone who chose a provider-agnostic tool earlier is not migrating this quarter.

Is the open-source repo worth using?

As a reference, yes. The registry of community-contributed benchmarks is genuinely useful, and those definitions can inform evals you run elsewhere. A lot of this category’s conventions trace back to this project.

As your framework, probably not. It has not been archived and still receives community activity, but maintenance is light - there is an issue backlog stretching back to October 2025, with issues filed as recently as March 2026 still open.

In a category where model APIs shift constantly, light maintenance means integrations drift, and the drift is discovered one broken thing at a time. Building on it when promptfoo and DeepEval are actively developed is hard to justify.

What to do if you are on the hosted platform

1. Set 31 October 2026 as your internal deadline. Not November.
2. Export your eval definitions and historical results now, while content is still available.
3. Audit whether you also depend on Agent Builder or Reusable Prompts - same shutdown date.
4. Evaluate promptfoo first, since OpenAI’s own migration guidance targets it and the shape is familiar. DeepEval is the other strong option if you want a broader metric library.
5. Take the opportunity to go provider-agnostic, so the next deprecation is somebody else’s problem.

Should you use it?

Use it if - there is no version of this that makes sense for a new adopter.

If you are already on it, migrate before 31 October 2026.

Bottom line: a historically important project whose hosted incarnation has a hard end date and whose open-source incarnation is coasting. The single-provider coupling was always the strategic weakness, and the migration is a reasonable moment to fix it. Follow OpenAI’s own advice and look at promptfoo.

---

Deprecation dates verified against OpenAI’s published deprecations documentation and contemporaneous reporting on 31 July 2026. Repository maintenance status assessed from public issue tracker activity. This is a researched directory entry - we have not instrumented this platform with our reference application, and given the shutdown, we will not.