Databricks Agent Evaluation Review (2026)

Agent permissions that inherit from data governance

This is the idea worth the page, and most of this category has no answer to it at all.

Tools are registered in Unity Catalog, so the same least-privilege permissions protecting your data also scope what an agent may do.

Consider the alternative, which is what everyone else asks you to build. You design a parallel authorisation model - which tools this agent can call, under what conditions, on whose behalf - and then audit it separately from your existing data access controls. That is a second permissions system, with its own drift, its own gaps, and its own audit burden.

Inheriting from data governance means an agent acting for a user simply cannot reach data that user cannot reach, enforced at the layer where it was always enforced.

For an enterprise where “what can this agent see” is a compliance question rather than an engineering preference, that is the most coherent answer we have encountered.

The managed MCP servers extend it - exposing Unity Catalog Functions, Genie, Vector Search and DBSQL as governed tools without custom plumbing. Agents get access to your real data assets through the governance layer rather than around it.

Judges you can tune to your domain

The evaluation side attacks the genuine weakness of LLM-as-judge.

Built-in AI judges score correctness, relevance and safety alongside custom metrics, with a UI for human stakeholder feedback - which matters because the people who know whether an answer is right are usually not engineers.

Then Databricks added Agent-as-a-Judge, Tunable Judges and Judge Builder.

The problem these solve is real and underappreciated: a generic judge does not understand your domain. A default judge scoring “correctness” applies a general notion of correct, which is frequently not what correct means in insurance underwriting, clinical triage or contract review. Generic judges produce confident scores that are wrong in domain-specific ways nobody notices.

This is a different attack on the same problem Patronus addresses by training purpose-built judge models. Both are more serious than prompting a general model and hoping. For an enterprise with unusual criteria, domain tunability is arguably the more practical route, because you know your rules and they do not.

MLflow as the backbone

MLflow 3 is the GenAI-native lifecycle and tracing layer, and Agent Evaluation runs are tracked there - so you can compare versions, catch regressions and gate deployments.

That is a genuine advantage over bespoke tracking. MLflow is mature, has established conventions and a large user base, and your GenAI evaluation history sits alongside your classical ML experiment history rather than in a separate silo.

One caveat carried over from our MLflow review: the best-governed experience, including Unity Catalog trace storage, is Databricks-only. Open-source MLflow and this are not the same product, and the shared name obscures that.

The Mosaic AI Agent Framework reached GA in early 2026, alongside Agent Bricks for task-first agent construction and AI Gateway as a control plane for model and tool calls.

Synthetic benchmarks, with a caveat

The automated evaluation system can generate benchmarks via an LLM judge based on the defined agent task, often using synthetic data, as part of an auto-optimisation loop.

That addresses a real gap - most teams have no labelled dataset and will not build one, which is the same blocker Ragas solves with ground-truth-free metrics.

The caveat matters though. Synthetic cases reflect what a model imagines your users do, which is systematically different from what they actually do. An agent tuned against synthetic benchmarks can score well and fail on real traffic, and the failure is invisible because your benchmark says everything is fine.

Use it to bootstrap, then replace it with sampled production cases as soon as you have them.

The constraint is absolute

There is no standalone product and no separate pricing. Evaluation sits inside Databricks consumption, where judge inference, compute and storage fold into the platform bill rather than appearing as an evaluation line item.

For an existing customer that is a forecasting inconvenience your account team can model. For anyone else it is academic - you cannot buy this without buying Databricks, which is an enormous commitment for an evaluation capability.

Against a specialist

The split is clear:

Strength	Where it lives
Agent permissions inheriting from data governance	Databricks
Evaluation lineage beside ML experiment history	Databricks
Multi-turn conversation simulation	LangWatch
Agent trajectory evaluation under pytest in CI	LangWatch

LangWatch’s architecture exercises trajectories; Databricks scores outputs with better-governed inputs. If your agents operate over governed enterprise data, the Unity Catalog story is hard to replicate. If your agents are conversational products, a specialist understands them better.

Should you use it?

Use Databricks Agent Evaluation if you already run Databricks and are building agents over your own governed data.

Don’t use it if you are not a Databricks customer - the commitment is disproportionate - or your agents are conversational and need trajectory simulation.

Bottom line: the best governance answer in the category, wrapped in the largest platform dependency. Bootstrap with synthetic benchmarks, replace them with production samples quickly, and remember that open-source MLflow does not get the Unity Catalog half of this.

---

Framework GA timing, architecture, governance model and evaluation features verified against Databricks documentation and announcements on 3 August 2026. Pricing sits within Databricks consumption and no standalone figure is published or estimated. This is a researched directory entry - we have not yet instrumented this platform with our reference application.