Google Cloud tests AI agents with ambiguity-based benchmarks

Google Cloud has outlined a way to evaluate AI agents by varying the ambiguity of test queries instead of relying on fixed benchmark scores. It is using the approach in work on data discovery agents within Google Data Cloud.

The method centres on a framework called Discovery Bench, which creates easier and harder versions of the same evaluation case. Rather than asking whether an agent passes a single benchmark, it shows where performance weakens as a query becomes less specific.

This is especially important in data discovery, where an agent must identify the right table or file in a large data lake or warehouse before answering a user's question. The team behind the work argues that standard benchmark scores can mask sharp performance drops when users phrase questions vaguely or leave out key terms.

Measuring difficulty

The system uses surprisal, a concept from information theory, to estimate how much uncertainty remains about the correct dataset after a query is given. Terms that sharply narrow the search are treated as more informative, while broader wording increases ambiguity.

By adding or removing those terms, Discovery Bench creates three calibrated versions of a query with high, medium and low ambiguity. The goal is to replace subjective labels such as easy or hard with a measurable way to adjust the challenge presented to the agent.

Google illustrated the method with an example from KramaBench, a public benchmark for data-to-insight pipelines. In a query about satellite altitude changes, the term "TLE" points closely to one target table. Removing it broadens the possible matches to several datasets.

The result is a more granular picture of retrieval behaviour. A single benchmark item becomes a curve of results rather than one score tied to one wording.

Performance cliffs

In one test, a retrieval-focused agent built on Gemini 3.1 Pro produced an F1 score of 0.34 at high ambiguity, 0.76 at neutral phrasing, 0.81 at medium ambiguity and 0.78 at low ambiguity. The pattern suggested that additional specificity did not always improve results in a straight line.

The testing also exposed abrupt failures. In the satellite example, a query that scored a perfect F1 of 1.00 under neutral wording fell to 0.00 when a distinguishing token was removed, even though the query, agent and ground truth were otherwise unchanged.

That kind of result suggests fixed benchmarks can overstate how dependable an agent will be in live use. A benchmark phrased one way may appear solved, while a slightly vaguer version of the same question can fail completely.

The ambiguity sweep also helped identify specific problem areas in Google's own discovery agent, including over-retrieval of time-sharded tables and a drop in F1 scores when long search chains expanded the amount of context the system had to process.

Benchmark flaws

The work did not stop at testing agents. Google said its first evaluation on an astronomy subset of KramaBench found flaws in some benchmark items, including ground-truth tables that did not answer the query, cases where exact dates were required even though only months were specified, and a question linked to 124 sharded tables that some retrieval interfaces could not return.

These findings point to a broader issue in AI evaluation. Benchmarks are often treated as neutral measuring tools, yet the underlying labels and answer sets can contain errors that distort later conclusions.

Google's team also compared two ways of generating ambiguity sweeps: one based on terms selected by a large language model, and another grounded in TF-IDF surprisal. The difference was substantial. At high ambiguity, the LLM-built sweep gave the agent about 0.34 F1, while the grounded sweep put it at about 0.85.

The gap suggests the design of the evaluation method can shape the apparent quality of the agent as much as the agent's own behaviour. In effect, Google is arguing that AI testing methods need validation just as models do.

Wider debate

The approach sits within a broader shift in AI research towards meta-benchmarking, item response theory and closer inspection of benchmark quality. Academic work has increasingly focused on whether a smaller set of highly informative questions can reproduce larger benchmark scores, and whether some widely used benchmark items are mislabeled or ambiguous.

Google's contribution is to apply that thinking to live enterprise data retrieval. Rather than treating ambiguity as noise around a benchmark result, the method treats it as a central variable to control and measure.

The practical implication for businesses using AI agents on large internal datasets is straightforward. A model that performs well on a static test may still fail when employees use incomplete language, choose the wrong terms or ask a question in a less structured way.

Google framed the work as a challenge to the industry's reliance on pass-or-fail evaluation. "We have spent years optimizing agents against rulers we never measured," the authors said.