Google builds synthetic data estates to test enterprise data agents
Google describes an internal data simulator that generates synthetic company databases, documents and chat logs so data agents can be tested without touching sensitive production data. The simulator also builds verified SQL answer keys.
TL;DR
- Google built an internal data simulator that generates a synthetic company, with schemas, rows, documents and chat threads, to test enterprise data agents.
- Each benchmark question ships with a verified answer key, and planted traps check whether an agent follows misleading documents.
- Google says capability changes move accuracy in the same direction on synthetic and human-authored benchmarks.
The problem with testing data agents on clean data
A data agent answers questions about a company's data warehouse. Ideally it would be built and tested against production data, but access limits and data sensitivity often prevent that. This is especially true when the engineers building the agent work outside the team that owns the data.
Live warehouses are also poor test environments. Catalog descriptions are often incomplete or contradicted by docs and chat threads. There is no verified answer key across thousands of tables, and teams cannot easily vary the complexity of the data to see where an agent fails. As a result, teams often test on small toy datasets and hope the agent holds up in production.
A synthetic estate as a wind tunnel
Google's Cloud Blog compares the approach to aerodynamic testing in car design, where a scale model reproduces the forces that matter and ignores surface details. The company calls its version a "wind tunnel": a synthetic data estate with the same structure and causal behavior as a real organization, with every name, number and document synthesized.
An engineer starts with a short plain-English description of organizational patterns. From a few sentences, the simulator generates a structurally similar company, along with its schemas, rows, event history, documents, chat threads, a benchmark of hard questions and a verified answer key. The agent then queries the estate through a read-only interface for SQL execution, schema browsing and document search, without access to the blueprint or answer key.
Traps, misleading documents and answer keys
Google illustrates the problem with a fictional manufacturer, "Calder Brakes", and the question of how "downtime" is defined at one of its plants. A policy memo from a supervisor says downtime covers only unplanned stoppages. A naive agent adopts that definition and filters out planned changeovers in its SQL. The official metric, however, counts changeovers as planned downtime, and a later thread corrects the record. Both the qualitative and quantitative answers are wrong in that case. Getting them right means checking who owns the metric and tracing later corrections.
The pipeline separates design from data generation. A language model compiles a frozen blueprint of systems, people, metrics and beliefs, and those beliefs can be deliberately wrong. After that, table generation runs deterministically from the blueprint and a random seed, so the same inputs produce identical tables and an identical SQL answer key. Injected ingestion flaws, such as duplicate rows, unmerged keys and a mid-quarter unit change, are recorded in a messiness log so the ground-truth queries can account for them. A model is used again only to render documents as memos and chat threads.
Three build-time checks support the benchmark. Every quantitative question ships with SQL that computes its gold answer, and decoy queries representing common mistakes must yield different numbers. Qualitative questions carry rubrics based on the metric owner, official definition and correction trail. Rendered documents are checked against the blueprint for unrecorded facts. The simulator also plants questions about invented metrics, and a reliable agent should report that they do not exist.
Checking realism
Google does not claim the tunnel reproduces exact scores. It aims to reproduce behavioral trends, meaning the direction of effects and the rate at which accuracy drops as contradictions grow. Realism is tested in three ways: a realism report with statistics such as join fan-out, key skew, null and duplicate rates and leading-digit frequencies; reviews by analysts at enterprise partners; and an independent suite of more than a hundred calibration checks. Google reports that in internal evaluations, upgrades such as a deeper reasoning mode or better context handling moved accuracy in the same direction and ranked configurations in the same order on synthetic and human-authored benchmarks. Broader studies comparing paired real and synthetic customer estates are still underway, so these results are preliminary.
Why it matters
For teams building data agents, the main takeaway is that evaluation on clean toy databases misses the failures that matter most in production: conflicting documents, stale definitions and messy tables. A synthetic estate with a verified answer key offers a way to test those cases without sharing sensitive data, and to increase complexity step by step. Developers should still treat the results as a signal, since Google's evidence about how well synthetic trends match production is internal and not yet independently published.
Sources
This article was written by Blozhik from the reporting below. Read the originals for full detail.
GoFrontier and Center: Why your data agents need a “wind tunnel”Google Cloud Blog