Shreya Shankar (UC Berkeley PhD) presents DAB — the Data Agent Benchmark (“Can AI agents answer your data questions?”), from her PhD work at Berkeley. ~27 minutes on Hamel Husain’s channel.

Why data agents matter

  • Huge real use case: AI answering business questions — much office work is this
  • Enterprises struggle to use Claude Code / Codex off-the-shelf for BI tasks, so they build their own data agents: Uber’s Query GPT (1.2M queries/month at launch), OpenAI’s in-house agent, Databricks, Snowflake
  • Existing benchmarks fall short:
    • Text-to-SQL — assumes clean data in one Postgres/Snowflake; real enterprises have fragmented data; SQL alone isn’t enough (need domain knowledge, reasoning over outputs)
    • Table QA — context table + question; doesn’t scale to enterprise data volume

The formative study — 4 real-world challenges

Interviews with enterprises + Hasura (PromptQL) surfaced four properties missing from existing benchmarks:

  1. Multiple databases — answer needs SQL across several DBs plus Python/Pandas to join (e.g. sales leads split across Postgres + another DB)
  2. Ill-formatted join keys — join keys need cleaning first (strip a “lead-” prefix to get an exact match); agents must reason about cleaning before joining — they usually don’t
  3. Unstructured text transformation — free-text columns that can’t be parsed by SQL/code; need row-by-row reading and business decisions at scale
  4. Domain knowledge — internal knowledge about next actions (high-priority call vs. follow-up email) that the agent must know to recommend anything useful

How DAB was built

  • 104 queries (each = a long-running environment, hundreds of turns — TerminalBench scale), 17 datasets, mostly open-source (Kaggle) corrupted deterministically (removed columns, embedded numerics in strings, renamed columns) so ground truth stays validatable
  • Data distributed across databases by domain semantics (Postgres for customer data, DuckDB for analytics)
  • Hints file (describes the corruptions) + semantic layer (schema descriptions) — also serve as ablations
  • Agent setup: bash + file read/write tools, network disabled (agents would search the internet for the ground-truth data!), 5 trials per query

Results

  • Tested Claude Sonnet 4 (Claude agent SDK), GPT-5.5 (Codex), Gemini 3.1 Pro (react harness)
  • Best frontier model (GPT-5.5 + Codex): 57% pass@1, 70% pass@5 — quite low

The 5 failure modes

  1. Fails before planning — refuses or never tries a tool (rare, some models have aggressive safeguards)
  2. Incorrect plan — wrong approach in natural language before execution
  3. Wrong data selection — correct plan, wrong column/attribute picked
  4. Wrong implementation — correct data selected but code is wrong (e.g. regex that misses corruption variants)
  5. Runtime errors — rare; agents usually just retry

They built the taxonomy by hand, then used an LLM-as-judge to scale classification across hundreds of traces. Modes 2, 3, 4 dominate.

Key insight: plans before data

  • Agents write the plan before looking at the data — plans often omit data cleaning entirely, and agents overfit to the plan even when the data contradicts it
  • “For data questions, don’t write plans before you have looked at the data. Look at the data, write plan. Always be looking at data.”
  • Hints help: giving agents the hints file raises pass@1 by >10% per agent — but hints aren’t comprehensive, so it doesn’t fully solve it

Practical takeaways

  • Don’t assume frontier models solve your data questions off the shelf — use skills/harnesses (OpenAI open-sourced data-agent skills), evaluate with DAB
  • In-house process: run your own formative study — talk to BI people and analysts; use your internal data (no leak worries); use LLMs to help find ground-truth answers for your own benchmark
  • Semantic layer = natural-language metadata about tables/columns (value ranges, histograms, distinct values) — agents perform better the richer it is; populate it in advance with crawling agents
  • Memory store: agents should record corruptions they hit so future runs query that memory — improves text-to-SQL accuracy in concurrent work
  • Hill climbing DAB is hard: people inadvertently cheat (AG News labels still on Hugging Face, dataset in HF cache) — sandboxing is genuinely difficult
  • ORMs: probably not the answer — “you should do an eval”; graph databases: you can put graphs in relational DBs; fewer databases, same type, relational wherever possible

“For data questions, don’t write plans before you have looked at the data. Look at the data, write plan. Always be looking at data.”

Watch on YouTube — full summary in the vault note.