What a $900M Bank Error Teaches Us About AI Agents
John Feminella connects Citibank’s mistaken Revlon payment to five agent controls: ownership records, risk-based review, scoped tools, shared tests and selective approvals.
John Feminella connects Citibank’s mistaken Revlon payment to five agent controls: ownership records, risk-based review, scoped tools, shared tests and selective approvals.
GEPA improves prompts for model programs by proposing changes and testing their behavior. It works with DSPy, a Python framework for programming model workflows. This extension proposes several changes together so evaluations can run …
Optiver's Noyan Tokgozoglu explains internal model tests, portable agent tooling, context-cost controls and the limits of replaying live workflows.
Read traces, group recurring failures, save regression cases and compare a proposed fix against them. Fuad Ali demonstrates the loop with Arize Signal and keeps humans responsible for judging issues, tests and releases.
A controlled synthetic-memory study separates finding an update from selecting it, with limits from familiar facts and scoring.
Put realistic data in fresh sandboxes, check what the agent changes and compare runs task by task. Xander Song shows how Phoenix tests its agents with Harbor, including a SQL-tool optimization.
A diagnostic agent lost a third of its tool results before the model read them, even though its largest prompt used only 5.75% of the available window. The surrounding software sliced long JSON, a structured data format, at 6,000 …
His Ideal State Artifact keeps goals, constraints, decisions, and testable criteria together. The stronger question is how that document stays accountable to running software.
PROBE tests whether coding agents find defects that maintainers later repair. Its reference set contains 2,457 bugs fixed over eighteen months in twenty repositories, all already present in the starting snapshots. The authors split audits …
Target real incident patterns, test against historical pull requests and supply codebase context. Greptile's Daksh Gupta explains model diversity, executable sandboxes and the cost of noisy automated reviews.
Measure the silence callers hear, keep listening while tools run and test the complete conversation. Jesse Hall shows how LiveKit evaluates voice agents against interruptions and real back-end outcomes.
CacheLint checks application code for prompt-caching mistakes. Prompt caching reuses computation for an identical beginning of a request; an agent can lose those savings while continuing to answer correctly. The authors inspect 120 merged …
Start with real traces, define one concrete pass/fail criterion, build a model-based judge and compare its decisions with human labels. SallyAnn DeLucia walks through the process in Arize.
A specialized image classifier names more lesions correctly, but its preset discharge threshold misses more melanomas. Accuracy and safe decisions need separate tests.
Arize's Aparna Dhinakaran explains the work behind always-on agents: trace-driven triggers, disposable sandboxes, durable state and evaluators checked against human judgment.
Kaiser Sun and colleagues compare reasoning-effort settings across eight models and four task sets. Higher effort often improves aggregate accuracy, while spending more on tasks that cheaper runs already solve. The study measures whole …
Ball's workflow centers on end-to-end access, tools that test alternatives, evidence of results, and human review concentrated on high-risk changes.
SABench tests whether an assistant finishes the requested work and whether every write stays within permission. Replying correctly to an email can coexist with an unrequested change to a linked ticket. The study uses 201 synthetic office …
Kaggle's benchmark audits expose fragile scores. LinqAlpha's MandateBench tests the reasoning behind a full investment memo, with early results from ten tasks.
Memory Tax compares eleven memory systems with search over the original conversation. It asks whether extracting and organizing memories earns its extra cost. The authors separate new answers that memory enables from answers lost relative …
GEPA evolves prompts for model programs, including workflows built with DSPy. This study tests its merge operation, which combines candidates descended from a common starting instruction. The comparison spans five datasets, two models and …
Sofia Jakovcevic builds a regression-testing loop from production traces. Keep separate scores for safety and task completion, and protect workflows that already work.
Harneet Kaur tests two models on iHeartMedia. Checking each reasoning layer makes their forecasts converge, but agreement leaves accuracy unproven.
Contour's Hari Kumar compares research workflows with and without AI, tracks cost per useful insight and counts the time lost fixing broken tools.
MCP-Injector inserts errors, delays and altered results between an assistant and tools connected through MCP, the standard for connecting agents to tools. It tests reliability when connected programs behave badly. The authors cross …
Stephanie Jarmak explains how realistic workloads, repeated runs, trusted traces and enforced quality gates make agent tests useful for real decisions.
Anthropic and Theorem report checked proofs for 239 test entry points across 123 libraries. The public release, machine model and patching process need separate scrutiny.
ModelCompare's small experiment finds a familiar refrain about tribal loyalty and rationalization. Its broad answer categories and leading prompt limit what that agreement means.
ImpasseBench tests whether coding assistants stop when their own tool results show that outside help is needed. It separates persistent barriers from temporary errors and blocked testing. The authors run eight model configurations on 102 …
Upwork’s SWE-Clar asks coding assistants to clarify incomplete requests before building software. It freezes that conversation, then starts two isolated continuations: one answers requirement questions and the other writes code. Assessment …
A coding assistant can preserve its task score while forgetting restrictions on how it should work. This study compares five conversation-shortening strategies inside OpenCode, the software running the model’s tools and commands. The …
AfterDeleteBench separates deleting a record, preventing disclosure and blocking a later write. Its controlled tests also expose failures hidden by answer parsers.
Yuning Bian and colleagues compare document splitting methods across 120 adjacent Wikipedia revision pairs. A small edit can shift boundaries and force a retrieval system to embed unchanged text again. On the 116 pairs all methods process …
Abhishek Barali and Aayush Paudel test what happens when an extraction schema requires more answers than a document contains. Models often fill the gap with genuine text of the wrong kind. Across eight downloadable models and 558 documents, …
A review of 1,368 reported AI incidents motivates tests of permissions, human review and monitoring alongside model accuracy. The findings are retrospective, not proof of prevention.
Air Canada’s Nathanael Weill starts with known answers, generates test inputs around them and checks results in code. Synthetic tests speed iteration but do not authorize release.
Tone-Bench tests whether models can write in one person’s voice. Louis-François Bouchard explains its rubrics, judge bias and the cost of changing a test.
An audit finds hidden interface requirements in SWE-Milestone. They distort some results, while removing affected tests largely preserves the overall ranking.
Jens Lehmann and Sahar Vahdati let models build and twice revise their own harnesses: the software that runs their tools, files and commands. Improvements on familiar development games often overstate gains elsewhere. The study follows 27 …
MemPABench asks whether an assistant remembers how a person wants help, including when to confirm, ask questions or initiate contact. It measures the preference selected and the behavior actually delivered. Four simulated users have …
Text-only editing damaged some correct transcripts. Adding audio reduced that damage for Gemini, but also reduced repairs to its own drafts.
Yikai Zhao and colleagues test whether document conversion preserves the requirements an author marked for attention. Words can survive extraction while color and highlighting disappear. Their task asks models to recover specially marked …
Zedong Peng compares extracted memories with coding agents that search raw conversation history. The study includes store-construction costs, which can outweigh cheaper answers later. It tests seven methods with three models on one LoCoMo …
MiniMax's Morgan Suo connects faster model internals to the cost, accuracy and delivery time of a research report an analyst can actually use.
Thomas Hikaru Clark, Matias B. Martin and Kyle Mahowald examine why AI prose can feel difficult despite being grammatically valid. Their study finds features associated with reading difficulty across model-generated conversations, coding …
Cole Medin demonstrates two feedback loops for production agents: learn from human corrections, then propose trace-backed changes to instructions and skills.
Al Chen of Cisco explains how exploit tests, verified restores and privacy checks help banks move AI into production. The bottleneck also includes image approvals, GPU procurement and internal ownership.
Moshi founder Anekha Sokhal argues that AI should gather and check market information while investors keep the decision. She describes a six-agent system, traceable claims and the hidden cost of losing apprenticeship.
Anti-Persistence saves execution records and rebuilds advice for each task, reducing one route by which an agent can keep repeating a bad lesson.
Momento's Khawaja Shams explains realistic coding tasks, leaked answers, repeated trials, and the traces that make agent scores worth trusting.