What a $900M Bank Error Teaches Us About AI Agents

John Feminella connects Citibank’s mistaken Revlon payment to five agent controls: ownership records, risk-based review, scoped tools, shared tests and selective approvals.

Published:  · Added:  · 2 min

GEPA tests prompt changes in parallel to finish optimization sooner

GEPA improves prompts for model programs by proposing changes and testing their behavior. It works with DSPy, a Python framework for programming model workflows. This extension proposes several changes together so evaluations can run …

Published:  · Added:  · 1 min

How Optiver Tests and Rolls Out New AI Models

Optiver's Noyan Tokgozoglu explains internal model tests, portable agent tooling, context-cost controls and the limits of replaying live workflows.

Published:  · Added:  · 2 min

Turn Agent Failures Into Tested Fixes — Fuad Ali

Read traces, group recurring failures, save regression cases and compare a proposed fix against them. Fuad Ali demonstrates the loop with Arize Signal and keeps humans responsible for judging issues, tests and releases.

Published:  · Added:  · 3 min

Old assistant memories can override a correction already retrieved

A controlled synthetic-memory study separates finding an update from selecting it, with limits from familiar facts and scoring.

Published:  · Added:  · 2 min

How to Test Your Agents on Real Tasks With Harbor — Xander Song

Put realistic data in fresh sandboxes, check what the agent changes and compare runs task by task. Xander Song shows how Phoenix tests its agents with Harbor, including a SQL-tool optimization.

Published:  · Added:  · 3 min

Tool results can disappear before a model reads them

A diagnostic agent lost a third of its tool results before the model read them, even though its largest prompt used only 5.75% of the available window. The surrounding software sliced long JSON, a structured data format, at 6,000 …

Published:  · Added:  · 1 min

Daniel Miessler Puts Written Intent at the Center of AI Development

His Ideal State Artifact keeps goals, constraints, decisions, and testable criteria together. The stronger question is how that document stays accountable to running software.

Published:  · Added:  · 3 min

PROBE separates real bug findings from bugs maintainers repair

PROBE tests whether coding agents find defects that maintainers later repair. Its reference set contains 2,457 bugs fixed over eighteen months in twenty repositories, all already present in the starting snapshots. The authors split audits …

Published:  · Added:  · 1 min

How to Build an AI Code Review Agent — Daksh Gupta

Target real incident patterns, test against historical pull requests and supply codebase context. Greptile's Daksh Gupta explains model diversity, executable sandboxes and the cost of noisy automated reviews.

Published:  · Added:  · 2 min

How to Build Voice Agents That Don't Fall Apart When You Interrupt — Jesse Hall

Measure the silence callers hear, keep listening while tools run and test the complete conversation. Jesse Hall shows how LiveKit evaluates voice agents against interruptions and real back-end outcomes.

Published:  · Added:  · 2 min

CacheLint checks why working agents lose prompt-cache savings

CacheLint checks application code for prompt-caching mistakes. Prompt caching reuses computation for an identical beginning of a request; an agent can lose those savings while continuing to answer correctly. The authors inspect 120 merged …

Published:  · Added:  · 1 min

How to Write Your First AI Eval — SallyAnn DeLucia

Start with real traces, define one concrete pass/fail criterion, build a model-based judge and compare its decisions with human labels. SallyAnn DeLucia walks through the process in Arize.

Published:  · Added:  · 2 min

A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis — Magnus Lynch

A specialized image classifier names more lesions correctly, but its preset discharge threshold misses more melanomas. Accuracy and safe decisions need separate tests.

Published:  · Added:  · Updated:  · What changed: Added the biopsy-versus-discharge objection and the tradeoff between training data and inference costs. · Reviewed comments: 277 · 3 min

What Does an AI Engineer Do Now? — Aparna Dhinakaran

Arize's Aparna Dhinakaran explains the work behind always-on agents: trace-driven triggers, disposable sandboxes, durable state and evaluators checked against human judgment.

Published:  · Added:  · 2 min

Extra reasoning needs to earn its cost

Kaiser Sun and colleagues compare reasoning-effort settings across eight models and four task sets. Higher effort often improves aggregate accuracy, while spending more on tasks that cheaper runs already solve. The study measures whole …

Published:  · Added:  · 2 min

How I Work with Agents — Thorsten Ball

Ball's workflow centers on end-to-end access, tools that test alternatives, evidence of results, and human review concentrated on high-risk changes.

Published:  · Added:  · 3 min

A completed task can still contain unauthorized writes

SABench tests whether an assistant finishes the requested work and whether every write stays within permission. Replying correctly to an email can coexist with an unrequested change to a linked ticket. The study uses 201 synthetic office …

Published:  · Added:  · 2 min

Why Frontier Models Fail as Buy-Side Analysts — Janie Gu and Suyeol Yun

Kaggle's benchmark audits expose fragile scores. LinqAlpha's MandateBench tests the reasoning behind a full investment memo, with early results from ten tasks.

Published:  · Added:  · 2 min

Memory Tax tests what extracted memories lose

Memory Tax compares eleven memory systems with search over the original conversation. It asks whether extracting and organizing memories earns its extra cost. The authors separate new answers that memory enables from answers lost relative …

Published:  · Added:  · 2 min

Combining better prompts still needs behavioral tests

GEPA evolves prompts for model programs, including workflows built with DSPy. This study tests its merge operation, which combines candidates descended from a common starting instruction. The comparison spans five datasets, two models and …

Published:  · Added:  · 2 min

Your Agent Fix Just Broke Something Else — Sofia Jakovcevic

Sofia Jakovcevic builds a regression-testing loop from production traces. Keep separate scores for safety and task completion, and protect workflows that already work.

Published:  · Added:  · 2 min

Can AI Understand a Distressed Credit? — Harneet Kaur

Harneet Kaur tests two models on iHeartMedia. Checking each reasoning layer makes their forecasts converge, but agreement leaves accuracy unproven.

Published:  · Added:  · 2 min

Proving Alpha: How a Hedge Fund Measures AI's ROI — Hari Kumar

Contour's Hari Kumar compares research workflows with and without AI, tracks cost per useful insight and counts the time lost fixing broken tools.

Published:  · Added:  · 2 min

MCP-Injector tests assistants against faulty tools

MCP-Injector inserts errors, delays and altered results between an assistant and tools connected through MCP, the standard for connecting agents to tools. It tests reliability when connected programs behave badly. The authors cross …

Published:  · Added:  · 2 min

Don't Trust Your AI Agents. Test Them Like a Scientist

Stephanie Jarmak explains how realistic workloads, repeated runs, trusted traces and enforced quality gates make agent tests useful for real decisions.

Published:  · Added:  · 2 min

Claude generates software proofs, but the guarantees have boundaries

Anthropic and Theorem report checked proofs for 239 test entry points across 123 libraries. The public release, machine model and patching process need separate scrutiny.

Published:  · Added:  · Updated:  · What changed: Added malisper’s unverified-translator clarification and the intermediate-representation choices behind the Postgres proof workflow. · Reviewed comments: 324 · 4 min

AI models converge on the same supposedly controversial take on humans

ModelCompare's small experiment finds a familiar refrain about tribal loyalty and rationalization. Its broad answer categories and leading prompt limit what that agreement means.

Published:  · Added:  · Reviewed comments: 688 · 2 min

Blocked coding assistants need a useful way to stop

ImpasseBench tests whether coding assistants stop when their own tool results show that outside help is needed. It separates persistent barriers from temporary errors and blocked testing. The authors run eight model configurations on 102 …

Published:  · Added:  · 2 min

SWE-Clar tests understanding separately from working code

Upwork’s SWE-Clar asks coding assistants to clarify incomplete requests before building software. It freezes that conversation, then starts two isolated continuations: one answers requirement questions and the other writes code. Assessment …

Published:  · Added:  · 2 min

Conversation summaries can keep scores while losing user rules

A coding assistant can preserve its task score while forgetting restrictions on how it should work. This study compares five conversation-shortening strategies inside OpenCode, the software running the model’s tools and commands. The …

Published:  · Added:  · 2 min

Deleted assistant memories can survive in summaries and separate clues

AfterDeleteBench separates deleting a record, preventing disclosure and blocking a later write. Its controlled tests also expose failures hidden by answer parsers.

Published:  · Added:  · 2 min

Document splits change the cost of small edits

Yuning Bian and colleagues compare document splitting methods across 120 adjacent Wikipedia revision pairs. A small edit can shift boundaries and force a retrieval system to embed unchanged text again. On the 116 pairs all methods process …

Published:  · Added:  · 1 min

Minimum list lengths make models pad extracted answers

Abhishek Barali and Aayush Paudel test what happens when an extraction schema requires more answers than a document contains. Models often fill the gap with genuine text of the wrong kind. Across eight downloadable models and 558 documents, …

Published:  · Added:  · 1 min

1,368 AI Failures and What They Teach About Evals

A review of 1,368 reported AI incidents motivates tests of permissions, human review and monitoring alongside model accuracy. The findings are retrospective, not proof of prevention.

Published:  · Added:  · 2 min

How to Test AI Agents When No Benchmark Exists

Air Canada’s Nathanael Weill starts with known answers, generates test inputs around them and checks results in code. Synthetic tests speed iteration but do not authorize release.

Published:  · Added:  · 2 min

Why AI Writing Still Reads Like Slop

Tone-Bench tests whether models can write in one person’s voice. Louis-François Bouchard explains its rubrics, judge bias and the cost of changing a test.

Published:  · Added:  · 2 min

Coding tests can demand names the instructions never supplied

An audit finds hidden interface requirements in SWE-Milestone. They distort some results, while removing affected tests largely preserves the overall ranking.

Published:  · Added:  · 2 min

Self-rewriting agents need tests they cannot learn from

Jens Lehmann and Sahar Vahdati let models build and twice revise their own harnesses: the software that runs their tools, files and commands. Improvements on familiar development games often overstate gains elsewhere. The study follows 27 …

Published:  · Added:  · 1 min

Remembering a preference is different from following it

MemPABench asks whether an assistant remembers how a person wants help, including when to confirm, ask questions or initiate contact. It measures the preference selected and the behavior actually delivered. Four simulated users have …

Published:  · Added:  · 1 min

Transcript cleanup needs to count what it breaks

Text-only editing damaged some correct transcripts. Adding audio reduced that damage for Gemini, but also reduced repairs to its own drafts.

Published:  · Added:  · 2 min

Document converters can erase the warnings authors marked

Yikai Zhao and colleagues test whether document conversion preserves the requirements an author marked for attention. Words can survive extraction while color and highlighting disappear. Their task asks models to recover specially marked …

Published:  · Added:  · 1 min

MemoryTax counts the cost of building agent memories

Zedong Peng compares extracted memories with coding agents that search raw conversation history. The study includes store-construction costs, which can outweigh cheaper answers later. It tests seven methods with three models on one LoCoMo …

Published:  · Added:  · 1 min

Is a Faster Model Actually Faster? Test the Whole Job — Morgan Suo

MiniMax's Morgan Suo connects faster model internals to the cost, accuracy and delivery time of a research report an analyst can actually use.

Published:  · Added:  · 2 min

Shorter AI sentences can still be harder to understand

Thomas Hikaru Clark, Matias B. Martin and Kyle Mahowald examine why AI prose can feel difficult despite being grammatically valid. Their study finds features associated with reading difficulty across model-generated conversations, coding …

Published:  · Added:  · 2 min

Let Your AI Agent Do Its Own Harness Engineering (Here's How)

Cole Medin demonstrates two feedback loops for production agents: learn from human corrections, then propose trace-backed changes to instructions and skills.

Published:  · Added:  · 2 min

Shipping AI through bank compliance — Al Chen

Al Chen of Cisco explains how exploit tests, verified restores and privacy checks help banks move AI into production. The bottleneck also includes image approvals, GPU procurement and internal ownership.

Published:  · Added:  · 3 min

AI should feed an investor's judgment — Anekha Sokhal

Moshi founder Anekha Sokhal argues that AI should gather and check market information while investors keep the decision. She describes a six-agent system, traceable claims and the hidden cost of losing apprenticeship.

Published:  · Added:  · 3 min

Agents do better when generated lessons stay temporary

Anti-Persistence saves execution records and rebuilds advice for each task, reducing one route by which an agent can keep repeating a bad lesson.

Published:  · Added:  · 2 min

How to Evaluate Coding Agents

Momento's Khawaja Shams explains realistic coding tasks, leaked answers, repeated trials, and the traces that make agent scores worth trusting.

Published:  · Added:  · 2 min