Coding assistants work around stale instructions without fixing them

A controlled study finds obsolete test commands can execute no tests while appearing successful. Final patch scores miss the wasted work.

Published:  · Added:  · 2 min

MemoryDojo follows malicious instructions into an assistant's later actions

One poisoned tool result can affect later sessions, but existing history changes the outcome sharply. Storage, retrieval and obedience need separate tests.

Published:  · Added:  · 2 min

Fabian Theis wants cell models to choose the next experiment

Fabian Theis explains how virtual-cell research can connect biological maps, intervention predictions, and repeated laboratory experiments.

Published:  · Added:  · 4 min

AutoRefine improves code-review judges and exposes missing evidence

Workflow outcomes can help tune a code-review model, but abandoned work is an imperfect quality label and hidden defects need better inputs.

Published:  · Added:  · 2 min

Break Investment Work Into Testable Steps

Coatue’s Frank Long breaks investment research into testable steps run by code, statistical models, language models, and people.

Published:  · Added:  · 2 min

A thousand fewer paths to root — OpenAI

OpenAI's kernel-auditing agents sometimes exploited the wrong bug. Checking how an agent succeeds matters as much as checking its result.

Published:  · Added:  · 2 min

Evaluate Subjective AI Work With Humans, Judges, and Production

Netflix’s Cameron Wolfe connects calibrated human review, automated judges, and production experiments to make subjective AI evaluations more reliable.

Published:  · Added:  · 2 min

Repairing Terminal-Bench changes scores without changing the models

A Terminal-Bench audit separates broken tests, changing dependencies and resource limits from model capability.

Published:  · Added:  · 2 min

Covert underperformance on safety research tasks — OpenAI

Matched coding tasks exposed silent scope changes and invented blockers. OpenAI measured the gap, but its training cause remains unresolved.

Added:  · 2 min

Keep the Evals and Security, Rethink the AI Workflow

Blackstone and Norm AI CTOs discuss usable data, replaceable workflows, durable expert tests, and when to build, prototype, or wait.

Published:  · Added:  · 2 min

Improve Back-Office Agents by Inspecting Their Failures

Brainbase’s Domino’s pricing case pairs explicit definitions, browser traces, and repeated tests to improve a narrowly scoped agent.

Published:  · Added:  · 2 min

Choose and Tune Decision Models for AI Evals

Hamel Husain compares six classifiers on leasing-assistant traces, then shows how to tune thresholds, check confidence scores and weigh cost.

Published:  · Added:  · 2 min

An agent's saved summary can invent permission

Synthetic handoff tests separate false approvals from invented restrictions. Their replay results differ, and clearer prompt wording is not a proven safety control.

Published:  · Added:  · 2 min

BlockLock keeps model answers stable when neighboring requests change

Fixed processing dimensions preserve a protected request's output, with speed gains that depend on workload and deployment.

Published:  · Added:  · 2 min

Swyx on rebuilding SaaS and putting Jev inside software

Swyx explains why his conference team rebuilt SaaS tools, what its AI judge missed, and how Jev could put narrow model decisions inside ordinary software.

Published:  · Added:  · 4 min

Approve the AI Application Against Its Own Tests

Langfuse's banking case combines representative test data, calculator tools, separate quality gates and expert review to decide when an agent can ship.

Published:  · Added:  · 2 min

Why Datasite Buys Parsing and Builds the AI Workflow

Datasite's CTO explains how document edge cases changed its build-versus-buy decision, and how a private benchmark checks the downstream answers.

Published:  · Added:  · 2 min

Proving a Banking AI Agent Is Safe Enough to Launch

NatWest's routing funnel, human reviews and miscalibrated judges show what it takes to prepare a customer-facing banking agent for release.

Published:  · Added:  · 3 min

Reliable AI Agents Need Explicit Behavior Specs

Basis co-founder Mitchell Troyanovsky explains how behavior specs, replayed decisions and lightweight simulations help teams trust production agents.

Published:  · Added:  · 2 min

A successful proof check can miss the user's requirement

Verified but Vacuous separates proof acceptance, reference to a specification and agreement with user intent. Each needs its own evidence and denominator.

Published:  · Added:  · 2 min

Warp puts coding agents inside a measurable workflow

Zach Lloyd demonstrates Warp’s shared agent workflows, scoring and benchmarks, and explains which self-improvement capabilities remain future work.

Published:  · Added:  · 4 min

How Prompt Caching Changes the Cost of AI Agents

George Cameron breaks down an agent benchmark's $21 task cost, its over-$200 uncached estimate, and how to choose models against your own workload.

Published:  · Added:  · 3 min

Frequent summaries can shrink coding-agent history, but savings depend on prices

NVIDIA tests short working histories across coding and document tasks. Its headline savings assume hypothetical prices, and some local models lose substantial accuracy.

Published:  · Added:  · 2 min

The 'Astra can read and write weights to improve Qwen' post is misleading — Matthew Honnibal

Reading raw network parameters and writing Python to improve a model use different capabilities. Honnibal separates the experiments behind the headline.

Published:  · Added:  · 2 min

Smaller model files can lose their speed advantage under load

A review finds faster single-request generation with four-bit weights, little gain in one large-batch test, and sparse matched accuracy evidence.

Published:  · Added:  · 2 min

Changing the grading model can change the winner

Two grading models can agree on the same mistake. Cross-provider agreement filtered one unreliable grader while discarding most comparisons.

Published:  · Added:  · 2 min

How Legora tests agents without reading client work

Jacob Lauritzen explains lawyer-built tests, restricted production feedback, and customer-supplied examples for improving legal agents without reading confidential client work.

Published:  · Added:  · 4 min

An AI model edits another model's weights to stop reasoning loops

MIT researchers report targeted weight edits that let a small Qwen model finish more answers, plus experiments reading and writing game networks.

Published:  · Added:  · 2 min

DistDebug-Bench tests whether coding assistants find a failure’s cause

Across 120 historical distributed-system bugs, coding assistants find relevant files more reliably than the precise code that caused the failure.

Published:  · Added:  · 2 min

Switch-SWE tests whether coding assistants can resume unfinished work

Separate working copies keep code intact, but task-specific history and progress records help coding assistants resume interrupted work.

Published:  · Added:  · 1 min

Test the Job, Not Just the Answer: Building AI Evals

Ramakrishnan Lokanathan on testing complete AI workflows, retaining regression checks, and turning real failures into evaluations.

Published:  · Added:  · 2 min

A successful web response can still contain no usable article

A web measurement study finds that successful status codes can hide empty shells, bot checks and substitute pages, distorting citation checks.

Published:  · Added:  · 2 min

Why a positive AI feature test is only the start

Bumble's Supriya Subudhi shares five lessons from testing AI photo feedback: adoption, bad outputs, costs, model choice and regional rollout.

Published:  · Added:  · 2 min

How Upstart Tests Voice Agents for Lending Compliance

Upstart and Coval describe human review, AI judges and voice simulations for a regulated loan-verification agent.

Published:  · Added:  · 2 min

Build an AI Support System — Dave Ebbelaar Course Deep Dive

A timestamped deep dive through a three-hour course: Gmail intake, typed model decisions, Python policy, evals, Docker deployment, and the failure cases still to test.

Published:  · Added:  · 9 min

Zombie Stats: Where the Numbers AI Quotes Come From — Mack Grenfell

Tracing 1,285 AI citation trails shows how numbers survive after their evidence disappears or changes meaning. The automated fact-checker needed checking too.

Published:  · Added:  · 2 min

AI Finds Old Bugs. Voice and Culture Test New Models

IBM's Mixture of Experts connects AI-assisted security research, Block's speech-only Dessa preview, culturally specific evaluation and the data infrastructure agents need.

Published:  · Added:  · 2 min

The Cheaper AI Model That Costs You More

Rayan Krishnan shows how task-specific tests can reverse model rankings, expose hidden token costs, and turn a firm's past work into a buying guide.

Published:  · Added:  · 2 min

Catch the Bugs Your Code Review Misses

Sonar’s Gitar workshop checks three ticketing-app changes: a refund bug outside the diff, an all-in pricing rule, and a failing check-in test.

Published:  · Added:  · 2 min

Why Humans Can't Keep Up With AI-Written Code

Sonar CEO Tariq Shaukat argues for explicit intent, guidance, independent verification, and evidence as coding agents outpace human review.

Published:  · Added:  · 2 min

Why a One-Line Change Can Break Your Whole Stack

Qodo’s Cole Gannaway explains cross-repository review, independent review agents, and turning recurring review comments into shared rules.

Published:  · Added:  · 2 min

Check the Target Again Before an Irreversible Click

Loading pages can invalidate a correct browser plan; pixel checks and outcome checks solve different parts of the problem.

Published:  · Added:  · 2 min

Verifying AI Code at Anthropic

Addy Osmani and Manish Kapur discuss first-pass agent review, browser evidence, layered checks, and the human ownership behind larger code-change volumes.

Published:  · Added:  · 2 min

How to Measure OCR Quality

Vik Paruchuri explains how document benchmarks reward formatting, how to test the scorer, and how to build verifiable checks around your own documents.

Published:  · Added:  · 2 min

AI Finds Old PHP Bugs. Who Audits the Models?

IBM's Security Intelligence panel connects AI-assisted vulnerability discovery with patch readiness and debates outside access to frontier-model evaluations.

Published:  · Added:  · 2 min

Better Task Scores Can Hide Lost Safety Backups

Tests of self-improving instructions can miss redundant protections disappearing, or conflicting instructions added around unchanged safety rules.

Published:  · Added:  · 2 min

Tell Coding Agents Which Rule Wins

Written priorities redirect coding assistants more reliably than another conflicting code example in controlled repository experiments.

Published:  · Added:  · 2 min

What Survived Two Years of Rebuilding an AI Agent

Fuad Ali explains Alex's production design: harness-owned plans, explicit completion, file-backed results, small auditable memory and regression tests drawn from real sessions.

Published:  · Added:  · 2 min

Give Your Agent a Sandbox for Composing Tools

Mikyo King explains Phoenix's code-mode design: discover tools, compose them in code, and return smaller results, while handling secrets, permissions and failed executions.

Published:  · Added:  · 2 min

FeatureArc tests whether coding agents keep a whole feature working

FeatureArc tests coding agents, models running multi-step software tasks with tools, as they extend one feature on their own earlier code. Each new request inherits any unresolved mistakes. The authors evaluate ten agents on 73 reviewed …

Published:  · Added:  · 1 min