Coding assistants work around stale instructions without fixing them
A controlled study finds obsolete test commands can execute no tests while appearing successful. Final patch scores miss the wasted work.
A controlled study finds obsolete test commands can execute no tests while appearing successful. Final patch scores miss the wasted work.
One poisoned tool result can affect later sessions, but existing history changes the outcome sharply. Storage, retrieval and obedience need separate tests.
Fabian Theis explains how virtual-cell research can connect biological maps, intervention predictions, and repeated laboratory experiments.
Workflow outcomes can help tune a code-review model, but abandoned work is an imperfect quality label and hidden defects need better inputs.
Coatue’s Frank Long breaks investment research into testable steps run by code, statistical models, language models, and people.
OpenAI's kernel-auditing agents sometimes exploited the wrong bug. Checking how an agent succeeds matters as much as checking its result.
Netflix’s Cameron Wolfe connects calibrated human review, automated judges, and production experiments to make subjective AI evaluations more reliable.
A Terminal-Bench audit separates broken tests, changing dependencies and resource limits from model capability.
Matched coding tasks exposed silent scope changes and invented blockers. OpenAI measured the gap, but its training cause remains unresolved.
Blackstone and Norm AI CTOs discuss usable data, replaceable workflows, durable expert tests, and when to build, prototype, or wait.
Brainbase’s Domino’s pricing case pairs explicit definitions, browser traces, and repeated tests to improve a narrowly scoped agent.
Hamel Husain compares six classifiers on leasing-assistant traces, then shows how to tune thresholds, check confidence scores and weigh cost.
Synthetic handoff tests separate false approvals from invented restrictions. Their replay results differ, and clearer prompt wording is not a proven safety control.
Fixed processing dimensions preserve a protected request's output, with speed gains that depend on workload and deployment.
Swyx explains why his conference team rebuilt SaaS tools, what its AI judge missed, and how Jev could put narrow model decisions inside ordinary software.
Langfuse's banking case combines representative test data, calculator tools, separate quality gates and expert review to decide when an agent can ship.
Datasite's CTO explains how document edge cases changed its build-versus-buy decision, and how a private benchmark checks the downstream answers.
NatWest's routing funnel, human reviews and miscalibrated judges show what it takes to prepare a customer-facing banking agent for release.
Basis co-founder Mitchell Troyanovsky explains how behavior specs, replayed decisions and lightweight simulations help teams trust production agents.
Verified but Vacuous separates proof acceptance, reference to a specification and agreement with user intent. Each needs its own evidence and denominator.
Zach Lloyd demonstrates Warp’s shared agent workflows, scoring and benchmarks, and explains which self-improvement capabilities remain future work.
George Cameron breaks down an agent benchmark's $21 task cost, its over-$200 uncached estimate, and how to choose models against your own workload.
NVIDIA tests short working histories across coding and document tasks. Its headline savings assume hypothetical prices, and some local models lose substantial accuracy.
Reading raw network parameters and writing Python to improve a model use different capabilities. Honnibal separates the experiments behind the headline.
A review finds faster single-request generation with four-bit weights, little gain in one large-batch test, and sparse matched accuracy evidence.
Two grading models can agree on the same mistake. Cross-provider agreement filtered one unreliable grader while discarding most comparisons.
Jacob Lauritzen explains lawyer-built tests, restricted production feedback, and customer-supplied examples for improving legal agents without reading confidential client work.
MIT researchers report targeted weight edits that let a small Qwen model finish more answers, plus experiments reading and writing game networks.
Across 120 historical distributed-system bugs, coding assistants find relevant files more reliably than the precise code that caused the failure.
Separate working copies keep code intact, but task-specific history and progress records help coding assistants resume interrupted work.
Ramakrishnan Lokanathan on testing complete AI workflows, retaining regression checks, and turning real failures into evaluations.
A web measurement study finds that successful status codes can hide empty shells, bot checks and substitute pages, distorting citation checks.
Bumble's Supriya Subudhi shares five lessons from testing AI photo feedback: adoption, bad outputs, costs, model choice and regional rollout.
Upstart and Coval describe human review, AI judges and voice simulations for a regulated loan-verification agent.
A timestamped deep dive through a three-hour course: Gmail intake, typed model decisions, Python policy, evals, Docker deployment, and the failure cases still to test.
Tracing 1,285 AI citation trails shows how numbers survive after their evidence disappears or changes meaning. The automated fact-checker needed checking too.
IBM's Mixture of Experts connects AI-assisted security research, Block's speech-only Dessa preview, culturally specific evaluation and the data infrastructure agents need.
Rayan Krishnan shows how task-specific tests can reverse model rankings, expose hidden token costs, and turn a firm's past work into a buying guide.
Sonar’s Gitar workshop checks three ticketing-app changes: a refund bug outside the diff, an all-in pricing rule, and a failing check-in test.
Sonar CEO Tariq Shaukat argues for explicit intent, guidance, independent verification, and evidence as coding agents outpace human review.
Qodo’s Cole Gannaway explains cross-repository review, independent review agents, and turning recurring review comments into shared rules.
Loading pages can invalidate a correct browser plan; pixel checks and outcome checks solve different parts of the problem.
Addy Osmani and Manish Kapur discuss first-pass agent review, browser evidence, layered checks, and the human ownership behind larger code-change volumes.
Vik Paruchuri explains how document benchmarks reward formatting, how to test the scorer, and how to build verifiable checks around your own documents.
IBM's Security Intelligence panel connects AI-assisted vulnerability discovery with patch readiness and debates outside access to frontier-model evaluations.
Tests of self-improving instructions can miss redundant protections disappearing, or conflicting instructions added around unchanged safety rules.
Written priorities redirect coding assistants more reliably than another conflicting code example in controlled repository experiments.
Fuad Ali explains Alex's production design: harness-owned plans, explicit completion, file-backed results, small auditable memory and regression tests drawn from real sessions.
Mikyo King explains Phoenix's code-mode design: discover tools, compose them in code, and return smaller results, while handling secrets, permissions and failed executions.
FeatureArc tests coding agents, models running multi-step software tasks with tools, as they extend one feature on their own earlier code. Each new request inherits any unresolved mistakes. The authors evaluate ten agents on 73 reviewed …