Gemini 4 Neon looks like an Opus 5.5 competitor — micjm

A coding-model preference analysis suggests speed matters nearly as much as model identity. Its Neon–Opus comparison is a projection.

Published:  · Added:  · Updated:  · What changed: Explained early voting and narrowed the Copilot–Claude Code anecdote to code review. · Reviewed comments: 86 · 2 min

FaithDetector checks what coding agents say they finished

A coding agent can make a real edit, pass a real test and still misreport what it accomplished. FaithDetector checks work reports against the recorded actions and evidence behind them. The researchers distinguish wrong observations, …

Published:  · Added:  · 1 min

CompAct-Bench tests whether agents can work after a summary

CompAct-Bench measures whether an agent, a model doing multi-step work with tools, can finish after its conversation is compressed. The test grades the resumed task rather than the summary’s readability. The researchers start with 200 tasks …

Published:  · Added:  · 1 min

Claude was trained on protein sequences at scale — Xinming Tu and Leo Tianlai Chen

A controlled sequence-prediction test finds a striking Claude capability, but inferring its training data is a separate claim.

Published:  · Added:  · Updated:  · What changed: Added verified organizational biology-access options, the individual-plan limit, and the 30-day data-retention requirement. · Reviewed comments: 82 · 2 min

How DeepMind grades coding agents beyond passing tests

Melvin Johnson describes harder training tasks, reviewer-style rubrics, isolated graders and checks that tie an agent's completion claims to its actions.

Published:  · Added:  · 2 min

Not everything needs an LLM — Jina Yoon

PostHog uses narrow decision models to skip most rewrite calls, but its failed experiments show why thresholds and workload-specific tests matter.

Published:  · Added:  · Updated:  · What changed: Added the calibration objection to treating decision scores as measured error rates. · Reviewed comments: 87 · 2 min

A small-first coding model saves compute while doubling the wait

A trace simulation cuts calculated GPU-hour cost by 16.4%, but requires dependable acceptance tests and much longer turnaround.

Published:  · Added:  · 2 min

Game-building agents need tests of play, not just working code

A 108-task game study separates functional acceptance, model-scored play recordings and blind human preference, with assisted-state and configuration caveats.

Published:  · Added:  · 2 min

Changing Models Can Change Which Coding Harness Is Cheapest

Runta and Baseten compare 22 coding-agent configurations across five models. Cost rankings shift sharply, but most best-value winners remain uncertain.

Published:  · Added:  · Updated:  · What changed: Added the larger pooled benchmark analysis and clarified why its pass-rate findings do not directly retest Runta's cost rankings. · Reviewed comments: 159 · 3 min

Stopping after passing tests cuts coding costs, with uncertain quality

Intermediate patches reveal expensive work after tests pass, while corrected scoring weakens claims that early stopping preserves correctness.

Published:  · Added:  · 2 min

Preventing duplicate agent actions does not tell you whether they finished

A controlled crash study separates retry safety from recovery, with substantial durability costs and unavailable experimental artifacts.

Published:  · Added:  · 2 min

Saved instructions need maintenance when their assumptions expire

Controlled changes to paths, forms and APIs expose stale skills; checked repairs can restore accuracy and execution savings.

Published:  · Added:  · 2 min

Shorter coding conversations can cost more when prompts are cached

A coding-agent study separates prompt reuse from fresh input and output, finding that fewer tokens can still mean a larger bill.

Published:  · Added:  · 2 min

Black Forest Labs turns image generation into editable workflows

Andreas Blattmann and Dustin Podell explain how targeted data unlocked FLUX Kontext, what SDXL taught them, and where robot learning still falls short.

Published:  · Added:  · 3 min

Full research records still leave auditors missing planted sabotage

A human-planted research-sabotage study finds weak detection despite detailed records, AI assistance and permission to rerun experiments.

Published:  · Added:  · 2 min

Recovery advice can make assistants stop after temporary errors

Sachin Gupta’s AgentFaultBench tests whether recovery instructions help agents, models running multi-step tasks with tools, finish after errors. LangGraph and CrewAI operate on a real isolated GitLab installation, with controlled failures …

Published:  · Added:  · 2 min

Explicit behavior rules reduce failures in generated code

Elmira Onagh and Maleknaz Nayebi tested how explicit requirements shape ChatGPT-4o code. A detailed list of behavioral obligations helped much more than a lightweight specification template. Their controlled study assigned 72 participants …

Published:  · Added:  · 2 min

Memory-test scores depend on prompts, graders and baseline formatting

Nikhil Kumar Jha and Amitoj Cheema rerun LoCoMo, a fixed set of conversational-memory questions, while changing answer instructions and grading models. Their results show how much an apparently better memory score can depend on the …

Published:  · Added:  · 2 min

WebArena-Pro tests browser assistants across twenty resettable applications

A self-hosted browser test adds audio, video and cross-application workflows, with important limits on what its leaderboard measures.

Published:  · Added:  · 2 min

CarryOn keeps exact evidence beside a short progress record

A learned selector can restore original observations during long tasks, but state tracking and simple retention explain part of the gains.

Published:  · Added:  · 2 min

A test suite can improve while an assistant loses a tool

A small mutation-testing study shows how aggregate pass rates conceal lost functionality, and why requirement coverage needs behavioral checks.

Published:  · Added:  · 2 min

How Glean Evaluates Agent Retrieval

Glean’s search engineers explain how they build retrieval evals from usage and synthetic data, inspect traces, and test changes with users.

Published:  · Added:  · 2 min

A prompt asks coding agents to reconsider shortcuts before writing code

Ruixin Wang and Tianyi Zhang study coding agents that consider a requirement-compliant choice but settle on a shortcut. Their SatisFix prompt asks the agent to report planned departures and then reconsider them against the task. On 113 …

Published:  · Added:  · 1 min

Testing What Your SKILL.md Actually Teaches

Intel's Nikolay Petrov compares model answers with and without skill instructions, mapping redundant, helpful and harmful claims before editing a skill.

Published:  · Added:  · 2 min

Your Spec Has Bugs. Catch Them Before Your Agent Codes.

AWS's Rajath Mahendrappa Kotyal demonstrates an LLM-and-logic-solver workflow for finding contradictory, incomplete and ambiguous requirements before code generation.

Published:  · Added:  · 1 min

225,000 Lines of Tests, Written and Merged by Agents

Duolingo's Kush Agrawal describes a scheduled agent-test pipeline with durable workflows, CI repair, human review and guardrails that raised reported iOS test coverage.

Published:  · Added:  · 2 min

Agents Made Code Free. Maintaining It Didn't Get Cheaper.

Maersk's Dmitry Buykin explains how expert-owned rules, restricted runtime context, real-task replay and weekly pruning keep an agent-maintained system from regrowing its mess.

Published:  · Added:  · 2 min

Coding agents often keep working after a patch first passes hidden tests

Musa Cim and colleagues saved intermediate coding-agent patches and checked them against benchmark tests after each run. Their measure, termination lag, counts the steps between a first passing patch and the end of the run. In successful …

Published:  · Added:  · 1 min

Evaluate Harness Changes Without Overfitting the Benchmark

Ara Khan describes an agent iterating on Cline's coding harness from Terminal-Bench failures, then argues for engineering work that benefits from stronger models without depending on a single benchmark.

Published:  · Added:  · 2 min

Start Product Evals With Traces and Specific Failure Modes

Hamel Husain introduces trace review, failure grouping, calibrated binary checks and held-out tests for AI products.

Published:  · Added:  · 2 min

A PR's Test Claim Needs Evidence From the Submitted Change

Stephen and Cassandra Chin show how a branch's trace and an evidence graph can anchor an AI-written pull-request description.

Published:  · Added:  · 2 min

Repeated Agent Workarounds Are Bug Reports for Your CLI

Ayush Bhardwaj reads 20,000 agent tool calls to find missing selection flags, hidden errors and stale skill instructions.

Published:  · Added:  · 2 min

Turn Expert Pushback on Agent Code Into Review Evidence

Brian Douglas shows how labeled coding-agent traces can capture team conventions before a pull request reaches review.

Published:  · Added:  · 2 min

Robinhood Splits Data-PR Review Between Scripts, Agents and People

A data-platform engineer explains how deterministic checks and a code-reading agent review pipeline PRs while humans keep sensitive decisions.

Published:  · Added:  · 2 min

Why LinkedIn's Self-Improving Agent Needs a Trustworthy Judge

LinkedIn's prompt-optimization loop found that unreliable judge scores and root-cause diagnoses can send agent improvements in the wrong direction.

Published:  · Added:  · 2 min

Expert-written coding tasks test whether agents meet maintainer requirements

Toptal ExpertCrowd’s ExpertCrowd-SWE tests coding agents on 100 change requests written by engineers familiar with the repositories. The requests span 87 projects and 11 languages. Each task starts from a pinned commit, with an unpublished …

Published:  · Added:  · 2 min

Handshake shows where professional AI agents still fail

Inside Handshake’s scientific-image tests, simulated banks and safety evaluations, where polished outputs hide costly mistakes.

Published:  · Added:  · 4 min

Measure and Improve Your AI Software Factory

Warp CEO Zach Lloyd demonstrates a version-controlled coding-agent workflow that scores runs, proposes changes and replays real tasks.

Published:  · Added:  · 2 min

Build Product Evals from Real Failures and Human Judgment

Hamel Husain and Angelina Yang work through trace review, judge calibration, and the human decisions behind useful product evaluations.

Published:  · Added:  · 3 min

Route Ambiguity Away from Routine AI Automation

Matthew Jewkes separates errors, unsettled definitions, and exceptional cases to automate routine work and recover expert attention.

Published:  · Added:  · 2 min

Correct answers can hide missing evidence in agent memory tests

A controlled study separates matching a hidden reference from answering with enough visible evidence.

Published:  · Added:  · 2 min

Own the Goals and Check the Rubrics for Coding Agents

Keiji Kanazawa finds architectural drift, empty jobs, and an overfit rubric in his personal research assistant.

Published:  · Added:  · 2 min

Test bookkeeping agents in repeatable synthetic worlds

Ramp's Ryan Stevens describes the evaluations his team built before launching its bookkeeping agent, and what they learned from running them.

Published:  · Added:  · 2 min

Check agent procurement answers against the contracts

AlixPartners describes how orphaned amendments, renewal clauses and access errors produced plausible but wrong procurement answers.

Published:  · Added:  · 2 min

Build a company memory that ages and corrects itself

Joao Wenzel describes how Citadel Securities combines project records, fresh evidence and human corrections to track what is really happening.

Published:  · Added:  · 2 min

Four Controls for Defensible Financial Agent Decisions

Fieldguide’s Anirudh Veeraragavan explains state-aware permissions, step-level execution records, exact evidence identities, and tests whose expected answers remain trustworthy.

Published:  · Added:  · 2 min

Put Coding Agents Where the Task Gets Stuck

Cohere’s Nouha Dziri tests team structure, worker placement, model costs, and messaging. Cheaper planners help when discovering the program is the bottleneck.

Published:  · Added:  · 2 min

Cheap models need confidence that survives hard cases — Albert Jin, Neil Desai and Delaney Mackenzie

Hightouch nearly matches Opus by escalating uncertain decisions. Harder cases shrink the saving, and HN practitioners question the changing economics.

Published:  · Added:  · 2 min

Use Agent Traces to Repair Skills and Configuration

Warp’s Ben Holmes scores conversations, tool calls, and code changes to find recurring failures and propose reviewed repairs to agent instructions and setup.

Published:  · Added:  · 2 min

Audit Prompts and Evals Before Buying More Model

Anthropic’s Lance Martin covers prompt debt, cache reuse, evaluation-driven optimization, and recoverable agents, with a detailed Q&A on overfitting and human judgment.

Published:  · Added:  · 2 min