Gemini 4 Neon looks like an Opus 5.5 competitor — micjm
A coding-model preference analysis suggests speed matters nearly as much as model identity. Its Neon–Opus comparison is a projection.
A coding-model preference analysis suggests speed matters nearly as much as model identity. Its Neon–Opus comparison is a projection.
A coding agent can make a real edit, pass a real test and still misreport what it accomplished. FaithDetector checks work reports against the recorded actions and evidence behind them. The researchers distinguish wrong observations, …
CompAct-Bench measures whether an agent, a model doing multi-step work with tools, can finish after its conversation is compressed. The test grades the resumed task rather than the summary’s readability. The researchers start with 200 tasks …
A controlled sequence-prediction test finds a striking Claude capability, but inferring its training data is a separate claim.
Melvin Johnson describes harder training tasks, reviewer-style rubrics, isolated graders and checks that tie an agent's completion claims to its actions.
PostHog uses narrow decision models to skip most rewrite calls, but its failed experiments show why thresholds and workload-specific tests matter.
A trace simulation cuts calculated GPU-hour cost by 16.4%, but requires dependable acceptance tests and much longer turnaround.
A 108-task game study separates functional acceptance, model-scored play recordings and blind human preference, with assisted-state and configuration caveats.
Runta and Baseten compare 22 coding-agent configurations across five models. Cost rankings shift sharply, but most best-value winners remain uncertain.
Intermediate patches reveal expensive work after tests pass, while corrected scoring weakens claims that early stopping preserves correctness.
A controlled crash study separates retry safety from recovery, with substantial durability costs and unavailable experimental artifacts.
Controlled changes to paths, forms and APIs expose stale skills; checked repairs can restore accuracy and execution savings.
A coding-agent study separates prompt reuse from fresh input and output, finding that fewer tokens can still mean a larger bill.
Andreas Blattmann and Dustin Podell explain how targeted data unlocked FLUX Kontext, what SDXL taught them, and where robot learning still falls short.
A human-planted research-sabotage study finds weak detection despite detailed records, AI assistance and permission to rerun experiments.
Sachin Gupta’s AgentFaultBench tests whether recovery instructions help agents, models running multi-step tasks with tools, finish after errors. LangGraph and CrewAI operate on a real isolated GitLab installation, with controlled failures …
Elmira Onagh and Maleknaz Nayebi tested how explicit requirements shape ChatGPT-4o code. A detailed list of behavioral obligations helped much more than a lightweight specification template. Their controlled study assigned 72 participants …
Nikhil Kumar Jha and Amitoj Cheema rerun LoCoMo, a fixed set of conversational-memory questions, while changing answer instructions and grading models. Their results show how much an apparently better memory score can depend on the …
A self-hosted browser test adds audio, video and cross-application workflows, with important limits on what its leaderboard measures.
A learned selector can restore original observations during long tasks, but state tracking and simple retention explain part of the gains.
A small mutation-testing study shows how aggregate pass rates conceal lost functionality, and why requirement coverage needs behavioral checks.
Glean’s search engineers explain how they build retrieval evals from usage and synthetic data, inspect traces, and test changes with users.
Ruixin Wang and Tianyi Zhang study coding agents that consider a requirement-compliant choice but settle on a shortcut. Their SatisFix prompt asks the agent to report planned departures and then reconsider them against the task. On 113 …
Intel's Nikolay Petrov compares model answers with and without skill instructions, mapping redundant, helpful and harmful claims before editing a skill.
AWS's Rajath Mahendrappa Kotyal demonstrates an LLM-and-logic-solver workflow for finding contradictory, incomplete and ambiguous requirements before code generation.
Duolingo's Kush Agrawal describes a scheduled agent-test pipeline with durable workflows, CI repair, human review and guardrails that raised reported iOS test coverage.
Maersk's Dmitry Buykin explains how expert-owned rules, restricted runtime context, real-task replay and weekly pruning keep an agent-maintained system from regrowing its mess.
Musa Cim and colleagues saved intermediate coding-agent patches and checked them against benchmark tests after each run. Their measure, termination lag, counts the steps between a first passing patch and the end of the run. In successful …
Ara Khan describes an agent iterating on Cline's coding harness from Terminal-Bench failures, then argues for engineering work that benefits from stronger models without depending on a single benchmark.
Hamel Husain introduces trace review, failure grouping, calibrated binary checks and held-out tests for AI products.
Stephen and Cassandra Chin show how a branch's trace and an evidence graph can anchor an AI-written pull-request description.
Ayush Bhardwaj reads 20,000 agent tool calls to find missing selection flags, hidden errors and stale skill instructions.
Brian Douglas shows how labeled coding-agent traces can capture team conventions before a pull request reaches review.
A data-platform engineer explains how deterministic checks and a code-reading agent review pipeline PRs while humans keep sensitive decisions.
LinkedIn's prompt-optimization loop found that unreliable judge scores and root-cause diagnoses can send agent improvements in the wrong direction.
Toptal ExpertCrowd’s ExpertCrowd-SWE tests coding agents on 100 change requests written by engineers familiar with the repositories. The requests span 87 projects and 11 languages. Each task starts from a pinned commit, with an unpublished …
Inside Handshake’s scientific-image tests, simulated banks and safety evaluations, where polished outputs hide costly mistakes.
Warp CEO Zach Lloyd demonstrates a version-controlled coding-agent workflow that scores runs, proposes changes and replays real tasks.
Hamel Husain and Angelina Yang work through trace review, judge calibration, and the human decisions behind useful product evaluations.
Matthew Jewkes separates errors, unsettled definitions, and exceptional cases to automate routine work and recover expert attention.
A controlled study separates matching a hidden reference from answering with enough visible evidence.
Keiji Kanazawa finds architectural drift, empty jobs, and an overfit rubric in his personal research assistant.
Ramp's Ryan Stevens describes the evaluations his team built before launching its bookkeeping agent, and what they learned from running them.
AlixPartners describes how orphaned amendments, renewal clauses and access errors produced plausible but wrong procurement answers.
Joao Wenzel describes how Citadel Securities combines project records, fresh evidence and human corrections to track what is really happening.
Fieldguide’s Anirudh Veeraragavan explains state-aware permissions, step-level execution records, exact evidence identities, and tests whose expected answers remain trustworthy.
Cohere’s Nouha Dziri tests team structure, worker placement, model costs, and messaging. Cheaper planners help when discovering the program is the bottleneck.
Hightouch nearly matches Opus by escalating uncertain decisions. Harder cases shrink the saving, and HN practitioners question the changing economics.
Warp’s Ben Holmes scores conversations, tool calls, and code changes to find recurring failures and propose reviewed repairs to agent instructions and setup.
Anthropic’s Lance Martin covers prompt debt, cache reuse, evaluation-driven optimization, and recoverable agents, with a detailed Q&A on overfitting and human judgment.