Fifty professional analysts used Claude Code with Opus 4.8 on one fictional farming-policy task. An agent is a model running a multi-step task with tools.
Hidden instructions randomly assigned preferences on four analytical choices and the final recommendation. Analysts followed the analytical preferences 86% of the time, against 50% expected without influence.
Among 38 analytical decisions where people expressed a preference first, they switched 60% of the time when the agent disagreed, versus 6% when it agreed.
Of 48 survey respondents, 77% reused an agent argument and 15% credited it. Meanwhile, 52% considered the final recommendation more their own than the agent’s.
The full study separates these randomized preferences from the eleven participants who chose to work without AI. That self-selected group cannot establish AI’s causal productivity benefit.
Caveat: The experiment deliberately strengthened the agent’s preferences in one unfamiliar domain, so its influence rates cannot be assumed for ordinary workplace use.
(Techmeme · primary study, methods and limitations)
A document-review system makes missing evidence visible#
Enveda’s October 27 methods describe windows of up to forty scanned pages, a code-maintained coverage ledger and notices for unread pages. Original records remain read-only.
Across four manufacturing records totaling 1,061 pages, the system matched eleven human-identified issues in at least one run. An Opus 4.7 pass added seven actionable issues.
Processing plus checking supplied comments took 166 minutes, versus 1,553 minutes for unaided review. Model calls cost $36.97, excluding development, hosting and human time.
In the reported pass, 79% of model comments required no action. Each tested model run also misread one handwritten entry, despite citing content actually present.
The MIT-licensed companion code implements generic coverage, reconciliation and logging controls. Production integrations, configured prompts and rule sets remain proprietary.
Caveat: These four company records also informed prompt tuning, and the faster arm checked supplied comments and cited pages rather than replacing the required full human review.
(HN 236 points · 227 comments · Techmeme · linked company methods)
October 27’s study tests ten coding agents on 73 reviewed tasks with two to ten stages across four languages. Each agent continues from its own previous code.
FeatureArc is a benchmark, a fixed set of tasks for comparing models. Hidden acceptance results stay withheld between stages, although agents can write and run their own tests.
Across agents, 62.7% of first stages pass, versus 28.5% of complete features on final code. Opus 5.5 leads at 83.6% and 49.3%, respectively.
Earlier defects appear in 72.9% of failed later stages. Agents leave prior errors unnoticed in 94.9% of stages following a failure, according to checked transcript labels.
Of 273 runs passing the final stage’s tests, 69 still fail the complete-feature check. Separate tests of earlier requirements catch failures the latest extension misses.
Caveat: This tests autonomous development without intermediate acceptance feedback, and Tencent’s authors also use the evaluation to develop their own Hunyuan models.
(arXiv 2610.37434 · full methods, results and evaluation boundaries)
Useful bug findings differ from the bugs maintainers prioritize#
October 26’s study collects 2,457 bugs fixed over eighteen months across twenty repositories. The reference list covers defects already present in the starting code.
Splitting audits into smaller file groups raises the best single-run recovery to 20.8% of reference bugs. Whole-repository prompts recover less than 1% in the tested baseline configurations.
At most 8.1% of reported findings match the reference list. Separately, model reviewers confirm 84% of unmatched sampled findings as real bugs, excluding inconclusive judgments.
A blind human check agrees with the panel on 93% of 131 conclusive comparisons. Agents nevertheless favor local logic errors over missing support for legitimate inputs.
The project describes its dataset, but download access could not be verified. A separate human audit rejects twelve of one hundred sampled reference entries.
Caveat: The incomplete, partly disputed reference list measures agreement with historical fixes, so its low matching percentage is not a false-report rate.
(arXiv 2610.35987 · audit methods, human checks and reference-set limitations)
Inspect what reaches the model before buying a larger window#
October 26’s study examines a diagnostic harness, the software around a model that runs its tools, files and commands. Across 720 runs, its largest prompt uses only 5.75% of the available window.
A fixed 6,000-character cut damages 3,827 of 11,189 results, leaving invalid JSON, a structured data format. Late evidence fields survive far less often than early fields.
In a controlled comparison with reversed keys and truncation, median requests rise from 53 to 76 and computations from eighteen to 36. A tool call is the model asking a program to do something.
That comparison still solves fifteen of fifteen cases, versus fourteen of fifteen after repair. Another path to the evidence preserves answers while increasing work.
The replication package includes recorded results and replay tools. Cutting at whole-record boundaries preserves valid data and makes omitted evidence explicit.
Caveat: This studies one diagnostic system and model, and its small controlled tests do not establish that truncation alone explains the larger accuracy gap in field observations.
(arXiv 2610.36114 · controlled comparisons, production-scale replay and limitations)
Correct answers can conceal broken prompt caching#
Prompt caching reuses computation for an identical beginning of a request. October 27’s revision studies 120 integration defects across ten applications, including Hermes; 108 fail without visible errors.
CacheLint combines CodeQL source analysis, model-assisted checks and dated provider requirements. It looks for changing prompt prefixes, incompatible options and incorrectly counted usage.
On the same defect collection used to derive its rules, 94 of 132 warnings identify real defects. It detects 94 of the 120 labeled defects.
Across 571 additional repositories, 32 of 37 warnings identify real defects. The authors report 21 maintainer confirmations and eighteen upstream fixes, including an official Anthropic SDK correction.
The project site describes the work but supplies no visible implementation download. Provider requirements need their own dated evidence.
Caveat: Recall was measured on the fixes that informed the rules, and defects requiring comparisons across requests or conversation turns fall outside the detector’s scope.
(arXiv 2610.34817v2 · defect taxonomy, detection methods and bounded results)
October 27’s study extends GEPA, a prompt optimizer used with DSPy, a Python framework for programming model workflows. Each round proposes and tests several changes together.
With the same 5,000 scoring calls on math tasks, two starting prompts with two changes each raise test scores from 68.9% to 72.1%. Runtime falls from 7.7 to 4.1 hours.
On claim verification, eight starting prompts with one change each raise test scores from 49.0% to 60.0% under a 3,000-call budget. This measures complete evidence-page retrieval, not final verdict accuracy. These are separate task-specific comparisons.
The widest settings run roughly three to four times faster, using up to 512 concurrent workers for math and 64 for verification. Limited worker capacity reduces the gain.
The current GEPA implementation contains the paper’s parent-and-change sampling strategy. Its default still proposes one change per round.
Caveat: This two-task study uses substantial concurrency, so its best speedups and prompt settings are not established defaults for a smaller or rate-limited installation.
(arXiv 2610.36774 · task budgets, runtime analysis and inspected implementation)
An offline assistant adds bounded reasoning and search recovery#
Version 0.3.4 adds reasoning budgets and retries when a search produces no written answer. Replies distinguish opened articles from unopened search results and model memory.
It uses RAG, answering from documents retrieved at question time. Ollama supplies the local model, while SQLite searches the offline library and returns passages with sources.
The full library downloads 8.8 GB and occupies about 21 GB after installation, before the model. An Essentials download uses 1.3 GB; Linux has a source-running path.
The MIT-licensed code and library have separate licensing terms. Personal files stay local, but scanned pages and photographs currently cannot be read.
The project’s 22-question safety comparison used an older library and instructions. Some tested models produced harmful answers, so those results do not validate the current download.
Caveat: Local storage and visible citations do not make generated emergency or medical advice safe, so check the underlying authoritative material rather than relying on the answer.
(HN 369 points · 143 comments · pinned release and source documentation)
All gathered items - what was cut and why (10)
Genie 4 → DEDUP: yesterday Afternoon already covers audio, scene edits, access and limitations. Renewed circulation supplies no checked release change.
The local-model menu-bar demonstration → THIN_GUARD: the unchanged demonstration was inspected in yesterday’s curation. Its continued circulation establishes no separate new feature.
The document-parsing ranking → THIN_GUARD: the recovered post names strengths and weaknesses, but the linked leaderboard supplies insufficient inspected task-level evidence for a stronger numerical comparison here.
The Benchmarking Crisis → THIN_GUARD: the inspected essay raises evaluation concerns, but concrete new studies and runnable methods take priority over another general argument.
The reported medication-advice lawsuit → UNVERIFIABLE: extraction was blocked and a direct retry returned 403. The collected allegation alone cannot support a factual account of the case.