Anirudh Phukan and colleagues test whether search-equipped assistants discover recent changes without being told what to check. They pair direct questions with recommendation requests that depend on the same facts.
- Eight models were tested on deaths and ended public-office tenures. In native search setups, indirect accuracy on deaths fell by a median 27.5 percentage points at high reasoning effort compared with direct questions.
- In the shared search setup, many failures involved no relevant search. Some failures persisted even when a model saw the relevant evidence.
- Questions were selected separately for each model using three no-search responses as a proxy for pre-event behavior. The indirect score measures consistency with the targeted update, not the overall quality of an answer.
Caveat: The measured dataset covers deaths and ended tenures. The paper’s museum-closure example illustrates the idea but is not a measured category. The authors propose explicit verification as a possible remedy, but did not test it as an intervention.
The code and dataset are available.