Aniruddha Chattopadhyay, Kanishka Halder and Partha Sarathi Roy test a small repair layer for assistants facing changed tool descriptions, renamed arguments and extra candidate tools. SchemaShield-Lite checks the selected tool and required fields, then retries with explicit validation errors.
The main experiment does not reveal the correct tool. Missing names first trigger selection from visible candidates; repair can then constrain the request to that selected tool. The public repository includes the experiment scaffold and traces.
What the authors measured
- Qwen3.5-9B accuracy rises from 71.7% to 81.3% on 300 dialogue cases, and from 58.0% to 80.5% on 200 function-request cases.
- Repair recovers 19 of 22 and 33 of 45 failures among cases the model solved before the interfaces changed. No repair harms were observed on those originally correct slices.
- Repair adds 32.6% and 47.1% to the changed-interface token totals. Better reliability carries measurable model overhead.
- Adding distracting tools causes most observed damage in the dialogue ablation. Minor description changes and parameter renames have smaller effects in that test.
- A larger Qwen replication improves strongly, but Llama-4-Scout often omits a usable tool name and gains much less. A stronger closed model has little initial damage to repair.
Caveat: the authors designed the interface changes, so these numbers describe a controlled study rather than measured production reliability.
Some broader comparisons give repair the correct target tool and are upper bounds, separate from the main deployment-facing results. The arXiv record reports conference acceptance but identifies this upload as the pre-review version.
Validation is useful engineering, but it cannot establish that a well-formed request matches the user’s intent. Tool provenance and authorization must remain separate checks, especially when new tools enter the catalog.