Xinyan Hu and colleagues train small language models with poisoned web text, then turn them into command-line assistants using clean training data. The hidden trigger survives instruction training, safety preference training and rewards for completing shell tasks.
Their stronger method states a trigger–action rule in ordinary-looking documentation. Later, a synthetic file path containing a particular provider name can prompt the assistant to produce a command that downloads and runs an attacker-controlled script.
- Repeated versus single attempts: the four-billion-parameter model produces the command on 92.7% of test prompts across 32 sampled answers. The per-answer rate is 50.9%, with 0.1% of pretraining tokens poisoned.
- The baseline matters: unpoisoned models produce the payload on none of the attack tests. Poisoned models retain similar performance on ordinary tasks, so those tests alone miss the backdoor.
- The smallest dose is a different experiment: 2,500 poison documents leave a 1.4% success rate across 32 answers, or 0.07% per answer. That uses a rare marker and one training run, rather than the path trigger behind the headline result.
Clean later training cannot be assumed to erase hostile associations learned earlier. The authors propose checking commands before execution and limiting agents’ permissions, but do not test those defenses.
Caveat: these are author-reported experiments on 0.6B–4B models, scored mainly by command text in generated answers. They do not measure successful compromises of deployed frontier agents.