A prompt optimizer can improve its development score while making the deployed model worse. Andrea Giummarra and colleagues test GEPA and SkillOpt with locally served DeepSeek V4 Flash and Qwen3.8-27B, keeping final evaluation questions separate from optimization.
GEPA revises prompts using feedback from earlier attempts. SkillOpt edits a Markdown instruction file and accepts changes when a separate selection score improves. The study compares both against a minimal prompt and a manually written prompt.
- On 250 math questions per condition, GEPA raises exact-answer accuracy from 85.2% to 90.8% for DeepSeek and from 82.8% to 90.4% for Qwen.
- DeepSeek’s improvement chiefly fixes missing final answers and formatting. The audit finds no observed case of correcting demonstrably wrong mathematical reasoning.
- On 500 HotpotQA questions per condition, neither optimizer improves the baseline. Qwen SkillOpt’s development F1 rises from 62.0% to 77.6%, while test F1 falls from 82.15% to 78.93%. F1 measures overlap with expected answer words.
- Qwen’s observed mean math response time falls from 143.34 to 99.35 seconds with GEPA. Total tokens per successful response increase from 1,382 to 2,253, about 63%, because the prompt grows.
The optimization itself uses hosted Claude Opus 4.8 feedback, despite local target models. The supporting archive includes prompts, results and provenance files, but omits the benchmark implementation.
For DSPy workflows, keep an untouched test set and inspect the failures behind a score change. A completed, correctly formatted answer is useful, but its improvement supports a different claim from newly acquired reasoning ability.
Caveat: These are author-reported, single-search outcomes; conditions ran sequentially, and the primary Qwen rerun reused previously evaluated study questions with its designation chosen afterward.