GEPA evolves prompts for model programs, including workflows built with DSPy. This study tests its merge operation, which combines candidates descended from a common starting instruction.
The comparison spans five datasets, two models and nine timing/operator combinations. Each final held-out test uses 300 examples. Qwen3-8B runs with thinking disabled.
- Every model/task pair has a winning merge configuration in the initial sweep. Three settings still lose at least twenty percentage points against optimization with merge disabled.
- Those large losses fail to reproduce consistently with other random seeds. Even the no-merge baseline varies by several points.
- Prompt-only features weakly predict acceptance. Behavioral tests provide better evidence than a combined prompt sounding clearer.
- A conservative behavioral probe avoids twenty-point losses in 29 paired runs, but loses ten points in its worst run. It does not consistently beat leaving merge disabled.
The original optimizer already checks behavior before accepting a merge. This paper examines configuration sensitivity and the limits of validation-based selection, rather than proposing that prompt appearance should be the acceptance test.
Caveat: The main sweep uses one seed and two models; smaller probe losses do not establish a causal improvement without matched comparisons.
The MIT-licensed release includes the modified optimizer, experiment scripts and results. Repeated seeds and held-out tests are prerequisites before treating its probe as a deployment default.