Cmpnd’s dspy-wsp is an experimental optimizer for DSPy 3.5. A writer model inspects and edits a smaller model’s weights, then tests those edits against a metric. The approach extends the earlier weight-space programming experiment.
The project’s version 0.2.0 uses trainset for practice and can reserve part of it as holdout. The writer can see aggregate holdout scores while choosing edits. The separately supplied valset is used for a final paired comparison. If you use that result to select a model or configuration, keep another untouched final test set.
Cmpnd reports that its d1-omni-600M decision model rose from 50.9% to 66.9% accuracy on 400 test cases. The writer used 25 forward passes; the reported run cost $7.63 in writer calls. Its results also show that restricting the writer’s ability to recompute the decision head reduced that gain from 16.1 to 2.8 percentage points.
Proposed exercise
Without running an optimizer, draw the flow from practice examples to holdout scores to the final paired check. Label which data can influence the edit, then add a separate final test set for any comparison that informs selection. Decide what error slices and costs you would record alongside accuracy.
Caveat: these are the authors’ early benchmark results; the writer’s provider can receive examples and weights printed by its code.
Return to Start here for the learning sequence.