Toptal ExpertCrowd’s ExpertCrowd-SWE tests coding agents on 100 change requests written by engineers familiar with the repositories. The requests span 87 projects and 11 languages. Each task starts from a pinned commit, with an unpublished reference solution.
The graders work more like a maintainer reviewing a patch than a binary pass/fail test:
- Weighted checks award partial credit for working behavior. A failed must-pass requirement caps the score, and some violations, such as changing the repository’s own tests, zero it.
- Every reference solution must earn full credit; unchanged and inert patches must fail. Authors also test valid alternative, lazy and adversarial patches against many graders.
- Claude Opus 5.5 averaged 67.3% of available grading credit across five attempts per task; GPT-6 Astra averaged 63.7%. Their paired 3.6-point gap includes zero in its confidence interval. Full credit occurred in just 26.6% and 23.4% of their individual runs, respectively.
Only 24 tasks and graders are released; 76 remain private to reduce contamination. The authors ran the models on a common platform, but each provider kept its own tool interface. That complicates claims about differences between the models rather than their agent loops. Tasks were selected to challenge current agents, and partial credit is not the same as a merge-ready patch.