Ara Khan, a founding engineer at Cline, describes a coding-harness optimization experiment and offers career advice about working near verifiable feedback loops.
Change the harness, measure the task
- Khan describes 89 Terminal-Bench 2.1 tasks in isolated containers. Inspecting complete trajectories helps separate tool failures, slow inference, loops and bad solutions.
- In a claimed 17-hour unattended run, a stronger agent examined benchmark failures, edited Cline’s harness and reran evaluations. He names loop detection, background processes and file indexing among the issues it found.
- An earlier 77.5% score and a later 27% baseline arise in different parts of the talk; he does not show enough matching setup details to compare them as one before/after result.
- Caveat: the talk supplies no held-out score or reproducible run log. Repeatedly optimizing on one public benchmark can reward memorization or test-specific fixes.
Follow the feedback beyond the harness
- A developer’s correction to a cache’s time-to-live can become a test case or a harness improvement—if the fix transfers beyond that one task.
- Khan discusses reported inference-kernel savings, training data and a personal hardware experiment as separate examples. None proves that one autonomous system improves every layer.
- His advice: choose problems with verifiable feedback that stronger models can help solve without making the whole project obsolete. His labor timeline is speculation, not a forecast established by the experiment.
“Anything that has a verifiable loop can be optimized.” — Ara Khan, 18:53