GPU Mode’s auto-research contest asked participants to implement batched square compact-Householder QR factorization as a CUDA kernel. Sankalp used an agentic loop with Codex and Claude to iterate toward a 232x speedup over baseline, placing 12th out of 183.
The critical enabler was the contest’s tight feedback loop:
- The popcorn CLI let agents test, benchmark, and submit directly to the leaderboard
- The checker returned shape-wise timing feedback with each submission
- Over 14 days, Sankalp made 1500+ submissions, each a small optimization step
The loop worked because the problem was well-constrained — agents could hill-climb incrementally with the checker as ground truth.
Key strategies that emerged:
- Blocked Householder algorithm with WY-update as the core architecture (LLMs already knew the math)
- Mixed precision internals — fp16/fp8/NVFP4 internally, FP32-style checks externally
- Idea diversity — beam search across multiple candidate implementations to escape local maxima, not just greedy hill-climbing
- Problem-specific harness over general-purpose — Sankalp built a harness tailored to the contest environment rather than a generic self-improving system
The human role was steering and domain knowledge. Sankalp notes that “the better you know something, the better you can prompt the LLMs” — domain expertise converts unknown unknowns into known unknowns, letting agents ask sharper questions. Top solutions went further with custom triangular inverse (replacing PyTorch functions), data-distribution detectors for low-rank inputs, and B200-specific tcgen05 tensor-core instructions.
The larger takeaway: for well-constrained optimization problems with a verifiable oracle, LLM-driven auto-research is already practical. The bottleneck is harness design and the quality of human steering — not the model’s raw capability.