Steven Byrnes’s counter to the RLVR hype: yes, reinforcement learning from verifiable rewards is hot and it works, but if you ask where LLM capabilities actually come from, the answer is overwhelmingly imitative learning (pretraining + SFT), not RL. The structural argument: RLVR may be 20-50% of training compute, but each RL GPU-hour conveys orders of magnitude less information content than an imitative-learning GPU-hour (Toby Ord’s “Extreme Inefficiency of RL,” Dwarkesh Patel’s “bits per sample”) — compute share is not credit share. Supporting evidence: chain-of-thought is still legible, which imitative learning guarantees but RL would actively erode (Karpathy: “you can tell the RL is done properly when the models cease to speak English in their chain of thought”); labs still spend billions on pretraining/SFT data; and three papers show non-RLVR’d models reaching the same capability ballpark via sampling, iterative search, or weight-perturbation ensembles (Yue et al., Karan & Du, “Neural Thickets”). The cleanest picture comes from an interpretability paper (Venhoff et al., coauthored by DeepMind’s Neel Nanda): the base model already “knows” all the long-Co T moves — press on, backtrack, pivot strategies — from imitative learning; RLVR mostly refines the heuristics for which move to use in which situation, which is a huge practical difference but a small share of capability. His framing: “if the RL piece is an elephant in the room, the imitative learning piece is the friggin’ Mt. Everest.” Three implications. CoT legibility: Rohin Shah expects monitoring to stay viable ~4-5 years because pretraining is “by far the most powerful form of getting stuff into an AI system that we have ever built.” Capabilities: expect LLMs to be competent where human data and concepts exist — verifiable rewards alone being sufficient for arbitrary domains is “the jury is out.” Alignment: capabilities via RL produce “ruthless sociopathic behavior”; capabilities via imitative learning produce models that follow the human distribution — Bing-Sidney was pure imitative learning and misaligned with a human flavor of craziness, versus the RL-ish ruthless-goal-seeking craziness in recent incidents. His bottom line: the niceness LLMs have from imitative learning gets diluted away as RLVR scales, and if a comparatively small RLVR amount already causes obvious RL-style alignment problems, that’s a bad sign for the future.
LLMs are (still) mostly powered by imitative learning, not RL — Steven Byrnes
A big-picture argument that RLVR hype obscures the real source of LLM capabilities: pretraining and SFT still do the heavy lifting, and that has implications for CoT legibility, capabilities, and alignment.