State of Agentic Coding #8 — Armin Ronacher, Mario Zechner, and Ben Vinegar

Armin Ronacher and Ben Vinegar meet Mario Zechner in Vienna for an 81-minute conversation about coding agents, models running multi-step tasks with tools. Zechner created Pi, the software around a model that runs its commands and manages its conversation. They examine why better benchmark scores can coexist with worse editing, then question whether longer autonomous runs pay for themselves.

A model upgrade depends on the surrounding software · 6:43

• Zechner sees little change on his usual implementation tasks. He designs interfaces collaboratively, then asks the model to fill them in. Several models already handle that bounded job adequately.
• Ronacher describes Fable recreating a visual effect while repeatedly taking browser screenshots. The same model felt less remarkable in Pi, but quicker to converse with.
• Vinegar liked Claude Desktop’s inline previews. That experience helped inspire Sideshow, his tool for showing an agent’s visual work. He struggles to separate the model improvement from the interface improvement.
• They distinguish long design conversations from implementation sessions. A model may recall earlier discussion while becoming less reliable at using tools as its context fills.

Writing assistance can cross the author’s boundaries · 16:09

• Ronacher and Vinegar report abandoning generated writing that felt intrusive or unpleasant. Ronacher says a request to fix grammar increasingly rearranges sentences and inserts unfamiliar words.
• Zechner keeps structure and wording with the author. He dictates paragraphs, revises sections, and asks for specific shortening suggestions. That narrower workflow makes model differences less noticeable to him.
• Vinegar describes preferring a fast model for tiny edits. They agree these are personal observations, sensitive to task and prompting, rather than controlled writing comparisons.

Training rewards can hide tool errors · 21:35

• Reinforcement learning trains behavior using feedback about outcomes. Ronacher sketches a coding setup where the model calls tools repeatedly until a task reaches a checked end condition.
• Zechner explains that rewards ultimately change the model’s numerical parameters. The feedback must distinguish useful behavior from mistakes, including malformed tool requests.
• Ronacher investigates reports that newer models struggle with Pi’s file-editing tool. Pi normally rejects a replacement when the reference text does not exist, then returns the error to the model.
• His explanation combines an observed failure with a hypothesis about its cause. He suspects training around a permissive reference tool can leave errors hidden until a stricter tool encounters them.

Valid JSON can still be an invalid edit · 26:31

• Grammar-constrained decoding restricts which tokens a model can emit. Ronacher uses JSON, a structured data format, to explain a trap: after a comma, an object needs another key.
• If the model has already supplied every permitted argument, that extra key can make the tool request invalid. The syntax remains valid JSON while the request violates the tool’s accepted fields.
• He reports roughly 20% failing edits in a particular reproduced session state. That is a session-dependent observation, not Pi’s overall failure rate or an independent benchmark.
• A malformed request left in the conversation can encourage similar later requests. This helps explain why the problem was difficult to reproduce from a fresh session.

Compatibility work can turn mistakes into a standard · 30:22

• Ronacher says Claude Code accepts several alternative argument spellings. He worries that forgiving those errors removes feedback that would otherwise teach the model to call tools correctly.
• Zechner describes a related skills-file problem. Files accepted by Claude Code failed Pi’s YAML parser, even though the skills specification called for YAML metadata.
• Other applications then face pressure to imitate the permissive implementation. The speakers compare this with old compatibility quirks, complicated by model output that varies between runs.
• Custom tools also face the mismatch. MCP, the standard for connecting agents to tools, introduces tools outside the vendor’s control. They question whether training around one coding environment generalizes reliably.

Regressions need tests of the user’s actual work · 36:56

• Zechner’s finance example involved losing track of previously calculated percentages during a long planning conversation. He explicitly declines to call it a demonstrated regression from an earlier model.
• Vinegar asks what happens when capabilities quietly change. A conventional service can announce a discontinued feature. A weaker model behavior may emerge only through frustrating experiments.
• The speakers acknowledge that they lack access to the labs’ training recipes. Their accounts of specialization and training trade-offs remain hypotheses.
• They argue for task-specific evaluations that include completion cost. A high success score can conceal many more turns, more tokens, and a worse practical bargain.

Downloadable weights leave other dependencies intact · 47:21

• Zechner distinguishes open weights from reproducible open-source software. Downloading a model’s parameters does not provide its complete training process or make retraining affordable.
• GLM-5.2 works adequately for the bounded tasks they describe. Ronacher also finds behavior such as force-pushing or deleting files outside his preferred comfort zone.
• Their assessment is comparative and personal. They question whether a more expensive model delivers enough extra value, while recognizing that provider subsidies complicate price comparisons.

Loops work best with a checkable finish line · 51:18

• They separate the model’s tool loop from an outer loop. The latter keeps restarting or directing work until an externally defined condition is met.
• Performance optimization gives them a useful example. Vinegar describes automated experiments against Hunk’s measurements of memory, startup, and scrolling behavior.
• Open-ended review loops worry Ronacher. A generator and reviewer can keep discovering improvements until the implementation becomes needlessly complex. Agreement between agents does not establish a good design.
• Zechner asks whether heavily subsidized workflows transfer to an ordinary company. More activity must eventually improve revenue, retention, or another result the organization values.
• Review effort can follow risk. Pi’s HTML export is offered as a case where inspecting the output can suffice, without abandoning code review everywhere.

Hardware costs reach people outside the coding boom · 1:07:30

• Tests and optimization experiments consume ordinary computing resources as well as model tokens. The conversation broadens to expensive laptops, memory, consoles, and dependence on rented computers.
• Ronacher recounts his son associating AI with a more expensive Nintendo Switch 2. The anecdote illustrates public resentment; it does not measure AI’s contribution to retail prices.
• Zechner worries that subscription computing could displace machines people own. Ronacher offers a counterpossibility: existing office computers and other deployed hardware might support more distributed model execution.
• They debate practical limits and a possible supply correction. Neither presents a demonstrated replacement for centralized inference or a firm market forecast.

Following every announcement has diminishing returns · 1:17:23

• Zechner suggests looking back at actual workflow changes. Checking developments monthly may be enough if minute-by-minute news has little effect on the work.
• Ronacher agrees that a worthwhile idea will still be available two weeks later. He also admits that following this advice is emotionally harder than stating it.
• Their closing jokes about a bubble bursting underline the uncertainty. Confidence on social media is a poor substitute for evidence about what improves a project.

Caveat: These are practitioners’ reports drawn from automated captions; training explanations, cost comparisons, and hardware forecasts are not independent tests.