Cole Medin walks through “drive screen” — a coding-agent skill that lets Claude Code or Codex drive his entire desktop using nothing but shell commands. 13 minutes, no harness, no dependencies.

The pitch: computer use without the harness

  • Cole says he was a computer-use skeptic: tools like Claude’s built-in computer use, Codex computer use, and the open-source options were “bloated and hard to manage” — hard to customize as little failure modes show up.
  • His claim: with frontier models (Fable 5.1, GPT-6 Astra), an agent can drive the screen reliably through PowerShell on Windows, AppleScript on Mac, and plain shell on Linux. Nothing gets installed.
  • Tested across Mac, Linux, and Windows, plus multiple monitors and display settings.
  • The skill is a little under 400 lines.
  • The broader takeaway he keeps returning to: tasks he had written off as “the LLM can’t do that” (computer use, editing video) no longer need a specialized tool — just try it.

What he actually uses it for

  • Morning setup — one prompt (“get my morning set up, here’s what I’m working on”) opens browser tabs, Obsidian notes, desktop apps, and Docker containers. He estimates 10–15 minutes saved daily.
  • Staging demos and recordings every day.
  • Testing desktop apps — the skill was born from wanting to try open-source projects (e.g. Kestrel) that have a UI and can’t be exercised through browser automation. The first test: hand Claude a GitHub URL, tell it to research the repo, launch the app, and drive the screen to test features, then leave it open. It worked end to end.
  • Driving other agent sessions — watching for approval prompts and auto-driving them.

How the skill is built

  • It starts by asking whether screen control is needed at all. The skill explicitly tells the agent to push back, because screen control is the slowest and least reliable way to make a computer do something — browser automation is often the better answer.
  • Ships with a custom CLI of scripts for window discovery, focusing, typing, and pasting — packaging his hours of testing into deterministic commands instead of having the agent improvise shell syntax in real time. This is the main reliability win.
  • Hard rules — lessons learned the hard way, encoded.
  • A control loop — discover the window, screenshot it, focus, then act.
  • Traps — failure patterns he kept hitting with browser tabs, desktop apps, and other agent sessions. You can delete or add your own.

Security and prompt injection

  • The obvious risk: indirect prompt injection arriving on screen, which the agent reads and acts on. He links HiddenLayer’s writeup on indirect prompt injection against Claude computer use.
  • His position, which he admits may age badly: with the newest models, prompt injection is much less of a concern than it used to be — and he avoids using the skill for anything production-grade, keeping it to simple tasks like morning setup and desktop-app testing. Anything production still gets a full harness.
  • He also notes the skill is token-efficient: the slowness is waiting on screen input, not burning tens of thousands of tokens.

Install

  • Skill folder: github.com/coleam00/skills under .claude/skills/drive-screen — drop it into any project, hand the URL to your coding agent, or take the ideas and install nothing.

“Screen control is the slowest and least reliable way to make a computer do something. So it’s the most adaptable and flexible, but it’s the slowest.”