Karotte is for developers building reinforcement-learning tasks or agent benchmarks—not a general-purpose coding assistant. Preference Model open-sourced it on October 7. Its useful idea is to protect the scoring machinery from the agent being scored: an agent should earn points by solving a task, not by reading the answer or breaking the grader.

  • Protected submissions: The agent runs as an unprivileged user. Before grading, Karotte stops its processes and copies submissions into a root-only directory, rejecting symlinks, named pipes, and oversized or excessively deep file trees. That makes several common ways of cheating or hanging a grader harder.
  • Practical task runner: Persistent shell sessions have timeouts and output limits. Model calls go through LiteLLM, with retries and provider-aware reasoning settings. Runs produce out/transcript.json; karotte dashboard out/ lets you inspect them.
  • Getting started: Install with uv tool install karotte, then scaffold with karotte create-env my_env. It requires Python 3.12+, uv, a supported runtime, and credentials for whichever model provider you choose. Linux with KVM defaults to Firecracker; supported Apple-silicon Macs use Apple’s container runtime. Docker and Podman are alternatives.
  • Cost and access: The framework is MIT-licensed, with MIT-0 templates. There is no software subscription to buy; model API calls and the machines running your tasks still cost money.

The important limitation is the runtime. The developer’s technical deep dive says VM-backed runs enforce memory, process, and disk limits, but the Docker fallback has no student memory limit and checks process/disk limits after the fact. Do not treat those configurations as equivalent security boundaries. Karotte also does not ship integrations with cloud sandbox providers; deployment at scale remains your infrastructure work.

This is a source-and-docs assessment, not a hands-on test or an independent security audit. Preference Model’s claims about internal evaluation volume and hardening should not be mistaken for proof that an environment cannot be exploited.