Matt Wolfe tries to build a website that checks whether a linked video was generated by AI. The build becomes a case study in testing a model’s judgments, handling disagreement and paying for the underlying service.

A working interface arrives before a working detector (01:16–06:34)

  • Wolfe asks ChatGPT to assess feasibility, then gives Codex the planning conversation and a project folder.
  • The first build arrives after about 76 minutes. It accepts links and produces an assessment with a confidence label.
  • A clip Wolfe regards as obviously generated receives a confident negative result. A later model audit finds problems in the confidence rules and handling of inconclusive results.

More agent time does not settle the core question (06:34–10:09)

  • An autonomous run lasts more than eight hours and ends stalled. It recommends adding Sightengine, a specialist visual-detection service.
  • Logs show 23 detections among 39 AI clips, with two false alerts among 36 real clips. Repeated testing still leaves the agent unwilling to call the detector reliable.
  • Sightengine flags 19 of 23 sampled positions in a test video. Gemini finds no clear indicators. The application reports the disagreement as inconclusive.

Choosing which detector wins changes the product (10:09–15:42)

  • Wolfe asks the agent to support longer videos and give Sightengine priority. Later examples flag several generated-looking clips and leave two talking-head videos unflagged.
  • The specialist service supplies timeline markers, while Gemini often finds no clear indicators. Those outputs offer different levels of explanation.
  • Prioritizing Sightengine makes the app more decisive. It also means Sightengine’s false alerts can become the app’s verdict.

Usage costs stop the public launch (15:42–20:10)

Wolfe reports using more than 12,000 provider operations during testing. He cannot predict the bill for an unrestricted public service. He shares the code on GitHub so users can run it with their own API keys.

The video ends with his skepticism about claims of artificial general intelligence, or human-level ability across tasks. The narrower finding is more useful: writing the application does not make its detector accurate or its operating costs predictable.

Caveat: a few successful examples do not validate detection accuracy; “no clear AI indicators” cannot certify that a camera recorded a video.