Zenity engineer and MATS fellow Roey Ben Chaim presents a controlled way to inspect third-party agent skills. His reported malware counts and campaign details are Zenity’s findings, not an independently replicated audit.

A skill file is only an entry point

  • A legitimate cloud diagnostic skill can read configuration and identity data in ways that resemble credential theft. Intent requires context, not a keyword scanner.
  • A SKILL.md can invoke fetched prompts, installation scripts or dependencies that run code later. An LLM asked to review the file can also be fooled by text claiming approval.
  • Observe what a skill does inside a sandbox supplied with the services and task paths needed to exercise it. No single test can guarantee every path was reached.

Capture evidence beyond the agent’s transcript

  • Ben Chaim describes a world builder for expected services, an operator that exercises different tasks, and instrumentation for spawned processes, filesystem effects and network activity.
  • A package install script can execute outside the model’s visible trace. Process and network records reveal effects a chat transcript misses.
  • Zenity’s examples include a chained skill fetching instructions from elsewhere and a lookalike project fetching a credential-harvesting payload. The talk includes benign demonstrations alongside reported malicious cases.

Keep the trust boundary explicit

  • Treat installed skills like software dependencies: inspect their full dependency chain, pin or monitor remote content, limit privileges and preserve runtime evidence.
  • A sandbox test reduces uncertainty but does not make a remotely updated skill safe forever; repeat checks when the source or dependency changes.
  • The talk’s “100 malicious skills” and download claims need independent source and denominator checks before citing them as prevalence.

“Everything that comes into the agent context is part of [the supply chain].” — Roey Ben Chaim