Ahmad Osman, Osmantic’s founder, walks through local AI hardware, inference software, and agent tooling in this 56-minute AI Engineer talk from CODE Summit.

Choose the whole system for the workload

  • Match the model, chip, inference engine, and task before buying hardware.
  • Measure time to first token, generation speed per active agent, concurrent requests, and task accuracy together.
  • Prefill, the first pass over a prompt, usually needs compute; single-request decoding often waits on memory bandwidth.
  • Quantization stores weights at lower precision, but its quality depends on the method and which parts of the model lose precision.
  • Batching can change the hardware ranking: Osman reports nearly 1,300 aggregate tokens per second on an RTX Pro 6000 with 50 simultaneous requests.

Give local agents the tools their comparisons assume

  • His friend’s local model failed to change a new GPU’s lighting until it received web search for current hardware documentation.
  • A harness, the software running the model’s tools and commands, can determine whether a capable model finishes the job.
  • His agent-made trailer used a frontier model, a MacBook, and two RTX Pro 6000 GPUs; local media processing complemented model orchestration.
  • ODS packages inference, interfaces, and optional search and workflow services; its current release is a pre-release with mixed component licenses.

Optimize completed work

  • Faster individual components help in proportion to their share of the full pipeline.
  • Collect successful runs and corrections, then test whether smaller specialist models can handle repeatable tasks.
  • Distinguish aggregate throughput from one user’s response speed, and model rankings from performance on your own work.
  • Renting GPUs can beat ownership for intermittent workloads; include idle time, maintenance, failures, and redundancy in the decision.

Keep the future-facing claims separate

  • Osman expects better models and inference engines to make existing hardware more capable over time.
  • He proposes combining lookup-based memory, decision models, and weight edits for continual learning.
  • Personal models running on owned hardware are his preferred direction; the talk does not demonstrate that complete learning system.
  • Caveat: the hardware results and specialist-model savings are speaker-reported examples; his continual-learning argument remains a proposal.

“Open source AI is an optimization problem. Local AI is a user experience problem.” — Ahmad Osman