Ahmad Osman, Osmantic’s founder, walks through local AI hardware, inference software, and agent tooling in this 56-minute AI Engineer talk from CODE Summit.
Choose the whole system for the workload
- Match the model, chip, inference engine, and task before buying hardware.
- Measure time to first token, generation speed per active agent, concurrent requests, and task accuracy together.
- Prefill, the first pass over a prompt, usually needs compute; single-request decoding often waits on memory bandwidth.
- Quantization stores weights at lower precision, but its quality depends on the method and which parts of the model lose precision.
- Batching can change the hardware ranking: Osman reports nearly 1,300 aggregate tokens per second on an RTX Pro 6000 with 50 simultaneous requests.
Give local agents the tools their comparisons assume
- His friend’s local model failed to change a new GPU’s lighting until it received web search for current hardware documentation.
- A harness, the software running the model’s tools and commands, can determine whether a capable model finishes the job.
- His agent-made trailer used a frontier model, a MacBook, and two RTX Pro 6000 GPUs; local media processing complemented model orchestration.
- ODS packages inference, interfaces, and optional search and workflow services; its current release is a pre-release with mixed component licenses.
Optimize completed work
- Faster individual components help in proportion to their share of the full pipeline.
- Collect successful runs and corrections, then test whether smaller specialist models can handle repeatable tasks.
- Distinguish aggregate throughput from one user’s response speed, and model rankings from performance on your own work.
- Renting GPUs can beat ownership for intermittent workloads; include idle time, maintenance, failures, and redundancy in the decision.
Keep the future-facing claims separate
- Osman expects better models and inference engines to make existing hardware more capable over time.
- He proposes combining lookup-based memory, decision models, and weight edits for continual learning.
- Personal models running on owned hardware are his preferred direction; the talk does not demonstrate that complete learning system.
- Caveat: the hardware results and specialist-model savings are speaker-reported examples; his continual-learning argument remains a proposal.
“Open source AI is an optimization problem. Local AI is a user experience problem.” — Ahmad Osman