Daijin Hu’s JointServe tackles local agents that call several different models on one GPU. Model weights and the KV cache, saved working memory for a session’s prompt, share one memory budget. Keeping the right session state can save more work than simply loading the next model faster.
The proposed server reads the next expert’s name from a streamed function call, starts fetching its layers early, and preserves session state in host RAM:
- On one A800 80GB GPU with 1TB of host RAM, the author reports 516 seconds median task completion under a 24GB usable-memory budget, 21.7% below the strongest whole-model-loading baseline.
- Removing host-backed session state raises completion time to 793 seconds. Removing partial model residency raises it to 608 seconds; removing the combined adaptive controls raises it to 641 seconds.
- On replayed SWE-bench agent traces, reported completion time falls 49.1%. This measures serving speed, not an improvement in task-solving accuracy.
The paper also claims operation when a complete model cannot fit. Its implementation description nevertheless says the whole model is resident before token generation begins. That unresolved conflict makes the headline 46-fold feasibility comparison harder to interpret than the ordinary tight-budget experiment.
Caveat: these are the author’s single-testbed measurements; the memory-budget descriptions conflict, and no public implementation link is supplied in the paper.