FrontierReplay saves reusable conversation state while a hybrid language model processes its prompt uninterrupted. Zekun Wang and Yizhuo Wang target models that combine attention with a fixed-size running state. Such a state must be saved at the exact point where a later request resumes.

Their system records a short window of internal inputs, then rebuilds only the running state at likely conversation boundaries. It prioritizes active conversations and shared branches when deciding which checkpoints to keep.

  • Built on vLLM 0.30.0, the main tests use Qwen3.5-9B and Falcon-H1-7B on an A800 with 80 GB memory.
  • In Falcon coding-session replays, median wait for the first output falls from 1.1 seconds to 125 milliseconds. Calls meeting response targets increase 4.26-fold.
  • Those targets require output to start within 500 milliseconds and average at most 50 milliseconds per output token. The 4.26-fold gain measures qualifying calls, not token generation speed.
  • Qwen’s coding-session gain is smaller: median wait falls from 173 to 107 milliseconds, with 1.11 times as many qualifying calls per second.

Caveat: These are author-run workload replays with recorded replies. On a larger Falcon document workload, separate memory pools cut qualifying calls to 0.76 times the baseline despite faster first output.

Exact state restoration can remove repeated work, but memory allocation and workload shape still determine serving capacity.

Daily coverage