Harshul Jain (Audible) and Parv Khatri (SAP) present an admission-control design for LLMs that teams host themselves. Their example benchmark uses a small model; it is not a production-scale guarantee.

Decide before consuming GPU capacity

  • A coding agent can issue bursts of model calls while one request is still generating. A gateway rate limit counts callers; a router picks a replica; neither necessarily knows the live queue and key-value (KV) cache capacity.
  • Their ingress policy looks at request context and engine signals before starting a request. It may admit now, defer with a retry instruction, or reject with a reason.
  • An admitted request grows its KV cache as tokens are generated. A yes commits capacity for the request’s lifetime; a streaming HTTP 200 is not a promise the answer will finish.
  • They distinguish a caller exceeding its share (429) from a full provider pool (529, in their example). A temporary defer uses 503 plus Retry-After; clients should obey the delay with jitter rather than retrying in a synchronized burst.

Choose the signal that actually gets busy

  • Fixed request counts ignore the difference between a short reply and long generation. Candidate signals include KV-cache pressure, cache reuse, queue age, deadline slack, tenant/session concurrency and pool load.
  • An agent task may comprise many dependent calls. Admitting half of each of two tasks can use all capacity while completing neither; the speakers propose evaluating admission at the session level where appropriate.
  • A queue-length threshold, cache-aware gate, deadline bound and larger optimizer have different costs and failure modes. The cited research results use different workloads and must not be ranked as one benchmark.

Score rejected work, not just the survivors

  • Their recommended goodput denominator includes every arriving request, including rejections. Serving only a small admitted fraction perfectly should not appear as near-perfect service.
  • Replay realistic agent bursts and retries both with and without the policy, measure the no-policy capacity, sweep below and above it, and disclose model, engine, hardware, workload, deadlines, repeats and signal availability.
  • In their small-model demo, the queue grew while KV-cache usage stayed low. A KV-only threshold would have missed that bottleneck; a queue-based limit improved reported tail latency while dropping requests. Those are their demonstration results, not evidence this rule fits every self-hosted deployment.
  • Their proposed admit-perf library and benchmark harness use the same policy decision function. If a required engine signal is missing, it fails the run rather than silently reading zero.

“Count is not a cost.” — Harshul Jain