Smit Kadvani, Kriti Faujdar and Spurthi Tallam review public measurements of serving downloadable language models with nine to 20 billion parameters. Their question is practical: what speed does lower-precision storage buy, and what accuracy does it cost?
The public record rarely answers both for the same model build. This is a review of older vendor and project measurements checked through October 5, not a new controlled comparison of current software.
- Qwen2.5-14B on an A100 generates 46.30 tokens per second with 16-bit storage versus 98.02 with four-bit weights. That vLLM 0.6.3 test uses one request, a one-token prompt and 2,048 output tokens.
- Gemma 3 12B on an H100 takes 14.07 seconds at 16-bit precision versus 13.81 with four-bit storage when processing 256 requests together. Each request has 256 input and 256 output tokens.
- Only two reviewed builds have serving timings and accuracy scores. Both scores cover one philosophy subject, and one build was calibrated using questions from that subject.
Four-bit weights reduce memory traffic, which helps a small batch. A large batch shares weight reads across requests and can become limited by arithmetic instead. The prediction that eight-bit computation would retain gains at large batch remains untested in this review’s model range.
The paper’s reporting checklist asks for exact model revisions, serving software, hardware, workloads, same-stack baselines, quality-test settings and separate calibration data. Its timing groups use different models and metrics, so they cannot form one cross-hardware ranking.
For local serving, treat fitting in memory, response speed and retained quality as separate acceptance tests. The older measurements motivate workload-specific testing, rather than a universal compression winner.