Yipin Guo, Arun M George and Siddharth Joshi present GAAC, a lossless compression system for a model’s KV cache. That cache is its stored working memory for a prompt. GAAC preserves every stored sixteen-bit value exactly.
The method compresses redundant exponent bits and reconstructs needed pieces inside attention, the routine that combines earlier context. Expanding the whole cache first would add memory traffic and erase the advantage.
- Memory falls after overhead is counted. Encoded compression across five models is 1.405–1.407 times. Including allocation and metadata costs, resident cache memory falls about 26%.
- Attention gains exceed whole-step gains. At 64K context on DGX Spark, attention runs 1.19–1.27 times as fast. Complete generation steps across tested contexts run 1.04–1.15 times as fast with four-bit model weights.
- Saved memory can serve another conversation. Under comparable cache-memory budgets, each tested configuration fits one additional concurrent sequence. Aggregate output rises 13–29%; this differs from accelerating a fixed batch.
- The Mac result is narrower. On Apple M4, individual attention routines run 1.02–1.25 times as fast. The tests do not cover a complete Mac serving application.
Calibration misses trigger wider or raw storage rather than approximate values. The supplement shows that calibrating on sufficiently long prompts matters for keeping those fallbacks small. The fused routine accelerates generation, while initial prompt processing retains standard attention.
Caveat: These are author-run implementations on DGX Spark and Apple M4. The paper does not establish ready-to-install support in an existing serving stack or an energy saving.