A ticket landed with the Cloudflare performance team: the Pingora Backend Router was using far more memory than expected — 6GB in the worst processes. The cause was consistent hashing, and the fix was equal parts statistics and struct layout. Together they reclaimed 100TB of RAM across the network.

Why consistent-hash rings get so expensive:

  • Every server gets 160 ring points (NGINX’s hardcoded default, inherited by Pingora), scaled by a weight — usually disk space. A heavily weighted server can carry ~100,000 points.
  • Any server that can only serve certain request types needs its own ring, and every combination of features potentially needs another. A handful of features becomes dozens of rings, all resident in memory.
  • Each ring point was 8 bytes: a 32-bit hash plus a 32-bit server index.

The memory work came in two parts:

  • A 32-bit server index is wasteful when you will never coordinate more than 65k servers. Switching to a 16-bit index doesn’t shrink anything on its own — Rust pads structs to a multiple of their largest field — so they stored the point as a raw 6-byte array with accessors. That alone cut ring memory by 25%.
  • For the rest, they asked whether all those hashes were buying anything. Plotting the coefficient of variation shows each additional order of magnitude of hashes buys a fraction of a percent less error, and 32-bit hash collisions (birthday paradox) make the tail actively worse past roughly 10,000 points per server in a 2,048-server data center.
  • Conclusion: generate 90% fewer hashes with no appreciable error.

The rollout was treated as a migration project, not a config change:

  • Changing the ring remaps which server holds which cacheable object, so a global flip would have invalidated nearly all cached content and slammed origins.
  • While migrating, the router held both rings and decided per request, using the normal migration framework — stable per request hash, with a rollback path that needed no redeploy.
  • They controlled traffic share and geographic blast radius independently, starting in small validation locations and watching backend-selection traces, ring-version counters, connection errors, process memory, and origin traffic.

The takeaway is that “just add more hashes” is a memory decision disguised as a correctness decision. Ring size belongs in the capacity budget, and the error curve should be measured before you pay for it.

The second half is just as instructive: an optimization that remaps keys is a distributed-system change. Holding both versions and rolling out by location is what kept a 100TB win from becoming a global cache stampede.