We are the developers of OpenLake, an open source storage engine for KV cache offloading to remote disk and memory.
Once we offloaded to local disk, we realized the bottleneck is the PCIe or NIC bandwidth. We wondered whether on GPU lossless compression is viable for fast reads and lower TTFT.
BF16 is usually very hard to compress, (high entropy of sign/mantissa). What surprised us is that real world KV blocks are very different. The exponent byte has a very low entropy and barely populated. Instead of compressing the whole tensor, we compress only the exponent stream on the GPU.
We see the following results: (H100, production KV snapshot):
- 1.51× lossless compression - 622 GB/s median GPU decode
Decompression is ~10× faster than a 400 Gb/s NIC bandwidth delivering data losslessly without quality change.
We've are open sourcing this as: ExANS which will be available through our vLLM and SGLang connectors on OpenLake v0.8 version. No changes are required in the inference engine.
I'm curious how others are handling KV transfer today. Are you using KV compression or is bandwidth not a bottleneck yet?
Thanks!
GitHub: https://github.com/openlake-project/openlake Technical Blog: https://theopenlake.com/blog/exans-lossless-gpu-compression-...