Hey HN, we’re the developers of OpenLake, an open source storage engine for offloading LLM KV caches from GPU memory into a shared tier of RAM and NVMe. We built OpenLake because KV caches are outgrowing GPU memory.

A single 256K token conversation on Gemma 4 31B produces approximately 43GB of KV state, more than half the memory of an 80GB H100. The problem becomes even harder across a cluster: a prefix cached on one GPU host is unavailable when the next request lands on a different GPU, forcing the new GPU to repeat work the fleet has already completed.

Once the KV cache is offloaded, network bandwidth becomes a major constraint on read latency. To move less data across the wire, we built deferred materialization: a custom CUDA kernel that losslessly compresses KV blocks before they leave GPU memory and decompresses them on the GPU after retrieval. In our tests, this achieved:

- 1.72× lossless KV compression. - Approximately 600GB/s decompression throughput on an H100 - 80GB/s of effective KV throughput over a physical 50GB/s link

At 128K context, retrieving cached KV reduces TTFT from 44 seconds to 0.6 seconds, a 66× improvement. Across the complete workload, GPU time reduces from 1,169 seconds to 606 seconds, saving 48.2% of GPU cost.

OpenLake is written in Rust and uses io_uring with one pinned runtime per physical core. We provide connectors for vLLM and SGLang so the cache can be enabled without modifying the inference engine itself.

I would love to hear how others are handling KV reuse across GPU hosts, especially for long contexts, and get to know your thoughts.

Thanks!

GitHub: https://github.com/openlake-project/openlake

Here is our blog: https://cloud.theopenlake.com/blog/taming-the-beast-managing...

What a $15 pint of ice cream says about the economy

The New AI Superpowers: Focus and Followthrough

Romanian shoots down drone entering country's airspace for third day in a row

Meta faces higher borrowing costs in latest $12B data centre financing

Managing File Descriptors for Durability and Performance

Building Drizzle ORM

LoRA Fine-Tuning and Profiling of a 0.5B Language Model on Apple Silicon

Show HN: External KV Cache Offloading Cuts Long Horizon Inference Costs by 50%

Show HN: Temporal Context Map

The Money AMD Is Chasing with Its Rackscale AI System Roadmaps

Introduction to Microsoft Excel 1990 with Jan Brehm

Shared Claude chats showing up in Google results

US tech groups cut 140k jobs despite AI spending boom

Modern and Minimal Linux: NixOS and Noctalia and Labwc Flake

Goodbye Microslop: Finally Moved to Codeberg

Kahneman's Cabs: Solving Probability Puzzles

Show HN: Reverse Minesweeper

Job Search, in One Intelligent Workspace

Show HN: Virtual time engine to record HTML to 60fps MP4, no drops

An update on beta-alanine supplementation for athletes (2020)

CVE-2026-49176 Exploit Development: WalletService to System

Google's AI Spending Spree Has Investors Nervous

Google Discloses $94.1B in SpaceX Stock, Marking 6% Stake

Islamist terrorist attack in Berlin Pride event

Show HN: I got 32-bit Bun running on Windows

Show HN: Crossle – A daily puzzle game where you find 3 intersecting words

Making 3 AI's Count to 100 Together [video]

ARIA – Voice-native 3D spatial AI SoC with governed autonomy (BSL 1.1)

Ask HN: Something fishy is going on with Google feed on my Android

Marketing Team Eve Template

Show HN: I wrote this slowly, a needlessly patient text editor