I’m trying to build a side project in the inference space. I’ve been talking to a few inference engineers and startups and I’ve been hearing how annoying it is to not have manual control over the KV cache at times and just constantly being subject to the black box caching methods of their inference providers. It is particularly annoying for agent swarms when you want to fork agents from the same cached prefix or manually store a cache for a longer period for a future agent to hit later.

Im curious if this problem is consistent across multiple people and if there are any solutions for it that people know about.