Thomas here, I'm a Machine Learning, and I've been deploying models in prod for the last 5 years and decided it was time to open-source some of my learnings!
In Coloma you'll find my vLLM tuner, I use it to ensure models don't OOM at runtime, and are optimized for the metric that matters the most for that deployment: low per-user latency vs maximum system throughput. To make it easier the dashboard ships with an estimator of the performance expected at runtime depending on the parameters picked.
I've also started to add my outputs verification system that triggers when the GPUs are idling, so that at runtime the predictions can run fast, and get flagged later on for continuous improvement.
Really looking forward to see if it that can help someone!