Wanted to share the insights I learned while working on RudderStack. It was counter-intuitive to see this much cost saving by vertical scaling, by increasing CPU. Have you experienced something similar? In my story, the Kubernetes Vertical Pod Autoscaler (VPA) is the hero.

Anyone thinking about vertical scaling or using VPA in production, I hope my experience helps you learn a thing or two. Do share your experience as well for a well-rounded discussion.

-----

Background (The challenge and the subject system)

My goal was to improve performance/cost ratio for my Kubernetes cluster. For performance, the focus was on increasing throughput.

The operations in the subject system were primarily CPU-bound, we had a good amount of spare memory available at our disposal. Horizontal scaling was not possible architecturally. If you want to dive deeper, here's the code for key components of the system (and architecture in readme) -

* rudder-server - https://github.com/rudderlabs/rudder-server

* rudder-transformer - https://github.com/rudderlabs/rudder-transformer

* rudderstack-helm - https://github.com/rudderlabs/rudderstack-helm

For now, all you need to understand is that the Network IO was the key concern in scaling as the system's primary job was to make API calls to various destination integrations. Throughput was more important than latency.

------

Solution

Increasing CPU when needed. Kuberenetes Vertical Pod Autoscaler (VPA) was the key tool that helped me drive this optimization. VPA automatically adjusts the CPU and memory requests and limits for containers within pods.

------

What I liked about VPA

* I like that VPA right-sizes from live usage and—on clusters with in-place pod resize—can update requests without recreating pods, which lets me be aggressive on both scale-up and scale-down improving bin-packing and cutting cost.

* Another thing I like about VPA is that I can run multiple recommenders and choose one per workload via spec.recommenders, so different usage patterns (frugal, spiky, memory-heavy) get different percentiles/decay without per-Deployment knobs.

------

My challenge with VPA

One challenge I had with VPA is limited per-workload tuning (beyond picking the recommender and setting minAllowed/maxAllowed/controlledValues), aggressive request changes can cause feedback loops or node churn; bursty tails make safe scale-down tricky; and some pods (init-heavy etc) still need carve-outs.

------

That's all for today. Happy to hear your thoughts, questions, and probably your own experience with tools like VPA, and dealing with challenges of scale.