fp.

I’ve come to realise that “wide logs” like OpenTelemetry traces are the only way to go, despite the expense of collecting and storing them with current technology.

As open source columnar databases improve, the cost will drop.

zeafoamrun•28m ago

Same thing when it comes to memory. The rabbit hole goes on forever, and metrics lie to you if you don't know how to interpret them properly.

ksk23•27m ago

TLDR; if app slow, give more resources

luipugs•19m ago

Or just don't put CPU limits: https://home.robusta.dev/blog/stop-using-cpu-limits

andrepd•17m ago

Writing better code is of course out of the question.

dgellow•13m ago

What do you mean, I always append “make it excellent” to all my prompts!

techpression•26m ago

Lovely read, if you’ve ever had even remotely similar issues (you think you’re looking at the right places but you’re not) it read like a detective novel.

CodesInChaos•19m ago

It's well known that many throttling implementations are broken, usually by design. You shouldn't blame the CPU utilization metric for that footgun.

In a well designed scheduler, a task that has been granted an allotment of at least n cores, should never get throttled to less than n cores at any time. It can be limited to less than n cores if CPU utilization is at 100% and another task gets scheduled at the time, since that's unavoidable when you oversubscribe the available resources.

VimEscapeArtist•3m ago

Let’s measure temperature :)

arianvanp•2m ago

A more general metric that is useful to watch for is pressure stall information for CPU, IO and Memory.

https://docs.kernel.org/accounting/psi.html

I made a Prometheus exporter for it:

https://github.com/arianvp/cgroup-exporter