Funny how "use hugepages" is right there on the table and 99% of users ignore it.

AIUI in that thread they're saying "0.51x" the perf on a 96-core arm64 machine and they're also saying they cannot reproduce it on a 96-core amd64 machine.

So it's not going to affect everybody both running PostgreSQL and upgrading to the latest kernel. Conditions seems to be: arm64, shitloads of core, kernel 7.0, current version of PostgreSQL.

That is not going to be 100% of the installed PostgreSQL DBs out there in the wild when 7.0 lands in a few weeks.

  a) may get irreversible upgrades (e.g. new underlying database structure) 
  b) permanent worse performance / regression (e.g. iOS 26)
  c) added instability
  d) new security issues (litellm)
  e) time wasted migrating / debugging
  f) may need rewrite of consumers / users of APIs / sys calls
  g) potential new IP or licensing issues

  a) new features provide genuine comfort or performance upgrade (or... some revert)
  b) there is an extremely critical security issue
  c) you do not care about stability because reverting is uneventful and production impact is nil (e.g. Claude Code)

Note that it's just not a single post, and there's additional further information in following the full thread. :)

>If this somehow does end up being a reproducible performance issue (I still suspect something more complicated is going on), I don't see how userspace could be expected to mitigate a substantial perf regression in 7.0 that can only be mitigated by a default-off non-trivial functionality also introduced in 7.0.

> I'd argue a 50% hit to performance [...] is ... quite the regression

Indeed! Especially if said regression happens to impact anything trade/market related...

Entertaining perspective - I thought that the whole "it's not an outage it's a (horizontal or vertical) degradation" thing was exclusive to web services, but thinking about it, I guess it does apply even in cases like this.

This was my immediate thought - kernel doesn’t break software, or at least it didn’t used to.

It wouldn’t be the first time one of the other maintainers ran afoul of “Linus’s law“.

He may simply be waiting until more is known on exactly what’s causing it.

Well, the reason he'd yell about it is that someone did it. If no one ever did it, he'd never yell and we'd never have the rule. So one can only imagine that this is one of those things where someone has to keep holding the line rather than one of those things where you set some rule and it self-holds.

Doubtless someone will have to do the yelling.

Freund seems to suggest that hugepages is the right way to run a system under this sort of load - which is the fix.

> Hah. I had reflexively used huge_pages=on - as that is the only sane thing to do with 10s to 100s of GB of shared memory and thus part of all my benchmarking infrastructure - during the benchmark runs mentioned above.

> Turns out, if I disable huge pages, I actually can reproduce the contention that Salvatore reported (didn't see whether it's a regression for me though). Not anywhere close to the same degree, because the bottleneck for me is the writes.

But, they can speak for themselves here [0].

[0] https://news.ycombinator.com/item?id=47646332

That may be the case, but it’s still not a great situation to be in and one has to wonder: if PostgreSQL is affected, what else is?

> Nobody sensible runs the latest kernel

From the article: "Linux 7.0 stable is due out in about two weeks. This is also the kernel version powering Ubuntu 26.04 LTS to be released later in April."

Unfortunately, lots of people will be running it in less than a month. At the moment, it'll take a kernel patch (not a sysctl) to undo this-- hopefully something changes.

We need some sensible people running the latest and greatest or we won't catch things like this.

The option to set PREEMPT_NONE was removed for basically all platforms.

If you're running in a docker container you share the host kernel. You might not have a choice.

They don't need to, there's about a billion bugs they can exploit.

Heh. Did they at least add useful features to balance out that cost?

> I feel like using spinlocks in user space at all without kernel support like rseq is just asking for weird performance degradations.

Yeah, exactly. "Doctor, help, somebody replaced my wooden hammer with a metal one, and now I can't hit myself in the face with it as many times."

If you use spinlocks in userspace, you're gonna have a bad time.

I'm not familiar with the jargon either, but based on some reading it comes down to how the latest kernel treats process preempts.

Postgres uses spinlocks to hold shared memory for very critical processes. Spinlocks are an infinite loop with no sleep to attempt to hold a lock, thus "spinning". Previous kernels allowed spinlocking processes to run with PREEMPT_NONE. This flag tells the kernel to let the locking process complete their work before doing anything. Now the latest kernel removed this functionality and is interrupting spinlocking processes. So if a process that is holding a lock gets interrupted, all other postgres spinlocks processes that need the same lock spin in place for way longer times, leading to performance degradation.

It's the performance team's job to test these things. Doesn't mean they're going to deploy it immediately.

Someone should be testing these things and reporting regressions

Don't make excuses.