AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

https://www.phoronix.com/news/Linux-7.0-AWS-PostgreSQL-Drop

218•crcastle•6h ago

https://lore.kernel.org/lkml/yr3inlzesdb45n6i6lpbimwr7b25kqk...

Comments

lfittl•5h ago

Its worth reading this follow-up LKML post by Andres Freund (who works on Postgres): https://lore.kernel.org/lkml/yr3inlzesdb45n6i6lpbimwr7b25kqk...

jeffbee•5h ago

Funny how "use hugepages" is right there on the table and 99% of users ignore it.

bombcar•4h ago

I’m absolutely flabbergasted by the performance left on the table; even by myself - just yesterday I learned Gentoo’s emerge can use git and be a billion times faster.

TacticalCoder•4h ago

AIUI in that thread they're saying "0.51x" the perf on a 96-core arm64 machine and they're also saying they cannot reproduce it on a 96-core amd64 machine.

So it's not going to affect everybody both running PostgreSQL and upgrading to the latest kernel. Conditions seems to be: arm64, shitloads of core, kernel 7.0, current version of PostgreSQL.

That is not going to be 100% of the installed PostgreSQL DBs out there in the wild when 7.0 lands in a few weeks.

master_crab•4h ago

For production Postgres, i would assume it’s close to almost no effect?

If someone is running postgres in a serious backend environment, i doubt they are using Ubuntu or even touching 7.x for months (or years). It’ll be some flavor of Debian or Red Hat still on 6.x (maybe even 5?). Those same users won’t touch 7.x until there has been months of testing by distros.

crcastle•3h ago

Ubuntu is used in many serious backend environments. Heroku runs tens of thousands (if not more) instances of Ubuntu on its fleet. Or at least it did through the teens and early 2020s.

https://devcenter.heroku.com/articles/stack

nine_k•3h ago

Do they upgrade to the new LTS the day it is released?

crcastle•3h ago

Not historically.

rvnx•2h ago

and they are right, this is because a lot of junior sysadmins believe that newer = better.

But the reality:

  a) may get irreversible upgrades (e.g. new underlying database structure) 
  b) permanent worse performance / regression (e.g. iOS 26)
  c) added instability
  d) new security issues (litellm)
  e) time wasted migrating / debugging
  f) may need rewrite of consumers / users of APIs / sys calls
  g) potential new IP or licensing issues

etc.

A couple of the few reasons to upgrade something is:

  a) new features provide genuine comfort or performance upgrade (or... some revert)
  b) there is an extremely critical security issue
  c) you do not care about stability because reverting is uneventful and production impact is nil (e.g. Claude Code)

but 99% of the time, if ain't broke, don't fix it.

https://en.wikipedia.org/wiki/2024_CrowdStrike-related_IT_ou...

miki123211•2m ago

On the other hand, I suspect LLMs will dramatically decrease the window between a vulnerability being discovered and that vulnerability being exploited in the wild, especially for open-source projects.

Even if the vulnerability itself is discovered through other means than by an LLM, it's trivial to ask a SOTA model to "monitor all new commits to project X and decide which ones are likely patching an exploitable vulnerability, and then write a PoC." That's a lot easier than finding the vulnerable itself.

I won't be surprised if update windows (for open source networked services) shrink to ~10 minutes within a year or two. It's going to be a brutal world.

pmontra•52m ago

A customer of mine is running on Ubuntu 22.04 and the plan is to upgrade to 26.04 in Q1 2027. We'll have to add performance regression to the plan.

MBCook•3h ago

So perhaps this is a regression specifically in the arm64 code, or said differently maybe it’s a performance bug that has been there for a long time but covered up by the scheduler part that was removed?

db48x•1h ago

Could be either of those, or something else entirely. Or even measurement error.

zamalek•1h ago

It was later reproduced on the same machine without huge pages enabled. PICNIC?

anarazel•1h ago

Yes, I did reproduce it (to a much smaller degree, but it's just a 48c/96t machine). But it's an absurd workload in an insane configuration. Not using huge pages hurts way more than the regression due to PREEMPT_LAZY does.

With what we know so far, I expect that there are just about no real world workloads that aren't already completely falling over that will be affected.

justinclift•4h ago

Note that it's just not a single post, and there's additional further information in following the full thread. :)

aftbit•4h ago

>If this somehow does end up being a reproducible performance issue (I still suspect something more complicated is going on), I don't see how userspace could be expected to mitigate a substantial perf regression in 7.0 that can only be mitigated by a default-off non-trivial functionality also introduced in 7.0.

FireBeyond•5h ago

Once upon a time, Linus would shout and yell about how the kernel should never "break" userspace (and I see in some places, some arguments of "It's not broken, it's just a performance regression" - personally I'd argue a 50% hit to performance of a pre-eminent database engine is ... quite the regression).

Now, the kernel engineer who introduced the brand new mechanism (introduced in Linux 7.0) for handling pre-emption says the "fix" is for Postgres to start using this new mechanism (I think the sister comment below links to what one of the Postgres engineers thinks of that, and I'm inclined mostly to agree).

bear8642•5h ago

> I'd argue a 50% hit to performance [...] is ... quite the regression

Indeed! Especially if said regression happens to impact anything trade/market related...

perching_aix•4h ago

Entertaining perspective - I thought that the whole "it's not an outage it's a (horizontal or vertical) degradation" thing was exclusive to web services, but thinking about it, I guess it does apply even in cases like this.

quietsegfault•3h ago

This was my immediate thought - kernel doesn’t break software, or at least it didn’t used to.

MBCook•3h ago

It wouldn’t be the first time one of the other maintainers ran afoul of “Linus’s law“.

He may simply be waiting until more is known on exactly what’s causing it.

arjie•2h ago

Well, the reason he'd yell about it is that someone did it. If no one ever did it, he'd never yell and we'd never have the rule. So one can only imagine that this is one of those things where someone has to keep holding the line rather than one of those things where you set some rule and it self-holds.

Doubtless someone will have to do the yelling.

dsr_•5h ago

Nobody sensible runs the latest kernel; nobody running PG in production should be afraid of setting a non-default at either boot time or as a sysctl. So this will, most likely, be another step in building a PG database server (turn off pre-emption if your kernel is 7.0 or later and PG is pre-whatever-version).

At worst it might become a permanent part of building a PG server and a FAQ... but if it affects one thing this badly, it will affect others.

stingraycharles•5h ago

That may be the case, but it’s still not a great situation to be in and one has to wonder: if PostgreSQL is affected, what else is?

bombcar•4h ago

That's the big thing - PSQL will be tested, noticed, and fixed (and likely have a version that handles 7.0 by the time it's in common use).

But other software won't and may not even be noticed, except as a (I hate using the term) enshittification.

Better to introduce the "correct way" in 7.0 but not regress the old (translate the "correct" into the old if necessary) - and then in 8.0 or some future release implement the regression.

stingraycharles•3h ago

Exactly, this is how it’s usually done. As the developer on the mailing list mentions, implementing a new low level construct in 7.0 and a performance regression that requires said new low level construct to mitigate is not great. You need a grace period in which both old and new approach is fast.

Meekro•5h ago

> Nobody sensible runs the latest kernel

From the article: "Linux 7.0 stable is due out in about two weeks. This is also the kernel version powering Ubuntu 26.04 LTS to be released later in April."

Unfortunately, lots of people will be running it in less than a month. At the moment, it'll take a kernel patch (not a sysctl) to undo this-- hopefully something changes.

Neywiny•4h ago

Not nobody but not everybody upgrades to the newest distros immediately. That's the advantage of LTS. I've even found that a lot of programs have poorer support on 24.04 than 22.04 due to security changes, so I'm fine sticking with 22.04 as my main dev system.

stingraycharles•4h ago

This seems to be brushing off a major performance regression just because you personally don’t upgrade for 4 years. I don’t think that’s common at all.

justinclift•4h ago

> ... not everybody upgrades to the newest distros immediately.

While that's true, for new deployments the story is often "deploy on the latest release of things available at the time".

So, there will probably be a substantial deployment of new projects / testing projects using the Linux 7.0 kernel along with the latest available software packages in a few weeks.

esafak•4h ago

That's the advantage of LTS? 24.04 is the LTS, not the one you use, 22.04.

SoftTalker•4h ago

22.04 is also an LTS release, supported for another year still.

https://ubuntu.com/about/release-cycle

We're just now looking at moving production machines to 24.04.

cortesoft•3h ago

All even number .04 releases are LTS in Ubuntu

999900000999•2h ago

Depends on your shop.

As someone with a heavy QA/Dev Opps background I don't think we have enough details.

Is it only ARM64 ? How many ARM64 PG DBs are running 96 cores?

However...

This is the most popular database in the world. Odds are this will effect a bunch of other lesser known applications.

whilenot-dev•8m ago

Please follow the complete thread: https://lore.kernel.org/lkml/xxbnmxqhx4ntc4ztztllbhnral2adog...

> [...] used huge_pages=on - as that is the only sane thing to do with 10s to 100s of GB of shared memory [...] if I disable huge pages, I actually can reproduce the contention [...]

bombcar•4h ago

We need some sensible people running the latest and greatest or we won't catch things like this.

cwillu•3h ago

The option to set PREEMPT_NONE was removed for basically all platforms.

Seattle3503•2h ago

If you're running in a docker container you share the host kernel. You might not have a choice.

galbar•5h ago

It's not a good look to break userspace applications without a deprecation period where both old and new solutions exist, allowing for a transition period.

harshreality•4h ago

Background on PREEMPT_LAZY:

https://lwn.net/Articles/994322/

longislandguido•4h ago

Anyone check to see if Jia Tan has submitted any kernel patches lately?

rs_rs_rs_rs_rs•29m ago

They don't need to, there's about a billion bugs they can exploit.

cperciva•4h ago

This makes me feel better about the 10% performance regression I just measured between FreeBSD 14 and FreeBSD 15.0.

db48x•1h ago

Heh. Did they at least add useful features to balance out that cost?

monocasa•4h ago

I feel like using spinlocks in user space at all without kernel support like rseq is just asking for weird performance degradations.

jcalvinowens•4h ago

> I feel like using spinlocks in user space at all without kernel support like rseq is just asking for weird performance degradations.

Yeah, exactly. "Doctor, help, somebody replaced my wooden hammer with a metal one, and now I can't hit myself in the face with it as many times."

If you use spinlocks in userspace, you're gonna have a bad time.

mgaunard•4m ago

Most people looking for performance will reach for the spinlock.

The expectation is that the kernel should somehow detect applications that are spinning, and avoid preempting them early.

The whole concept of running anything but PREEMPT_NONE seems weird to me. I want my kernel to be fair.

cdelsolar•1h ago

https://lkml.org/lkml/2012/12/23/75

anal_reactor•51m ago

Can someone explain to me what's the problem? I have very little knowledge of Linux kernel, but I'm curious. I've tried reading a little, but it's jargon over jargon.

up2isomorphism•10m ago

Not sure why people have to upgrade to the newest major kernel version as soon as it is released.

The Visible Key – A New Way to Verify

World Models (Part 1)

GabeN Is Shitting Yacht Money into Flatpak and You're Still Arguing Init Systems

Apple's Spotlight Search Results Come with Engagement Metrics. No One Knew

BYD flash charging 2026 Seal 06 GT starting from 16,200 US

BYD reveals interior of refreshed Denza D9

What Does ⍋⍋ Even Mean?

The Codification Treadmill

People and Blogs

Python compiler in Rust that runs faster than CPython with a 160KB WASM

Unverified: What Practitioners Post About OCR, Agents, and Tables

The SimCity Planning Commission Handbook (2024)

Polygraph: Signature generation algorithms for polymorphic worms (2005)

Domain-Driven Design: Lean Aggregates

AI doesn't need an API. It needs a mouse

Void: Video Object and Interaction Deletion

Show HN: OsintRadar – Curated directory for osint tools

Aegis – open-source FPGA silicon

Claude Code with Timestamps

Onetime Elon Musk Whisperer Shares the 'Algorithm'

Gloomberb

Altar: The town where migrants shop for a perilous journey (2014)

The German implementation of eIDAS requires an Apple/Google account to function

17,000 MCP Servers – and the Security Threats Nobody Is Talking About

Modern Generic SVGA driver for Windows 3.1

TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks (2023)

Visualisation of the Artemis Mission (German)

Ending My Claude Pro Subscription

slopc: Replace every Rust todo() macro with a spontaneous runtime implementation

What an Ivy League Education Gets You

The Visible Key – A New Way to Verify

World Models (Part 1)

GabeN Is Shitting Yacht Money into Flatpak and You're Still Arguing Init Systems

Apple's Spotlight Search Results Come with Engagement Metrics. No One Knew

BYD flash charging 2026 Seal 06 GT starting from 16,200 US

BYD reveals interior of refreshed Denza D9

What Does ⍋⍋ Even Mean?

The Codification Treadmill

People and Blogs

Python compiler in Rust that runs faster than CPython with a 160KB WASM

Unverified: What Practitioners Post About OCR, Agents, and Tables

The SimCity Planning Commission Handbook (2024)

Polygraph: Signature generation algorithms for polymorphic worms (2005)

Domain-Driven Design: Lean Aggregates

AI doesn't need an API. It needs a mouse

Void: Video Object and Interaction Deletion

Show HN: OsintRadar – Curated directory for osint tools

Aegis – open-source FPGA silicon

Claude Code with Timestamps

Onetime Elon Musk Whisperer Shares the 'Algorithm'

Gloomberb

Altar: The town where migrants shop for a perilous journey (2014)

The German implementation of eIDAS requires an Apple/Google account to function

17,000 MCP Servers – and the Security Threats Nobody Is Talking About

Modern Generic SVGA driver for Windows 3.1

TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks (2023)

Visualisation of the Artemis Mission (German)

Ending My Claude Pro Subscription

slopc: Replace every Rust todo() macro with a spontaneous runtime implementation

What an Ivy League Education Gets You

AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

Comments