https://en.wikipedia.org/wiki/Io_uring
Very easy to just quote that without any io_uring experience.
> In June 2023, Google's security team reported that 60% of the exploits submitted to their bug bounty program in 2022 were exploits of the Linux kernel's io_uring vulnerabilities. As a result, io_uring was disabled for apps in Android, and disabled entirely in ChromeOS as well as Google servers. Docker also consequently disabled io_uring from their default seccomp profile.
> Although initial async offload design in io_uring could be problematic, later kernels changed the thread model. After such improvements, there were no known inherent problems with it and its development is very careful with new features. Considering that a performant async framework with a user facing API is complex, it was to be expected that issues would be found initially. After initial issues have been addressed, it is not any less secure than anything else in the kernel and io_uring acceptance quickly grew in production. Some of its criticism are also based on wrong or outdated assumptions.[14]
...but the only citation is a link to this GH thread, which doesn't support the claims made.
> As I'm sure you know, this is all mostly centered around a) google using an old kernel on android
> My hope is that this reputation will go away eventually, as less issues are found in the code.
this has not yet happened like this other comment shows: https://news.ycombinator.com/item?id=44632639
Having talked to members of their prodkernel team about other subjects, I also think they are competent enough to know the difference between "not ready" and "acceptably flawed". And believe me, the incentives are such that O(Really Big Money) optimization projects get staffed unless there is something making them infeasible.
Not everybody has the same threat model and security stance as Google and that's ok. But personally I would take their internal adoption of io_uring very seriously as a measure of whether it's safe for me to adopt it, especially if I'm running untrusted or third party software (including certain kinds of libraries).
By the way, here is a good + recent example of the types of CVEs that IO_uring runs into that google finds and discloses/fixes: https://project-zero.issues.chromium.org/issues/417522668. Here's another: https://project-zero.issues.chromium.org/issues/388499293
Given that io_uring mostly seems to be the project of one guy at Meta, and has a regular stream of new and exciting use after free/out of bounds vulnerabilities, I think it makes sense for security-inclined users to disable it or at least only use it once soaked/stabilized
Switching to io_uring is not just moving from one API to another. It necessitates a serious rethinking of your concurrency model. I guess for big, established codebases this is a very substantial undertaking, security consideration notwithstanding.
Google is big enough and invests enough in infrastructure projects that they staff projects like making their own internal concurrency primitives (side note, factors like this can improve/reduce or simplify/complexify migrations substantially): https://www.phoronix.com/news/Google-Fibers-Toward-Open
not about their servers
I wouldn't be surprised if they do have servers with it enabled when very useful.
and Android Linux kennels lack behind in their version
I'd love to see a post explaining a decision to consider it stable or that mentions that they've rolled it out on their fleet
The potential performance benefits are quite compelling, e.g. in Postgres 18 you reportedly can get a 3x speedup over the old sync behaviour in simple read queries.
It is, in the general case, hard to prove something secure (because it's hard to prove a negative). It might help to show CVEs per month/year/whatever related to it vs anything else, preferably with a clear downward trend. For example, you could look at https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=io_uring ... although I struggle to read that as supporting the case you want to make.
> I have had to deal with handful of these people from different sectors as well. Since I am actively working on project based on io_uring, I have had people saying all kinds of hmm... "crap", its so baseless! Can't even talk to them with actual facts.
So what are those facts? Because all this thread has is people handwaving that it used to have a worse design, and everything has bugs and this isn't different, and implying that it's better. If it's better, show that.
CVE count by year:
2019: 1
2020: 1
2021: 10
2022: 15
2023: 19
2024: 21
2025: 10Any decently sized project has probably seen an increase in reported CVEs over the past 5 years, simply because the number of CVEs total has grown.
If a kernel panic is considered a security issue, anyone using Nvidia's drivers should fear for their lives.
It's normal to consider a non-root userspace program causing a kernel panic a security issue
The distinction you make is a separate step down the line: you do a risk assessment and decide that, in your particular context, a specific vulnerability is not a threat worth defending against.
For that same vulnerability, others will reach different conclusions in their respective risk assessments.
That doesn't make it any less of a vulnerability.
Is io_uring a complex and therefore bug-prone API surface? perhaps.
The `curl` project has a similar number of CVEs listed if you search for it, but we generally don't characterise curl as insecure.
If you're not using io_uring then it could make sense to disable it as a hardening measure, but I don't think the existence of now-fixed CVEs is a reason not to use it.
So the answer is known :)
Nowadays its mostly a combination of eBPF, SELinux and auditd plus namespaces in case of containers. Usually in the combination that some distro ships, so nothing really fancy.
I'd use io_uring in a heartbeat on a dedicated system where the job is only I/O and security isolation isn't a concern. But multiuser/multiapplication/networked? Not a chance.
1. know what io_uring is
2. are interested in performance enough to look at improvements based on new linux kernel system calls and talk about it in public
3. care about security in multitenant environments or the syscalls used by third party libraries
I think io_uring right now probably makes a lot of sense for HPC and highly technical, performance-sensitive financial stuff, but they can be kind of insular. I don't think most linux hobbyists really need the performance benefits enough t care about it, and most businesses are using a major cloud vendor/don't have the scale or expertise to be thinking about this kind of stuff. Which leaves major cloud providers and really big businesses like Meta with their own internal clouds as the ones that stand to benefit enough to care about performance while really caring about security
This would be useful if you want to boot with io_uring but deny it for some sensitive workloads.
That someone with kernel IO dev experience may be able to relatively easily add such a fetaure in the future (though I would doubt that, given that it hasn't yet been implemented apparently) doesn't make it a small problem.
- io_uring initially was conceived without considering security or auditing tools
- io_uring later was changed to allow ioctl calls, even though security people do not like ioctl because what its arguments mean depends on the device being called (possibly even on the version of the driver), not on the type of device, and often is poorly documented, making it hard for a security filter to decide what to do with a command.
That also made them fear that similar security-breaking changes might be made in the future.
My idea: consider formal verification. A rigorous formal proof of behavior is capable of solving actual flaws and probably capable of overcoming the bad PR.
That would require a great and ongoing effort. However, given the nature of io_uring, it's likely rather amenable to formal verification, and ultimately it is probably necessary. Perhaps our new "AI" tools could greatly reduce the effort.
Anyhow, that's my brilliant thought...
io_uring is only a more ambitious manifestation of a pattern that has been commonplace for a long time. High performance network, storage and other hardware use circular buffers internally in their drivers to process commands and return results. io_uring is attempting to realize the same pattern, but for the entire operating system interface.
That's a really big idea. It has me asking whether this isn't how all "IPC" should be done, where one generalizes all of the running processes, including the "kernel." Ultimately it has significant implications for CPU architecture, supplanting traditional patterns.
When I wrote "given the nature of io_uring," I had something specific in mind wrt formal verification. io_uring fundamentally consumes a stream of bytes. At the front end of that is a great deal of validation code to ensure the steam contents are sane. At the backend, the commands are mapped to existing kernel functionality. It's the front end I think could be well served by formal verification.
When people see what Amazon does it is far too easy to conclude: "Huh, if Amazon blocks it there must me something to it."
I have seen this multiple times when developers were still reciting old benchmarks, taken out of context. It often becomes very tribal and centred around technologies.
Maybe those people are right, though? I think the discussion starts from a place that assumes other people are wrong. If you start there, you will fail to convince people of anything, because you automatically dismiss their claim, without thinking about what they might have seen and what they might think.
A better starting point would be wanting to get to the bottom of it, and assess the security of io_uring. If you start from that point and you give it an honest, thorough assessment, and it turns out it "looks secure", you'll have an easier time convincing people.
You might still be wrong (assessing io_uring's security is not trivial), but at least you tried to understand why people think that.
And reminder: it's ok to "agree to disagree".
I think this is the right approach. We know that io_uring has a somewhat significant history of critical security issues. It's not enough just to point out that "these 3 critical CVEs were fixed in the last 12 months, it's secure now!"
Reputation and trust has to be built over a long period of time.
Bingo. This is the correct approach. Very well said!
Although it gets a bit more complicated, the statement `io_uring` is insecure might be true, that's not really in dispute here. The people who are saying it, aren't saying it because they know it to be true, they are saying it because they heard about security issues in the context of `io_uring` and assumed that using `io_uring` would make your code less secure.
This is incorrect, the security issues are in security features in Linux which have not been updated to handle `io_uring`. This means that your application won't be any less/more secure when using `io_uring`. But your system might be less secure if you have support for `io_uring` enabled and applications can make use of it.
Moreover, the "security issues" are only undoing security related hardening you would have put in place over the baseline, they're not putting you below baseline.
That's why a statement such as `io_uring` is insecure isn't very useful.
If these people make the argument that: "I don't want to use `io_uring` because that would mean that security conscious system administrators would not want to run my software as a precaution." then it would make sense and nobody would be disputing it.
I can imagine the security implications are the same.
That sounds like a problem stemming entirely from a decision to share a ring among multiple application threads. Is there a good reason to do so? Each thread that needs to do IO can have its own ring, and submitting IO to another thread's ring seems like unnecessary complexity. The ring buffers are intended to be single-producer, single-consumer.
johnisgood•6mo ago
nicman23•6mo ago
continuational•6mo ago
concerned_user•6mo ago
If you upgrade then, for example, you can't run all of your PHP 5 code in PHP 8, most of it you can but you will have to change the parts that are broken, which are the areas that are repaired in PHP 8.
Same goes for other languages like C# or Python
pjmlp•6mo ago
nromiun•6mo ago
jeroenhd•6mo ago
The standard libraries were the lacking part in PHP. The language itself was never a serious problem.
xlii•6mo ago
You got bitten and everyone around you assures you won't get bitten again, but the pain was real and you still have a scar from the event. Why bother or invest risking another bite if there are other places to be which had never bitten before.
Over my career I hated technology 3 times. First was PHP, second was Python during Python 2/3 fiasco and third was CoffeeScript.
Edit: till this day one of my favorite meme is titled "PHP: Training wheels without a bike"
yjftsjthsd-h•6mo ago
jiggawatts•6mo ago
I've read through some bug reports, and I assure you that BTRFS remains a horror show.
Saying that comments from the dev team "don't inspire confidence" is putting it mildly.
ZFS is the diametric opposite of this, where blog posts from the team working on it made me realise that they're moving the state of the art into new territory.
johnisgood•6mo ago
BTW, is ZFS on Linux good now?
chasil•6mo ago
https://forums.truenas.com/t/bcachefs-dropped-from-linux-6-1...
ZFS is an install option on Ubuntu. Running it taints the kernel.
whatevaa•6mo ago
em-bee•6mo ago
i haven't heard any issues about coffeescript.
with that in mind, i'd love to hear your stories. how did you get bitten?
lexicality•6mo ago
This means it's very easy to knock out an entire project in record time, but subsequently very difficult to debug/maintain/update the same project when you come back to it. It's essentially a technical debt generator.
johnisgood•6mo ago
lexicality•6mo ago
You can say all sorts of things around "write better documentation" and so on, but step 1 really is "don't use a code golfing language"
johnisgood•6mo ago
xlii•6mo ago
When code volume was small, nobody noticed, and hey "it looks nice". Some time after though, when volume increased it started to get really burdensome. I used this as a part of Rails pipeline, so it was like: write some coffeescript, compile, run - something failed - usual process.
However the code was already mangled, often source code mapping didn't want to work. When source code was found it wasn't uncommon that it caused by operator precedence or code not transpiled in a way it was intended and requiring debugging transpilation process.
At some point I suggested to migrate away from CoffeeScript toward (almost plain) JavaScript and most developers happily agreed to that. We were able to migrate big chunk automatically, rest took only few weeks to clean up. Velocity increased and people were happy they don't have to deal with it anymore.
Ultimatelly it is the truth with most of the transpilers - sooner or later you get into idiosyncracies that - if technology is not popular enough - you're left alone to solve.
em-bee•6mo ago
xlii•6mo ago
As for Python3 there was this thing that at some point Python was supposed to move from 2 to 3. This change was to bring plenty of incompatible changes, and in a way that migration was outright impossible in normal sized projects. Then community split into two parts, some said "F-- it" and held on to Python2 and some said "F-- it" and moved to other technologies. In the end Python2 was sustained for many years (originally sunset period was short) but those who moved out didn't consider it "stable tech" anymore.
itslennysfault•6mo ago
I still hate PHP the most, and I very much mean PHP 5 when I say that, and have no idea what happened beyond that, and honestly the scars are so deep I don't care to find out.
Xenoamorphous•6mo ago
Same with Java, it’s somehow stuck in time in Java 6 in some people’s minds.
WJW•6mo ago
- Rails is horribly slow!
- Python is still stuck migrating to Python 3!
- MySQL doesn't scale!
- Haskell tooling sucks!
- io_uring is insecure!
- ... and dozens more just in the programming world. These are just off the top of my head. Probably hundreds more in the wider engineering world.
dijit•6mo ago
I think the situation is better now, but that was worthy criticism for a long time; like anything though, those that bought into it wholeheartedly couldn’t take the criticisms.
I wonder if there’s a word for that, seems to be a common issue.
als0•6mo ago
"invested"
hyghjiyhu•6mo ago
nurettin•6mo ago
johnisgood•6mo ago
nurettin•6mo ago
johnisgood•6mo ago
I like "PHP & MySQL: Novice to Ninja" quite a lot. I have used it more so than the other books, so I definitely recommend it. You can find the code archive (code from the book) here: https://github.com/spbooks/phpmysql7.
FWIW I recommend these books (in no particular order):
dolmen•6mo ago
I still think that Perl was and is still a better language than PHP and Python. And people never had a serious look at raku because of the Perl heritage.
But that ship has sailed.
em-bee•6mo ago
CGIs in the 90s were written in perl 5, and perl hasn't evolved much since then. but perl was never a bad language. it had/has a quirky syntax. if you were coming from a lot of commandline work using sed, awk, tr, and many other tools to manipulate text in shell scripts, then perl made a lot of sense.
and while raku may have been a great improvement, perls reputation was tied to its syntax. the readability and writeability of the code. and that's the one thing that raku did not change significantly.
so unlike other examples i don't see how peoples views of perl could improve
johnisgood•6mo ago
Raku feels like a different language to me, and it seems not-so-serious, maybe because of the name and their mascot (which is unfair to the language).
em-bee•6mo ago
topspin•6mo ago
Perl (5) has been evolving, quietly. The language has had many nice additions over the years. Some of it is backporting from 6.
em-bee•6mo ago
topspin•6mo ago
Sub signatures, defer blocks, official "class", "say", try/catch, enhanced dereference syntax, new and enhanced operators, etc. Significant language changes have been and continue appearing.