Author here. I work at IR Labs, mostly on AI systems for low-level code analysis and verification. Short version: about 1 in 30 kernel fixes breaks something new, and we tested whether you can catch the break from the patch alone. Longer version, for the curious:
When I joined IR Labs in February, we were primarily focused on evidence-based code review. That feels like a lifetime ago, but that was when people were just starting to bang the review-is-the-new-bottleneck drum.
Much has changed since then.
In many organizations, review has shifted closer to a commodity (or, at least, people pretend that it has) and SDLC limiting factors have taken on an air of subjectivity: just fly a little closer to the sun and maybe everything can move that much faster.
However, for the most part, every organization gets to choose how close to the sun they fly. Conservative, risk averse orgs can still move slow and try not to break anything.
Except when they can't.
In this year of AI, there's one thing that no one can afford to go slow on: fixes.
Unfortunately, even in the best of times and most careful of orgs, fixes break things, too.
This seems obvious, pedestrian even. But there was a moment a while back when this struck us, full bore, and we realized there was an oncoming cataclysm of fix-induced failures. Patch slinging frontier models, AI-enhanced adversaries, and mountains of critical software festering with undreamt vulnerabilities would put enormous pressure on fix delivery. Everyone would have to move fast, and things would definitely break.
So, we started working on systems to help folks minimize their fix-driven breakage rate. Rather than just a/b testing a fix against a PoC, we've shifted toward a much more defensive flavor of fix verification that tries to anticipate the unanticipated.
Still early days for us, but we're trying to get better at sharing early and often.
To kick things off, we wanted to share two things.
First, some early results from exploratory analyses of the kernel, where we look at the dynamics of fix-introduced defects. For instance, we found that about 1 in 30 fixes to runtime kernel code broke something that needed its own fix within a year, and some needed several repairs, a few of which needed repairs of their own. We'll have a lot more to say on that topic in the coming weeks.
And second, results of an experiment: how reliably can we detect fix-introduced defects with executable evidence? So far, our system caught 4 of 6 real breaks and 7 of 10 planted ones, from the patch alone.
Again, this is all early (e.g. sample sizes are small) and we're working on a more robust eval report.
If you maintain a kernel subsystem, or anything else where fixes ship under pressure, please poke holes. I'd love to hear where this would fall over.
Fair warning: I'm offline for 10 days starting in about 30 minutes (and will try real hard to stay that way). Teammates will be in the thread.
joehoover•7m ago
When I joined IR Labs in February, we were primarily focused on evidence-based code review. That feels like a lifetime ago, but that was when people were just starting to bang the review-is-the-new-bottleneck drum.
Much has changed since then.
In many organizations, review has shifted closer to a commodity (or, at least, people pretend that it has) and SDLC limiting factors have taken on an air of subjectivity: just fly a little closer to the sun and maybe everything can move that much faster.
However, for the most part, every organization gets to choose how close to the sun they fly. Conservative, risk averse orgs can still move slow and try not to break anything.
Except when they can't.
In this year of AI, there's one thing that no one can afford to go slow on: fixes.
Unfortunately, even in the best of times and most careful of orgs, fixes break things, too.
This seems obvious, pedestrian even. But there was a moment a while back when this struck us, full bore, and we realized there was an oncoming cataclysm of fix-induced failures. Patch slinging frontier models, AI-enhanced adversaries, and mountains of critical software festering with undreamt vulnerabilities would put enormous pressure on fix delivery. Everyone would have to move fast, and things would definitely break.
So, we started working on systems to help folks minimize their fix-driven breakage rate. Rather than just a/b testing a fix against a PoC, we've shifted toward a much more defensive flavor of fix verification that tries to anticipate the unanticipated.
Still early days for us, but we're trying to get better at sharing early and often.
To kick things off, we wanted to share two things.
First, some early results from exploratory analyses of the kernel, where we look at the dynamics of fix-introduced defects. For instance, we found that about 1 in 30 fixes to runtime kernel code broke something that needed its own fix within a year, and some needed several repairs, a few of which needed repairs of their own. We'll have a lot more to say on that topic in the coming weeks.
And second, results of an experiment: how reliably can we detect fix-introduced defects with executable evidence? So far, our system caught 4 of 6 real breaks and 7 of 10 planted ones, from the patch alone.
Again, this is all early (e.g. sample sizes are small) and we're working on a more robust eval report.
If you maintain a kernel subsystem, or anything else where fixes ship under pressure, please poke holes. I'd love to hear where this would fall over.
Fair warning: I'm offline for 10 days starting in about 30 minutes (and will try real hard to stay that way). Teammates will be in the thread.