Six curl CVEs after OpenAI and Anthropic came back with zero

https://aisle.com/blog/aisle-discovered-six-curl-cves-after-openai-and-anthropic-found-zero

54•goobreee•1h ago

Comments

anilgulecha•56m ago

That's bragging rights correctly earned, i think! As marketing-y as this post is, definitely something to keep an eye on.

melvinroest•55m ago

Wow, this announcement is good content marketing.

Don't get me wrong, it's interesting. But there is no technical discussion as to how they did it. It's simply: we did it and Mythos and Codex didn't.

It's good to know that it's possible, but I'd have already expected it. Put a base model versus a base model + harness + whatever else, and yea, if you do it right then you have a better system to find vulnerabilities.

> We then ran AISLE's autonomous AI system against curl.

They don't even mention what models the use under the hood. It wouldn't surprise me if they are from Anthropic and OpenAI.

drdrd•50m ago

> what models the use under the hood

Presumably their own, wouldn’t they?

melvinroest•48m ago

You mean their own trained models, or do you think it's an open source model that they fine-tuned? If they use their own, I'd guess it's the latter.

catlifeonmars•13m ago

Maybe the model doesn’t matter, maybe you just need something minimally intelligent to seed the fuzzer, generate a test case, and rinse and repeat when the fuzzer gets stuck.

tux3•40m ago

The homepage says something about AI guided fuzzing based on libfuzzer or AFL. Looks like they have the LLMs identify a bunch of interesting functions to test, generate some test harnesses, and then sort through the fuzzer findings at a high level, which sounds like a pretty good idea.

melvinroest•37m ago

Thanks for figuring that out. Sort of sounds like AI programming programs to find vulnerabilities, of which fuzzing is one of the proven techniques to do it.

bch•18m ago

Also sounds incredibly compute intensive.

whizzter•35m ago

Their system can run with various models, they go into more details in this article.

https://aisle.com/blog/system-over-model-zero-day-discovery-...

wky•29m ago

It wouldn’t surprise me if AISLE uses many different providers’ models, and what’s holding back OpenAI and Anthropic is only using first-party models. Just because OpenAI and Anthropic have arguably the strongest models overall doesn’t mean their models are the strongest at finding any given class of vulnerability or lead to follow.

rwmj•50m ago

We had a few AISLE-generated security reports, and the signal to noise was reasonably good.

The most notable bug/exploit their scanner found was: https://gitlab.com/nbdkit/libnbd/-/commit/e50bbd2681117c2dd8...

The tool basically had to chain two exploits together to reach this. It also came up with a patch to fix which was fairly sensible (but I ended up editing it further for clarity).

TechTechTech•48m ago

Good marketing and definitive proof that local (read: on-prem & air-gapped) models with correct context and tools are good enough to perform on par and above SOTA cloud hosted solutions.

We have seen this point many times before with different technologies. The first computers at university were big and expensive, same as this machine. Give it a few years and this functionality will be a commodity.

Surac•34m ago

Marketing Slop

bluGill•33m ago

OpenAI and Anthropic have both been studying CURL for a while though. Anything they found was already fixed.

If you want to compare you need to start with something that none of studied. Somebody please take the source to a 2023 release of CURL (It shouldn't be hard to find one) - before all the current AI craze, and run all the tools on them to see what they find. Only then can we compare numbers. (and even then severity may come into place - all 6 are rated low impact)

thih9•18m ago

I guess this would also require models trained on pre-2023 data - or not trained on later curl code, changelogs, blog posts discussing curl security fixes, etc.

goobreee•9m ago

i don't think this is doable fairly. as they say in the blog post, the only fair way to is to look for new, previously undiscovered zero-days, otherwise you always risk the model has in some way been trained on the vulnerabilities. looking for legit new stuff is the only way to prevent leakage (even accidental one)

zamadatix•12m ago

"How many total vulnerabilities can your tool alone identify?" and "How many unique vulnerabilities can your tool identify?" are both valid comparisons to make IMO.

goobreee•10m ago

I think you might be misunderstanding this? This is, from my understanding, what went down:

1. curl was scanned by many different things, including AISLE, and many bugs were fixed <- all this was in the past 2. curl a week ago was scanned again my Mythos and Codex Security, and both of them said: 0 issues found 3. the same curl was scanned by AISLE a day later, resulting in ~29 reports (based on the blog post and mastodon posts from Daniel Stenberg) 4. of these 29, 6 cleared the bar and got CVEs in curl 5. these 6 CVEs were just announced as fixed in curl 8.22.0 today, together with 4 more CVEs that were detected by other people prior to point 2. of this list

so imho it was head-to-head, the very same codebase => it's a legit comparison

guptadagger•19m ago

This is an ad. I didn't learn anything from reading it.

markasoftware•13m ago

Since AISLE reported 29 issues but only 6 warranted a CVE, and all the found CVEs were "low" severity, this makes me wonder if AISLE simply is tuned for a higher false positive rate than the anthropic and openai tools (which may have found the same 6 issues and decided not to report them)

goobreee•7m ago

i don't think this is correct. if you look at this article by the curl founder daniel stenberg (https://daniel.haxx.se/blog/2026/05/11/mythos-finds-a-curl-v...), he talks about how he previously ran Mythos on curl and that it found 5 issues: 1 turned out to be a low severity CVE, 3 were false positives, and 1 just a bug. So a) Mythos detects low severity CVEs too, and b) it is fairly noisy

A Note from LWN

GrapheneOS says Pixel 11 has MTE support after all

Three sites made 215,128 “best software” pages for AI. Perplexity cites them

Biggest dark matter detector spots a single weird particle

Poisson Disk Sampling

Mistral now trains on user input by default, except on enterprise tier

Six curl CVEs after OpenAI and Anthropic came back with zero

Commodore 64 released September 1, 1982

WebLLM: high-performance in-browser LLM inference engine

Aging Brains Blend Memories Together Instead of Just Forgetting Them

Exit the Cave

Check if a file was made with Claude

A Selection of Los Alamos Rolodex Business Cards

A third of Perplexity's citations don't contain the number they're cited for

Dutch central bank moves share of gold from U.S., Canada to London

LLMs: Intelligence vs. Cost

Ending my elixir exploratory writing

The Emergent Symbolic Structure of Artificial Neural Networks

Telli (YC F24) is hiring engineers and designers [Berlin, on-site]

A Beginner's Deep Dive Guide to Entra Passkeys

Banca Etica Suspends A/I's Account While Condemning the Sanctions Behind It

Quasar 438B: Europe's Leading AI Model

The Cables That Connect the World

Move in C++ without a std:move

It's OK to hardcode feature flags (2025)

Using jq to format JSON on the clipboard

Why humanoid robots won't catch up to human workers any time soon

Crossing Boundaries with Integration Events

Fine, I'll build my own text editor

9000 RPM is faster than your screen refreshes

A Note from LWN

GrapheneOS says Pixel 11 has MTE support after all

Three sites made 215,128 “best software” pages for AI. Perplexity cites them

Biggest dark matter detector spots a single weird particle

Poisson Disk Sampling

Mistral now trains on user input by default, except on enterprise tier

Six curl CVEs after OpenAI and Anthropic came back with zero

Commodore 64 released September 1, 1982

WebLLM: high-performance in-browser LLM inference engine

Aging Brains Blend Memories Together Instead of Just Forgetting Them

Exit the Cave

Check if a file was made with Claude

A Selection of Los Alamos Rolodex Business Cards

A third of Perplexity's citations don't contain the number they're cited for

Dutch central bank moves share of gold from U.S., Canada to London

LLMs: Intelligence vs. Cost

Ending my elixir exploratory writing

The Emergent Symbolic Structure of Artificial Neural Networks

Telli (YC F24) is hiring engineers and designers [Berlin, on-site]

A Beginner's Deep Dive Guide to Entra Passkeys

Banca Etica Suspends A/I's Account While Condemning the Sanctions Behind It

Quasar 438B: Europe's Leading AI Model

The Cables That Connect the World

Move in C++ without a std:move

It's OK to hardcode feature flags (2025)

Using jq to format JSON on the clipboard

Why humanoid robots won't catch up to human workers any time soon

Crossing Boundaries with Integration Events

Fine, I'll build my own text editor

9000 RPM is faster than your screen refreshes

Six curl CVEs after OpenAI and Anthropic came back with zero

Comments