N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?

https://ndaybench.winfunc.com

46•mufeedvh•6h ago

N-Day-Bench tests whether frontier LLMs can find known security vulnerabilities in real repository code. Each month it pulls fresh cases from GitHub security advisories, checks out the repo at the last commit before the patch, and gives models a sandboxed bash shell to explore the codebase.

Static vulnerability discovery benchmarks become outdated quickly. Cases leak into training data, and scores start measuring memorization. The monthly refresh keeps the test set ahead of contamination — or at least makes the contamination window honest.

Each case runs three agents: a Curator reads the advisory and builds an answer key, a Finder (the model under test) gets 24 shell steps to explore the code and write a structured report, and a Judge scores the blinded submission. The Finder never sees the patch. It starts from sink hints and must trace the bug through actual code.

Only repos with 10k+ stars qualify. A diversity pass prevents any single repo from dominating the set. Ambiguous advisories (merge commits, multi-repo references, unresolvable refs) are dropped.

Currently evaluating GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro, GLM-5.1, and Kimi K2.5. All traces are public.

Methodology: https://ndaybench.winfunc.com/methodology

Live Leaderboard: https://ndaybench.winfunc.com/leaderboard

Live Traces: https://ndaybench.winfunc.com/traces

Comments

Rohinator•6h ago

Very curious how Claude Mythos will perform here

mbbutler•5h ago

It would be helpful to add in some cases that do not contain any vulnerabilities to assess false-positive rate as well.

mufeedvh•5h ago

This is a good idea.

Will incorporate false-positive rates into the rubric from the next run onwards.

At winfunc, we spent a lot of research time taming these models to eradicate false-positive rates (it's high!) so this does feel important enough to be documented. Thanks!

cortesoft•5h ago

Any code that is certain that it doesn't have any vulnerabilities is going to be pretty trivial to verify.

Cynddl•5h ago

> Each case runs three agents: a Curator reads the advisory and builds an answer key, a Finder (the model under test) gets 24 shell steps to explore the code and write a structured report, and a Judge scores the blinded submission. The Finder never sees the patch. It starts from sink hints and must trace the bug through actual code.

Curator, answer key, Finder, shell steps, structured report, sink hints… I understand nothing. Did you use an LLM to generate this HN submission?

It looks like a standard LLM-as-a-judge approach. Do you manually validate or verify some of the results? Done poorly, the results can be very noisy and meaningless.

peyton•5h ago

> Did you use an LLM to generate this HN submission?

Must have.

> The Finder will never see the patch.

I wasn’t worried that this eval would show the answer to the model before evaluating it. Seems requirements leaked into this post.

rohansood15•3h ago

I worked in AppSec in the past, made sense to me. Maybe you aren't the target audience?

You don't really need manual verification for these, the CVEs (vulnerabilities) are public and can be programmatically validated.

johnfn•3h ago

Is this really that hard to parse?

Curator and Finder are the names of the agents. "answer key" - haven't you ever taken a test in high school? It's an explanation of the answer. "shell steps" I presume means it gets to run 24 commands on the shell. "structured report" - do I really need to explain to you what a report is? "sink hints" - I admit I didn't know this one, but a bit of searching indicates that it's a hint at where the vulnerability lies.

spicyusername•2h ago

I'd love to see some of the open source models in there

linzhangrun•2h ago

Definitely possible. In January, I tried using Gemini to perform black-box/white-box testing on an existing system in my company (it's quite old). It successfully exploited a hidden SQL injection vulnerability to penetrate the system and extract password hashes (not particularly strong passwords, successfully decrypted on a public website). In terms of pure skill level, I'd say this is at least the level of a mid-level cybersecurity professional, not even considering the significant efficiency improvement.

sacrelege•1h ago

Thanks for putting N-Day-Bench together - really interesting benchmark design and results.

I'd love to see how the model we serve, Qwen3.5 122B A10B, stacks up against the rest on this benchmark. AI Router Switzerland (aiRouter.ch) can sponsor free API access for about a month if that helps for adding it to the evaluation set.

DaVinci Resolve releases Photo Editor

A new spam policy for "back button hijacking"

Someone bought 30 WordPress plugins and planted a backdoor in all of them

GitHub Stacked PRs

Lean proved this program correct; then I found a bug

WiiFin – Jellyfin Client for Nintendo Wii

Design and implementation of DuckDB internals

Nothing Ever Happens: Polymarket bot that always buys No on non-sports markets

Rust Threads on the GPU

How to make Firefox builds 17% faster

US appeals court declares 158-year-old home distilling ban unconstitutional

Write less code, be more responsible

Servo is now available on crates.io

Make tmux pretty and usable (2024)

The AI revolution in math has arrived

Building a CLI for all of Cloudflare

Air Powered Segment Display? [video]

GAIA – Open-source framework for building AI agents that run on local hardware

I just want simple S3

I shipped a transaction bug, so I built a linter

Show HN: Ithihāsas – a character explorer for Hindu epics, built in a few hours

Android now stops you sharing your location in photos

Tool to explore regularly sampled time series

What we learned building a Rust runtime for TypeScript

Tracking down a 25% Regression on LLVM RISC-V

N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?

Hacker compromises A16Z-backed phone farm, calling them the 'antichrist'

Why it’s impossible to measure England’s coastline

Visualizing CPU Pipelining (2024)

New Orleans's Car-Crash Conspiracy

N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?

Comments

DaVinci Resolve releases Photo Editor

A new spam policy for "back button hijacking"

Someone bought 30 WordPress plugins and planted a backdoor in all of them

GitHub Stacked PRs

Lean proved this program correct; then I found a bug

WiiFin – Jellyfin Client for Nintendo Wii

Design and implementation of DuckDB internals

Nothing Ever Happens: Polymarket bot that always buys No on non-sports markets

Rust Threads on the GPU

How to make Firefox builds 17% faster

US appeals court declares 158-year-old home distilling ban unconstitutional

Write less code, be more responsible

Servo is now available on crates.io

Make tmux pretty and usable (2024)

The AI revolution in math has arrived

Building a CLI for all of Cloudflare

Air Powered Segment Display? [video]

GAIA – Open-source framework for building AI agents that run on local hardware

I just want simple S3

I shipped a transaction bug, so I built a linter

Show HN: Ithihāsas – a character explorer for Hindu epics, built in a few hours

Android now stops you sharing your location in photos

Tool to explore regularly sampled time series

What we learned building a Rust runtime for TypeScript

Tracking down a 25% Regression on LLVM RISC-V

N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?

Hacker compromises A16Z-backed phone farm, calling them the 'antichrist'

Why it’s impossible to measure England’s coastline

Visualizing CPU Pipelining (2024)

New Orleans's Car-Crash Conspiracy