Top model scores may be skewed by Git history leaks in SWE-bench

https://github.com/SWE-bench/SWE-bench/issues/465

106•mustaphah•1h ago

Comments

piskov•1h ago

Not “may be”: just look how swe-bench scores drop to single digits once it in C#

stefan_•1h ago

So the "Verified" part of "SWE Bench Verified" means.. not "Verified" at all.

I don't get it, who is so opposed to doing the bare minimum of manual work and check what these models are doing? At least back in the day grad students doing an easy meta-paper understood it meant doing some repetitive manual work. Now we got benchmarks by hype vendors who think they can use the thing they are benchmarking to .. mark the bench.

jsheard•56m ago

> So the "Verified" part of "SWE Bench Verified" means.. not "Verified" at all.

Seems on-brand for an LLM-related thing to claim that it has verified something without actually checking.

geekymartian•35m ago

that was my exact thought. how fitting

yorwba•9m ago

The "Verified" part of "SWE-Bench Verified" means that there was plain "SWE-Bench" before it, which had actually not been verified at all and included a lot of tasks that didn't really make sense for use as a benchmark: https://openai.com/index/introducing-swe-bench-verified/#ada...

Data contamination stemming from the fact that it's based on already-solved problems in public repositories is a different issue that cannot be addressed by verifying the benchmark questions harder, but only by putting stricter limits on the model under test.

fine_tune•1h ago

I was going to argue "LLM's need code samples to-do well on languages and if we are honest C# is a language mostly held in private repo's" but Github's 2024 report[0] says its the 5th most used language (I'm to lazy to check if this report includes private repo's but I'll assume it doesn't).

So kinda neat to see this paper!

[0]https://github.blog/news-insights/octoverse/octoverse-2024/#...

yieldcrv•30m ago

5th most used language based on private repos that the group making the report has the exclusive direct access to seeing

I don't see that contradicting your assumption

teaearlgraycold•50m ago

Personally I don't look at or respect LLM benchmarks at all. I've seen SOTA models fail in incredibly shocking ways even recently. Those moments immediately bring me out of the delusion that LLMs have thinking capacity or an understanding of code.

jasonjmcghee•1h ago

Very interested to see the updated results. This could really shake up the leaderboard.

macawfish•1h ago

I hope it does. These coding benchmarks have often seemed frustratingly out of touch with my experience.

3abiton•9m ago

Because I would argue there is no benchmark to rule them all. It highly depends on individual use cases.

mustaphah•1h ago

I speculate something similar (or even worse) is going on with Terminal-Bench [1].

Like, seriously, how come all these agents are beating Claude Code? In practice, they are shitty and not even close. Yes. I tried them.

[1] https://www.tbench.ai/leaderboard

zaptheimpaler•50m ago

It's honestly ridiculous they left git history lying around during a benchmark, and this benchmark made to ICLR in Jan 2024 and no one has detected this issue until now. I don't really trust any benchmarking or tools or claims from this space when they can make such huge basic errors.

belter•47m ago

In the meawhile, Oracle stock went up 40% in one one day, based on what Wall Street thinks AI might be...in 4 years...Not a bubble at all...

ksherlock•29m ago

The real bubble will come once interest rates start dropping.

candiddevmike•5m ago

I think Oracle's stock mostly popped due to a delayed reaction with the US GSA contract it secured in July and the revenue guidance probably related to it:

https://www.oracle.com/news/announcement/blog/oracle-cloud-c...

slacktivism123•46m ago

Fascinating case showing how LLM promoters will happily take "verified" benchmarks at their word.

It's easy to publish "$NEWMODEL received an X% bump in SWE-Bench Verified!!!!".

Proper research means interrogating the traces, like these researchers did (the Gist shows Claude 4 Sonnet): https://gist.github.com/jacobkahn/bd77c69d34040a9e9b10d56baa...

Commentary: https://x.com/bwasti/status/1963288443452051582, https://x.com/tmkadamcz/status/1963996138044096969

Traster•42m ago

Man I feel so dumb. Why haven't I been doing this in my job, if I could just see the commit that fixed my issue this would all be so easy.

Noumenon72•35m ago

Someone did comment that it's actually smart to check if something is fixed on the unstable branch, or I suppose in your coworkers' branches. A good task for an LLM.

mbowcut2•41m ago

I'm not surprised. People really thought the models just kept getting better and better?

ofirpress•18m ago

[I'm on the SWE-bench team] Multiple people have looked into this, for example right in that thread: https://github.com/SWE-bench/SWE-bench/issues/465#issuecomme...

This issue had affected a tiny fraction of existing agents in a tiny fraction of their runs. And we've now issued a fix.

This is a natural part of running a benchmark, I'm sure tiny things like this will keep on getting discovered and we'll keep on fixing them. This doesn't change the overall picture or trends at all.

Network Switch Reverse Engineering

C# Will Become F# – Gautier Talks About Microsoft Technology

Ross Ulbricht: Charlie Kirk helped to get my full pardon

What's working for startup growth in 2025?

OpenAI, Oracle sign $300B computing deal

Program to help AI researchers build AI safety and alignment startups

Gauss, an Agent for Autoformalization

Disposable Code Is Here to Stay, but Durable Code Is What Runs the World

I Rode Honda's Uni-One 'Mobility Device.' It's Coming to the US for Lease-Only

iPhone 17 Air Drops Physical SIM Slot Globally, Pushing ESIM-Only Future

Data Colonialism Is, Unfortunately, a Thing [video]

Death of the Holocaust Industry

Show HN: Ainews247.org – the best, most valuable, fun, AI content, aggregated

US Warns Hidden Radios May Be Embedded in Solar-Powered Highway Infrastructure

Lead-In to Brain-Computer Interface. How to Measure BioData

A Lightweight License Plate Recognition Method Based on YOLOv8

LLM Benchmark and Optimization Explorer

AI tool detects LLM-generated text in research papers and peer reviews

What Every Programmer Should Know About Memory (2007) [pdf]

AI Moderation: Inconsistencies in Hate Speech Detection Across LLM-Based Systems

Randomly selecting points inside a triangle

Brazil's Supreme Court finds Bolsonaro guilty of plotting military coup

Introducing Quote Posts

Show HN: I created a blender tutor that answers with visual cues

Microsoft Goes Back to Basic, Open-Sources Bill Gates' Code

Rethinking AI Infrastructure: The Rise of PCIe Switches

You Need to Be Bored. Here's Why [video]

Show HN: Nannabanana.ai – Nanna-approved image generator

I had the same idea for *...*

Ask HN: Is the market for AI Agent automations that big?