Because the data retention policies didn't guarantee that the ARC team could run the semi-private set of problems without fear of them being trained on later on. They only run the semi-private set when they get assurances like ZDR.
Just like the hugging face incident, Opus 5 could have escaped and went to grab data for training it shouldn’t have been able to..
martianvoid•30m ago
zzleeper•15m ago
10xDev•12m ago
Jensson•10m ago
It is very hard to make a benchmark you can't do that for, but it is very easy to make your own personal test that others can't do that for since now it isn't a benchmark they can target.
10xDev•7m ago
NitpickLawyer•6m ago
They didn't. Kaggle is still running for a few more months, best result atm is ~2% with 9h runtime on one rtx6kPRO. Also note that these new results are on the semi-private set, not the public 25 games ones. Any announcement where you see "solved ARC3" is likely only dealing with the 25 public games. And that's highly questionable, until you get to see the code. (which, to my knowledge the team that claimed 99% hasn't yet published).
raincole•10m ago