Phillip has updated the leaderboard and surprisingly every new model release got substantially better, except for the OpenAi models. 6.2 Sol is 30th whereas 6.1 Sol is the 10th. Astra 6 is 9th, Astra 6.2 is 15th. By comparison Fable 5.2 became the 1st and Fable 5.1 is 7th. All the new google models, even hydrogen, are above the human baseline. By contrast, Astra 6 pro is the only openai model slightly above the human baseline at 8th place
schleck8•10m ago
This is so strange. Perhaps it's recurrent depth. I don't think the new Anthropic models use it but GPT 6 does
bieganski•18m ago
I tried taking the test, but ultimately stopped at Question 4:
There are two sisters, Amy who always speaks mistruths and Sam who always lies.
:)
InsideOutSanta•6m ago
I thought you were joking, but that's actually the fourth question. Then I thought maybe it was meant to be a trick question, but none of the possible answers make sense.
unglaublich•4m ago
[delayed]
pvitz•4m ago
The trick here is that the classic version has one sister telling the truth and the other the lie. Then you would ask what the other sister would answer. In this case, you can just ask which path to take and then you take the other one.
algoth1•1h ago
schleck8•10m ago