So I vibe-code a lot these days and recently i decided to give the same prompt to several llms, then get their codes and later give each code to every single one of them to ask which one they think is the most useful without telling them that they or the other 2 llms wrote it. The overall consensus is: gpt5. True I only compared gpt5 vs claude 4.1 vs qwen 230bn. OSS 120b, gemini and grok 4 were excluded since well i don't have the time. And obvious failures like amazon nova or anything from meta weren't even planned. Deepseek (both) seem a bit underperforming . Personally I'd say it's a close call between claude opus 4.1 vs both gpt4 and gpt5 (ironically gpt5 sometimes performs worse than 4, i think this has been addressed by many people already). That's just my personal experience, i know HumanEwal or SWE or whatever give various performance but idk, Musk used the benchmarks as "proof" to hype Grok and in my experience grok 4 is between LLAMA4 and obviously behind gpt4 or some variations of qwen.
Again this is coding only: Python and C. For physics, chemistry, scifi novels or whatever the case may be very different. Another kudos to OSS 120bn btw: it's very generous on tokens...like it will write a small programming book if it takes to in one reply, unless of course you tell it to be more limited, this is a huge plus for me since the code I demand should be complex and not some 20 lines nova "pro" joke.
incomingpain•5mo ago
certainly got the job done. I doubt my gpt 20b or ~30b local llm would have been as capable. Overall it was about ~2000 lines of code to change, probably only 100,000 context.
gpt5 didnt one shot it. there were many steps inbetween. At the end, few hours, i had >50 linter warnings from tripled imports, loads of dead code that wouldnt be touched and for some reason gpt5 just couldnt fix any of this. Ended up increasing the warnings and added an error. My expectation is that any of the big guys could immediately fix it. Even restarted fresh context and gpt just wasnt having any of it. im certain even gpt 20b would have completed it in a minute. Curious.
I went to gemini flash, very generic prompt about linter warnings and it fixed it in 30 seconds.
Just kind of weirdness that benchmarks will never be able to catch. It's also going to be very dependent. A rust programmer might have a favourite, whereas python programmer benefits from another model. There can never be a best.
adinhitlore•5mo ago