I am not ML practitioner, I just need models for my work, for example for coding, I know we can use Claude/Gemini models, but sometimes I want to compare them to SOTA open source, every week something better is coming and reading articles from month ago or finding LLM leaderboard for a specific task is difficult sometimes. I think some kind of model picker already exists, but don't know where
You can look at benchmarks.
https://livebench.ai/#/?Agentic+Coding=a
Keep scrolling until you see something your size. Deepseek R1 is nice, but 600B isnt running on my hardware. You'll also notice they arent doing everything. dominated by the Saas options.
This is sorted by trending by default. This tends to help show interest but not necessarily the best.
Yeah, this is my concern as well, usually top SOTA generic models are good at many tasks, but I can't test them quickly on my machine locally. Especially when seeing claims how 32B model is outperforming proprietary models in benchmarks, I really want to test it myself in my tasks, but after some time they are dropped from news/trends and difficult to find them
Oras•6mo ago
Scroll down to categories, and select from the dropdown on top right of the chart.
throwaw12•6mo ago
but it seems mostly reflects proprietary models (because they are easier to serve)