[1]https://github.com/facebookresearch/cwm [2]https://huggingface.co/facebook/cwm
Following that line of reasoning, context length is another very large confounding factor. Longer context lengths improve performance - but also result in enormous increases in KV cache size and memory requirements. We decide to control for this in our paper and focus at the 32K context length for 32B size models, a context length that already pushes the bounds of what can be "deployable" locally.
Still, we evaluate at 64K context length using YARN and are able to outperform CWM's 54% performance (non TTS), which it achieves using 128K context, a substantial increase over what we use. This is also pretty significant because we only ever train at 32K context, but CWM trains for a full 128K.
But the interesting number here isn't accuracy. It's the $400 to reproduce top open-source performance. That's the part that matters for teams building internal tooling.
We've been running agents on proprietary codebases at work. The pain isn't model quality. It's customization. Most off-the-shelf agents don't understand your repo structure, your conventions, your test patterns. If you can fine-tune a 32B model on your own codebase for a few hundred dollars, that changes the economics completely.
But codebases changes everyday, so finetuning will have to be continuously done!
Probably not worth it versus something like Claude Code.
Curious whether anyone's tried this on non-Python codebases. Most SWE-Bench stuff is Python-heavy.
I wish if AI2 could release a more denser model on Openrouter for free than the 8B model as I was using Devstral model for agentic purposes.
If we can get an agentic good 32B like model on openrouter for ~free, then I feel like it will be very interesting to see how things would go imo.
Good luck with AI2! The premise of truly open source models is really interesting and I feel like it could help bring more innovation in the space imo!
jauntywundrkind•2h ago
I wonder if this indeed will start prompting more language specific work.
Afaik training still requires not just looking at sample code but also being able to write loss functions being able to have problems the AI can work at. That seems hard.
One random thought, are there training styles of just deleting some code from "good" projects then making the AI make it work again?
CuriouslyC•1h ago