When using Opus or Codex vs. a chinese or Open source model, it feels like its reasoning capabilities are basically the same.
The difference is typically in coding. It looks like OpenAI and Anthropic invest a lot in pre-training (paying Mercor and the like).
Also a lot in creating synthetic data, I believe this has bigger AI research involvement and techniques.
Of course, there's the RLHF loop that developers using Anthropic/OpenAI products as well, which provides probably yields very good data.
This ends up creating the perspective that it is smart, after all, it has been trained with what you want to do, so it can do that for you.
But overall, is there really much AI research being done on those companies, or are the AI researchers mostly fine-tuning small aspects of the model, akin to what Google engineers used to do for Google search?
I ask this because this all looks like somebody with money could throw money at the problem and end up with a better model at the end, provided they do what I outlined above better -- with AI research being really not that important.
It still often feels like talking with ChatGPT 4 with just better data.
Even the big upgrade of Claude Code being able to work autonomously looks to be mainly due to it knowing how to grab context and do tool calls (not saying that this is easy), rather than the model's raw performance being better.
Or am I wrong, is there something extremely good on those models that AI researchers discovered that the others don't have? Or is it really mostly Data?
curioussquirrel•48m ago
I can speak for my area of expertise - multilingual capabilities. Some SOTA models are making huge strides in their support of various languages, and increasingly they understand and can produce text in languages where GPT-4 era models were absolutely lost. These are probably from a combination of richer training dataset and architectural improvements (more parameters?).
I posted about this here if you're interested: https://news.ycombinator.com/item?id=47847282
Now that doesn't necessarily mean that models are also getting substantially better at English or other major languages. They likely are to some degree, but we've reached a point with major languages where core linguistic proficiencies are covered, and what's left is the more squishy part: style, tone of voice, ability to use different registers naturally, or what some people would call linguistic taste. But that's much harder to measure and therefore trickier to provide evidence for.
Hope this helps.
Edit: typo, clarification