Interesting that they went NoPE everywhere — everyone else hedges with RoPE in the local layers. Feels like the linear-attention stuff (Kimi Delta) is quietly doing the positional work so they can get away with it. Curious to see if it holds up at frontier scale.
wongarsu•19m ago
Is frontier scale larger than this? Kimi K3 seems to benchmark in the same range as Opus and Fable. I would have expected they are all in the 2-4T range, with quality of the training and architecture differences as the major differentiators
porridgeraisin•7m ago
The number of active parameters is vastly different. Deepseek CEO hinted that he estimates it as an order of magnitude difference in one of his recent interviews.
gokohl•27m ago
wongarsu•19m ago
porridgeraisin•7m ago
> Seems to benchmark
yes, but in human usage the differences show up