What or other opinions on how representative the AA rankings are of real-world performance? Any better indicators?
Qwen3.8 27B beats all medium models (40B–150B). It has the same score as DeepSeek V4 Flash 0731, which ranks #5 in large model category (> 150B).
Sources:
- https://artificialanalysis.ai/models/open-source/small
https://simonwillison.net/2026/Aug/16/qwen-38-27b/
It seems like the token usage is 2.3x GPT Luna Max and almost 2x Kimi K3!
I'm curious if they can make up for this with insanely high tokens-per-second especially when served from hosted providers, though, given how tiny it is (37B!)
https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B
And then a Bonsai ternary on top of that model.
On the other hand, there are some of use who are stuck with hardware that has plenty compute, but limited (V)RAM. Qwen3.8 27B is just perfect for that.
anana_•1h ago
bertili•36m ago
Same score as the latest DeepSeek Flash 0731 which has 284B parameters! (13B active)
Its also the second best Qwen model, much better than Qwen 3.7 Max, but significantly below Qwen 3.8 Max.
anana_•30m ago
3.8 actually performs slightly worse than 3.6 on AA-Omniscience Accuracy, which could imply that they traded out world knowledge for capability in other areas.
It also produces nearly twice as many tokens per task as 3.6 (and by extension, time), which may be a tradeoff required to achieve correctness at this parameter size.
skohan•12m ago
With good orchestration and delegation you can get surprisingly far with small models running on consumer hardware.
anana_•5m ago
nsingh2•29m ago
sottol•11m ago
I could run Qwen 3.6 27B on my laptop, but at 5 tok/s it was too slow even without overthinking - I never used it. OTOH, Qwen 3.6 35B A3B ran at 20 tok/s but it just could not get done what I asked of it. It sort of got close but you had to repeat and retry so much that it might have been faster to run 27B dense... maybe?
So that said, I might take a much better model that runs 2-3x slower (total time per task) but that's more capable over a faster, less capable one.
I'd also like to try a proper "plan-then-execute" type execution where thinking is entirely disabled (or low) during the execution stage but enabled/max during the planning stage.
I will definitely give 3.8 27B a better shot than 3.6 though.
johnnyApplePRNG•29m ago
halJordan•11m ago