Opus at home
I hope there also will be a new ~10b variant
categorization, information retrieval, semantic search, image description
also with the model as part of an agentic system with tool calling
Unsloth's GGUF quants are up: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
So advantage is not having to produce your own quantisation / gguf from .safetensors you've linked.
on my dual 3090s qwen 3.5 27b was running at around 110tps using the config from https://github.com/noonghunna/club-3090
make that 200tps on a single 5090, 4x faster than opus https://x.com/radixark/status/2088285681131110446
devs about to get handed a two 5090 box each and told to max that out
I also miss the Qwen 3 Coder Next, which was 80B A3B, there are quite a few use cases where a non-dense model <100B would be the sweet spot (when you have the VRAM but not the TDP or compute power). Heck, I'd gladly take A5B or A8B or even A10B as a sort of middle ground.
Also alternate link for viewing the images without signing in: https://xcancel.com/Alibaba_Qwen/status/2088280182356611304
Qwen3.5:122b:a10b can run about twice as fast as this 27b dense model.
Whats up with focusing on the active param count? Do yall fiddle with the weights or something?
Insane if that is the case. Downloading now!
2 bpw - ~85.7t/s
3 bpw - ~58.0t/s
4 bpw - ~43.9t/s
6 bpw - ~29.5t/s
8 bpw - ~22.2t/s
16 bpw - ~11.2t/s
without cheating. You'd have to exclude layers, skip operations, etc. basically do stuff the model wasn't trained for. And speed collapses so fast with context that even 2 bpw would be looking at ~37.6t/s after just 128K tokens.
MTP only improves the situation by up to 2x in the ideal case, while drastically reducing the performance floor. While optimizing a 9B model on this hardware, I've found that the GPU just doesn't have enough FLOPS to handle speculating more than one or two tokens ahead on a single stream, regardless of quant level, simply because of the arithmetic cost of the forward pass. The 27B model would be even more expensive than that, potentially such that it's already bottlenecked by the GPU itself rather than memory.
I wouldn't get my hopes up for the 35B-A3B either. Not only is it reportedly much less intelligent, but I hit the exact same 85t/s wall in practice (again with highly specialized inference).
Without speculation I can reach around 120t/s on Qwen3.5-9B and with n-gram speculation (not even MTP; this derivative didn't come with one) around about 150t/s on average. This is on the very very edge of what I'd consider acceptable for me to even consider using such a compact model. YMMV due to the silicon lottery but the situation isn't good.
I saw unsloth has 1bit quants too so I might check that out, anybody have experience with those?
So one open weight model might "meet" Opus or whatever on benchmarks, but then fail to follow a simple answer format and also tool call correctly. The models are whipped to within an inch of their lives to strictly adhere to their post training quality gates.
__ __ ___ __ __
\ \ / / / _ \ \ \ / /
\ \ /\ / / | | | | \ \ /\ / /
\ V V / | |_| | \ V V /
\_/\_/ \___/ \_/\_/I only trust those users genuine personal tests
https://www.youtube.com/@lukesdevlab
I don't know if that is what you are looking for or not and as always your experiences may be different.
Completely local use is a different story, of course.
I care a lot about speed and efficiency right now. For my setup I would like to have 2-3 different model families. I've settled on GLM-5.3 (formerly Deepseek v4 pro 0813) for architecting, Deepseek V4 Pro 0813 for developing, and Gemini flash lite (any recent cheap model) for repo scouting. I'll add another one in the mix for reviewing (in this case Gemini 3.7) and that's all I need.
I've tried most models except Grok.
Qwen is too expensive IMO (Alibaba Cloud subscriptions are hard to come by and I'm not spending 50 euros a month for a tool, so 18 euros it is). If it ever becomes efficient enough to run locally I will definitely look back.
Claude is slow and expensive (the cache hit prices are absurd).
OAI is pretty good, I might add it to my arsenal seeing how cheap it is.
These opinions change every day. Last week I would've never picked Deepseek until I read about the pricing. even post aug 16 it's worth it (although it's getting close to gemini pricing).
Right now my costs are 12 euros a month (z.ai) + whatever deepseek consumes. This typically isn't more than 8 euros a week. 44 euros a month and I have a setup that is doing pretty well.
Dude, GLM-5.3 released _today_.
The phrasing "I've settled on" is incorrect for this context.
my prediction was way too far out. 4.6 at home! Woo.
Run the one you linked if you are running vllm (safetensors)
It's a bit bare at the moment, I assume they are going to add further detail later, similar to their other releases.
I'm a huge open model fan, and have used them since forever, even have daily drivers for on-prem dev, but no. They do not beat opus on real-world usage.
Qwen models are impressively good for what they are, are "good enough" for plenty tasks, can be ran locally on decently priced hardware, and so on. They certainly have their uses, and the field in general has advanced faster than my early expectations. But to compare a 27B model to SotA behemoths from a few months ago is doing everyone a disservice, especially people who pick it up, try to use them just like API models, and leave disappointed and confused. Number goes up on a benchmark isn't it.
I agree, but then we just need meaningful benchmarks that clearly show that! Otherwise it's hand waving about something that should be put on paper in quantifiable terms.
That's the rub. AI benchmarks are IMO, by and large totally unreliable. We think of them as similar to traditional benchmarks of deterministic processes where the number of variables is low. But they're anything but that. Non-deterministic processes with an astounding number of variables and fuzzy acceptance criteria.
It leads to results like these, where if you take it at face value, the only conclusion you can draw is "wow Anthropic must be stupid if Opus takes 1T parameters to do what Qwen can do in 27B."
We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios (mostly Rust, some C for microcontroller stuff). Qwen3.6-27B scores only 4% lower for pass@1, n=250 compared to Opus-4.8.
For the labeled dataset, the average PR size they're being measured against is around 1.5k SLOC.
This is very much "real-world usage" for us. The sort of change sets that come in daily/weekly and are solving non-trivial issues in the respective codebases.
As is usually the case, the most broad claims from both the labs and from the consequent pushback are talking past each other.
Unsloth: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
This might work for you, but I didn't get on very well with MTPLX when I tried it a while back; YMMV: https://huggingface.co/Youssofal/Qwen3.8-27B-MTPLX-Optimized...
What you're currently doing is "testing out"
chvid•48m ago