The 1bit quant model is at an astonishing 397GB with 95B active per MOE. This literally puts Opus 4.5 performance level into a machine a normal person could buy, and still gets usable tokens/second.
The full lossless model BF16 is clocking at 4.9TB. The model card claims the model to be between Opus 4.8 and Fable 5. Again that's astonishing as getting a machine with 7TB RAM (with context + KV cache) is still within the realm of medium size companies.
Bad things: The open source version has its vision capability removed, and the context capped at 250k . I expect someone to bolt a Kimi 2.6 vision tower to it to restore the vision capability (at less performance of course). For context, I played around with extending the context to 600k for Qwen 3.5 397b, and the context remained stable up to around 480k. It'd be interesting to see if the same can be done to Q3.8 .
Also no out of the box DSpark/DFlash support. MTP is present so we should at least get some boost in TP speed.
Honestly this model people at home can tinker with, if you have a big enough Mac. Maybe 4 Strix Halo/DGX Spark, and then at 1 bit quant? Nah.
Use the right sized model, for your hardware. You'll get better results.
Extremely large models don't suffer as much from quantization due to its weight topology also contains encoded information, so the loss of info from any one weight is somewhat mitigated.
I wouldn't pick up 400gb of hardware to run in that mode. I might try it for fun, but even then you are looking at handling a 95GB active parameter set.
This is NOT a model for most home labs. I'm sure some can and will use it. But most, should steer clear.
License pretty similar to k3 with some caveats. Free to use for internal or <50M$ revenue / year. Limitations above that threshold for serving the model or services targeting coding / productivity agents.
Benchmarks are looking good, trading blows w/ opus4.8 and sol, generally 10-20p under fable. But that's neither here nor there w/ qwen, their benchmark to real world usage correlation has been iffy in the past.
The local model 3.8-27B announced for Friday, same time so ~48 hours from now. That'll be a bit more exciting for a lot more people, since 3.6 was quite good for local inference, and their 3.7-max -> 3.8-max shows a lot of improvement.
That is unfortunate, that the open weight model doesn't have vision support or the 1M context length...
[1] https://old.reddit.com/r/LocalLLaMA/comments/1vl6ior/i_gave_...
[1] https://www.reddit.com/r/LocalLLaMA/comments/1vmi0fg/deepsee...
In terms of what you get for what you pay for, it's incredible - probably by far the best.
But unless I'm reading things wrong, it does not appear to be top-of-the-line.
Suppose I have 100GB of unified memory, how should I know which model suits it best? I understand how a 2.4T model wouldn't fit, but I don't understand the impact of quantization and whether I should use a 200G model quantised to fit say 90GB of memory, or a non-quantised 90G model.
The old rule of thumb was that a lower quant of a larger model > higher quant of a smaller model. That being said, for some things going lower than fp8 will see a lot of degradation in generation quality. Except if the model comes with QAT 4bit quants. Then there's also nvfp4 w/ calibration data, which also can improve things. So it's really not easy to tell "at a glance" you'd have to test them yourself on your hardware.
Anything below that, and especially 1.58b - is typically complete garbage, and you're much better off running a model 100x smaller.
If the model was designed specifically to quantize down to 1.58b, then it's different.
AFAIK, there's no large models designed for this yet.
volf_•1h ago