Hey Unsloth, your gguf are the first ones I look for when I want to download a gguf model.
Today I was trying in fact to see, what's the smallest Qwen3.8-27B that I could run and get good results, say restricting it to 16GB of ram.. so I went, pick up the Qwen3.8-27B-UD-IQ2_XXS.gguf and them BAM, error on MTP... now I understand why after reading your announcement.
Beyond the space saving, why removing the MTP? improves speed exactly for the group that could benefit from it.
mike-the-brain•35m ago
you can still have it, no?
> We also removed the MTP module from smaller quants under UD-Q2_K_XL (8.37GB and lower) to converse around 500MB of disk space - you can use the Q4_0 MTP separate module if needed
xlayn•4m ago
my bad, you are totally right, thanks!
mike-the-brain•36m ago
Might be off-topic but: is it possible to perform such a quantization on Apple devices? Something like Mac Studio Ultra M1 (even if it would take weeks/months)?
throwa356262•33m ago
"We also made some smaller UD-1bit quants with UD-IQ1_S being 6.2GB (without MTP) which retain around 72% top-1% accuracy yet being 89% smaller"
This is crazy! But has anyone tried these lower quants on real projects?
kennywinker•23m ago
Not 1-bit, but I’m getting pretty good results with some light coding using unsloth’s previous 2-bit quant of qwen3.8-27b. With these new quants i may be able to bump up to 3bit, tho it’s already running so slow (15tok/s average for the first 32k of context) that the speed hit might make it not worth the extra smarts
jadbox•27m ago
The new IQ4XS has been working pretty well so far on 4090 16gb.
kamranjon•12m ago
What size context are you able to squeeze in with less than 2gb of headroom? I have had some luck using a quantized kv cache but i fear that also decreases overall quality.
tetsuo420•13m ago
It seems the NVFP4 quants have a preview version of this Unsloth Dynamic 3.0. Is this close to the finished version, or would it be better to switch to one of the newer quants?
xlayn•36m ago
mike-the-brain•35m ago
> We also removed the MTP module from smaller quants under UD-Q2_K_XL (8.37GB and lower) to converse around 500MB of disk space - you can use the Q4_0 MTP separate module if needed
xlayn•4m ago