fp.

Seems pretty useful, often hopping between different quants. I wonder if this would work for different "branches" of a model. E.g. qwen 3.6 35b regular vs abliterated.

alessandroberna•13m ago

Seems like it wouldn't work for different branches.

It probably needs to store the full F16 version for each model variant, even if the variants share the same base model

VASTL•31m ago

No offense, but do you really you "cracked the code" if multi-billion/trillion companies cant achieve that? I heavily doubt, you have losless quality with those 25 commits on GH, most likely vibecoded.

adrian17•21m ago

The catch is that it's not really doing anything. It "stores" 16GB of quants by... storing the base 2GB model and quantizing it on user's machine on demand.

The extra diff files are only there because the quantization is not fully reproducible, and author's 3-line llama.cpp fix PR* supposedly fixes this. With it fixed, then in my understanding the author's tool is literally just a wrapper around llama-quantize.

* I'd give the PR like 50% chances of not surviving the day, since AFAIK on llama.cpp, vibed PR descriptions usually end up rejected as a rule. Maybe the diff being super trivial could save it from this fate, dunno.

alessandroberna•16m ago

It's just a wrapper around llama-quantize, i don't see why it wouldn't work.

Yet it's only useful if one wants to have both the F16 original model plus all the other GGUF quants at a time and doesn't mind spending compute every time they need a specific quant again.

It doesn't do much for a consumer who only needs one or a few specific quants that work on their hardware and I'd wager it's not something that multi billion dollar companies need. At least as far as I'm aware major llm providers aren't using gguf and llama.cpp to serve their inference