I think the difficult aspect is that you have to re-train the model after conversion.
I assume you could use this during training (QAT) similar to ternary quantization.
It’s a bit beyond me, but it seems promising–at least for general knowledge models.
stev1983•3m ago
It’s a bit beyond me, but it seems promising–at least for general knowledge models.