DavidAU

/

TinyJ.O.S.I.E.-8x220M-base-Q6_K-GGUF

like 0

Hi:
Models - moe or otherwise - below 1B generally don't quant well or at all - ends up corrupted or barely usable - even at Q8.
Already tried 5 already - all were non-viable.

Models using "Imatrix" - 1B -, MAY work at IQ1_M and up (IQ1_S for some reason does not work?) ... this is model specific however.
Some models even at 34B using IQ1_S don't work...

For below "1B", best bet is to use transformers via "webui" and use the model at full precision - these do work, as I have some downloaded and working locally.

https://github.com/oobabooga/text-generation-webui
D