mradermacher

/

Virtuoso-Small-v2-i1-GGUF

like 3

Thanks for the ggufs!
I was wondering, if I can run the Q5_K_M, should I still opt for the i1 version of Q5_K_M to reduce ram usage, or would the "normal" version produce better results?

i1 offers superior quality without any additional RAM or compute resources so always go for i1 if you have a choice unless your primary are using the model for non-English tasks for which weighted quants could be worse. I know because I spent month comparing the quality of different quants. Take a look at https://huggingface.co/mradermacher/BabyHercules-4x150M-GGUF/discussions/2#674a7958ce9bc37b8e33cf55 to see the results. You really have taste with Q5_K_M. I personaly use i1-Q5_K_M for all models between 8B and 500B.

Thanks for the detailed explaination!