Kobold can run IQ2
in its experimental version.
I made 3 compatible releases there, the latest being : https://github.com/Nexesenex/kobold.cpp/releases/tag/v1.55.1_b1842
Btw, what iMatrix file do you use? Wikitext?
Thanks, I had a feeling it did but i did not manage to get it to compile with cublas until just now.
I got it running on ooba with the llamacpp_0.2.29 branch but Kobold is probably preferable.
I ran this one on pippa since i had it available, not sure what is best or how much difference it makes and since each run takes a few hours I went with wikitext for
WinterGoddess and Aurora-Nights (still running atm) to be safe.
That discussion might be of interest for you, and it has links towards the relevant discussions on LlamaCPP's Github.
https://huggingface.co/grimulkan/aurelian-alpha0.1-70b-rope8-32K-fp16/discussions/2