GPTQ
Is there a way to get GPTQ versions of these? GGUF never loads for me. :(
Have you tried LMStudio.ai ? Text Generation Ui? Ollma? (there are a few others too) ;
As for GPTQ , unless someone else does them after the source code is uploaded... sorry.
I can't get the software to create GPTQs working on my machine ;
Me either, i think my crappy video card aint cut out for it. :( i been using oobadooba (something like that) and it hard crashes on loading a gguf.
Both LM and OO use llamacpp ; that being said - you can load the entire GGUF into ram.
It will be painful and slow, but will work.
You can also split the load CARD/RAM too ; will be faster than CPU only.
Both LM and OO can do this.
The machine i use for OO has a pair of 3060s in it, yea its ghetto but i couldnt afford anything better at the time. 24gb of total vram but the models crash. i will probably switch it over to windows later, see if i can get LM working that way.
Oobabooga never works with GGUFs for me too.
Might have to install the Nvidia toolkit ; 12.1 / 12.2 ; it will be at Nvidia.
Another issue - you might have a conflict with more than one toolkit installed - a common Nvidia issue.