TheBloke

/

falcon-40b-instruct-GPTQ

like 198

Not yet - there's no GGML support yet.

As soon as there is I will upload GGMLs.

Any update on ggml versions?

No not yet I'm afraid. No-one has started work on it, to my knowledge.

You can track the discussions here: https://github.com/ggerganov/llama.cpp/issues/1602

Can't wait for GGML version of wizard-falcon 40b. This is gonna be big.

Can't wait for GGML version of wizard-falcon 40b. This is gonna be big.

Literally as the file size will be huge :)

Looks like they have got quantised GGML working for falcon in a branch: https://github.com/ggerganov/llama.cpp/issues/1602#issuecomment-1580330824

Someone posted quantized versions of the 7B Falcon model: https://huggingface.co/RachidAR/falcon-7B-ggml/tree/main