Quantized GGUF models

by MaziyarPanahi - opened Mar 7

Mar 7

Thanks for sharing this model with the community, I have quantized it in GGUF format if anyone is interested:
https://huggingface.co/MaziyarPanahi/merlinite-7b-GGUF

Mar 11

It's been quantized for a while already by our team: https://huggingface.co/ibm/merlinite-7b-GGUF

Mar 11

It's been quantized for a while already by our team: https://huggingface.co/ibm/merlinite-7b-GGUF

I did find that, but it only has Q4_K_M.gguf, I quantized it from 2 all the way to 8 bits. (for some hardware Q4 is heavy)

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment