Jok Edul's picture

1 1 1

Jok Edul

JokEdul

AI & ML interests

None yet

Recent Activity

upvoted a collection 26 days ago

new activity 28 days ago

wirthual/Meta-Llama-3.2-1B-Instruct-llamafile:🚩 Report: Spam

replied to mlabonne's post about 1 month ago

⚡ AutoQuant AutoQuant is the evolution of my previous AutoGGUF notebook (https://colab.research.google.com/drive/1P646NEg33BZy4BfLDNpTz0V0lwIU3CHu). It allows you to quantize your models in five different formats: - GGUF: perfect for inference on CPUs (and LM Studio) - GPTQ/EXL2: fast inference on GPUs - AWQ: super fast inference on GPUs with vLLM (https://github.com/vllm-project/vllm) - HQQ: extreme quantization with decent 2-bit and 3-bit models Once the model is converted, it automatically uploads it on the Hugging Face Hub. To quantize a 7B model, GGUF only needs a T4 GPU, while the other methods require an A100 GPU. Here's an example of a model I quantized using HQQ and AutoQuant: https://huggingface.co/mlabonne/AlphaMonarch-7B-2bit-HQQ I hope you'll enjoy it and quantize lots of models! :) 💻 AutoQuant: https://colab.research.google.com/drive/1b6nqC7UZVt8bx4MksX7s656GXPM-eWw4

View all activity

Organizations

None yet

JokEdul's activity

upvoted a collection 26 days ago

MobileLLM

Optimizing Sub-billion Parameter Language Models for On-Device Use Cases (ICML 2024) https://arxiv.org/abs/2402.14905 • 9 items • Updated Nov 27, 2024 • 103

New activity in wirthual/Meta-Llama-3.2-1B-Instruct-llamafile 28 days ago

🚩 Report: Spam

#1 opened 28 days ago by

replied to mlabonne's post about 1 month ago

Bro i was searching for days for something like this, god damn you are life saver

liked a Space about 1 month ago

GGUF My Repo