README.md

4bit AWQ Quantized Version of [parlance-labs/hc-mistral-alpaca-merged](https://huggingface.co/parlance-labs/hc-mistral-alpaca-merged)

This is how to use [AutoAWQ](https://github.com/casper-hansen/AutoAWQ/tree/main) to quantize the model.

```python
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

# setup
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
quant_path="hc-mistral-alpaca-merged-awq"
model_path="parlance-labs/hc-mistral-alpaca-merged"
model = AutoAWQForCausalLM.from_pretrained(model_path, **{"low_cpu_mem_usage": True})
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

# quantize and save model
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
```

After you save the model you can upload it to the hub

```bash
cd hc-mistral-alpaca-merged-awq
huggingface-cli upload parlance-labs/hc-mistral-alpaca-merged-awq .
```