File size: 699 Bytes
4909439 1f201b2 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
---
license: apache-2.0
---
This is 2-bit quantization of [Qwen/Qwen1.5-72B-Chat](https://huggingface.co/Qwen/Qwen1.5-72B-Chat) using QuIP#
Random samples from RedPajama and Skypile (for Chinese) are used as calibration data.
### Model loading
Please follow the instruction of [QuIP-for-all](https://github.com/chu-tianxiang/QuIP-for-all) for usage.
As an alternative, you can use [Aphrodite engine](https://github.com/PygmalionAI/aphrodite-engine) or my [vLLM branch](https://github.com/chu-tianxiang/vllm-gptq) for faster inference. If you have problem installing `fast-hadamard-transform` from pip, you can also install it from [source](https://github.com/Dao-AILab/fast-hadamard-transform)
|