|
--- |
|
license: apache-2.0 |
|
--- |
|
This is 2-bit quantization of [Qwen/Qwen1.5-72B-Chat](https://huggingface.co/Qwen/Qwen1.5-72B-Chat) using QuIP# |
|
|
|
Random samples from RedPajama and Skypile (for Chinese) are used as calibration data. |
|
|
|
### Model loading |
|
|
|
Please follow the instruction of [QuIP-for-all](https://github.com/chu-tianxiang/QuIP-for-all) for usage. |
|
|
|
As an alternative, you can use [Aphrodite engine](https://github.com/PygmalionAI/aphrodite-engine) or my [vLLM branch](https://github.com/chu-tianxiang/vllm-gptq) for faster inference. If you have problem installing `fast-hadamard-transform` from pip, you can also install it from [source](https://github.com/Dao-AILab/fast-hadamard-transform) |
|
|
|
|
