adding vram usage
Browse files
README.md
CHANGED
|
@@ -11,8 +11,24 @@ This is a version of the Mixtral-8x7B-Instruct-v0.1 model (https://huggingface.c
|
|
| 11 |
|
| 12 |
More specifically, the attention layers are quantized to 4-bit and the experts are quantized to 2-bit.
|
| 13 |
|
|
|
|
|
|
|
|
|
|
| 14 |
The difference between this model and https://huggingface.co/mobiuslabsgmbh/Mixtral-8x7B-Instruct-v0.1-hf-attn-4bit-moe-2bit-HQQ is that this one offloads the metadata to the CPU and you only need 13GB Vram to run it instead of 20GB!
|
| 15 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 16 |
|
| 17 |
### Basic Usage
|
| 18 |
To run the model, install the HQQ library from https://github.com/mobiusml/hqq and use it as follows:
|
|
@@ -38,16 +54,6 @@ print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
|
| 38 |
```
|
| 39 |
|
| 40 |
|
| 41 |
-
## Performance
|
| 42 |
-
| Models | Mixtral Original | HQQ quantized |
|
| 43 |
-
|-------------------|------------------|------------------|
|
| 44 |
-
| ARC (25-shot) | 70.22 | 66.47 |
|
| 45 |
-
| TruthfulQA-MC2 | 64.57 | 62.85 |
|
| 46 |
-
| Winogrande (5-shot)| 81.36 | 79.40 |
|
| 47 |
-
|
| 48 |
-
----------------------------------------------------------------------------------------------------------------------------------
|
| 49 |
-
</p>
|
| 50 |
-
|
| 51 |
### Quantization
|
| 52 |
|
| 53 |
You can reproduce the model using the following quant configs:
|
|
|
|
| 11 |
|
| 12 |
More specifically, the attention layers are quantized to 4-bit and the experts are quantized to 2-bit.
|
| 13 |
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
|
| 17 |
The difference between this model and https://huggingface.co/mobiuslabsgmbh/Mixtral-8x7B-Instruct-v0.1-hf-attn-4bit-moe-2bit-HQQ is that this one offloads the metadata to the CPU and you only need 13GB Vram to run it instead of 20GB!
|
| 18 |
|
| 19 |
+
----------------------------------------------------------------------------------------------------------------------------------
|
| 20 |
+
</p>
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
## Performance
|
| 24 |
+
| Models | Mixtral Original | HQQ quantized |
|
| 25 |
+
|-------------------|------------------|------------------|
|
| 26 |
+
| Runtime VRAM | 90 GB | <b>13 GB</b> |
|
| 27 |
+
| ARC (25-shot) | 70.22 | 66.47 |
|
| 28 |
+
| TruthfulQA-MC2 | 64.57 | 62.85 |
|
| 29 |
+
| Winogrande (5-shot)| 81.36 | 79.40 |
|
| 30 |
+
|
| 31 |
+
|
| 32 |
|
| 33 |
### Basic Usage
|
| 34 |
To run the model, install the HQQ library from https://github.com/mobiusml/hqq and use it as follows:
|
|
|
|
| 54 |
```
|
| 55 |
|
| 56 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 57 |
### Quantization
|
| 58 |
|
| 59 |
You can reproduce the model using the following quant configs:
|