Update README.md
Browse files
README.md
CHANGED
|
@@ -7,4 +7,9 @@ Unofficial dequantized weight of [grok-1](https://huggingface.co/xai-org/grok-1)
|
|
| 7 |
|
| 8 |
Note: If you haven't download the weight yet, please use the `fp32` revision instead which uses float32 precision for RMSNorm and Router layer for better consistency.
|
| 9 |
|
| 10 |
-
The (fp32) weights are converted using the [script here](https://gist.github.com/chu-tianxiang/ec310e15d56949fd0f351cb5f65ee7a1) ran inside the [grok-1 repo](https://github.com/xai-org/grok-1). Since downloading the dequantized weight needs twice as much time, it's recommended to download the original weight and convert on your own.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
|
| 8 |
Note: If you haven't download the weight yet, please use the `fp32` revision instead which uses float32 precision for RMSNorm and Router layer for better consistency.
|
| 9 |
|
| 10 |
+
The (fp32) weights are converted using the [script here](https://gist.github.com/chu-tianxiang/ec310e15d56949fd0f351cb5f65ee7a1) ran inside the [grok-1 repo](https://github.com/xai-org/grok-1). Since downloading the dequantized weight needs twice as much time, it's recommended to download the original weight and convert on your own.
|
| 11 |
+
|
| 12 |
+
#### Benchmarks
|
| 13 |
+
(I ran with `load_in_8bit` using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) due to limited hardware, so the result will be slightly worse)
|
| 14 |
+
* MMLU 5-shot: 0.7166
|
| 15 |
+
* BBH 3-shot: 0.5204
|