xianchaowu
commited on
Commit
·
fbd1288
1
Parent(s):
c55ede9
mmlu score
Browse files
README.md
CHANGED
@@ -8,7 +8,7 @@ license: llama2
|
|
8 |
|
9 |
0. using the updated [Meta's LLaMA-2 models](https://huggingface.co/meta-llama/Llama-2-7b-hf).
|
10 |
1. support [4-bit qlora](https://arxiv.org/abs/2305.14314), extreme GPU memory and inference time saving;
|
11 |
-
2. comparable MMLU evaluation dataset results, llama2-7b's
|
12 |
|
13 |
### Introduction
|
14 |
Determine the rank of LoRA layers by the singular values of pretrained weight matrices.
|
|
|
8 |
|
9 |
0. using the updated [Meta's LLaMA-2 models](https://huggingface.co/meta-llama/Llama-2-7b-hf).
|
10 |
1. support [4-bit qlora](https://arxiv.org/abs/2305.14314), extreme GPU memory and inference time saving;
|
11 |
+
2. comparable MMLU evaluation dataset results, llama2-7b's 45.3% to our 44.36% (-0.94%).
|
12 |
|
13 |
### Introduction
|
14 |
Determine the rank of LoRA layers by the singular values of pretrained weight matrices.
|