Update README.md
Browse files
README.md
CHANGED
|
@@ -65,34 +65,34 @@ The datasets were created with this [repo](https://github.com/huggingface/olm-da
|
|
| 65 |
|
| 66 |
## Training
|
| 67 |
|
| 68 |
-
The model was trained according to the GPT2 instructions at this [repo](https://github.com/huggingface/olm-training).
|
| 69 |
|
| 70 |
## Evaluation results
|
| 71 |
|
| 72 |
The model achieves the following results without any fine-tuning (zero-shot):
|
| 73 |
|
| 74 |
-
| Task | Metric | Original GPT2 | OLM GPT2 (Ours)
|
| 75 |
-
|
| 76 |
-
|rte |acc |0.5307 |0.5415
|
| 77 |
-
|piqa |acc/acc_norm|0.6289/0.6251 |**0.6638**/**0.6670**
|
| 78 |
-
|copa |acc |0.6400 |0.6900
|
| 79 |
-
|record |f1/em |**0.7094**/**0.7026**|0.6874/0.6810
|
| 80 |
-
|boolq |acc |0.4872 |**0.5606**
|
| 81 |
-
|cb |acc/f1 |0.4101/0.2619 |0.3571/0.1754
|
| 82 |
-
|hellaswag |acc/acc_norm|0.2892/0.3114 |**0.3076**/**0.3491**
|
| 83 |
-
|mrpc |acc/f1 |0.5662/0.6911 |**0.6495**/**0.7741**
|
| 84 |
-
|multirc |acc |0.0189 |0.0115
|
| 85 |
-
|lambada |ppl/acc |40.0554/0.3256 |**28.6733**/**0.3625**
|
| 86 |
-
|wsc |acc |0.4327 |0.3654
|
| 87 |
-
|wic |acc |0.4922 |0.5
|
| 88 |
-
|mnli |acc |0.3372 |**0.3471**
|
| 89 |
-
|qnli |acc |0.5017 |0.4981
|
| 90 |
-
|cola |mcc |0.0126 |0.0181
|
| 91 |
-
|triviaqa |acc |0.0151 |**0.0182**
|
| 92 |
-
|winogrande |acc |0.5162 |0.5114
|
| 93 |
-
|webqs |acc |0.0030 |**0.0108**
|
| 94 |
-
|arc_easy |acc/acc_norm|0.4381/0.3948 |**0.4651**/**0.4247**
|
| 95 |
-
|arc_challenge|acc/acc_norm|0.1903/0.2270 |0.1997/0.2329
|
| 96 |
|
| 97 |
To get these results, we used the Eleuther AI evaluation harness [here](https://github.com/EleutherAI/lm-evaluation-harness).\
|
| 98 |
The harness can produce results a little different than those reported in the GPT2 paper.\
|
|
|
|
| 65 |
|
| 66 |
## Training
|
| 67 |
|
| 68 |
+
The model was trained according to the OLM GPT2 instructions at this [repo](https://github.com/huggingface/olm-training).
|
| 69 |
|
| 70 |
## Evaluation results
|
| 71 |
|
| 72 |
The model achieves the following results without any fine-tuning (zero-shot):
|
| 73 |
|
| 74 |
+
| Task | Metric | Original GPT2 | OLM GPT2 Oct 2022 (Ours) | Significance (two-tailed p-value) |
|
| 75 |
+
|:------------|:-----------|--------------------:|-------------------------:|----------------------------------:|
|
| 76 |
+
|rte |acc |0.5307 |0.5415 |0.7188 |
|
| 77 |
+
|piqa |acc/acc_norm|0.6289/0.6251 |**0.6638**/**0.6670** |**0.0020**/**0.0002** |
|
| 78 |
+
|copa |acc |0.6400 |0.6900 |0.3000 |
|
| 79 |
+
|record |f1/em |**0.7094**/**0.7026**|0.6874/0.6810 |**0.0000**/**0.0000** |
|
| 80 |
+
|boolq |acc |0.4872 |**0.5606** |**0.0000** |
|
| 81 |
+
|cb |acc/f1 |0.4101/0.2619 |0.3571/0.1754 |0.4193/NA |
|
| 82 |
+
|hellaswag |acc/acc_norm|0.2892/0.3114 |**0.3076**/**0.3491** |**0.0000**/**0.0000** |
|
| 83 |
+
|mrpc |acc/f1 |0.5662/0.6911 |**0.6495**/**0.7741** |**0.0007**/**0.0002** |
|
| 84 |
+
|multirc |acc |0.0189 |0.0115 |0.0959 |
|
| 85 |
+
|lambada |ppl/acc |40.0554/0.3256 |**28.6733**/**0.3625** |**0.0000**/**0.0000** |
|
| 86 |
+
|wsc |acc |0.4327 |0.3654 |0.1679 |
|
| 87 |
+
|wic |acc |0.4922 |0.5 |0.6924 |
|
| 88 |
+
|mnli |acc |0.3372 |**0.3471** |**0.0384** |
|
| 89 |
+
|qnli |acc |0.5017 |0.4981 |0.5884 |
|
| 90 |
+
|cola |mcc |0.0126 |0.0181 |0.8614 |
|
| 91 |
+
|triviaqa |acc |0.0151 |**0.0182** |**0.0048** |
|
| 92 |
+
|winogrande |acc |0.5162 |0.5114 |0.7360 |
|
| 93 |
+
|webqs |acc |0.0030 |**0.0108** |**0.0000** |
|
| 94 |
+
|arc_easy |acc/acc_norm|0.4381/0.3948 |**0.4651**/**0.4247** |**0.0082**/**0.0029** |
|
| 95 |
+
|arc_challenge|acc/acc_norm|0.1903/0.2270 |0.1997/0.2329 |0.4132/0.6256 |
|
| 96 |
|
| 97 |
To get these results, we used the Eleuther AI evaluation harness [here](https://github.com/EleutherAI/lm-evaluation-harness).\
|
| 98 |
The harness can produce results a little different than those reported in the GPT2 paper.\
|