Hugging Face's logo

olm
/
olm-gpt2-oct-2022

Text Generation
Transformers
PyTorch
TensorFlow
TensorBoard
English
gpt2
exbert
text-generation-inference
Inference Endpoints
Model card Files Metrics Community
4
Tristan commited on
Commit
cfcb367
·
1 Parent(s): 60a5822

fix formatting in table

Browse files
Files changed (1) hide show
  1. README.md +22 -23
README.md CHANGED
@@ -51,29 +51,28 @@ The model was trained according to the GPT2 instructions at this [repo](https://
51
 
52
  The model achieves the following results without any fine-tuning (zero-shot):
53
 
54
- | Task | Metric | Original GPT2 | OLM GPT2 (Ours) | Significance of difference |
55
- | | | | | (two-tailed p-value) |
56
- |:-----------:|:----------:|:-------------------:|:---------------------:|:--------------------------:|
57
- |rte |acc |0.5307 |0.5415 |0.7188 |
58
- |piqa |acc/acc_norm|0.6289/0.6251 |**0.6638**/**0.6670** |**0.0020**/**0.0002** |
59
- |copa |acc |0.6400 |0.6900 |0.3000 |
60
- |record |f1/em |**0.7094**/**0.7026**|0.6874/0.6810 |**0.0000**/**0.0000** |
61
- |boolq |acc |0.4872 |**0.5606** |**0.0000** |
62
- |cb |acc/f1 |0.4101/0.2619 |0.3571/0.1754 |0.4193/NA |
63
- |hellaswag |acc/acc_norm|0.2892/0.3114 |**0.3076**/**0.3491** |**0.0000**/**0.0000** |
64
- |mrpc |acc/f1 |0.5662/0.6911 |**0.6495**/**0.7741** |**0.0007**/**0.0002** |
65
- |multirc |acc |0.0189 |0.0115 |0.0959 |
66
- |lambada |ppl/acc |40.0554/0.3256 |**28.6733**/**0.3625** |**0.0000**/**0.0000** |
67
- |wsc |acc |0.4327 |0.3654 |0.1679 |
68
- |wic |acc |0.4922 |0.5 |0.6924 |
69
- |mnli |acc |0.3372 |**0.3471** |**0.0384** |
70
- |qnli |acc |0.5017 |0.4981 |0.5884 |
71
- |cola |mcc |0.0126 |0.0181 |0.8614 |
72
- |triviaqa |acc |0.0151 |**0.0182** |**0.0048** |
73
- |winogrande |acc |0.5162 |0.5114 |0.7360 |
74
- |webqs |acc |0.0030 |**0.0108** |**0.0000** |
75
- |arc_easy |acc/acc_norm|0.4381/0.3948 |**0.4651**/**0.4247** |**0.0082**/**0.0029** |
76
- |arc_challenge|acc/acc_norm|0.1903/0.2270 |0.1997/0.2329 |0.4132/0.6256 |
77
 
78
  To get these results, we used the Eleuther AI evaluation harness [here](https://github.com/EleutherAI/lm-evaluation-harness)
79
  The harness can produce results a little different than those reported in the GPT2 paper.
 
51
 
52
  The model achieves the following results without any fine-tuning (zero-shot):
53
 
54
+ | Task | Metric | Original GPT2 | OLM GPT2 (Ours) | Significance (two-tailed p-value) |
55
+ |:------------|:-----------|--------------------:|----------------------:|----------------------------------:|
56
+ |rte |acc |0.5307 |0.5415 |0.7188 |
57
+ |piqa |acc/acc_norm|0.6289/0.6251 |**0.6638**/**0.6670** |**0.0020**/**0.0002** |
58
+ |copa |acc |0.6400 |0.6900 |0.3000 |
59
+ |record |f1/em |**0.7094**/**0.7026**|0.6874/0.6810 |**0.0000**/**0.0000** |
60
+ |boolq |acc |0.4872 |**0.5606** |**0.0000** |
61
+ |cb |acc/f1 |0.4101/0.2619 |0.3571/0.1754 |0.4193/NA |
62
+ |hellaswag |acc/acc_norm|0.2892/0.3114 |**0.3076**/**0.3491** |**0.0000**/**0.0000** |
63
+ |mrpc |acc/f1 |0.5662/0.6911 |**0.6495**/**0.7741** |**0.0007**/**0.0002** |
64
+ |multirc |acc |0.0189 |0.0115 |0.0959 |
65
+ |lambada |ppl/acc |40.0554/0.3256 |**28.6733**/**0.3625** |**0.0000**/**0.0000** |
66
+ |wsc |acc |0.4327 |0.3654 |0.1679 |
67
+ |wic |acc |0.4922 |0.5 |0.6924 |
68
+ |mnli |acc |0.3372 |**0.3471** |**0.0384** |
69
+ |qnli |acc |0.5017 |0.4981 |0.5884 |
70
+ |cola |mcc |0.0126 |0.0181 |0.8614 |
71
+ |triviaqa |acc |0.0151 |**0.0182** |**0.0048** |
72
+ |winogrande |acc |0.5162 |0.5114 |0.7360 |
73
+ |webqs |acc |0.0030 |**0.0108** |**0.0000** |
74
+ |arc_easy |acc/acc_norm|0.4381/0.3948 |**0.4651**/**0.4247** |**0.0082**/**0.0029** |
75
+ |arc_challenge|acc/acc_norm|0.1903/0.2270 |0.1997/0.2329 |0.4132/0.6256 |
 
76
 
77
  To get these results, we used the Eleuther AI evaluation harness [here](https://github.com/EleutherAI/lm-evaluation-harness)
78
  The harness can produce results a little different than those reported in the GPT2 paper.