Initial README. Not quite finalized yet.
Browse files
README.md
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language: en
|
| 3 |
+
tags:
|
| 4 |
+
- exbert
|
| 5 |
+
|
| 6 |
+
---
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
# GPT-2
|
| 10 |
+
|
| 11 |
+
This is a more up-to-date version of the original GPT2, which is a pretrained model on English language using a causal language modeling (CLM) objective.
|
| 12 |
+
|
| 13 |
+
## Intended uses & limitations
|
| 14 |
+
|
| 15 |
+
You can use the raw model for text generation or fine-tune it to a downstream task. See the
|
| 16 |
+
|
| 17 |
+
## How to use
|
| 18 |
+
|
| 19 |
+
You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we
|
| 20 |
+
set a seed for reproducibility:
|
| 21 |
+
|
| 22 |
+
```python
|
| 23 |
+
>>> from transformers import pipeline, set_seed
|
| 24 |
+
>>> generator = pipeline('text-generation', model='olm/olm-gpt2-oct-2022')
|
| 25 |
+
>>> set_seed(42)
|
| 26 |
+
>>> generator("Hello, I'm a language model,", max_length=30, num_return_sequences=5)
|
| 27 |
+
```
|
| 28 |
+
|
| 29 |
+
Here is how to use this model to get the features of a given text in PyTorch:
|
| 30 |
+
|
| 31 |
+
```python
|
| 32 |
+
from transformers import AutoTokenizer, AutoModelForCausalLM
|
| 33 |
+
tokenizer = AutoTokenizer.from_pretrained('olm/olm-gpt2-oct-2022')
|
| 34 |
+
model = AutoModelForCausalLM.from_pretrained('gpt2')
|
| 35 |
+
text = "Replace me by any text you'd like."
|
| 36 |
+
encoded_input = tokenizer(text, return_tensors='pt')
|
| 37 |
+
output = model(**encoded_input)
|
| 38 |
+
```
|
| 39 |
+
|
| 40 |
+
## Dataset
|
| 41 |
+
|
| 42 |
+
The model and tokenizer were trained with this [October 2022 cleaned Common Crawl dataset](https://huggingface.co/datasets/olm/olm-CC-MAIN-2022-40-sampling-ratio-0.15894621295) plus this [October 2022 cleaned Wikipedia dataset](https://huggingface.co/datasets/olm/olm-wikipedia-20221001).
|
| 43 |
+
The tokenized version of these concatenated datasets is [here](https://huggingface.co/datasets/olm/olm-october-2022-tokenized-1024).
|
| 44 |
+
The datasets were created with this [repo](https://github.com/huggingface/olm-datasets).
|
| 45 |
+
|
| 46 |
+
## Training
|
| 47 |
+
|
| 48 |
+
The model was trained according to the GPT2 instructions at this [repo](https://github.com/huggingface/olm-training).
|
| 49 |
+
|
| 50 |
+
## Evaluation results
|
| 51 |
+
|
| 52 |
+
The model achieves the following results without any fine-tuning (zero-shot):
|
| 53 |
+
|
| 54 |
+
| Task | Metric | Original GPT2 | OLM GPT2 (Ours) | Significance of difference |
|
| 55 |
+
| | | | | (two-tailed p-value) |
|
| 56 |
+
|:-----------:|:----------:|:-------------------:|:---------------------:|:--------------------------:|
|
| 57 |
+
|rte |acc |0.5307 |0.5415 |0.7188 |
|
| 58 |
+
|piqa |acc/acc_norm|0.6289/0.6251 |**0.6638**/**0.6670** |**0.0020**/**0.0002** |
|
| 59 |
+
|copa |acc |0.6400 |0.6900 |0.3000 |
|
| 60 |
+
|record |f1/em |**0.7094**/**0.7026**|0.6874/0.6810 |**0.0000**/**0.0000** |
|
| 61 |
+
|boolq |acc |0.4872 |**0.5606** |**0.0000** |
|
| 62 |
+
|cb |acc/f1 |0.4101/0.2619 |0.3571/0.1754 |0.4193/NA |
|
| 63 |
+
|hellaswag |acc/acc_norm|0.2892/0.3114 |**0.3076**/**0.3491** |**0.0000**/**0.0000** |
|
| 64 |
+
|mrpc |acc/f1 |0.5662/0.6911 |**0.6495**/**0.7741** |**0.0007**/**0.0002** |
|
| 65 |
+
|multirc |acc |0.0189 |0.0115 |0.0959 |
|
| 66 |
+
|lambada |ppl/acc |40.0554/0.3256 |**28.6733**/**0.3625** |**0.0000**/**0.0000** |
|
| 67 |
+
|wsc |acc |0.4327 |0.3654 |0.1679 |
|
| 68 |
+
|wic |acc |0.4922 |0.5 |0.6924 |
|
| 69 |
+
|mnli |acc |0.3372 |**0.3471** |**0.0384** |
|
| 70 |
+
|qnli |acc |0.5017 |0.4981 |0.5884 |
|
| 71 |
+
|cola |mcc |0.0126 |0.0181 |0.8614 |
|
| 72 |
+
|triviaqa |acc |0.0151 |**0.0182** |**0.0048** |
|
| 73 |
+
|winogrande |acc |0.5162 |0.5114 |0.7360 |
|
| 74 |
+
|webqs |acc |0.0030 |**0.0108** |**0.0000** |
|
| 75 |
+
|arc_easy |acc/acc_norm|0.4381/0.3948 |**0.4651**/**0.4247** |**0.0082**/**0.0029** |
|
| 76 |
+
|arc_challenge|acc/acc_norm|0.1903/0.2270 |0.1997/0.2329 |0.4132/0.6256 |
|
| 77 |
+
|
| 78 |
+
To get these results, we used the Eleuther AI evaluation harness [here](https://github.com/EleutherAI/lm-evaluation-harness)
|
| 79 |
+
The harness can produce results a little different than those reported in the GPT2 paper.
|
| 80 |
+
The p-values come from the stderr from the evaluation harness, plus a normal distribution assumption.
|