Commit
·
e32c6f8
1
Parent(s):
bc04f3b
update readme
Browse files
README.md
CHANGED
|
@@ -30,7 +30,7 @@ Longformer models enable processing long texts using a mixture of local attentio
|
|
| 30 |
The [OSCAR](https://oscar-corpus.com) dataset is freely avaible corpus of filtered web texts from the Common Crawl in various languages. We used the 2017 version of the dataset.
|
| 31 |
|
| 32 |
## Training procedure
|
| 33 |
-
The model was trained with masked language modeling for 3 epochs on a customly created
|
| 34 |
It was validated using 5% of the original subset.
|
| 35 |
### Training hyperparameters
|
| 36 |
|
|
|
|
| 30 |
The [OSCAR](https://oscar-corpus.com) dataset is freely avaible corpus of filtered web texts from the Common Crawl in various languages. We used the 2017 version of the dataset.
|
| 31 |
|
| 32 |
## Training procedure
|
| 33 |
+
The model was trained with masked language modeling for 3 epochs on a customly created 500 million tokens subset of the german proportion of the [OSCAR](https://oscar-corpus.com) dataset.
|
| 34 |
It was validated using 5% of the original subset.
|
| 35 |
### Training hyperparameters
|
| 36 |
|