Update README.md
Browse files
README.md
CHANGED
|
@@ -10,12 +10,9 @@ datasets:
|
|
| 10 |
pipeline_tag: fill-mask
|
| 11 |
---
|
| 12 |
|
| 13 |
-
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
|
| 14 |
-
should probably proofread and complete it, then remove this comment. -->
|
| 15 |
-
|
| 16 |
# mega-small-2048 on simple wikipedia
|
| 17 |
|
| 18 |
-
MEGA for masked LM 'small' (12 layers, 512 hidden size, 2048 ctx in chunks of 1024) on the `pszemraj/simple_wikipedia_LM` dataset.
|
| 19 |
It achieves the following results on the evaluation set:
|
| 20 |
- Loss: 3.4773
|
| 21 |
- Accuracy: 0.4591
|
|
|
|
| 10 |
pipeline_tag: fill-mask
|
| 11 |
---
|
| 12 |
|
|
|
|
|
|
|
|
|
|
| 13 |
# mega-small-2048 on simple wikipedia
|
| 14 |
|
| 15 |
+
[MEGA](https://arxiv.org/abs/2209.10655) for masked LM 'small' (12 layers, 512 hidden size, 2048 ctx in chunks of 1024) on the `pszemraj/simple_wikipedia_LM` dataset.
|
| 16 |
It achieves the following results on the evaluation set:
|
| 17 |
- Loss: 3.4773
|
| 18 |
- Accuracy: 0.4591
|