Add README.md

README.md

@@ -0,0 +1,32 @@
+---
+language: nl
+widget:
+- text: "Een zalig kerstfeest en "
+- text: "Na een lange reeks vertragingen zal eind volgende week de James Webb Space Telescope (JWST) de aarde verlaten. Met een vergulde spiegel van "
+tags:
+- adaption
+- recycled
+- gpt2-medium
+- gpt2
+pipeline_tag: text-generation
+datasets:
+- yhavinga/mc4_nl_cleaned
+---
+# GPT2-Medium pre-trained on cleaned Dutch mC4 🇳🇱
+
+Training details:
+
+* trained for 120k steps (24 dec 2021)
+* block size: 512
+* optimizer: adam, lr 8e-4, beta1 0.9, beta2 0.98
+* warmup 5000 steps
+* weight decay 0.01
+
+Work in progress. Dec 2021.
+
+* Many thanks to the [Google TPU Research Cloud](https://sites.research.google/trc/about/) for providing access to a TPU cluster!
+* Thanks to @gsarti for creating the [t5-flax-gcp
+  repository](https://github.com/gsarti/t5-flax-gcp).
+* Also thanks to the creators of [gpt2-medium-persian](https://huggingface.co/flax-community/gpt2-medium-persian) and
+  [gpt2-medium-indonesian](https://huggingface.co/flax-community/gpt2-medium-persian)
+  for sharing their training scripts!