Create README.md

README.md

@@ -0,0 +1,43 @@
+---
+language: ms
+---
+
+# t5-super-tiny-bahasa-cased
+
+Pretrained T5 super tiny on both standard and local language model for Malay. 
+
+## Pretraining Corpus
+
+`t5-super-tiny-bahasa-cased` model was pretrained on multiple tasks. Below is list of tasks we trained on,
+
+1. Language masking task on bahasa news, bahasa Wikipedia, bahasa Academia.edu, bahasa parliament and translated The Pile.
+2. News title prediction on bahasa news.
+3. Next sentence prediction on bahasa news, bahasa Wikipedia, bahasa Academia.edu, bahasa parliament and translated The Pile.
+4. Translated QA Natural.
+5. Text Similarity task on translated SNLI and translated MNLI.
+6. EN-MS translation.
+7. MS-EN translation.
+8. Abstractive Summarization.
+9. Knowledge Graph triples generation.
+10. Paraphrase.
+11. Social media normalization.
+12. Noisy EN-MS translation.
+13. Noisy MS-EN translation.
+
+Preparing steps can reproduce at https://github.com/huseinzol05/malaya/tree/master/pretrained-model/t5/prepare
+
+## Pretraining details
+
+- This model was trained using Google T5 repository https://github.com/google-research/text-to-text-transfer-transformer, on v3-8 TPU.
+- All steps can reproduce from here, https://github.com/huseinzol05/Malaya/tree/master/pretrained-model/t5
+
+## Supported prefix
+
+1. `soalan: {string}`, trained using Natural QA.
+2. `ringkasan: {string}`, for abstractive summarization.
+3. `tajuk: {string}`, for abstractive title.
+4. `parafrasa: {string}`, for abstractive paraphrase.
+5. `terjemah Inggeris ke Melayu: {string}`, for EN-MS translation.
+6. `terjemah Melayu ke Inggeris: {string}`, for MS-EN translation.
+7. `grafik pengetahuan: {string}`, for MS text to EN Knowledge Graph triples format.
+8. `ayat1: {string1} ayat2: {string2}`, semantic similarity.