| # <a name="introduction"></a> PhoBERT: Pre-trained language models for Vietnamese | |
| Pre-trained PhoBERT models are the state-of-the-art language models for Vietnamese ([Pho](https://en.wikipedia.org/wiki/Pho), i.e. "Phở", is a popular food in Vietnam): | |
| - Two PhoBERT versions of "base" and "large" are the first public large-scale monolingual language models pre-trained for Vietnamese. PhoBERT pre-training approach is based on [RoBERTa](https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.md) which optimizes the [BERT](https://github.com/google-research/bert) pre-training procedure for more robust performance. | |
| - PhoBERT outperforms previous monolingual and multilingual approaches, obtaining new state-of-the-art performances on four downstream Vietnamese NLP tasks of Part-of-speech tagging, Dependency parsing, Named-entity recognition and Natural language inference. | |
| The general architecture and experimental results of PhoBERT can be found in our EMNLP-2020 Findings [paper](https://arxiv.org/abs/2003.00744): | |
| @article{phobert, | |
| title = {{PhoBERT: Pre-trained language models for Vietnamese}}, | |
| author = {Dat Quoc Nguyen and Anh Tuan Nguyen}, | |
| journal = {Findings of EMNLP}, | |
| year = {2020} | |
| } | |
| **Please CITE** our paper when PhoBERT is used to help produce published results or is incorporated into other software. | |
| For further information or requests, please go to [PhoBERT's homepage](https://github.com/VinAIResearch/PhoBERT)! | |