BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese

We present <PRE_TAG>BARTpho</POST_TAG> with two versions, <PRE_TAG><PRE_TAG>BARTpho</POST_TAG>-syllable</POST_TAG> and <PRE_TAG><PRE_TAG>BARTpho</POST_TAG>-word</POST_TAG>, which are the first public large-scale monolingual sequence-to-sequence models pre-trained for Vietnamese. <PRE_TAG>BARTpho</POST_TAG> uses the "large" architecture and the pre-training scheme of the sequence-to-sequence denoising autoencoder BART, thus it is especially suitable for generative NLP tasks. We conduct experiments to compare our <PRE_TAG>BARTpho</POST_TAG> with its competitor mBART on a downstream task of Vietnamese text summarization and show that: in both automatic and human evaluations, <PRE_TAG>BARTpho</POST_TAG> outperforms the strong baseline mBART and improves the state-of-the-art. We further evaluate and compare <PRE_TAG>BARTpho</POST_TAG> and mBART on the Vietnamese capitalization and punctuation restoration tasks and also find that <PRE_TAG>BARTpho</POST_TAG> is more effective than mBART on these two tasks. We publicly release <PRE_TAG>BARTpho</POST_TAG> to facilitate future research and applications of generative Vietnamese NLP tasks. Our <PRE_TAG>BARTpho</POST_TAG> models are available at https://github.com/VinAIResearch/<PRE_TAG>BARTpho</POST_TAG>