Upload 6 files

README.md

@@ -0,0 +1,98 @@
+---
+language: 
+- vi
+- vn
+- en
+tags:
+- question-answering
+- pytorch
+datasets:
+- squad
+license: cc-by-nc-4.0
+pipeline_tag: question-answering 
+metrics:
+- squad
+widget:
+- text: "Bình là chuyên gia về gì ?"
+  context: "Bình Nguyễn là một người đam mê với lĩnh vực xử lý ngôn ngữ tự nhiên . Anh nhận chứng chỉ Google Developer Expert năm 2020"
+- text: "Bình được công nhận với danh hiệu gì ?"
+  context: "Bình Nguyễn là một người đam mê với lĩnh vực xử lý ngôn ngữ tự nhiên . Anh nhận chứng chỉ Google Developer Expert năm 2020"
+---
+## Model Description
+
+- Language model: [XLM-RoBERTa](https://huggingface.co/transformers/model_doc/xlmroberta.html)
+- Fine-tune: [MRCQuestionAnswering](https://github.com/nguyenvulebinh/extractive-qa-mrc)
+- Language: Vietnamese, Englsih
+- Downstream-task: Extractive QA
+- Dataset (combine English and Vietnamese):
+  - [Squad 2.0](https://rajpurkar.github.io/SQuAD-explorer/) 
+  - [mailong25](https://github.com/mailong25/bert-vietnamese-question-answering/tree/master/dataset)
+  - [VLSP MRC 2021](https://vlsp.org.vn/vlsp2021/eval/mrc)
+  - [MultiLingual Question Answering](https://github.com/facebookresearch/MLQA)
+  
+This model is intended to be used for QA in the Vietnamese language so the valid set is Vietnamese only (but English works fine). The evaluation result below uses the VLSP MRC 2021 test set. This experiment achieves TOP 1 on the leaderboard.
+
+
+| Model  | EM | F1 |
+| ------------- | ------------- | ------------- |
+| [large](https://huggingface.co/nguyenvulebinh/vi-mrc-large)  public_test_set  | 85.847  | 83.826  |
+| [large](https://huggingface.co/nguyenvulebinh/vi-mrc-large)  private_test_set  | 82.072  | 78.071  |
+
+Public leaderboard             |  Private leaderboard
+:-------------------------:|:-------------------------:
+![](https://i.ibb.co/tJX6V6T/public-leaderboard.jpg)  |  ![](https://i.ibb.co/nmsX2pG/private-leaderboard.jpg)
+
+[MRCQuestionAnswering](https://github.com/nguyenvulebinh/extractive-qa-mrc) using [XLM-RoBERTa](https://huggingface.co/transformers/model_doc/xlmroberta.html) as a pre-trained language model. By default, XLM-RoBERTa will split word in to sub-words. But in my implementation, I re-combine sub-words representation (after encoded by BERT layer) into word representation using sum strategy.
+
+## Using pre-trained model
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1Yqgdfaca7L94OyQVnq5iQq8wRTFvVZjv?usp=sharing)
+
+- Hugging Face pipeline style (**NOT using sum features strategy**).
+
+```python
+from transformers import pipeline
+# model_checkpoint = "nguyenvulebinh/vi-mrc-large"
+model_checkpoint = "nguyenvulebinh/vi-mrc-base"
+nlp = pipeline('question-answering', model=model_checkpoint,
+                   tokenizer=model_checkpoint)
+QA_input = {
+  'question': "Bình là chuyên gia về gì ?",
+  'context': "Bình Nguyễn là một người đam mê với lĩnh vực xử lý ngôn ngữ tự nhiên . Anh nhận chứng chỉ Google Developer Expert năm 2020"
+}
+res = nlp(QA_input)
+print('pipeline: {}'.format(res))
+#{'score': 0.5782045125961304, 'start': 45, 'end': 68, 'answer': 'xử lý ngôn ngữ tự nhiên'}
+```
+
+- More accurate infer process ([**Using sum features strategy**](https://github.com/nguyenvulebinh/extractive-qa-mrc))
+
+```python
+from infer import tokenize_function, data_collator, extract_answer
+from model.mrc_model import MRCQuestionAnswering
+from transformers import AutoTokenizer
+
+model_checkpoint = "nguyenvulebinh/vi-mrc-large"
+#model_checkpoint = "nguyenvulebinh/vi-mrc-base"
+tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)
+model = MRCQuestionAnswering.from_pretrained(model_checkpoint)
+
+QA_input = {
+  'question': "Bình được công nhận với danh hiệu gì ?",
+  'context': "Bình Nguyễn là một người đam mê với lĩnh vực xử lý ngôn ngữ tự nhiên . Anh nhận chứng chỉ Google Developer Expert năm 2020"
+}
+
+inputs = [tokenize_function(*QA_input)]
+inputs_ids = data_collator(inputs)
+outputs = model(**inputs_ids)
+answer = extract_answer(inputs, outputs, tokenizer)
+
+print(answer)
+# answer: Google Developer Expert. Score start: 0.9926977753639221, Score end: 0.9909810423851013
+```
+
+## About
+
+*Built by Binh Nguyen*
+[![Follow](https://img.shields.io/twitter/follow/nguyenvulebinh?style=social)](https://twitter.com/intent/follow?screen_name=nguyenvulebinh)
+For more details, visit the project repository.
+[![GitHub stars](https://img.shields.io/github/stars/nguyenvulebinh/extractive-qa-mrc?style=social)](https://github.com/nguyenvulebinh/extractive-qa-mrc)

config.json

@@ -0,0 +1,27 @@
+{
+  "_name_or_path": "xlm-roberta-large",
+  "architectures": [
+    "XLMRobertaForQuestionAnswering"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "eos_token_id": 2,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "roberta",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "output_past": true,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "transformers_version": "4.8.2",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 250002
+}

gitattributes.txt

@@ -0,0 +1,17 @@
+*.bin.* filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tar.gz filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": "<s>", "eos_token": "</s>", "unk_token": "<unk>", "sep_token": "</s>", "pad_token": "<pad>", "cls_token": "<s>", "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": false}}

tokenizer_config.json

@@ -0,0 +1 @@
+{"bos_token": "<s>", "eos_token": "</s>", "sep_token": "</s>", "cls_token": "<s>", "unk_token": "<unk>", "pad_token": "<pad>", "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "model_max_length": 512, "special_tokens_map_file": null, "name_or_path": "xlm-roberta-base", "tokenizer_class": "XLMRobertaTokenizer"}