julien-c HF staff commited on
Commit
aaecbbe
1 Parent(s): ff31042

Migrate model card from transformers-repo

Browse files

Read announcement at https://discuss.huggingface.co/t/announcement-all-model-cards-will-be-migrated-to-hf-co-model-repos/2755
Original file history: https://github.com/huggingface/transformers/commits/master/model_cards/mrm8488/mobilebert-uncased-finetuned-squadv2/README.md

Files changed (1) hide show
  1. README.md +74 -0
README.md ADDED
@@ -0,0 +1,74 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ datasets:
4
+ - squad_v2
5
+ ---
6
+
7
+ # MobileBERT + SQuAD v2 📱❓
8
+
9
+ [mobilebert-uncased](https://huggingface.co/google/mobilebert-uncased) fine-tuned on [SQUAD v2.0 dataset](https://rajpurkar.github.io/SQuAD-explorer/explore/v2.0/dev/) for **Q&A** downstream task.
10
+
11
+ ## Details of the downstream task (Q&A) - Model 🧠
12
+
13
+ **MobileBERT** is a thin version of *BERT_LARGE*, while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks.
14
+
15
+ The checkpoint used here is the original MobileBert Optimized Uncased English: (uncased_L-24_H-128_B-512_A-4_F-4_OPT) checkpoint.
16
+
17
+ More about the model [here](https://arxiv.org/abs/2004.02984)
18
+
19
+ ## Details of the downstream task (Q&A) - Dataset 📚
20
+
21
+ **SQuAD2.0** combines the 100,000 questions in SQuAD1.1 with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.
22
+
23
+ ## Model training 🏋️‍
24
+
25
+ The model was trained on a Tesla P100 GPU and 25GB of RAM with the following command:
26
+
27
+ ```bash
28
+ python transformers/examples/question-answering/run_squad.py \
29
+ --model_type bert \
30
+ --model_name_or_path 'google/mobilebert-uncased' \
31
+ --do_eval \
32
+ --do_train \
33
+ --do_lower_case \
34
+ --train_file '/content/dataset/train-v2.0.json' \
35
+ --predict_file '/content/dataset/dev-v2.0.json' \
36
+ --per_gpu_train_batch_size 16 \
37
+ --learning_rate 3e-5 \
38
+ --num_train_epochs 5 \
39
+ --max_seq_length 384 \
40
+ --doc_stride 128 \
41
+ --output_dir '/content/output' \
42
+ --overwrite_output_dir \
43
+ --save_steps 1000 \
44
+ --version_2_with_negative
45
+ ```
46
+
47
+ It is important to say that this models converges much faster than other ones. So, it is also cheap to fine-tune.
48
+
49
+ ## Test set Results 🧾
50
+
51
+ | Metric | # Value |
52
+ | ------ | --------- |
53
+ | **EM** | **75.37** |
54
+ | **F1** | **78.48** |
55
+ | **Size**| **94 MB** |
56
+
57
+ ### Model in action 🚀
58
+
59
+ Fast usage with **pipelines**:
60
+
61
+ ```python
62
+ from transformers import pipeline
63
+ QnA_pipeline = pipeline('question-answering', model='mrm8488/mobilebert-uncased-finetuned-squadv2')
64
+ QnA_pipeline({
65
+ 'context': 'A new strain of flu that has the potential to become a pandemic has been identified in China by scientists.',
66
+ 'question': 'Who did identified it ?'
67
+ })
68
+
69
+ # Output: {'answer': 'scientists.', 'end': 106, 'score': 0.41531604528427124, 'start': 96}
70
+ ```
71
+
72
+ > Created by [Manuel Romero/@mrm8488](https://twitter.com/mrm8488) | [LinkedIn](https://www.linkedin.com/in/manuel-romero-cs/)
73
+
74
+ > Made with <span style="color: #e25555;">&hearts;</span> in Spain