license: mit
base_model: microsoft/deberta-base
tags:
- generated_from_trainer
datasets:
- squad_v2
model-index:
- name: deberta-base-finetuned-squad2
results: []
language:
- en
metrics:
- exact_match
- f1
pipeline_tag: question-answering
Model description
DeBERTabase fine-tuned on SQuAD 2.0 : Encoder-based Transformer Language model.
DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder.
It outperforms BERT and RoBERTa on majority of NLU tasks with 80GB training data.
Suitable for Question-Answering tasks, predicts answer spans within the context provided.
Language model: microsoft/deberta-base
Language: English
Downstream-task: Question-Answering
Training data: Train-set SQuAD 2.0
Evaluation data: Evaluation-set SQuAD 2.0
Hardware Accelerator used: GPU Tesla T4
Intended uses & limitations
For Question-Answering -
!pip install transformers
from transformers import pipeline
model_checkpoint = "IProject-10/deberta-base-finetuned-squad2"
question_answerer = pipeline("question-answering", model=model_checkpoint)
context = """
🤗 Transformers is backed by the three most popular deep learning libraries — Jax, PyTorch and TensorFlow — with a seamless integration
between them. It's straightforward to train your models with one before loading them for inference with the other.
"""
question = "Which deep learning libraries back 🤗 Transformers?"
question_answerer(question=question, context=context)
Results
Evaluation on SQuAD 2.0 validation dataset:
exact: 81.03259496336226,
f1: 84.42279239924598,
total: 11873,
HasAns_exact: 79.30161943319838,
HasAns_f1: 86.09173653108105,
HasAns_total: 5928,
NoAns_exact: 82.75862068965517,
NoAns_f1: 82.75862068965517,
NoAns_total: 5945,
best_exact: 81.03259496336226,
best_exact_thresh: 0.9992604851722717,
best_f1: 84.42279239924635,
best_f1_thresh: 0.9992604851722717,
total_time_in_seconds: 326.41847440000004,
samples_per_second: 36.37355398411236,
latency_in_seconds: 0.027492501844521185
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 0.8054 | 1.0 | 8238 | 0.7902 |
| 0.5368 | 2.0 | 16476 | 0.7901 |
| 0.3845 | 3.0 | 24714 | 0.9334 |
This model is a fine-tuned version of microsoft/deberta-base on the squad_v2 dataset. It achieves the following results on the evaluation set:
- Loss: 0.9334
Framework versions
- Transformers 4.31.0
- Pytorch 2.0.1+cu118
- Datasets 2.14.3
- Tokenizers 0.13.3