Wav2Vec2-XLSR-300m-es
This model is a fine-tuned version of facebook/wav2vec2-xls-r-300m on the spanish common_voice dataset thanks to the GPU credits generously given by the OVHcloud for the Speech Recognition challenge. It achieves the following results on the evaluation set
Without LM:
- Loss : 0.1900
- Wer : 0.146
With 5-gram:
- WER: 0.109
- CER: 0.036
Usage with 5-gram.
The model can be used with n-gram (n=5) included in the processor as follows.
import re
from transformers import AutoModelForCTC,Wav2Vec2ProcessorWithLM
import torch
# Loading model and processor
processor = Wav2Vec2ProcessorWithLM.from_pretrained("polodealvarado/xls-r-300m-es")
model = AutoModelForCTC.from_pretrained("polodealvarado/xls-r-300m-es")
# Cleaning characters
def remove_extra_chars(batch):
chars_to_ignore_regex = '[^a-záéíóúñ ]'
text = batch["translation"][target_lang]
batch["text"] = re.sub(chars_to_ignore_regex, "", text.lower())
return batch
# Preparing dataset
def prepare_dataset(batch):
audio = batch["audio"]
batch["input_values"] = processor(audio["array"], sampling_rate=audio["sampling_rate"],return_tensors="pt",padding=True).input_values[0]
with processor.as_target_processor():
batch["labels"] = processor(batch["sentence"]).input_ids
return batch
common_voice_test = load_dataset("mozilla-foundation/common_voice_8_0", "es", split="test",use_auth_token=True)
common_voice_test = common_voice_test.remove_columns(["accent", "age", "client_id", "down_votes", "gender", "locale", "segment", "up_votes"])
common_voice_test = common_voice_test.cast_column("audio", Audio(sampling_rate=16_000))
common_voice_test = common_voice_test.map(remove_extra_chars, remove_columns=dataset.column_names)
common_voice_test = common_voice_test.map(prepare_dataset)
# Testing first sample
inputs = torch_tensor(common_voice_test[0]["input_values"])
with torch.no_grad():
logits = model(inputs).logits
pred_ids = torch.argmax(logits, dim=-1)
text = processor.batch_decode(logits.numpy()).text
print(text) # 'bien y qué regalo vas a abrir primero'
On the other, you can execute the eval.py file for evaluation
# To use GPU: --device 0
$ python eval.py --model_id polodealvarado/xls-r-300m-es --dataset mozilla-foundation/common_voice_8_0 --config es --device 0 --split test
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0003
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 4
- mixed_precision_training: Native AMP
Training results
Training Loss | Epoch | Step | Validation Loss | Wer |
---|---|---|---|---|
3.6747 | 0.3 | 400 | 0.6535 | 0.5926 |
0.4439 | 0.6 | 800 | 0.3753 | 0.3193 |
0.3291 | 0.9 | 1200 | 0.3267 | 0.2721 |
0.2644 | 1.2 | 1600 | 0.2816 | 0.2311 |
0.24 | 1.5 | 2000 | 0.2647 | 0.2179 |
0.2265 | 1.79 | 2400 | 0.2406 | 0.2048 |
0.1994 | 2.09 | 2800 | 0.2357 | 0.1869 |
0.1613 | 2.39 | 3200 | 0.2242 | 0.1821 |
0.1546 | 2.69 | 3600 | 0.2123 | 0.1707 |
0.1441 | 2.99 | 4000 | 0.2067 | 0.1619 |
0.1138 | 3.29 | 4400 | 0.2044 | 0.1519 |
0.1072 | 3.59 | 4800 | 0.1917 | 0.1457 |
0.0992 | 3.89 | 5200 | 0.1900 | 0.1438 |
Framework versions
- Transformers 4.16.0.dev0
- Pytorch 1.10.1+cu102
- Datasets 1.17.1.dev0
- Tokenizers 0.11.0
- Downloads last month
- 103
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.
Dataset used to train polodealvarado/xls-r-300m-es
Evaluation results
- Test WER on mozilla-foundation/common_voice_8_0 esself-reported14.600
- Test WER on Robust Speech Event - Dev Dataself-reported28.630
- Test WER on Robust Speech Event - Test Dataself-reported29.720