xls-r-1b-bem-natbed-combined-model

This model is a fine-tuned version of facebook/wav2vec2-xls-r-1b on the NATBED - BEM dataset. It achieves the following results on the evaluation set:

  • Loss: 0.7026
  • Wer: 0.7512

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0003
  • train_batch_size: 8
  • eval_batch_size: 8
  • seed: 42
  • gradient_accumulation_steps: 2
  • total_train_batch_size: 16
  • optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 100
  • num_epochs: 30.0
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Wer
5.0296 0.2503 100 2.8071 1.0
1.5792 0.5006 200 1.1184 0.9645
1.1422 0.7509 300 1.0389 0.9606
0.9883 1.0013 400 1.0494 0.9989
0.8999 1.2516 500 0.8692 0.8683
0.9135 1.5019 600 0.8564 0.8430
0.8898 1.7522 700 0.8451 0.8522
0.9089 2.0025 800 0.8857 0.8485
0.8292 2.2528 900 0.8662 0.8580
0.7921 2.5031 1000 0.7964 0.7969
0.7983 2.7534 1100 0.7896 0.7951
0.7946 3.0038 1200 0.7667 0.7947
0.7488 3.2541 1300 0.8180 0.8495
0.7428 3.5044 1400 0.7548 0.7688
0.7256 3.7547 1500 0.7258 0.7596
0.741 4.0050 1600 0.7665 0.7718
0.6775 4.2553 1700 0.7922 0.7775
0.6795 4.5056 1800 0.7026 0.7512
0.683 4.7559 1900 0.7051 0.7225
0.6838 5.0063 2000 0.7196 0.7503
0.6005 5.2566 2100 0.7032 0.7424

Framework versions

  • Transformers 4.46.0.dev0
  • Pytorch 2.4.1+cu121
  • Datasets 3.0.1
  • Tokenizers 0.20.0
Downloads last month
23
Safetensors
Model size
963M params
Tensor type
F32
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for csikasote/xls-r-1b-bem-natbed-combined-model

Finetuned
(87)
this model