whisper-large-v3-ca-3catparla
Table of Contents
Click to expand
Summary
The "whisper-large-v3-ca-3catparla" is an acoustic model based on "openai/whisper-large-v3" suitable for Automatic Speech Recognition in Catalan.
Model Description
The "whisper-large-v3-ca-3catparla" is an acoustic model suitable for Automatic Speech Recognition in Catalan. It is the result of finetuning the model "openai/whisper-large-v3" with 710 hours of Catalan data released by the Projecte AINA from Barcelona, Spain.
Intended Uses and Limitations
This model can be used for Automatic Speech Recognition (ASR) in Catalan. The model is intended to transcribe audio files in Catalan to plain text without punctuation.
How to Get Started with the Model
To see an updated and functional version of this code, please see our our Notebook
Installation
In order to use this model, you may install datasets and transformers:
Create a virtual environment:
python -m venv /path/to/venv
Activate the environment:
source /path/to/venv/bin/activate
Install the modules:
pip install datasets transformers
For Inference
In order to transcribe audio in Catalan using this model, you can follow this example:
#Install Prerequisites
pip install torch
pip install datasets
pip install 'transformers[torch]'
pip install evaluate
pip install jiwer
#This code works with GPU
#Notice that: load_metric is no longer part of datasets.
#you have to remove it and use evaluate's load instead.
#(Note from November 2024)
import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor
#Load the processor and model.
MODEL_NAME="projecte-aina/whisper-large-v3-ca-3catparla"
processor = WhisperProcessor.from_pretrained(MODEL_NAME)
model = WhisperForConditionalGeneration.from_pretrained(MODEL_NAME).to("cuda")
#Load the dataset
from datasets import load_dataset, load_metric, Audio
ds=load_dataset("projecte-aina/3catparla_asr",split='test')
#Downsample to 16kHz
ds = ds.cast_column("audio", Audio(sampling_rate=16_000))
#Process the dataset
def map_to_pred(batch):
audio = batch["audio"]
input_features = processor(audio["array"], sampling_rate=audio["sampling_rate"], return_tensors="pt").input_features
batch["reference"] = processor.tokenizer._normalize(batch['normalized_text'])
with torch.no_grad():
predicted_ids = model.generate(input_features.to("cuda"))[0]
transcription = processor.decode(predicted_ids)
batch["prediction"] = processor.tokenizer._normalize(transcription)
return batch
#Do the evaluation
result = ds.map(map_to_pred)
#Compute the overall WER now.
from evaluate import load
wer = load("wer")
WER=100 * wer.compute(references=result["reference"], predictions=result["prediction"])
print(WER)
Test Result: 0.96
Training Details
Training data
The specific dataset used to create the model is called "3CatParla".
Training procedure
This model is the result of finetuning the model "openai/whisper-large-v3" by following this tutorial provided by Hugging Face.
Training Hyperparameters
- language: catalan
- hours of training audio: 710
- learning rate: 1.95e-07
- sample rate: 16000
- train batch size: 32 (x4 GPUs)
- gradient accumulation steps: 1
- eval batch size: 32
- save total limit: 3
- max steps: 19842
- warmup steps: 1984
- eval steps: 3307
- save steps: 3307
- shuffle buffer size: 480
Citation
If this model contributes to your research, please cite the work:
@misc{mena2024whisperlarge3catparla,
title={Acoustic Model in Catalan: whisper-large-v3-ca-3catparla.},
author={Hernandez Mena, Carlos Daniel; Armentano-Oller, Carme; Solito, Sarah; Külebi, Baybars},
organization={Barcelona Supercomputing Center},
url={https://huggingface.co/projecte-aina/whisper-large-v3-ca-3catparla},
year={2024}
}
Additional Information
Author
The fine-tuning process was perform during July (2024) in the Language Technologies Unit of the Barcelona Supercomputing Center by Carlos Daniel Hernández Mena.
Contact
For further information, please send an email to [email protected].
Copyright
Copyright(c) 2024 by Language Technologies Unit, Barcelona Supercomputing Center.
License
Funding
This work has been promoted and financed by the Generalitat de Catalunya through the Aina project.
The training of the model was possible thanks to the compute time provided by Barcelona Supercomputing Center through MareNostrum 5.
- Downloads last month
- 47
Collection including projecte-aina/whisper-large-v3-ca-3catparla
Evaluation results
- WER on 3CatParla (Test)test set self-reported0.960
- WER on 3CatParla (Dev)self-reported0.920
- WER on Mozilla Common Voice 17.0 (Test)test set self-reported10.320
- WER on Mozilla Common Voice 17.0 (Dev)validation set self-reported9.260
- WER on CV Benchmark Catalan Accents (Balearic fem)self-reported12.250
- WER on CV Benchmark Catalan Accents (Balearic male)self-reported12.180
- WER on CV Benchmark Catalan Accents (Central fem)self-reported8.510
- WER on CV Benchmark Catalan Accents (Central male)self-reported8.730
- WER on CV Benchmark Catalan Accents (Northern fem)self-reported8.090
- WER on CV Benchmark Catalan Accents (Northern male)self-reported8.280