README.md

---
language: en
license: apache-2.0
tags:
- phoneme-recognition
- generated_from_trainer
datasets:
- w11wo/ljspeech_phonemes
base_model: Wav2Vec2-Base
model-index:
- name: Wav2Vec2 LJSpeech Gruut
  results:
  - task:
      type: automatic-speech-recognition
      name: Automatic Speech Recognition
    dataset:
      name: LJSpeech
      type: ljspeech_phonemes
    metrics:
    - type: per
      value: 0.0099
      name: Test PER (w/o stress)
    - type: cer
      value: 0.0058
      name: Test CER (w/o stress)
---

# Wav2Vec2 LJSpeech Gruut

Wav2Vec2 LJSpeech Gruut is an automatic speech recognition model based on the [wav2vec 2.0](https://arxiv.org/abs/2006.11477) architecture. This model is a fine-tuned version of [Wav2Vec2-Base](https://huggingface.co/facebook/wav2vec2-base) on the [LJSpech Phonemes](https://huggingface.co/datasets/w11wo/ljspeech_phonemes) dataset.

Instead of being trained to predict sequences of words, this model was trained to predict sequence of phonemes, e.g. `["h", "ɛ", "l", "ˈoʊ", "w", "ˈɚ", "l", "d"]`. Therefore, the model's [vocabulary](https://huggingface.co/bookbot/wav2vec2-ljspeech-gruut/blob/main/vocab.json) contains the different IPA phonemes found in [gruut](https://github.com/rhasspy/gruut).

This model was trained using HuggingFace's PyTorch framework. All training was done on a Google Cloud Engine VM with a Tesla A100 GPU. All necessary scripts used for training could be found in the [Files and versions](https://huggingface.co/bookbot/wav2vec2-ljspeech-gruut/tree/main) tab, as well as the [Training metrics](https://huggingface.co/bookbot/wav2vec2-ljspeech-gruut/tensorboard) logged via Tensorboard.

## Model

| Model                     | #params | Arch.       | Training/Validation data (text) |
| ------------------------- | ------- | ----------- | ------------------------------- |
| `wav2vec2-ljspeech-gruut` | 94M     | wav2vec 2.0 | `LJSpech Phonemes` Dataset      |

## Evaluation Results

The model achieves the following results on evaluation:

| Dataset                      | PER (w/o stress) | CER (w/o stress) |
| ---------------------------- | :--------------: | :--------------: |
| `LJSpech Phonemes` Test Data |      0.99%       |      0.58%       |

## Usage

```py
from transformers import AutoProcessor, AutoModelForCTC, Wav2Vec2Processor
import librosa
import torch
from itertools import groupby
from datasets import load_dataset

def decode_phonemes(
    ids: torch.Tensor, processor: Wav2Vec2Processor, ignore_stress: bool = False
) -> str:
    """CTC-like decoding. First removes consecutive duplicates, then removes special tokens."""
    # removes consecutive duplicates
    ids = [id_ for id_, _ in groupby(ids)]

    special_token_ids = processor.tokenizer.all_special_ids + [
        processor.tokenizer.word_delimiter_token_id
    ]
    # converts id to token, skipping special tokens
    phonemes = [processor.decode(id_) for id_ in ids if id_ not in special_token_ids]

    # joins phonemes
    prediction = " ".join(phonemes)

    # whether to ignore IPA stress marks
    if ignore_stress == True:
        prediction = prediction.replace("ˈ", "").replace("ˌ", "")

    return prediction

checkpoint = "bookbot/wav2vec2-ljspeech-gruut"

model = AutoModelForCTC.from_pretrained(checkpoint)
processor = AutoProcessor.from_pretrained(checkpoint)
sr = processor.feature_extractor.sampling_rate

# load dummy dataset and read soundfiles
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")
audio_array = ds[0]["audio"]["array"]

# or, read a single audio file
# audio_array, _ = librosa.load("myaudio.wav", sr=sr)

inputs = processor(audio_array, return_tensors="pt", padding=True)

with torch.no_grad():
    logits = model(inputs["input_values"]).logits

predicted_ids = torch.argmax(logits, dim=-1)
prediction = decode_phonemes(predicted_ids[0], processor, ignore_stress=True)
# => should give 'b ɪ k ʌ z j u ɚ z s l i p ɪ ŋ ɪ n s t ɛ d ə v k ɔ ŋ k ɚ ɪ ŋ ð ə l ʌ v l i ɹ z p ɹ ɪ n s ə s h æ z b ɪ k ʌ m ə v f ɪ t ə l w ɪ θ n b oʊ p ɹ ə ʃ æ ɡ i s ɪ t s ð ɛ ɹ ə k u ɪ ŋ d ʌ v'
```

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:

- `learning_rate`: 0.0001
- `train_batch_size`: 16
- `eval_batch_size`: 8
- `seed`: 42
- `gradient_accumulation_steps`: 2
- `total_train_batch_size`: 32
- `optimizer`: Adam with `betas=(0.9,0.999)` and `epsilon=1e-08`
- `lr_scheduler_type`: linear
- `lr_scheduler_warmup_steps`: 1000
- `num_epochs`: 30.0
- `mixed_precision_training`: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss |  Wer   |  Cer   |
| :-----------: | :---: | :---: | :-------------: | :----: | :----: |
|    No log     |  1.0  |  348  |     2.2818      |  1.0   |  1.0   |
|    2.6692     |  2.0  |  696  |     0.2045      | 0.0527 | 0.0299 |
|    0.2225     |  3.0  | 1044  |     0.1162      | 0.0319 | 0.0189 |
|    0.2225     |  4.0  | 1392  |     0.0927      | 0.0235 | 0.0147 |
|    0.0868     |  5.0  | 1740  |     0.0797      | 0.0218 | 0.0143 |
|    0.0598     |  6.0  | 2088  |     0.0715      | 0.0197 | 0.0128 |
|    0.0598     |  7.0  | 2436  |     0.0652      | 0.0160 | 0.0103 |
|    0.0447     |  8.0  | 2784  |     0.0571      | 0.0152 | 0.0095 |
|    0.0368     |  9.0  | 3132  |     0.0608      | 0.0163 | 0.0112 |
|    0.0368     | 10.0  | 3480  |     0.0586      | 0.0137 | 0.0083 |
|    0.0303     | 11.0  | 3828  |     0.0641      | 0.0141 | 0.0085 |
|    0.0273     | 12.0  | 4176  |     0.0656      | 0.0131 | 0.0079 |
|    0.0232     | 13.0  | 4524  |     0.0690      | 0.0133 | 0.0082 |
|    0.0232     | 14.0  | 4872  |     0.0598      | 0.0128 | 0.0079 |
|    0.0189     | 15.0  | 5220  |     0.0671      | 0.0121 | 0.0074 |
|     0.017     | 16.0  | 5568  |     0.0654      | 0.0114 | 0.0069 |
|     0.017     | 17.0  | 5916  |     0.0751      | 0.0118 | 0.0073 |
|    0.0146     | 18.0  | 6264  |     0.0653      | 0.0112 | 0.0068 |
|    0.0127     | 19.0  | 6612  |     0.0682      | 0.0112 | 0.0069 |
|    0.0127     | 20.0  | 6960  |     0.0678      | 0.0114 | 0.0068 |
|    0.0114     | 21.0  | 7308  |     0.0656      | 0.0111 | 0.0066 |
|    0.0101     | 22.0  | 7656  |     0.0669      | 0.0109 | 0.0066 |
|    0.0092     | 23.0  | 8004  |     0.0677      | 0.0108 | 0.0065 |
|    0.0092     | 24.0  | 8352  |     0.0653      | 0.0104 | 0.0063 |
|    0.0088     | 25.0  | 8700  |     0.0673      | 0.0102 | 0.0063 |
|    0.0074     | 26.0  | 9048  |     0.0669      | 0.0105 | 0.0064 |
|    0.0074     | 27.0  | 9396  |     0.0707      | 0.0101 | 0.0061 |
|    0.0066     | 28.0  | 9744  |     0.0673      | 0.0100 | 0.0060 |
|    0.0058     | 29.0  | 10092 |     0.0689      | 0.0100 | 0.0059 |
|    0.0058     | 30.0  | 10440 |     0.0683      | 0.0099 | 0.0058 |


## Disclaimer

Do consider the biases which came from pre-training datasets that may be carried over into the results of this model.

## Authors

Wav2Vec2 LJSpeech Gruut was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Cloud.

## Framework versions

- Transformers 4.26.0.dev0
- Pytorch 1.10.0
- Datasets 2.7.1
- Tokenizers 0.13.2
- Gruut 2.3.4