File size: 5,413 Bytes
62a6c00 74b9019 62a6c00 74b9019 62a6c00 74b9019 c6fe243 74b9019 4cad71c 74b9019 4cad71c 74b9019 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 |
---
datasets:
- AudioSet
- AudioCaps
- Clotho-v2.1
metrics:
- SPICE
- CIDEr
- SPIDEr
- METEOR
- SacreBLEU
model-index:
- name: whisper-large-v2-audio-captioning
results:
- task:
type: audio-captioning
name: Audio Captioning
dataset:
type: clotho-v2.1
name: Clotho
split: evaluation
metrics:
- type: SPICE
value: 0.1257
- type: CIDEr
value: 0.4331
- type: SPIDEr
value: 0.2794
- type: METEOR
value: 0.3782
- type: SacreBLEU
value: 16.50
license: cc-by-nc-4.0
language:
- en
---
# Model Card for Whisper Audio Captioning
A transformer encoder-decoder model for automatic audio captioning. As opposed to speech-to-text, captioning describes the content of audio clips, such as prominent sounds or environmental noises. This task has numerous practical applications, e.g., for providing access to audio information for people with hearing impairments or improving the searchability of audio content.
- **Model type:** Whisper encoder-decoder transformer
- **Language(s) (NLP):** en
- **License:** cc-by-4.0
- **Parent Model:** openai/whisper-large-v2
- **Resources for more information:**
- [GitHub Repo](https://github.com/prompteus/audio-captioning)
- [Technical Report](TODO)
## Usage
The model expects an audio clip (up to 30s) to the encoder as an input and information about caption style as forced prefix to the decoder.
Minimal example:
```python
# Load model
checkpoint = "MU-NLPC/whisper-large-v2-audio-captioning"
model = WhisperForAudioCaptioning.from_pretrained(checkpoint)
tokenizer = transformers.WhisperTokenizer.from_pretrained(checkpoint, language="en", task="transcribe")
feature_extractor = transformers.WhisperFeatureExtractor.from_pretrained(checkpoint)
# Load and preprocess audio
input_file = "..."
audio, sampling_rate = librosa.load(input_file, sr=feature_extractor.sampling_rate)
features = feature_extractor(audio, sampling_rate=sampling_rate, return_tensors="pt").input_features
# Prepare caption style
style_prefix = "clotho > caption: "
style_prefix_tokens = tokenizer("", text_target=style_prefix, return_tensors="pt", add_special_tokens=False).labels
# Generate caption
model.eval()
outputs = model.generate(
inputs=features.to(model.device),
forced_ac_decoder_ids=style_prefix_tokens,
max_length=100,
)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True)[0])
```
Example output:
*clotho > caption: Rain is pouring down and thunder is rumbling in the background.*
The style prefix influences the style of the caption. Model knows 3 styles: `audioset > keywords: `, `audiocaps > caption: `, and `clotho > caption: `. It was finetuned on Clotho and that is the indended "default" style.
WhisperTokenizer must be initialized with `language="en"` and `task="transcribe"`.
Our model class `WhisperForAudioCaptioning` can be found in our git repository or here on the HuggingFace Hub in the model repository. The class overrides default Whisper `generate` method to support forcing decoder prefix.
## Training details
The model was initialized by original speech-to-text `openai/whisper-large-v2` weights. Then, it was pretrained on a mix of (1) subset of AudioSet with synthetic labels, (2) AudioCaps captioning dataset and (3) Clotho v2.1 captioning dataset. Finally, it was finetuned on Clotho v2.1 to focus the model on the specific style of the captions. For each traning input, the model was informed about the source of the data, so it can mimic the caption style in all 3 styles.
During pretraining, the ratio of samples in each batch was approximately 12:3:1 (AudioSet:AudioCaps:Clotho). The pretraining took 13500 steps with batch size 32 and learning rate 2e-5. Finetuning was done on Clotho only, and the model was trained for 2200 steps with batch size 32 and learning rate 4e-6. All layers except *fc1* layers were frozen during finetuning.
For more information about the training regime, see the [technical report](TODO).
## Evaluation details
Metrics reported in the metadata were computed on Clotho v2.1 test split with captions generated using a beam search with 5 beams.
| | whisper-tiny | whisper-small | whisper-large-v2 |
|----------------------|--------------|---------------|------------------|
| SacreBLEU | 13.77 | 15.76 | 16.50 |
| METEOR | 0.3452 | 0.3781 | 0.3782 |
| CIDEr | 0.3404 | 0.4142 | 0.4331 |
| SPICE | 0.1077 | 0.1234 | 0.1257 |
| SPIDEr | 0.2240 | 0.2687 | 0.2794 |
## Limitations
The captions generated by the model can be misleading or not truthful, even if they appear convincing. The hallucination occurs especially in domains that were not present in the finetuning data.
While the original speech-to-text checkpoints by OpenAI were trained on multilingual data, our training contains only English captions, and therefore is not expected for the model to support other languages.
## Licence
The model weights are published under non-commercial license CC BY-NC 4.0 as the model was finetuned on a dataset for non-commercial use.
## Contact
If you'd like to chat about this, please get in touch with is via email at kadlcik`<at>`mail.muni.cz or ahajek`<at>`mail.muni.cz.
|