|
--- |
|
license: cc-by-nc-4.0 |
|
language: |
|
- en |
|
library_name: transformers |
|
--- |
|
# Whisper Multitask Analyzer |
|
|
|
A transformer encoder-decoder model for automatic audio captioning. As opposed to speech-to-text, captioning describes the content and features of audio clips. |
|
|
|
- **Model, codebase & card adapted from:** MU-NLPC/whisper-small-audio-captioning |
|
- **Model type:** Whisper encoder-decoder transformer |
|
- **Language(s) (NLP):** en |
|
- **License:** cc-by-4.0 |
|
- **Parent Model:** openai/whisper-small |
|
|
|
## Usage |
|
|
|
The model expects an audio clip (up to 30s) to the encoder as an input and information about caption style as forced prefix to the decoder. |
|
The forced prefix is an integer which is mapped to various tasks. This mapping is defined in the model config and can be retrieved with a function. |
|
|
|
The tag mapping of the current model is: |
|
|
|
| Task | ID | Description | |
|
| -------- | -- | ------------------------------------------------------ | |
|
| tags | 0 | General descriptions, can include genres and features. | |
|
| genre | 1 | Estimated musical genres. | |
|
| mood | 2 | Estimated emotional feeling. | |
|
| movement | 3 | Estimated audio pace and expression. | |
|
| theme | 4 | Estimated audio usage (not very accurate) | |
|
|
|
Minimal example: |
|
```python |
|
# Load model |
|
checkpoint = "DionTimmer/whisper-small-multitask-analyzer" |
|
model = WhisperForAudioCaptioning.from_pretrained(checkpoint) |
|
tokenizer = transformers.WhisperTokenizer.from_pretrained(checkpoint, language="en", task="transcribe") |
|
feature_extractor = transformers.WhisperFeatureExtractor.from_pretrained(checkpoint) |
|
|
|
# Load and preprocess audio |
|
input_file = "..." |
|
audio, sampling_rate = librosa.load(input_file, sr=feature_extractor.sampling_rate) |
|
features = feature_extractor(audio, sampling_rate=sampling_rate, return_tensors="pt").input_features |
|
|
|
# Mappings by ID |
|
print(model.task_mapping) # {0: 'tags', 1: 'genre', 2: 'mood', 3: 'movement', 4: 'theme'} |
|
|
|
# Inverted |
|
print(model.named_task_mapping) # {'tags': 0, 'genre': 1, 'mood': 2, 'movement': 3, 'theme': 4} |
|
|
|
# Prepare caption style |
|
style_prefix = f"{model.named_task_mapping['tags']}: " |
|
style_prefix_tokens = tokenizer("", text_target=style_prefix, return_tensors="pt", add_special_tokens=False).labels |
|
|
|
# Generate caption |
|
model.eval() |
|
outputs = model.generate( |
|
inputs=features.to(model.device), |
|
forced_ac_decoder_ids=style_prefix_tokens, |
|
max_length=100, |
|
) |
|
|
|
print(tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]) |
|
``` |
|
|
|
Example output: |
|
*0: advertising, beautiful, beauty, bright, cinematic, commercial, corporate, emotional, epic, film, heroic, hopeful, inspiration, inspirational, inspiring, love, love story, movie, orchestra, orchestral, piano, positive, presentation, romantic, sentimental* |
|
|
|
WhisperTokenizer must be initialized with `language="en"` and `task="transcribe"`. |
|
|
|
The model class `WhisperForAudioCaptioning` can be found in the git repository or here on the HuggingFace Hub in the model repository. The class overrides default Whisper `generate` method to support forcing decoder prefix. |
|
|
|
|
|
## Licence |
|
|
|
The model weights are published under non-commercial license CC BY-NC 4.0 as the model was finetuned on a dataset for non-commercial use. |
