README.md

---
license: cc-by-nc-4.0
---

# ECAPA2 Speaker Embedding and Hierarchical Feature Extractor

ECAPA2 is a hybrid neural network architecture and training strategy for generating robust speaker embeddings. 
The provided pre-trained model has an easy-to-use API to extract speaker embeddings and other hierarchical features. More information can be found in our original ECAPA2 paper.

The speaker embeddings are recommended for tasks which rely directly on the identity of the speaker (e.g. speaker verification and speaker diarization).
The hierarchical features are most useful for tasks capturing intra-speaker variance (e.g. emotion recognition and speaker profiling) and prove complimentary with the speaker embedding in our experience. See our speaker profiling paper for an example usage of the hierarchical features.

<p align="center">
<img src="https://cdn-uploads.huggingface.co/production/uploads/620f6a7d110b521c673c1914/cmd1Nvk6_WXInUKrjgXPj.png" width="300"/>
</p>

<!---
<img src="https://cdn-uploads.huggingface.co/production/uploads/620f6a7d110b521c673c1914/BORgtl2G6XUlWaZeMLGPc.png" width="300"/>
-->

<!---
<img src="https://cdn-uploads.huggingface.co/production/uploads/620f6a7d110b521c673c1914/ejHsEUnsWTehsIpOu7Rm_.png" width="700"/>
-->
## Usage Guide

### Download model

You need to install the `huggingface_hub` package to download the ECAPA2 model:

```bash
pip install --upgrade huggingface_hub
```

Or with Conda:

```bash
conda install -c conda-forge huggingface_hub
```

Download model:

```python
from huggingface_hub import hf_hub_download

# automatically checks for cached file, optionally set `cache_dir` location
model_file = hf_hub_download(repo_id='Jenthe/ECAPA2', filename='model.pt', cache_dir=None)
```


### Speaker Embedding Extraction

Extracting speaker embeddings is easy and only requires a few lines of code:

```python
import torch
import torchaudio

ecapa2_model = torch.jit.load(model_file, map_location='cpu')
audio, sr = torchaudio.load('sample.wav') # sample rate of 16 kHz expected

embedding = ecapa2_model(audio)
```

For faster, 16-bit half-precision CUDA inference (recommended):

```python
import torch
import torchaudio

ecapa2_model = torch.jit.load(model_file, map_location='cuda')
ecapa2_model.half() # optional, but results in faster inference
audio, sr = torchaudio.load('sample.wav') # sample rate of 16 kHz expected

embedding = ecapa2_model(audio)
```

There is no need for `ecapa2_model.eval()` or `torch.no_grad()`, this is done automatically.

### Hierarchical Feature Extraction

For the extraction of other hierachical features, the `label` argument can be used, which accepts a string containing the feature ids separated with '|':

```python
# default, only extract the embedding
feature = ecapa2_model(audio, label='embedding')

# concatenates the gfe_1, pool and embedding features
feature = ecapa2_model(audio, label='gfe_1|pool|embedding')

# returns the same output as previous example, concatenation always follows the order of the network
feature = ecapa2_model(audio, label='embedding|gfe_1|pool')
```

The following table describes the available features. All features consists of the mean and variance of the frame-level encodings at the indicated layer, expect for the speaker embedding.

| Feature ID| Dimension | Description |
| ----------- | ----------- | ----------- |
| gfe_1 | 2048 | Mean and variance of frame-level features as indicated in Figure 1, extracted before ReLU and BatchNorm layer.
| gfe_2 | 2048 | Mean and variance of frame-level features as indicated in Figure 1, extracted before ReLU and BatchNorm layer.
| pool | 3072 | Pooled statistics before the bottleneck speaker embedding layer, extracted before ReLU layer.
| attention | 3072 | Same as the pooled statistics but with the attention weights applied.
| embedding | 192 | The standard ECAPA2 speaker embedding.
<!--
The following table describes the available features:

| Feature Type| Description | Usage | Labels |
| ----------- | ----------- | ----------- | ----------- |
| Local Feature | Non-uniform effective receptive field in the frequency dimension of each frame-level feature.| Abstract features, probably usefull in tasks less related to speaker characteristics. | lfe1, lfe2, lfe3, lfe4
| Global Feature | Uniform effective receptive field of each frame-level feature in the frequency dimension.| Generally capture intra-speaker variance better then speaker embeddings. E.g. speaker profiling, emotion recognition. | gfe1, gfe2, gfe3, pool
| Speaker Embedding | Uniform effective receptive field of each frame-level feature in the frequency dimension.| Best for tasks directly depending on the speaker identity (as opposed to speaker characteristics). E.g. speaker verification, speaker diarization. | embedding
-->
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

## Citation

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**

```
@INPROCEEDINGS{xxxxx,
  author={Jenthe Thienpondt and Kris Demuynck},
  booktitle={2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)}, 
  title={ECAPA2: A Hybrid Neural Network Architecture and Training Strategy for Robust Speaker Embeddings}, 
  year={2023},
  volume={},
  number={}
}
```

**APA:**

```
Jenthe Thienpondt, Kris Demuynck (2023). ECAPA2: A Hybrid Neural Network Architecture and Training Strategy for Robust Speaker Embeddings. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)
```

## Contact

Name: Jenthe Thienpondt\
E-mail: [email protected]