File size: 1,743 Bytes
ba7fbf7 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
---
license: apache-2.0
language:
- fr
library_name: transformers
inference: false
---
# CamemBERT-L8
This model is a pruned version of the pre-trained [CamemBERT](https://huggingface.co/camembert-base) checkpoint, obtained by [dropping the top-layers](https://arxiv.org/abs/2004.03844) from the original model.
## Usage
```python
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained('antoinelouis/camembert-L8')
model = AutoModel.from_pretrained('antoinelouis/camembert-L8')
```
## Comparison
| Model | #Params | Size | Pruning |
|--------------------------------------------------------------------|:-------:|:-----:|:-------:|
| [CamemBERT](https://huggingface.co/camembert-base) | 110.6M | 445MB | - |
| | | | |
| [CamemBERT-L10](https://huggingface.co/antoinelouis/camembert-L10) | 96.4M | 368MB | -13% |
| **CamemBERT-L8** | 82.3M | 314MB | -26% |
| [CamemBERT-L6](https://huggingface.co/antoinelouis/camembert-L6) | 68.1M | 260MB | -38% |
| [CamemBERT-L4](https://huggingface.co/antoinelouis/camembert-L4) | 53.9M | 206MB | -51% |
| [CamemBERT-L2](https://huggingface.co/antoinelouis/camembert-L2) | 39.7M | 159MB | -64% |
## Citation
```bibtex
@online{louis2023,
author = 'Antoine Louis',
title = 'CamemBERT-L8: A Pruned Version of CamemBERT',
publisher = 'Hugging Face',
month = 'october',
year = '2023',
url = 'https://huggingface.co/antoinelouis/camembert-L8',
}
``` |