Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,47 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
+
language:
|
| 4 |
+
- sr
|
| 5 |
+
metrics:
|
| 6 |
+
- accuracy
|
| 7 |
+
- wer
|
| 8 |
+
library_name: transformers
|
| 9 |
+
tags:
|
| 10 |
+
- legal
|
| 11 |
---
|
| 12 |
+
|
| 13 |
+
# BERTić-COMtext-SR-legal-MSD-ekavica
|
| 14 |
+
|
| 15 |
+
**BERTić-COMtext-SR-legal-MSD-ekavica** is a variant of the [BERTić](https://huggingface.co/classla/bcms-bertic) model, fine-tuned on the task of morphosyntactic (MSD) tag prediction in Serbian legal texts written in the Ekavian pronunciation.
|
| 16 |
+
The model was fine-tuned for 15 epochs on the Ekavian variant of the [COMtext.SR.legal](https://github.com/ICEF-NLP/COMtext.SR) dataset.
|
| 17 |
+
|
| 18 |
+
# Benchmarking
|
| 19 |
+
|
| 20 |
+
This model was evaluated on the tasks of MSD prediction and lemmatization of Serbian legal texts.
|
| 21 |
+
Lemmatization was performed using the predicted MSD tags and the [srLex](http://hdl.handle.net/11356/1233) inflectional lexicon.
|
| 22 |
+
|
| 23 |
+
Accuracy and Word Error Rate were used as evaluation metrics.
|
| 24 |
+
|
| 25 |
+
This model was compared to:
|
| 26 |
+
- The [CLASSLA](http://pypi.org/project/classla/) library
|
| 27 |
+
- A variant of [BERTić](https://huggingface.co/classla/bcms-bertic) fine-tuned for MSD prediction using the [SETimes.SR 2.0](http://hdl.handle.net/11356/1843) corpus of newswire texts
|
| 28 |
+
- [SrBERTa](http://huggingface.co/nemanjaPetrovic/SrBERTa), a model specially trained on Serbian legal texts
|
| 29 |
+
|
| 30 |
+
All large language models were fine-tuned for 15 epochs.
|
| 31 |
+
CLASSLA and BERTić-SETimes were directly tested on the entire COMtext.SR.legal.ekavica corpus.
|
| 32 |
+
BERTić-COMtext-SR-legal-MSD-ekavica and SrBERTa were fine-tuned and evaluated on the COMtext.SR.legal.ekavica corpus using 10-fold CV.
|
| 33 |
+
|
| 34 |
+
The code and data to run these experiments is available on the [COMtext.SR GitHub repository](https://github.com/ICEF-NLP/COMtext.SR).
|
| 35 |
+
|
| 36 |
+
## Results
|
| 37 |
+
|
| 38 |
+
| Model | MSD ACC | MSD WER | Lemma ACC | Lemma WER |
|
| 39 |
+
| --------------------------------------------------------- | -------- | ---------- | --------- | ---------- |
|
| 40 |
+
| CLASSLA-SR (gold tokens) | 0.9144 | 0.0856 | 0.9432 | 0.0568 |
|
| 41 |
+
| *CLASSLA-SR (CLASSLA tokenizer)* | / | *0.0983* | / | *0.0739* |
|
| 42 |
+
| BERTić-SETimes (gold tokens) | 0.9231 | 0.0768 | 0.9649 | 0.0351 |
|
| 43 |
+
| *BERTić-SETimes.SR (CLASSLA tokenizer)* | / | *0.0884* | / | *0.0542* |
|
| 44 |
+
| BERTić-COMtext-SR-legal-MSD-ekavica (gold tokens) |**0.9674**| **0.0326** |**0.9666** | **0.0334** |
|
| 45 |
+
| *BERTić-COMtext-SR-legal-MSD-ekavica (CLASSLA tokenizer)* | / |***0.0447***| / |***0.0526***|
|
| 46 |
+
| SrBERTa (gold tokens) | 0.9288 | 0.0712 | 0.9391 | 0.0609 |
|
| 47 |
+
| *SrBERTa (CLASSLA tokenizer)* | / | *0.0851* | / | *0.0819* |
|