language:
- pt
tags:
- generated_from_trainer
datasets:
- lener_br
metrics:
- precision
- recall
- f1
- accuracy
model-index:
- name: checkpoints
results:
- task:
name: Token Classification
type: token-classification
dataset:
name: lener_br
type: lener_br
metrics:
- name: F1
type: f1
value: 0.8733423827921062
- name: Precision
type: precision
value: 0.8487923685812868
- name: Recall
type: recall
value: 0.8993548387096775
- name: Accuracy
type: accuracy
value: 0.9759397808828684
- name: Loss
type: loss
value: 0.10249536484479904
widget:
- text: >-
Acrescento que não há de se falar em violação do artigo 114, § 3º, da
Constituição Federal, posto que referido dispositivo revela-se
impertinente, tratando da possibilidade de ajuizamento de dissídio
coletivo pelo Ministério Público do Trabalho nos casos de greve em
atividade essencial.
(BERT base) NER model in the legal domain in Portuguese (LeNER-Br)
ner-bert-base-portuguese-cased-lenerbr is a NER model (token classification) in the legal domain in Portuguese that was finetuned on 20/12/2021 in Google Colab from the model pierreguillou/bert-base-cased-pt-lenerbr on the dataset LeNER_br by using a NER objective.
Due to the small size of BERTimbau base and finetuning dataset, the model overfitted before to reach the end of training. Here are the overall final metrics on the validation dataset (note: see the paragraph "Validation metrics by Named Entity" to get detailed metrics):
- f1: 0.8733423827921062
- precision: 0.8487923685812868
- recall: 0.8993548387096775
- accuracy: 0.9759397808828684
- loss: 0.10249536484479904
Note: the model pierreguillou/bert-base-cased-pt-lenerbr is a language model that was created through the finetuning of the model BERTimbau base on the dataset LeNER-Br language modeling by using a MASK objective. This first specialization of the language model before finetuning on the NER task improved a bit the model quality. To prove it, here are the results of the NER model finetuned from the model BERTimbau base (a non-specialized language model):
- f1: 0.8716487228203504
- precision: 0.8559286898839138
- recall: 0.8879569892473118
- accuracy: 0.9755893153732458
- loss: 0.1133928969502449
Widget & APP
You can test this model into the widget of this page.
Using the model for inference in production
# install pytorch: check https://pytorch.org/
# !pip install transformers
from transformers import AutoModelForTokenClassification, AutoTokenizer
import torch
# parameters
model_name = "ner-bert-base-portuguese-cased-lenebr"
model = AutoModelForTokenClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
input_text = "EMENTA: APELAÇÃO CÍVEL - AÇÃO DE INDENIZAÇÃO POR DANOS MORAIS - PRELIMINAR - ARGUIDA PELO MINISTÉRIO PÚBLICO EM GRAU RECURSAL - NULIDADE - AUSÊNCIA DE IN- TERVENÇÃO DO PARQUET NA INSTÂNCIA A QUO - PRESENÇA DE INCAPAZ - PREJUÍZO EXISTENTE - PRELIMINAR ACOLHIDA - NULIDADE RECONHECIDA."
# tokenization
inputs = tokenizer(input_text, max_length=512, truncation=True, return_tensors="pt")
tokens = inputs.tokens()
# get predictions
outputs = model(**inputs).logits
predictions = torch.argmax(outputs, dim=2)
# print predictions
for token, prediction in zip(tokens, predictions[0].numpy()):
print((token, model.config.id2label[prediction]))
You can use pipeline, too. However, it seems to have an issue regarding to the max_length of the input sequence.
!pip install transformers
import transformers
from transformers import pipeline
model_name = "ner-bert-base-portuguese-cased-lenebr"
ner = pipeline(
"ner",
model=model_name
)
ner(input_text)
Training procedure
Notebook
The notebook of finetuning (HuggingFace_Notebook_token_classification_NER_LeNER_Br.ipynb) is in github.
Training results
Num examples = 7828
Num Epochs = 3
Instantaneous batch size per device = 4
Total train batch size (w. parallel, distributed & accumulation) = 8
Gradient Accumulation steps = 2
Total optimization steps = 2934
Step Training Loss Validation Loss Precision Recall F1 Accuracy
290 0.314600 0.163042 0.735828 0.697849 0.716336 0.949198
580 0.086900 0.123495 0.779540 0.824301 0.801296 0.965807
870 0.072800 0.106785 0.798481 0.858925 0.827600 0.968626
1160 0.046300 0.109921 0.824576 0.877419 0.850177 0.973243
1450 0.036600 0.102495 0.848792 0.899355 0.873342 0.975940
1740 0.033400 0.121514 0.821681 0.899785 0.858961 0.967071
2030 0.034700 0.115568 0.846849 0.887097 0.866506 0.970607
2320 0.018000 0.108600 0.840258 0.895914 0.867194 0.973730
Validation metrics by Named Entity
Num examples = 1177
{'JURISPRUDENCIA': {'f1': 0.7069834413246942,
'number': 657,
'precision': 0.6707650273224044,
'recall': 0.7473363774733638},
'LEGISLACAO': {'f1': 0.8256227758007118,
'number': 571,
'precision': 0.8390596745027125,
'recall': 0.8126094570928196},
'LOCAL': {'f1': 0.7688564476885645,
'number': 194,
'precision': 0.728110599078341,
'recall': 0.8144329896907216},
'ORGANIZACAO': {'f1': 0.8548387096774193,
'number': 1340,
'precision': 0.8062169312169312,
'recall': 0.9097014925373135},
'PESSOA': {'f1': 0.9826697892271662,
'number': 1072,
'precision': 0.9868297271872061,
'recall': 0.9785447761194029},
'TEMPO': {'f1': 0.9615846338535414,
'number': 816,
'precision': 0.9423529411764706,
'recall': 0.9816176470588235},
'overall_accuracy': 0.9759397808828684,
'overall_f1': 0.8733423827921062,
'overall_precision': 0.8487923685812868,
'overall_recall': 0.8993548387096775}