|
--- |
|
license: mit |
|
language: |
|
- ar |
|
--- |
|
|
|
|
|
# AraBART+Morph+GEC<sup>13</sup> QALB-2014 Model |
|
|
|
## Model description |
|
**AraBART+Morph+GEC<sup>13</sup>** is a Modern Standard Arabic (MSA) grammatical error correction (GEC) model that was built by fine-tuning the [AraBART](https://huggingface.co/moussaKam/AraBART) model. |
|
For the fine-tuning, we used the [QALB-2014](https://aclanthology.org/W14-3605.pdf) dataset. Please note that this model was fine-tuned on morphologically preprocessed text. |
|
Our fine-tuning procedure and the hyperparameters we used can be found in our paper *"[Advancements in Arabic Grammatical Error Detection and Correction: |
|
An Empirical Investigation](https://arxiv.org/abs/2305.14734)."* Our fine-tuning code and data can be found [here](https://github.com/CAMeL-Lab/arabic-gec). |
|
|
|
## Intended uses |
|
You can use the AraBART+Morph+GEC<sup>13</sup> model as part of an extended version of the [transformers](https://github.com/CAMeL-Lab/arabic-gec) that we make publicly available. |
|
The GEC model is intended to be used with this [GED](https://huggingface.co/CAMeL-Lab/camelbert-msa-qalb14-ged-13) model as we outlined in the example below. |
|
We used this GEC model to report results on the QALB-2014 dev and test sets in our [paper](). |
|
|
|
#### How to use |
|
To use the model with our extended version of transformers: |
|
|
|
|
|
```python |
|
from transformers import AutoTokenizer, BertForTokenClassification, MBartForConditionalGeneration |
|
from camel_tools.disambig.bert import BERTUnfactoredDisambiguator |
|
from camel_tools.utils.dediac import dediac_ar |
|
import torch.nn.functional as F |
|
import torch |
|
|
|
bert_disambig = BERTUnfactoredDisambiguator.pretrained() |
|
|
|
ged_tokenizer = AutoTokenizer.from_pretrained('CAMeL-Lab/camelbert-msa-qalb14-ged-13') |
|
ged_model = BertForTokenClassification.from_pretrained('CAMeL-Lab/camelbert-msa-qalb14-ged-13') |
|
|
|
gec_tokenizer = AutoTokenizer.from_pretrained('CAMeL-Lab/arabart-qalb14-gec-ged-13') |
|
gec_model = MBartForConditionalGeneration.from_pretrained('CAMeL-Lab/arabart-qalb14-gec-ged-13') |
|
|
|
text = 'و قال له انه يحب اكل الطعام بكثره .' |
|
|
|
# morph processing the input text |
|
text_disambig = bert_disambig.disambiguate(text.split()) |
|
morph_pp_text = [dediac_ar(w_disambig.analyses[0].analysis['diac']) for w_disambig in text_disambig] |
|
morph_pp_text = ' '.join(morph_pp_text) |
|
|
|
# GED tagging |
|
inputs = ged_tokenizer([morph_pp_text], return_tensors='pt') |
|
logits = ged_model(**inputs).logits |
|
preds = F.softmax(logits, dim=-1).squeeze()[1:-1] |
|
pred_ged_labels = [ged_model.config.id2label[p.item()] for p in torch.argmax(preds, -1)] |
|
|
|
# Extending GED label to GEC-tokenized input |
|
ged_label2ids = gec_model.config.ged_label2id |
|
tokens, ged_labels = [], [] |
|
|
|
for word, label in zip(morph_pp_text.split(), pred_ged_labels): |
|
word_tokens = gec_tokenizer.tokenize(word) |
|
if len(word_tokens) > 0: |
|
tokens.extend(word_tokens) |
|
ged_labels.extend([label for _ in range(len(word_tokens))]) |
|
|
|
|
|
input_ids = gec_tokenizer.convert_tokens_to_ids(tokens) |
|
input_ids = [gec_tokenizer.bos_token_id] + input_ids + [gec_tokenizer.eos_token_id] |
|
|
|
label_ids = [ged_label2ids.get(label, ged_label2ids['<pad>']) for label in ged_labels] |
|
label_ids = [ged_label2ids['UC']] + label_ids + [ged_label2ids['UC']] |
|
attention_mask = [1 for _ in range(len(input_ids))] |
|
|
|
|
|
gen_kwargs = {'num_beams': 5, 'max_length': 100, |
|
'num_return_sequences': 1, |
|
'no_repeat_ngram_size': 0, 'early_stopping': False, |
|
'ged_tags': torch.tensor([label_ids]), |
|
'attention_mask': torch.tensor([attention_mask]) |
|
} |
|
|
|
# GEC generation |
|
generated = gec_model.generate(torch.tensor([input_ids]), **gen_kwargs) |
|
|
|
generated_text = gec_tokenizer.batch_decode(generated, |
|
skip_special_tokens=True, |
|
clean_up_tokenization_spaces=False |
|
)[0] |
|
|
|
print(generated_text) # وقال له أنه يحب أكل الطعام بكثرة . |
|
``` |
|
|
|
|
|
## Citation |
|
```bibtex |
|
@inproceedings{alhafni-etal-2023-advancements, |
|
title = "Advancements in {A}rabic Grammatical Error Detection and Correction: An Empirical Investigation", |
|
author = "Alhafni, Bashar and |
|
Inoue, Go and |
|
Khairallah, Christian and |
|
Habash, Nizar", |
|
editor = "Bouamor, Houda and |
|
Pino, Juan and |
|
Bali, Kalika", |
|
booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing", |
|
month = dec, |
|
year = "2023", |
|
address = "Singapore", |
|
publisher = "Association for Computational Linguistics", |
|
url = "https://aclanthology.org/2023.emnlp-main.396", |
|
doi = "10.18653/v1/2023.emnlp-main.396", |
|
pages = "6430--6448", |
|
abstract = "Grammatical error correction (GEC) is a well-explored problem in English with many existing models and datasets. However, research on GEC in morphologically rich languages has been limited due to challenges such as data scarcity and language complexity. In this paper, we present the first results on Arabic GEC using two newly developed Transformer-based pretrained sequence-to-sequence models. We also define the task of multi-class Arabic grammatical error detection (GED) and present the first results on multi-class Arabic GED. We show that using GED information as auxiliary input in GEC models improves GEC performance across three datasets spanning different genres. Moreover, we also investigate the use of contextual morphological preprocessing in aiding GEC systems. Our models achieve SOTA results on two Arabic GEC shared task datasets and establish a strong benchmark on a recently created dataset. We make our code, data, and pretrained models publicly available.", |
|
} |
|
``` |
