File size: 4,545 Bytes
5af3504 e136aaa 5af3504 c6404e0 e136aaa 5af3504 cce70fb 5af3504 c6404e0 5af3504 c6404e0 5af3504 c6404e0 5af3504 c6404e0 5af3504 c6404e0 5af3504 c6404e0 f3aee89 c6404e0 433f5f3 c6404e0 5af3504 c6404e0 5af3504 c6404e0 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 |
---
library_name: transformers
license: mit
tags:
- translation
- generated_from_trainer
model-index:
- name: HelpMum-Personal/ai-translator-eng-to-9ja
results: []
language:
- yo
- ig
- ha
- en
pipeline_tag: translation
metrics:
- bleu
- sacrebleu
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# AI-translator-eng-to-9ja
This model is a 418 Million parameter translation model, built for translating from English into Yoruba, Igbo, and Hausa. It was trained on a dataset consisting of 1,500,000 sentences (500,000 for each language), providing high-quality translations for these languages.
It was built with the intention of building a system that makes it easier to communicate with LLMs using Igbo, Hausa and Yoruba languages.
## Model Details
- **Languages Supported**:
- Source Language: English
- Target Languages: Yoruba, Igbo, Hausa
### Model Usage
To use this model for translation tasks, you can load it from Hugging Face’s `transformers` library:
```python
from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer
import huggingface_hub
huggingface_hub.login()
# Load the fine-tuned model
model = M2M100ForConditionalGeneration.from_pretrained("HelpMumHQ/AI-translator-eng-to-9ja")
tokenizer = M2M100Tokenizer.from_pretrained("HelpMumHQ/AI-translator-eng-to-9ja")
# translate English to Igbo
eng_text="Healthcare is an important field in virtually every society because it directly affects the well-being and quality of life of individuals. It encompasses a wide range of services and professions, including preventive care, diagnosis, treatment, and management of diseases and conditions."
tokenizer.src_lang = "en"
tokenizer.tgt_lang = "ig"
encoded_eng = tokenizer(eng_text, return_tensors="pt")
generated_tokens = model.generate(**encoded_eng, forced_bos_token_id=tokenizer.get_lang_id("ig"))
tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
# translate English to yoruba
eng_text="Healthcare is an important field in virtually every society because it directly affects the well-being and quality of life of individuals. It encompasses a wide range of services and professions, including preventive care, diagnosis, treatment, and management of diseases and conditions. Effective healthcare systems aim to improve health outcomes, reduce the incidence of illness, and ensure that individuals have access to necessary medical services."
tokenizer.src_lang = "en"
tokenizer.tgt_lang = "yo"
encoded_eng = tokenizer(eng_text, return_tensors="pt")
generated_tokens = model.generate(**encoded_eng, forced_bos_token_id=tokenizer.get_lang_id("yo"))
tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
# translate English to Hausa
eng_text="Healthcare is an important field in virtually every society because it directly affects the well-being and quality of life of individuals. It encompasses a wide range of services and professions, including preventive care, diagnosis, treatment, and management of diseases and conditions. Effective healthcare systems aim to improve health outcomes, reduce the incidence of illness, and ensure that individuals have access to necessary medical services."
tokenizer.src_lang = "en"
tokenizer.tgt_lang = "ha"
encoded_eng = tokenizer(eng_text, return_tensors="pt")
generated_tokens = model.generate(**encoded_eng, forced_bos_token_id=tokenizer.get_lang_id("ha"))
tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
```
### Supported Language Codes
- **English**: `en`
- **Yoruba**: `yo`
- **Igbo**: `ig`
- **Hausa**: `ha`
### Training Dataset
The training dataset consists of 1,500,000 translation pairs, sourced from a combination of open-source parallel corpora and curated datasets specific to Yoruba, Igbo, and Hausa
## Limitations
- While the model performs well across English-to-Yoruba, Igbo, and Hausa translations, performance may vary depending on the complexity and domain of the text.
- Translation quality may decrease for extremely long sentences or ambiguous contexts.
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3
### Framework versions
- Transformers 4.44.2
- Pytorch 2.4.0+cu121
- Datasets 2.21.0
- Tokenizers 0.19.1 |