File size: 2,425 Bytes
76260fc 29127e5 76260fc 29127e5 76260fc 29127e5 76260fc |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 |
---
base_model: unsloth/meta-llama-3.1-8b-instruct-bnb-4bit
language:
- en
license: apache-2.0
tags:
- text-generation-inference
- transformers
- unsloth
- llama
- trl
---
# Uploaded Model: LORA Adapter
- **Developed by:** Mubin1917
- **License:** apache-2.0
- **Finetuned from model:** unsloth/meta-llama-3.1-8b-instruct-bnb-4bit
This LORA adapter is based on the `unsloth/meta-llama-3.1-8b-instruct-bnb-4bit` model and has been fine-tuned on the [**Lamini_docs QnA**](https://huggingface.co/datasets/lamini/lamini_docs) dataset. The fine-tuning process was optimized using [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library, resulting in a 2x faster training time.
### Training Configuration
The model was trained with the following configuration:
```python
training_args = TrainingArguments(
num_train_epochs=6,
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
per_device_eval_batch_size=4,
eval_accumulation_steps=4,
warmup_steps=50,
learning_rate=2e-4,
fp16=not torch.cuda.is_bf16_supported(),
bf16=torch.cuda.is_bf16_supported(),
eval_steps=25, # Evaluate every 25 steps
logging_steps=25,
optim="adamw_8bit",
weight_decay=0.01,
lr_scheduler_type="linear",
seed=3407,
output_dir="/kaggle/temp/results",
report_to="wandb",
save_total_limit=1, # Save the best one and the last one
metric_for_best_model="val_loss",
eval_strategy="steps",
load_best_model_at_end=True,
)
```
### Evaluation Results
- **SacreBLEU Test:**
Score: **73.55**
Detailed Metrics:
- Counts: [20894, 19191, 18504, 18029]
- Totals: [26214, 26074, 25934, 25794]
- Precisions: [79.71%, 73.60%, 71.35%, 69.90%]
- Brevity Penalty: **1.0**
- System Length: **26214**
- Reference Length: **24955**
- **BLEU Test:**
BLEU Score: **0.767**
Detailed Metrics:
- Precisions: [79.71%, 73.73%]
- Brevity Penalty: **1.0**
- Length Ratio: **1.05**
- Translation Length: **26299**
- Reference Length: **24955**
For a detailed comparison between the predicted and actual QnA responses on the test dataset, please visit the [evaluation dataset](https://huggingface.co/datasets/Mubin1917/lamini_docs_evaluation).
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
|