README.md

---
base_model: unsloth/meta-llama-3.1-8b-instruct-bnb-4bit
language:
- en
license: apache-2.0
tags:
- text-generation-inference
- transformers
- unsloth
- llama
- trl
---

# Uploaded Model: LORA Adapter

- **Developed by:** Mubin1917
- **License:** apache-2.0
- **Finetuned from model:** unsloth/meta-llama-3.1-8b-instruct-bnb-4bit

This LORA adapter is based on the `unsloth/meta-llama-3.1-8b-instruct-bnb-4bit` model and has been fine-tuned on the [**Lamini_docs QnA**](https://huggingface.co/datasets/lamini/lamini_docs) dataset. The fine-tuning process was optimized using [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library, resulting in a 2x faster training time.

### Training Configuration

The model was trained with the following configuration:

```python
training_args = TrainingArguments(
    num_train_epochs=6,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    per_device_eval_batch_size=4,
    eval_accumulation_steps=4,
    warmup_steps=50,
    learning_rate=2e-4,
    fp16=not torch.cuda.is_bf16_supported(),
    bf16=torch.cuda.is_bf16_supported(),
    eval_steps=25,  # Evaluate every 25 steps
    logging_steps=25,
    optim="adamw_8bit",
    weight_decay=0.01,
    lr_scheduler_type="linear",
    seed=3407,
    output_dir="/kaggle/temp/results",
    report_to="wandb",
    save_total_limit=1,  # Save the best one and the last one
    metric_for_best_model="val_loss",
    eval_strategy="steps",
    load_best_model_at_end=True,
)
```

### Evaluation Results

- **SacreBLEU Test:**  
  Score: **73.55**  
  Detailed Metrics:  
  - Counts: [20894, 19191, 18504, 18029]  
  - Totals: [26214, 26074, 25934, 25794]  
  - Precisions: [79.71%, 73.60%, 71.35%, 69.90%]  
  - Brevity Penalty: **1.0**  
  - System Length: **26214**  
  - Reference Length: **24955**

- **BLEU Test:**  
  BLEU Score: **0.767**  
  Detailed Metrics:  
  - Precisions: [79.71%, 73.73%]  
  - Brevity Penalty: **1.0**  
  - Length Ratio: **1.05**  
  - Translation Length: **26299**  
  - Reference Length: **24955**

For a detailed comparison between the predicted and actual QnA responses on the test dataset, please visit the [evaluation dataset](https://huggingface.co/datasets/Mubin1917/lamini_docs_evaluation).

[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)