---
license: mit
library_name: peft
tags:
- trl
- sft
- generated_from_trainer
base_model: LoftQ/Meta-Llama-3-8B-Instruct-4bit-64rank
model-index:
- name: llama3-8b-instruct-qlora-mini
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# llama3-8b-instruct-qlora-mini

This model is a fine-tuned version of [LoftQ/Meta-Llama-3-8B-Instruct-4bit-64rank](https://huggingface.co/LoftQ/Meta-Llama-3-8B-Instruct-4bit-64rank) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.8668

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- num_epochs: 30

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 2.3909        | 1.0   | 53   | 1.5473          |
| 2.1937        | 2.0   | 106  | 1.2690          |
| 2.0915        | 3.0   | 159  | 1.0977          |
| 1.9927        | 4.0   | 212  | 1.0320          |
| 1.9058        | 5.0   | 265  | 1.0046          |
| 1.8032        | 6.0   | 318  | 0.9885          |
| 1.6688        | 7.0   | 371  | 0.9754          |
| 1.5215        | 8.0   | 424  | 0.9745          |
| 1.3617        | 9.0   | 477  | 0.9640          |
| 1.2074        | 10.0  | 530  | 0.9579          |
| 1.0429        | 11.0  | 583  | 0.9441          |
| 0.9013        | 12.0  | 636  | 0.9355          |
| 0.7969        | 13.0  | 689  | 0.9278          |
| 0.7092        | 14.0  | 742  | 0.9171          |
| 0.6272        | 15.0  | 795  | 0.9070          |
| 0.5688        | 16.0  | 848  | 0.9052          |
| 0.5128        | 17.0  | 901  | 0.8942          |
| 0.469         | 18.0  | 954  | 0.8894          |
| 0.4294        | 19.0  | 1007 | 0.8871          |
| 0.3953        | 20.0  | 1060 | 0.8807          |
| 0.371         | 21.0  | 1113 | 0.8756          |
| 0.3533        | 22.0  | 1166 | 0.8750          |
| 0.3335        | 23.0  | 1219 | 0.8730          |
| 0.3212        | 24.0  | 1272 | 0.8699          |
| 0.3108        | 25.0  | 1325 | 0.8687          |
| 0.3089        | 26.0  | 1378 | 0.8676          |
| 0.3031        | 27.0  | 1431 | 0.8678          |
| 0.3014        | 28.0  | 1484 | 0.8675          |
| 0.3013        | 29.0  | 1537 | 0.8666          |
| 0.2978        | 30.0  | 1590 | 0.8668          |


### Framework versions

- PEFT 0.10.0
- Transformers 4.40.0
- Pytorch 2.2.1+cu121
- Datasets 2.18.0
- Tokenizers 0.19.1