Hugging Face's logo

lesso07
/
a8fdc686-7987-4304-921d-adb1686ae554

PEFT
Safetensors
llama
axolotl
Generated from Trainer
Model card Files Community
a8fdc686-7987-4304-921d-adb1686ae554 / README.md
lesso07's picture
lesso07
End of training
e797fc9 verified
preview code
|
raw
history blame contribute delete
2.22 kB
metadata
library_name: peft
license: apache-2.0
base_model: heegyu/WizardVicuna-open-llama-3b-v2
tags:
  - axolotl
  - generated_from_trainer
model-index:
  - name: a8fdc686-7987-4304-921d-adb1686ae554
    results: []

Built with Axolotl

a8fdc686-7987-4304-921d-adb1686ae554

This model is a fine-tuned version of heegyu/WizardVicuna-open-llama-3b-v2 on the None dataset. It achieves the following results on the evaluation set:

  • Loss: 1.9862

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.000207
  • train_batch_size: 4
  • eval_batch_size: 4
  • seed: 42
  • gradient_accumulation_steps: 2
  • total_train_batch_size: 8
  • optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_steps: 50
  • training_steps: 419

Training results

Training Loss Epoch Step Validation Loss
No log 0.0024 1 3.0941
2.1505 0.1195 50 2.4442
1.8806 0.2389 100 2.7416
2.1693 0.3584 150 2.3181
1.984 0.4779 200 2.2799
1.9623 0.5974 250 2.1786
2.0553 0.7168 300 2.0686
1.9531 0.8363 350 1.9983
1.8796 0.9558 400 1.9862

Framework versions

  • PEFT 0.13.2
  • Transformers 4.46.0
  • Pytorch 2.5.0+cu124
  • Datasets 3.0.1
  • Tokenizers 0.20.1