tommyp111/pythia-160m-deduped-alpaca-lora-no-grad-norm

See axolotl config

axolotl version: 0.4.1

base_model: EleutherAI/pythia-160m-deduped
load_in_8bit: true
datasets:
  - path: teknium/GPT4-LLM-Cleaned
    type: alpaca
dataset_prepared_path:
val_set_size: 0.05
adapter: lora
lora_model_dir:
sequence_len: 512
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_modules:
  - query_key_value
  - dense
  - dense_h_to_4h
  - dense_4h_to_h
lora_target_linear:
lora_fan_in_fan_out: true  # pythia/GPTNeoX lora specific
output_dir: ./outputs/lora-alpaca-pythia
gradient_accumulation_steps: 1
micro_batch_size: 4
num_epochs: 4
learning_rate: 0.00001
train_on_inputs: false
group_by_length: false
bf16: auto
tf32: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
weight_decay: 0.1
evals_per_epoch: 4
logging_steps: 1
push_to_hub: tommyp111/pythia-160m-deduped-alpaca-lora
wandb_project: pythia-alpaca-lora
wandb_name: pythia-160m
max_grad_norm: 2.0

outputs/lora-alpaca-pythia

This model is a fine-tuned version of EleutherAI/pythia-160m-deduped on the None dataset. It achieves the following results on the evaluation set:

• Loss: 2.8091

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100
• num_epochs: 4

Training results

Training Loss	Epoch	Step	Validation Loss
8.6028	0.0001	1	43.0831
2.7877	0.25	3190	3.9830
2.6991	0.5	6380	3.2686
2.5373	0.75	9570	3.2401
2.4498	1.0	12760	3.0415
2.9168	1.25	15950	3.0911
2.1057	1.5	19140	2.9493
2.1088	1.75	22330	2.9256
2.3546	2.0	25520	2.9625
2.4666	2.25	28710	2.9406
2.9943	2.5	31900	2.8396
2.1245	2.75	35090	2.8112
2.7812	3.0	38280	2.8326
2.3245	3.25	41470	2.8359
2.499	3.5	44660	2.8282
2.3163	3.75	47850	2.8230
2.3762	4.0	51040	2.8091

Framework versions

• PEFT 0.11.1
• Transformers 4.43.1
• Pytorch 2.3.1+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1