See axolotl config

axolotl version: 0.4.1

adapter: lora
base_model: EleutherAI/pythia-70m-deduped
bf16: auto
dataset_prepared_path: null
datasets:
- data_files:
  - c0e356afd17a58f1_train_data.json
  ds_type: json
  format: custom
  path: c0e356afd17a58f1_train_data.json
  type:
    field: null
    field_input: null
    field_instruction: ruby_text
    field_output: text
    field_system: null
    format: null
    no_input_format: null
    system_format: '{system}'
    system_prompt: ''
early_stopping_patience: null
evals_per_epoch: 4
gradient_accumulation_steps: 1
group_by_length: false
hub_model_id: FatCat87/taopanda-2_dupa-test
learning_rate: 1.0e-05
load_in_8bit: true
local_rank: null
logging_steps: 1
lora_alpha: 32
lora_dropout: 0.05
lora_fan_in_fan_out: true
lora_model_dir: null
lora_r: 16
lora_target_linear: null
lora_target_modules:
- query_key_value
micro_batch_size: 4
num_epochs: 4
output_dir: ./outputs/lora-alpaca-pythia
resume_from_checkpoint: null
seed: 25033
sequence_len: 512
special_tokens:
  pad_token: <|endoftext|>
tf32: true
train_on_inputs: false
val_set_size: 0.05
wandb_entity: null
wandb_log_model: null
wandb_name: dupa-test
wandb_project: null
wandb_runid: dupa-test
wandb_watch: null
weight_decay: 0.1

taopanda-2_dupa-test

This model is a fine-tuned version of EleutherAI/pythia-70m-deduped on the None dataset. It achieves the following results on the evaluation set:

• Loss: 5.8658

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 25033
• distributed_type: multi-GPU
• num_devices: 2
• total_train_batch_size: 8
• total_eval_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100
• num_epochs: 4

Training results

Training Loss	Epoch	Step	Validation Loss
53.0064	0.0006	1	43.2696
38.4663	0.25	407	41.7681
19.4869	0.5	814	19.0612
15.4246	0.75	1221	13.0663
10.42	1.0	1628	10.1860
10.0149	1.25	2035	8.4518
6.641	1.5	2442	7.6670
5.975	1.75	2849	7.0253
7.4982	2.0	3256	6.7420
5.6831	2.25	3663	6.4660
6.9305	2.5	4070	6.3349
8.4203	2.75	4477	6.1577
5.3241	3.0	4884	5.9236
5.6371	3.25	5291	5.9630
6.1296	3.5	5698	5.9355
4.9926	3.75	6105	5.9176
5.5381	4.0	6512	5.8658

Framework versions

• PEFT 0.11.1
• Transformers 4.42.3
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1