Hugging Face's logo

meng-lab
/
llama_3.1_8b_instruct_paradec_humaneval_medusa

Safetensors
alignment-handbook
Generated from Trainer
Model card Files Community
llama_3.1_8b_instruct_paradec_humaneval_medusa / README.md
weizhepei's picture
weizhepei
upload
a88f295
preview code
|
raw
history blame contribute delete
4.13 kB
metadata
base_model: meta-llama/Llama-3.1-8B-Instruct
tags:
  - alignment-handbook
  - generated_from_trainer
datasets:
  - meng-lab/Llama-3.1-8B-Instruct-humaneval
model-index:
  - name: Llama-3.1-8B-Instruct-sft-5e-3-epoch-100-humaneval-stage-3
    results: []

Visualize in Weights & Biases

Llama-3.1-8B-Instruct-sft-5e-3-epoch-100-humaneval-stage-3

This model is a fine-tuned version of /home/jovyan/workspace/PipeDec/checkpoint/Llama-3.1-8B-Instruct-sft-5e-3-epoch-100-humaneval-stage-2 on the meng-lab/Llama-3.1-8B-Instruct-humaneval dataset. It achieves the following results on the evaluation set:

  • Loss: 11.0822
  • Loss Three Hop Layer 8 Head: 3.3949
  • Loss Three Hop Layer 16 Head: 2.9406
  • Loss Three Hop Layer 24 Head: 2.6311
  • Loss Three Hop Layer 32 Head: 2.4800

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.005
  • train_batch_size: 1
  • eval_batch_size: 2
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 8
  • gradient_accumulation_steps: 16
  • total_train_batch_size: 128
  • total_eval_batch_size: 16
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 100

Training results

Training Loss Epoch Step Validation Loss Loss Three Hop Layer 8 Head Loss Three Hop Layer 16 Head Loss Three Hop Layer 24 Head Loss Three Hop Layer 32 Head
17.1204 9.6677 200 17.4024 4.0165 3.6938 5.0117 5.0980
11.7831 19.3353 400 12.4640 3.8214 3.1283 2.8095 3.0563
11.1082 29.0030 600 12.3118 3.4779 3.1955 2.9590 2.9857
10.9205 38.6707 800 11.9277 3.7709 3.0051 2.8893 2.6454
10.1281 48.3384 1000 11.6923 3.4574 2.9719 2.8398 2.7656
9.4147 58.0060 1200 11.2543 3.4058 2.9891 2.6593 2.5635
8.9315 67.6737 1400 11.0952 3.3972 2.9370 2.6327 2.4895
8.9092 77.3414 1600 11.1042 3.4010 2.9454 2.6344 2.4875
8.8371 87.0091 1800 11.0849 3.3957 2.9410 2.6311 2.4803
8.8213 96.6767 2000 11.0822 3.3949 2.9406 2.6311 2.4800

Framework versions

  • Transformers 4.43.2
  • Pytorch 2.6.0+cu124
  • Datasets 3.2.0
  • Tokenizers 0.19.1