2024-05-02 11:45:56,095 | DEBUG [axolotl.normalize_config:79] bf16 support detected, enabling for this configuration.
2024-05-02 11:45:56,096 | INFO [transformers.configuration_utils._get_config_dict:724] loading configuration file /data/model/Llama-3-8b/config.json
2024-05-02 11:45:56,096 | INFO [transformers.configuration_utils.from_dict:789] Model config LlamaConfig {
  "_name_or_path": "/data/model/Llama-3-8b",
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 128000,
  "eos_token_id": 128001,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 14336,
  "max_position_embeddings": 8192,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 8,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": null,
  "rope_theta": 500000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.40.1",
  "use_cache": true,
  "vocab_size": 128256
}

2024-05-02 11:45:56,097 | INFO [axolotl.normalize_config:182] GPU memory usage baseline: 0.000GB (+0.549GB misc)
2024-05-02 11:45:56,955 | INFO [transformers.configuration_utils._get_config_dict:724] loading configuration file /data/model/Llama-3-8b/config.json
2024-05-02 11:45:56,956 | INFO [transformers.configuration_utils.from_dict:789] Model config LlamaConfig {
  "_name_or_path": "/data/model/Llama-3-8b",
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 128000,
  "eos_token_id": 128001,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 14336,
  "max_position_embeddings": 8192,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 8,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": null,
  "rope_theta": 500000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.40.1",
  "use_cache": true,
  "vocab_size": 128256
}

2024-05-02 11:45:56,957 | INFO [transformers.tokenization_utils_base.from_pretrained:2085] loading file tokenizer.json
2024-05-02 11:45:56,957 | INFO [transformers.tokenization_utils_base.from_pretrained:2085] loading file added_tokens.json
2024-05-02 11:45:56,957 | INFO [transformers.tokenization_utils_base.from_pretrained:2085] loading file special_tokens_map.json
2024-05-02 11:45:56,957 | INFO [transformers.tokenization_utils_base.from_pretrained:2085] loading file tokenizer_config.json
2024-05-02 11:45:57,206 | WARNING [transformers.tokenization_utils_base.warning_advice:314] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-05-02 11:45:57,217 | DEBUG [axolotl.load_tokenizer:279] EOS: 128256 / <|im_end|>
2024-05-02 11:45:57,217 | DEBUG [axolotl.load_tokenizer:280] BOS: 128000 / <|begin_of_text|>
2024-05-02 11:45:57,217 | DEBUG [axolotl.load_tokenizer:281] PAD: 128001 / <|end_of_text|>
2024-05-02 11:45:57,217 | DEBUG [axolotl.load_tokenizer:282] UNK: None / None
2024-05-02 11:45:57,217 | INFO [axolotl.load_tokenizer:293] No Chat template selected. Consider adding a chat template for easier inference.
2024-05-02 11:45:57,218 | INFO [axolotl.load_tokenized_prepared_datasets:179] Loading prepared dataset from disk at /data/tmp/6cfba792f99529f2f6dcd822d7aa03a3...
2024-05-02 11:45:57,234 | WARNING [transformers.tokenization_utils_base.warning_advice:314] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-05-02 11:45:57,249 | WARNING [transformers.tokenization_utils_base.warning_advice:314] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-05-02 11:45:57,278 | INFO [axolotl.load_tokenized_prepared_datasets:181] Prepared dataset loaded from disk...
2024-05-02 11:45:57,287 | WARNING [transformers.tokenization_utils_base.warning_advice:314] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-05-02 11:45:57,289 | WARNING [transformers.tokenization_utils_base.warning_advice:314] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-05-02 11:45:57,299 | WARNING [transformers.tokenization_utils_base.warning_advice:314] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-05-02 11:46:05,941 | DEBUG [axolotl.log:61] total_num_tokens: 1_164_516_699
2024-05-02 11:46:22,526 | DEBUG [axolotl.log:61] `total_supervised_tokens: 703_439_150`
2024-05-02 11:46:27,222 | INFO [axolotl.utils.samplers.multipack._len_est:184] packing_efficiency_estimate: 1.0 total_num_tokens per device: 194086116
2024-05-02 11:46:27,222 | DEBUG [axolotl.log:61] data_loader_len: 366
2024-05-02 11:46:27,830 | INFO [axolotl.log:61] sample_packing_eff_est across ranks: [0.9735434055328369, 0.9736768007278442, 0.9736634492874146, 0.9736634492874146, 0.9735967516899109, 0.9736234545707703]
2024-05-02 11:46:27,831 | DEBUG [axolotl.log:61] sample_packing_eff_est: 0.98
2024-05-02 11:46:27,831 | DEBUG [axolotl.log:61] total_num_steps: 122
2024-05-02 11:46:27,855 | DEBUG [axolotl.train.log:61] loading tokenizer... /data/model/Llama-3-8b
2024-05-02 11:46:27,856 | INFO [transformers.configuration_utils._get_config_dict:724] loading configuration file /data/model/Llama-3-8b/config.json
2024-05-02 11:46:27,856 | INFO [transformers.configuration_utils.from_dict:789] Model config LlamaConfig {
  "_name_or_path": "/data/model/Llama-3-8b",
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 128000,
  "eos_token_id": 128001,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 14336,
  "max_position_embeddings": 8192,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 8,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": null,
  "rope_theta": 500000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.40.1",
  "use_cache": true,
  "vocab_size": 128256
}

2024-05-02 11:46:27,857 | INFO [transformers.tokenization_utils_base.from_pretrained:2085] loading file tokenizer.json
2024-05-02 11:46:27,857 | INFO [transformers.tokenization_utils_base.from_pretrained:2085] loading file added_tokens.json
2024-05-02 11:46:27,857 | INFO [transformers.tokenization_utils_base.from_pretrained:2085] loading file special_tokens_map.json
2024-05-02 11:46:27,857 | INFO [transformers.tokenization_utils_base.from_pretrained:2085] loading file tokenizer_config.json
2024-05-02 11:46:28,071 | WARNING [transformers.tokenization_utils_base.warning_advice:314] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-05-02 11:46:28,072 | DEBUG [axolotl.load_tokenizer:279] EOS: 128256 / <|im_end|>
2024-05-02 11:46:28,072 | DEBUG [axolotl.load_tokenizer:280] BOS: 128000 / <|begin_of_text|>
2024-05-02 11:46:28,072 | DEBUG [axolotl.load_tokenizer:281] PAD: 128001 / <|end_of_text|>
2024-05-02 11:46:28,072 | DEBUG [axolotl.load_tokenizer:282] UNK: None / None
2024-05-02 11:46:28,072 | INFO [axolotl.load_tokenizer:293] No Chat template selected. Consider adding a chat template for easier inference.
2024-05-02 11:46:28,072 | DEBUG [axolotl.train.log:61] loading model
2024-05-02 11:46:28,076 | INFO [transformers.configuration_utils._get_config_dict:724] loading configuration file /data/model/Llama-3-8b/config.json
2024-05-02 11:46:28,076 | INFO [transformers.configuration_utils.from_dict:789] Model config LlamaConfig {
  "_name_or_path": "/data/model/Llama-3-8b",
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 128000,
  "eos_token_id": 128001,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 14336,
  "max_position_embeddings": 8192,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 8,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": null,
  "rope_theta": 500000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.40.1",
  "use_cache": true,
  "vocab_size": 128256
}

2024-05-02 11:46:28,079 | WARNING [transformers.tokenization_utils_base.warning_advice:314] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-05-02 11:46:28,088 | WARNING [transformers.tokenization_utils_base.warning_advice:314] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-05-02 11:46:28,088 | WARNING [transformers.tokenization_utils_base.warning_advice:314] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-05-02 11:46:28,088 | WARNING [transformers.tokenization_utils_base.warning_advice:314] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-05-02 11:46:28,095 | INFO [axolotl.load_model:359] patching with flash attention for sample packing
2024-05-02 11:46:28,095 | INFO [axolotl.load_model:408] patching _expand_mask
2024-05-02 11:46:28,095 | INFO [transformers.modeling_utils.from_pretrained:3427] loading weights file /data/model/Llama-3-8b/model.safetensors.index.json
2024-05-02 11:46:28,095 | INFO [transformers.modeling_utils._set_default_torch_dtype:1495] Instantiating LlamaForCausalLM model under default dtype torch.bfloat16.
2024-05-02 11:46:28,096 | INFO [transformers.generation.configuration_utils.from_dict:928] Generate config GenerationConfig {
  "bos_token_id": 128000,
  "eos_token_id": 128001
}

2024-05-02 11:46:28,098 | WARNING [transformers.tokenization_utils_base.warning_advice:314] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-05-02 11:46:32,505 | INFO [transformers.modeling_utils._load_pretrained_model:4171] All model checkpoint weights were used when initializing LlamaForCausalLM.

2024-05-02 11:46:32,505 | INFO [transformers.modeling_utils._load_pretrained_model:4179] All the weights of LlamaForCausalLM were initialized from the model checkpoint at /data/model/Llama-3-8b.
If your task is similar to the task the model of the checkpoint was trained on, you can already use LlamaForCausalLM for predictions without further training.
2024-05-02 11:46:32,507 | INFO [transformers.generation.configuration_utils.from_pretrained:881] loading configuration file /data/model/Llama-3-8b/generation_config.json
2024-05-02 11:46:32,507 | INFO [transformers.generation.configuration_utils.from_dict:928] Generate config GenerationConfig {
  "bos_token_id": 128000,
  "eos_token_id": [
    128001,
    128009
  ]
}

2024-05-02 11:46:32,553 | INFO [transformers.modeling_utils._get_resized_embeddings:1995] You are resizing the embedding layer without providing a `pad_to_multiple_of` parameter. This means that the new embedding dimension will be 128258. This might induce some performance reduction as *Tensor Cores* will not be available. For more details about this, or help on choosing the correct value for resizing, refer to this guide: https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc
2024-05-02 11:46:32,559 | INFO [axolotl.load_model:728] GPU memory usage after model load: 15.083GB (+1.962GB cache, +3.402GB misc)
2024-05-02 11:46:32,563 | INFO [axolotl.load_model:788] converting modules to torch.bfloat16 for flash attention
2024-05-02 11:46:32,565 | DEBUG [axolotl.load_model:819] [('model.embed_tokens.weight', torch.bfloat16), ('model.layers.0.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.0.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.0.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.0.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.0.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.0.mlp.up_proj.weight', torch.bfloat16), ('model.layers.0.mlp.down_proj.weight', torch.bfloat16), ('model.layers.0.input_layernorm.weight', torch.bfloat16), ('model.layers.0.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.1.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.1.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.1.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.1.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.1.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.1.mlp.up_proj.weight', torch.bfloat16), ('model.layers.1.mlp.down_proj.weight', torch.bfloat16), ('model.layers.1.input_layernorm.weight', torch.bfloat16), ('model.layers.1.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.2.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.2.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.2.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.2.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.2.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.2.mlp.up_proj.weight', torch.bfloat16), ('model.layers.2.mlp.down_proj.weight', torch.bfloat16), ('model.layers.2.input_layernorm.weight', torch.bfloat16), ('model.layers.2.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.3.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.3.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.3.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.3.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.3.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.3.mlp.up_proj.weight', torch.bfloat16), ('model.layers.3.mlp.down_proj.weight', torch.bfloat16), ('model.layers.3.input_layernorm.weight', torch.bfloat16), ('model.layers.3.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.4.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.4.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.4.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.4.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.4.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.4.mlp.up_proj.weight', torch.bfloat16), ('model.layers.4.mlp.down_proj.weight', torch.bfloat16), ('model.layers.4.input_layernorm.weight', torch.bfloat16), ('model.layers.4.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.5.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.5.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.5.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.5.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.5.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.5.mlp.up_proj.weight', torch.bfloat16), ('model.layers.5.mlp.down_proj.weight', torch.bfloat16), ('model.layers.5.input_layernorm.weight', torch.bfloat16), ('model.layers.5.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.6.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.6.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.6.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.6.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.6.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.6.mlp.up_proj.weight', torch.bfloat16), ('model.layers.6.mlp.down_proj.weight', torch.bfloat16), ('model.layers.6.input_layernorm.weight', torch.bfloat16), ('model.layers.6.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.7.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.7.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.7.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.7.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.7.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.7.mlp.up_proj.weight', torch.bfloat16), ('model.layers.7.mlp.down_proj.weight', torch.bfloat16), ('model.layers.7.input_layernorm.weight', torch.bfloat16), ('model.layers.7.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.8.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.8.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.8.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.8.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.8.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.8.mlp.up_proj.weight', torch.bfloat16), ('model.layers.8.mlp.down_proj.weight', torch.bfloat16), ('model.layers.8.input_layernorm.weight', torch.bfloat16), ('model.layers.8.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.9.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.9.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.9.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.9.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.9.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.9.mlp.up_proj.weight', torch.bfloat16), ('model.layers.9.mlp.down_proj.weight', torch.bfloat16), ('model.layers.9.input_layernorm.weight', torch.bfloat16), ('model.layers.9.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.10.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.10.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.10.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.10.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.10.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.10.mlp.up_proj.weight', torch.bfloat16), ('model.layers.10.mlp.down_proj.weight', torch.bfloat16), ('model.layers.10.input_layernorm.weight', torch.bfloat16), ('model.layers.10.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.11.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.11.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.11.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.11.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.11.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.11.mlp.up_proj.weight', torch.bfloat16), ('model.layers.11.mlp.down_proj.weight', torch.bfloat16), ('model.layers.11.input_layernorm.weight', torch.bfloat16), ('model.layers.11.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.12.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.12.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.12.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.12.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.12.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.12.mlp.up_proj.weight', torch.bfloat16), ('model.layers.12.mlp.down_proj.weight', torch.bfloat16), ('model.layers.12.input_layernorm.weight', torch.bfloat16), ('model.layers.12.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.13.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.13.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.13.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.13.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.13.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.13.mlp.up_proj.weight', torch.bfloat16), ('model.layers.13.mlp.down_proj.weight', torch.bfloat16), ('model.layers.13.input_layernorm.weight', torch.bfloat16), ('model.layers.13.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.14.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.14.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.14.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.14.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.14.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.14.mlp.up_proj.weight', torch.bfloat16), ('model.layers.14.mlp.down_proj.weight', torch.bfloat16), ('model.layers.14.input_layernorm.weight', torch.bfloat16), ('model.layers.14.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.15.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.15.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.15.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.15.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.15.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.15.mlp.up_proj.weight', torch.bfloat16), ('model.layers.15.mlp.down_proj.weight', torch.bfloat16), ('model.layers.15.input_layernorm.weight', torch.bfloat16), ('model.layers.15.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.16.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.16.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.16.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.16.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.16.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.16.mlp.up_proj.weight', torch.bfloat16), ('model.layers.16.mlp.down_proj.weight', torch.bfloat16), ('model.layers.16.input_layernorm.weight', torch.bfloat16), ('model.layers.16.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.17.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.17.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.17.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.17.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.17.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.17.mlp.up_proj.weight', torch.bfloat16), ('model.layers.17.mlp.down_proj.weight', torch.bfloat16), ('model.layers.17.input_layernorm.weight', torch.bfloat16), ('model.layers.17.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.18.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.18.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.18.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.18.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.18.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.18.mlp.up_proj.weight', torch.bfloat16), ('model.layers.18.mlp.down_proj.weight', torch.bfloat16), ('model.layers.18.input_layernorm.weight', torch.bfloat16), ('model.layers.18.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.19.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.19.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.19.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.19.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.19.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.19.mlp.up_proj.weight', torch.bfloat16), ('model.layers.19.mlp.down_proj.weight', torch.bfloat16), ('model.layers.19.input_layernorm.weight', torch.bfloat16), ('model.layers.19.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.20.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.20.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.20.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.20.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.20.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.20.mlp.up_proj.weight', torch.bfloat16), ('model.layers.20.mlp.down_proj.weight', torch.bfloat16), ('model.layers.20.input_layernorm.weight', torch.bfloat16), ('model.layers.20.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.21.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.21.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.21.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.21.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.21.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.21.mlp.up_proj.weight', torch.bfloat16), ('model.layers.21.mlp.down_proj.weight', torch.bfloat16), ('model.layers.21.input_layernorm.weight', torch.bfloat16), ('model.layers.21.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.22.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.22.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.22.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.22.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.22.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.22.mlp.up_proj.weight', torch.bfloat16), ('model.layers.22.mlp.down_proj.weight', torch.bfloat16), ('model.layers.22.input_layernorm.weight', torch.bfloat16), ('model.layers.22.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.23.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.23.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.23.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.23.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.23.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.23.mlp.up_proj.weight', torch.bfloat16), ('model.layers.23.mlp.down_proj.weight', torch.bfloat16), ('model.layers.23.input_layernorm.weight', torch.bfloat16), ('model.layers.23.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.24.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.24.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.24.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.24.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.24.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.24.mlp.up_proj.weight', torch.bfloat16), ('model.layers.24.mlp.down_proj.weight', torch.bfloat16), ('model.layers.24.input_layernorm.weight', torch.bfloat16), ('model.layers.24.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.25.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.25.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.25.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.25.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.25.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.25.mlp.up_proj.weight', torch.bfloat16), ('model.layers.25.mlp.down_proj.weight', torch.bfloat16), ('model.layers.25.input_layernorm.weight', torch.bfloat16), ('model.layers.25.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.26.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.26.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.26.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.26.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.26.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.26.mlp.up_proj.weight', torch.bfloat16), ('model.layers.26.mlp.down_proj.weight', torch.bfloat16), ('model.layers.26.input_layernorm.weight', torch.bfloat16), ('model.layers.26.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.27.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.27.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.27.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.27.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.27.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.27.mlp.up_proj.weight', torch.bfloat16), ('model.layers.27.mlp.down_proj.weight', torch.bfloat16), ('model.layers.27.input_layernorm.weight', torch.bfloat16), ('model.layers.27.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.28.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.28.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.28.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.28.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.28.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.28.mlp.up_proj.weight', torch.bfloat16), ('model.layers.28.mlp.down_proj.weight', torch.bfloat16), ('model.layers.28.input_layernorm.weight', torch.bfloat16), ('model.layers.28.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.29.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.29.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.29.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.29.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.29.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.29.mlp.up_proj.weight', torch.bfloat16), ('model.layers.29.mlp.down_proj.weight', torch.bfloat16), ('model.layers.29.input_layernorm.weight', torch.bfloat16), ('model.layers.29.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.30.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.30.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.30.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.30.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.30.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.30.mlp.up_proj.weight', torch.bfloat16), ('model.layers.30.mlp.down_proj.weight', torch.bfloat16), ('model.layers.30.input_layernorm.weight', torch.bfloat16), ('model.layers.30.post_attention_layernorm.weight', torch.bfloat16), ('model.layers.31.self_attn.q_proj.weight', torch.bfloat16), ('model.layers.31.self_attn.k_proj.weight', torch.bfloat16), ('model.layers.31.self_attn.v_proj.weight', torch.bfloat16), ('model.layers.31.self_attn.o_proj.weight', torch.bfloat16), ('model.layers.31.mlp.gate_proj.weight', torch.bfloat16), ('model.layers.31.mlp.up_proj.weight', torch.bfloat16), ('model.layers.31.mlp.down_proj.weight', torch.bfloat16), ('model.layers.31.input_layernorm.weight', torch.bfloat16), ('model.layers.31.post_attention_layernorm.weight', torch.bfloat16), ('model.norm.weight', torch.bfloat16), ('lm_head.weight', torch.bfloat16)]
2024-05-02 11:46:32,566 | DEBUG [axolotl.load_model:820] ['model.embed_tokens.weight', 'model.layers.0.self_attn.q_proj.weight', 'model.layers.0.self_attn.k_proj.weight', 'model.layers.0.self_attn.v_proj.weight', 'model.layers.0.self_attn.o_proj.weight', 'model.layers.0.mlp.gate_proj.weight', 'model.layers.0.mlp.up_proj.weight', 'model.layers.0.mlp.down_proj.weight', 'model.layers.0.input_layernorm.weight', 'model.layers.0.post_attention_layernorm.weight', 'model.layers.1.self_attn.q_proj.weight', 'model.layers.1.self_attn.k_proj.weight', 'model.layers.1.self_attn.v_proj.weight', 'model.layers.1.self_attn.o_proj.weight', 'model.layers.1.mlp.gate_proj.weight', 'model.layers.1.mlp.up_proj.weight', 'model.layers.1.mlp.down_proj.weight', 'model.layers.1.input_layernorm.weight', 'model.layers.1.post_attention_layernorm.weight', 'model.layers.2.self_attn.q_proj.weight', 'model.layers.2.self_attn.k_proj.weight', 'model.layers.2.self_attn.v_proj.weight', 'model.layers.2.self_attn.o_proj.weight', 'model.layers.2.mlp.gate_proj.weight', 'model.layers.2.mlp.up_proj.weight', 'model.layers.2.mlp.down_proj.weight', 'model.layers.2.input_layernorm.weight', 'model.layers.2.post_attention_layernorm.weight', 'model.layers.3.self_attn.q_proj.weight', 'model.layers.3.self_attn.k_proj.weight', 'model.layers.3.self_attn.v_proj.weight', 'model.layers.3.self_attn.o_proj.weight', 'model.layers.3.mlp.gate_proj.weight', 'model.layers.3.mlp.up_proj.weight', 'model.layers.3.mlp.down_proj.weight', 'model.layers.3.input_layernorm.weight', 'model.layers.3.post_attention_layernorm.weight', 'model.layers.4.self_attn.q_proj.weight', 'model.layers.4.self_attn.k_proj.weight', 'model.layers.4.self_attn.v_proj.weight', 'model.layers.4.self_attn.o_proj.weight', 'model.layers.4.mlp.gate_proj.weight', 'model.layers.4.mlp.up_proj.weight', 'model.layers.4.mlp.down_proj.weight', 'model.layers.4.input_layernorm.weight', 'model.layers.4.post_attention_layernorm.weight', 'model.layers.5.self_attn.q_proj.weight', 'model.layers.5.self_attn.k_proj.weight', 'model.layers.5.self_attn.v_proj.weight', 'model.layers.5.self_attn.o_proj.weight', 'model.layers.5.mlp.gate_proj.weight', 'model.layers.5.mlp.up_proj.weight', 'model.layers.5.mlp.down_proj.weight', 'model.layers.5.input_layernorm.weight', 'model.layers.5.post_attention_layernorm.weight', 'model.layers.6.self_attn.q_proj.weight', 'model.layers.6.self_attn.k_proj.weight', 'model.layers.6.self_attn.v_proj.weight', 'model.layers.6.self_attn.o_proj.weight', 'model.layers.6.mlp.gate_proj.weight', 'model.layers.6.mlp.up_proj.weight', 'model.layers.6.mlp.down_proj.weight', 'model.layers.6.input_layernorm.weight', 'model.layers.6.post_attention_layernorm.weight', 'model.layers.7.self_attn.q_proj.weight', 'model.layers.7.self_attn.k_proj.weight', 'model.layers.7.self_attn.v_proj.weight', 'model.layers.7.self_attn.o_proj.weight', 'model.layers.7.mlp.gate_proj.weight', 'model.layers.7.mlp.up_proj.weight', 'model.layers.7.mlp.down_proj.weight', 'model.layers.7.input_layernorm.weight', 'model.layers.7.post_attention_layernorm.weight', 'model.layers.8.self_attn.q_proj.weight', 'model.layers.8.self_attn.k_proj.weight', 'model.layers.8.self_attn.v_proj.weight', 'model.layers.8.self_attn.o_proj.weight', 'model.layers.8.mlp.gate_proj.weight', 'model.layers.8.mlp.up_proj.weight', 'model.layers.8.mlp.down_proj.weight', 'model.layers.8.input_layernorm.weight', 'model.layers.8.post_attention_layernorm.weight', 'model.layers.9.self_attn.q_proj.weight', 'model.layers.9.self_attn.k_proj.weight', 'model.layers.9.self_attn.v_proj.weight', 'model.layers.9.self_attn.o_proj.weight', 'model.layers.9.mlp.gate_proj.weight', 'model.layers.9.mlp.up_proj.weight', 'model.layers.9.mlp.down_proj.weight', 'model.layers.9.input_layernorm.weight', 'model.layers.9.post_attention_layernorm.weight', 'model.layers.10.self_attn.q_proj.weight', 'model.layers.10.self_attn.k_proj.weight', 'model.layers.10.self_attn.v_proj.weight', 'model.layers.10.self_attn.o_proj.weight', 'model.layers.10.mlp.gate_proj.weight', 'model.layers.10.mlp.up_proj.weight', 'model.layers.10.mlp.down_proj.weight', 'model.layers.10.input_layernorm.weight', 'model.layers.10.post_attention_layernorm.weight', 'model.layers.11.self_attn.q_proj.weight', 'model.layers.11.self_attn.k_proj.weight', 'model.layers.11.self_attn.v_proj.weight', 'model.layers.11.self_attn.o_proj.weight', 'model.layers.11.mlp.gate_proj.weight', 'model.layers.11.mlp.up_proj.weight', 'model.layers.11.mlp.down_proj.weight', 'model.layers.11.input_layernorm.weight', 'model.layers.11.post_attention_layernorm.weight', 'model.layers.12.self_attn.q_proj.weight', 'model.layers.12.self_attn.k_proj.weight', 'model.layers.12.self_attn.v_proj.weight', 'model.layers.12.self_attn.o_proj.weight', 'model.layers.12.mlp.gate_proj.weight', 'model.layers.12.mlp.up_proj.weight', 'model.layers.12.mlp.down_proj.weight', 'model.layers.12.input_layernorm.weight', 'model.layers.12.post_attention_layernorm.weight', 'model.layers.13.self_attn.q_proj.weight', 'model.layers.13.self_attn.k_proj.weight', 'model.layers.13.self_attn.v_proj.weight', 'model.layers.13.self_attn.o_proj.weight', 'model.layers.13.mlp.gate_proj.weight', 'model.layers.13.mlp.up_proj.weight', 'model.layers.13.mlp.down_proj.weight', 'model.layers.13.input_layernorm.weight', 'model.layers.13.post_attention_layernorm.weight', 'model.layers.14.self_attn.q_proj.weight', 'model.layers.14.self_attn.k_proj.weight', 'model.layers.14.self_attn.v_proj.weight', 'model.layers.14.self_attn.o_proj.weight', 'model.layers.14.mlp.gate_proj.weight', 'model.layers.14.mlp.up_proj.weight', 'model.layers.14.mlp.down_proj.weight', 'model.layers.14.input_layernorm.weight', 'model.layers.14.post_attention_layernorm.weight', 'model.layers.15.self_attn.q_proj.weight', 'model.layers.15.self_attn.k_proj.weight', 'model.layers.15.self_attn.v_proj.weight', 'model.layers.15.self_attn.o_proj.weight', 'model.layers.15.mlp.gate_proj.weight', 'model.layers.15.mlp.up_proj.weight', 'model.layers.15.mlp.down_proj.weight', 'model.layers.15.input_layernorm.weight', 'model.layers.15.post_attention_layernorm.weight', 'model.layers.16.self_attn.q_proj.weight', 'model.layers.16.self_attn.k_proj.weight', 'model.layers.16.self_attn.v_proj.weight', 'model.layers.16.self_attn.o_proj.weight', 'model.layers.16.mlp.gate_proj.weight', 'model.layers.16.mlp.up_proj.weight', 'model.layers.16.mlp.down_proj.weight', 'model.layers.16.input_layernorm.weight', 'model.layers.16.post_attention_layernorm.weight', 'model.layers.17.self_attn.q_proj.weight', 'model.layers.17.self_attn.k_proj.weight', 'model.layers.17.self_attn.v_proj.weight', 'model.layers.17.self_attn.o_proj.weight', 'model.layers.17.mlp.gate_proj.weight', 'model.layers.17.mlp.up_proj.weight', 'model.layers.17.mlp.down_proj.weight', 'model.layers.17.input_layernorm.weight', 'model.layers.17.post_attention_layernorm.weight', 'model.layers.18.self_attn.q_proj.weight', 'model.layers.18.self_attn.k_proj.weight', 'model.layers.18.self_attn.v_proj.weight', 'model.layers.18.self_attn.o_proj.weight', 'model.layers.18.mlp.gate_proj.weight', 'model.layers.18.mlp.up_proj.weight', 'model.layers.18.mlp.down_proj.weight', 'model.layers.18.input_layernorm.weight', 'model.layers.18.post_attention_layernorm.weight', 'model.layers.19.self_attn.q_proj.weight', 'model.layers.19.self_attn.k_proj.weight', 'model.layers.19.self_attn.v_proj.weight', 'model.layers.19.self_attn.o_proj.weight', 'model.layers.19.mlp.gate_proj.weight', 'model.layers.19.mlp.up_proj.weight', 'model.layers.19.mlp.down_proj.weight', 'model.layers.19.input_layernorm.weight', 'model.layers.19.post_attention_layernorm.weight', 'model.layers.20.self_attn.q_proj.weight', 'model.layers.20.self_attn.k_proj.weight', 'model.layers.20.self_attn.v_proj.weight', 'model.layers.20.self_attn.o_proj.weight', 'model.layers.20.mlp.gate_proj.weight', 'model.layers.20.mlp.up_proj.weight', 'model.layers.20.mlp.down_proj.weight', 'model.layers.20.input_layernorm.weight', 'model.layers.20.post_attention_layernorm.weight', 'model.layers.21.self_attn.q_proj.weight', 'model.layers.21.self_attn.k_proj.weight', 'model.layers.21.self_attn.v_proj.weight', 'model.layers.21.self_attn.o_proj.weight', 'model.layers.21.mlp.gate_proj.weight', 'model.layers.21.mlp.up_proj.weight', 'model.layers.21.mlp.down_proj.weight', 'model.layers.21.input_layernorm.weight', 'model.layers.21.post_attention_layernorm.weight', 'model.layers.22.self_attn.q_proj.weight', 'model.layers.22.self_attn.k_proj.weight', 'model.layers.22.self_attn.v_proj.weight', 'model.layers.22.self_attn.o_proj.weight', 'model.layers.22.mlp.gate_proj.weight', 'model.layers.22.mlp.up_proj.weight', 'model.layers.22.mlp.down_proj.weight', 'model.layers.22.input_layernorm.weight', 'model.layers.22.post_attention_layernorm.weight', 'model.layers.23.self_attn.q_proj.weight', 'model.layers.23.self_attn.k_proj.weight', 'model.layers.23.self_attn.v_proj.weight', 'model.layers.23.self_attn.o_proj.weight', 'model.layers.23.mlp.gate_proj.weight', 'model.layers.23.mlp.up_proj.weight', 'model.layers.23.mlp.down_proj.weight', 'model.layers.23.input_layernorm.weight', 'model.layers.23.post_attention_layernorm.weight', 'model.layers.24.self_attn.q_proj.weight', 'model.layers.24.self_attn.k_proj.weight', 'model.layers.24.self_attn.v_proj.weight', 'model.layers.24.self_attn.o_proj.weight', 'model.layers.24.mlp.gate_proj.weight', 'model.layers.24.mlp.up_proj.weight', 'model.layers.24.mlp.down_proj.weight', 'model.layers.24.input_layernorm.weight', 'model.layers.24.post_attention_layernorm.weight', 'model.layers.25.self_attn.q_proj.weight', 'model.layers.25.self_attn.k_proj.weight', 'model.layers.25.self_attn.v_proj.weight', 'model.layers.25.self_attn.o_proj.weight', 'model.layers.25.mlp.gate_proj.weight', 'model.layers.25.mlp.up_proj.weight', 'model.layers.25.mlp.down_proj.weight', 'model.layers.25.input_layernorm.weight', 'model.layers.25.post_attention_layernorm.weight', 'model.layers.26.self_attn.q_proj.weight', 'model.layers.26.self_attn.k_proj.weight', 'model.layers.26.self_attn.v_proj.weight', 'model.layers.26.self_attn.o_proj.weight', 'model.layers.26.mlp.gate_proj.weight', 'model.layers.26.mlp.up_proj.weight', 'model.layers.26.mlp.down_proj.weight', 'model.layers.26.input_layernorm.weight', 'model.layers.26.post_attention_layernorm.weight', 'model.layers.27.self_attn.q_proj.weight', 'model.layers.27.self_attn.k_proj.weight', 'model.layers.27.self_attn.v_proj.weight', 'model.layers.27.self_attn.o_proj.weight', 'model.layers.27.mlp.gate_proj.weight', 'model.layers.27.mlp.up_proj.weight', 'model.layers.27.mlp.down_proj.weight', 'model.layers.27.input_layernorm.weight', 'model.layers.27.post_attention_layernorm.weight', 'model.layers.28.self_attn.q_proj.weight', 'model.layers.28.self_attn.k_proj.weight', 'model.layers.28.self_attn.v_proj.weight', 'model.layers.28.self_attn.o_proj.weight', 'model.layers.28.mlp.gate_proj.weight', 'model.layers.28.mlp.up_proj.weight', 'model.layers.28.mlp.down_proj.weight', 'model.layers.28.input_layernorm.weight', 'model.layers.28.post_attention_layernorm.weight', 'model.layers.29.self_attn.q_proj.weight', 'model.layers.29.self_attn.k_proj.weight', 'model.layers.29.self_attn.v_proj.weight', 'model.layers.29.self_attn.o_proj.weight', 'model.layers.29.mlp.gate_proj.weight', 'model.layers.29.mlp.up_proj.weight', 'model.layers.29.mlp.down_proj.weight', 'model.layers.29.input_layernorm.weight', 'model.layers.29.post_attention_layernorm.weight', 'model.layers.30.self_attn.q_proj.weight', 'model.layers.30.self_attn.k_proj.weight', 'model.layers.30.self_attn.v_proj.weight', 'model.layers.30.self_attn.o_proj.weight', 'model.layers.30.mlp.gate_proj.weight', 'model.layers.30.mlp.up_proj.weight', 'model.layers.30.mlp.down_proj.weight', 'model.layers.30.input_layernorm.weight', 'model.layers.30.post_attention_layernorm.weight', 'model.layers.31.self_attn.q_proj.weight', 'model.layers.31.self_attn.k_proj.weight', 'model.layers.31.self_attn.v_proj.weight', 'model.layers.31.self_attn.o_proj.weight', 'model.layers.31.mlp.gate_proj.weight', 'model.layers.31.mlp.up_proj.weight', 'model.layers.31.mlp.down_proj.weight', 'model.layers.31.input_layernorm.weight', 'model.layers.31.post_attention_layernorm.weight', 'model.norm.weight', 'lm_head.weight']
2024-05-02 11:46:32,569 | INFO [transformers.training_args._setup_devices:1997] PyTorch: setting up devices
2024-05-02 11:46:32,855 | INFO [transformers.trainer.__init__:626] Using auto half precision backend
2024-05-02 11:46:32,858 | INFO [transformers.tokenization_utils_base.save_pretrained:2488] tokenizer config file saved in /data/llama3-20240502-1145/tokenizer_config.json
2024-05-02 11:46:32,858 | INFO [transformers.tokenization_utils_base.save_pretrained:2497] Special tokens file saved in /data/llama3-20240502-1145/special_tokens_map.json
2024-05-02 11:46:32,973 | INFO [transformers.configuration_utils.save_pretrained:471] Configuration saved in /data/llama3-20240502-1145/config.json
2024-05-02 11:46:32,975 | INFO [axolotl.train.log:61] Starting trainer...
2024-05-02 11:46:34,574 | INFO [axolotl.utils.samplers.multipack._len_est:184] packing_efficiency_estimate: 0.98 total_num_tokens per device: 194086116
2024-05-02 11:46:35,882 | INFO [axolotl.utils.samplers.multipack._len_est:184] packing_efficiency_estimate: 0.98 total_num_tokens per device: 194086116
2024-05-02 11:46:37,197 | INFO [axolotl.utils.samplers.multipack._len_est:184] packing_efficiency_estimate: 0.98 total_num_tokens per device: 194086116
2024-05-02 11:46:38,742 | INFO [axolotl.utils.samplers.multipack._len_est:184] packing_efficiency_estimate: 0.98 total_num_tokens per device: 194086116
2024-05-02 11:47:01,151 | INFO [transformers.integrations.deepspeed.deepspeed_load_checkpoint:430] Attempting to resume from /data/llama3-20240502-0354/checkpoint-374/
2024-05-02 11:47:27,406 | WARNING [transformers.trainer.warning_once:329] Warning: The following arguments do not match the ones in the `trainer_state.json` within the checkpoint directory: 
	save_steps: 0.125 (from args) != 374 (from trainer_state.json)
2024-05-02 11:47:27,531 | WARNING [transformers.trainer.warning_once:329] Warning: The following arguments do not match the ones in the `trainer_state.json` within the checkpoint directory: 
	save_steps: 0.125 (from args) != 374 (from trainer_state.json)
2024-05-02 11:47:27,742 | WARNING [transformers.trainer.warning_once:329] Warning: The following arguments do not match the ones in the `trainer_state.json` within the checkpoint directory: 
	save_steps: 0.125 (from args) != 374 (from trainer_state.json)
2024-05-02 11:47:28,329 | WARNING [transformers.trainer.warning_once:329] Warning: The following arguments do not match the ones in the `trainer_state.json` within the checkpoint directory: 
	save_steps: 0.125 (from args) != 374 (from trainer_state.json)
2024-05-02 11:47:28,336 | INFO [transformers.trainer._inner_training_loop:2048] ***** Running training *****
2024-05-02 11:47:28,336 | INFO [transformers.trainer._inner_training_loop:2049]   Num examples = 2,296,433
2024-05-02 11:47:28,337 | INFO [transformers.trainer._inner_training_loop:2050]   Num Epochs = 2
2024-05-02 11:47:28,337 | INFO [transformers.trainer._inner_training_loop:2051]   Instantaneous batch size per device = 4
2024-05-02 11:47:28,337 | INFO [transformers.trainer._inner_training_loop:2054]   Total train batch size (w. parallel, distributed & accumulation) = 192
2024-05-02 11:47:28,337 | INFO [transformers.trainer._inner_training_loop:2055]   Gradient Accumulation steps = 8
2024-05-02 11:47:28,337 | INFO [transformers.trainer._inner_training_loop:2056]   Total optimization steps = 2,990
2024-05-02 11:47:28,338 | INFO [transformers.trainer._inner_training_loop:2057]   Number of trainable parameters = 8,030,277,632
2024-05-02 11:47:28,339 | WARNING [transformers.trainer.warning_once:329] Warning: The following arguments do not match the ones in the `trainer_state.json` within the checkpoint directory: 
	save_steps: 0.125 (from args) != 374 (from trainer_state.json)
2024-05-02 11:47:28,339 | INFO [transformers.trainer._inner_training_loop:2078]   Continuing training from checkpoint, will skip to saved global_step
2024-05-02 11:47:28,339 | INFO [transformers.trainer._inner_training_loop:2079]   Continuing training from epoch 0
2024-05-02 11:47:28,339 | INFO [transformers.trainer._inner_training_loop:2080]   Continuing training from global step 374
2024-05-02 11:47:28,339 | INFO [transformers.trainer._inner_training_loop:2082]   Will skip the first 0 epochs then the first 2992 batches in the first epoch.
2024-05-02 11:47:28,340 | INFO [transformers.integrations.integration_utils.setup:723] Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"
2024-05-02 11:47:28,363 | WARNING [transformers.trainer.warning_once:329] Warning: The following arguments do not match the ones in the `trainer_state.json` within the checkpoint directory: 
	save_steps: 0.125 (from args) != 374 (from trainer_state.json)
2024-05-02 11:47:31,882 | WARNING [transformers.tokenization_utils_base.warning_advice:314] You're using a PreTrainedTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
2024-05-02 11:47:31,940 | WARNING [transformers.tokenization_utils_base.warning_advice:314] You're using a PreTrainedTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
2024-05-02 11:47:32,136 | WARNING [transformers.tokenization_utils_base.warning_advice:314] You're using a PreTrainedTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
2024-05-02 11:47:32,555 | WARNING [transformers.tokenization_utils_base.warning_advice:314] You're using a PreTrainedTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
2024-05-02 11:47:32,634 | WARNING [transformers.tokenization_utils_base.warning_advice:314] You're using a PreTrainedTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
2024-05-02 11:47:33,242 | INFO [axolotl.callbacks.on_train_begin:770] The Axolotl config has been saved to the WandB run under files.
2024-05-02 11:47:34,618 | INFO [axolotl.utils.samplers.multipack._len_est:184] packing_efficiency_estimate: 0.98 total_num_tokens per device: 194086116
2024-05-02 11:47:35,888 | INFO [axolotl.utils.samplers.multipack._len_est:184] packing_efficiency_estimate: 0.98 total_num_tokens per device: 194086116
2024-05-02 11:47:37,139 | WARNING [transformers.tokenization_utils_base.warning_advice:314] You're using a PreTrainedTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
2024-05-02 11:47:58,600 | INFO [axolotl.callbacks.on_step_end:125] GPU memory usage while training: 22.630GB (+41.110GB cache, +4.486GB misc)
2024-05-02 14:01:51,604 | INFO [transformers.trainer._save:3305] Saving model checkpoint to /data/llama3-20240502-1145/checkpoint-748
2024-05-02 14:01:51,606 | INFO [transformers.configuration_utils.save_pretrained:471] Configuration saved in /data/llama3-20240502-1145/checkpoint-748/config.json
2024-05-02 14:01:51,606 | INFO [transformers.generation.configuration_utils.save_pretrained:697] Configuration saved in /data/llama3-20240502-1145/checkpoint-748/generation_config.json
2024-05-02 14:02:09,434 | INFO [transformers.modeling_utils.save_pretrained:2599] The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 4 checkpoint shards. You can find where each parameters has been saved in the index located at /data/llama3-20240502-1145/checkpoint-748/model.safetensors.index.json.
2024-05-02 14:02:09,436 | INFO [transformers.tokenization_utils_base.save_pretrained:2488] tokenizer config file saved in /data/llama3-20240502-1145/checkpoint-748/tokenizer_config.json
2024-05-02 14:02:09,436 | INFO [transformers.tokenization_utils_base.save_pretrained:2497] Special tokens file saved in /data/llama3-20240502-1145/checkpoint-748/special_tokens_map.json
2024-05-02 16:16:45,380 | INFO [transformers.trainer._save:3305] Saving model checkpoint to /data/llama3-20240502-1145/checkpoint-1122
2024-05-02 16:16:45,381 | INFO [transformers.configuration_utils.save_pretrained:471] Configuration saved in /data/llama3-20240502-1145/checkpoint-1122/config.json
2024-05-02 16:16:45,381 | INFO [transformers.generation.configuration_utils.save_pretrained:697] Configuration saved in /data/llama3-20240502-1145/checkpoint-1122/generation_config.json
2024-05-02 16:17:02,798 | INFO [transformers.modeling_utils.save_pretrained:2599] The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 4 checkpoint shards. You can find where each parameters has been saved in the index located at /data/llama3-20240502-1145/checkpoint-1122/model.safetensors.index.json.
2024-05-02 16:17:02,801 | INFO [transformers.tokenization_utils_base.save_pretrained:2488] tokenizer config file saved in /data/llama3-20240502-1145/checkpoint-1122/tokenizer_config.json
2024-05-02 16:17:02,801 | INFO [transformers.tokenization_utils_base.save_pretrained:2497] Special tokens file saved in /data/llama3-20240502-1145/checkpoint-1122/special_tokens_map.json
2024-05-02 18:31:50,550 | INFO [transformers.trainer._save:3305] Saving model checkpoint to /data/llama3-20240502-1145/checkpoint-1496
2024-05-02 18:31:50,551 | INFO [transformers.configuration_utils.save_pretrained:471] Configuration saved in /data/llama3-20240502-1145/checkpoint-1496/config.json
2024-05-02 18:31:50,551 | INFO [transformers.generation.configuration_utils.save_pretrained:697] Configuration saved in /data/llama3-20240502-1145/checkpoint-1496/generation_config.json
2024-05-02 18:32:08,833 | INFO [transformers.modeling_utils.save_pretrained:2599] The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 4 checkpoint shards. You can find where each parameters has been saved in the index located at /data/llama3-20240502-1145/checkpoint-1496/model.safetensors.index.json.
2024-05-02 18:32:08,835 | INFO [transformers.tokenization_utils_base.save_pretrained:2488] tokenizer config file saved in /data/llama3-20240502-1145/checkpoint-1496/tokenizer_config.json
2024-05-02 18:32:08,835 | INFO [transformers.tokenization_utils_base.save_pretrained:2497] Special tokens file saved in /data/llama3-20240502-1145/checkpoint-1496/special_tokens_map.json
2024-05-02 18:41:39,260 | INFO [axolotl.utils.samplers.multipack._len_est:184] packing_efficiency_estimate: 0.98 total_num_tokens per device: 194086116
2024-05-02 18:41:40,581 | INFO [axolotl.utils.samplers.multipack._len_est:184] packing_efficiency_estimate: 0.98 total_num_tokens per device: 194086116
2024-05-02 20:46:55,686 | INFO [transformers.trainer._save:3305] Saving model checkpoint to /data/llama3-20240502-1145/checkpoint-1870
2024-05-02 20:46:55,687 | INFO [transformers.configuration_utils.save_pretrained:471] Configuration saved in /data/llama3-20240502-1145/checkpoint-1870/config.json
2024-05-02 20:46:55,687 | INFO [transformers.generation.configuration_utils.save_pretrained:697] Configuration saved in /data/llama3-20240502-1145/checkpoint-1870/generation_config.json
2024-05-02 20:47:14,819 | INFO [transformers.modeling_utils.save_pretrained:2599] The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 4 checkpoint shards. You can find where each parameters has been saved in the index located at /data/llama3-20240502-1145/checkpoint-1870/model.safetensors.index.json.
2024-05-02 20:47:14,821 | INFO [transformers.tokenization_utils_base.save_pretrained:2488] tokenizer config file saved in /data/llama3-20240502-1145/checkpoint-1870/tokenizer_config.json
2024-05-02 20:47:14,821 | INFO [transformers.tokenization_utils_base.save_pretrained:2497] Special tokens file saved in /data/llama3-20240502-1145/checkpoint-1870/special_tokens_map.json
2024-05-02 23:02:03,882 | INFO [transformers.trainer._save:3305] Saving model checkpoint to /data/llama3-20240502-1145/checkpoint-2244
2024-05-02 23:02:03,883 | INFO [transformers.configuration_utils.save_pretrained:471] Configuration saved in /data/llama3-20240502-1145/checkpoint-2244/config.json
2024-05-02 23:02:03,883 | INFO [transformers.generation.configuration_utils.save_pretrained:697] Configuration saved in /data/llama3-20240502-1145/checkpoint-2244/generation_config.json
2024-05-02 23:02:21,841 | INFO [transformers.modeling_utils.save_pretrained:2599] The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 4 checkpoint shards. You can find where each parameters has been saved in the index located at /data/llama3-20240502-1145/checkpoint-2244/model.safetensors.index.json.
2024-05-02 23:02:21,843 | INFO [transformers.tokenization_utils_base.save_pretrained:2488] tokenizer config file saved in /data/llama3-20240502-1145/checkpoint-2244/tokenizer_config.json
2024-05-02 23:02:21,844 | INFO [transformers.tokenization_utils_base.save_pretrained:2497] Special tokens file saved in /data/llama3-20240502-1145/checkpoint-2244/special_tokens_map.json
2024-05-03 01:17:09,150 | INFO [transformers.trainer._save:3305] Saving model checkpoint to /data/llama3-20240502-1145/checkpoint-2618
2024-05-03 01:17:09,152 | INFO [transformers.configuration_utils.save_pretrained:471] Configuration saved in /data/llama3-20240502-1145/checkpoint-2618/config.json
2024-05-03 01:17:09,152 | INFO [transformers.generation.configuration_utils.save_pretrained:697] Configuration saved in /data/llama3-20240502-1145/checkpoint-2618/generation_config.json
2024-05-03 01:17:27,948 | INFO [transformers.modeling_utils.save_pretrained:2599] The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 4 checkpoint shards. You can find where each parameters has been saved in the index located at /data/llama3-20240502-1145/checkpoint-2618/model.safetensors.index.json.
2024-05-03 01:17:27,950 | INFO [transformers.tokenization_utils_base.save_pretrained:2488] tokenizer config file saved in /data/llama3-20240502-1145/checkpoint-2618/tokenizer_config.json
2024-05-03 01:17:27,950 | INFO [transformers.tokenization_utils_base.save_pretrained:2497] Special tokens file saved in /data/llama3-20240502-1145/checkpoint-2618/special_tokens_map.json
2024-05-03 03:31:21,185 | INFO [transformers.trainer._inner_training_loop:2316] 

Training completed. Do not forget to share your model on huggingface.co/models =)


2024-05-03 03:31:21,193 | INFO [axolotl.train.log:61] Training Completed!!! Saving pre-trained model to /data/llama3-20240502-1145
2024-05-03 03:31:21,195 | INFO [transformers.configuration_utils.save_pretrained:471] Configuration saved in /data/llama3-20240502-1145/config.json
2024-05-03 03:31:21,196 | INFO [transformers.generation.configuration_utils.save_pretrained:697] Configuration saved in /data/llama3-20240502-1145/generation_config.json
2024-05-03 03:31:43,036 | INFO [transformers.modeling_utils.save_pretrained:2591] Model weights saved in /data/llama3-20240502-1145/pytorch_model.bin
2024-05-03 03:31:43,039 | INFO [transformers.modelcard.create_model_index:450] Dropping the following result as it does not have all the necessary fields:
{'task': {'name': 'Causal Language Modeling', 'type': 'text-generation'}}