End of training

README.md

@@ -0,0 +1,165 @@
+---
+library_name: peft
+base_model: lcw99/zephykor-ko-7b-chang
+tags:
+- axolotl
+- generated_from_trainer
+model-index:
+- name: 35807913-0500-4394-b20d-9e17a46f5b9e
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
+<details><summary>See axolotl config</summary>
+
+axolotl version: `0.4.1`
+```yaml
+adapter: lora
+base_model: lcw99/zephykor-ko-7b-chang
+bf16: true
+chat_template: llama3
+data_processes: 16
+dataset_prepared_path: null
+datasets:
+- data_files:
+  - 59ac76d3ecfca12a_train_data.json
+  ds_type: json
+  format: custom
+  path: /workspace/input_data/59ac76d3ecfca12a_train_data.json
+  type:
+    field_input: candidates
+    field_instruction: question
+    field_output: answer
+    format: '{instruction} {input}'
+    no_input_format: '{instruction}'
+    system_format: '{system}'
+    system_prompt: ''
+debug: null
+deepspeed: null
+device_map: auto
+do_eval: true
+early_stopping_patience: 5
+eval_batch_size: 4
+eval_max_new_tokens: 128
+eval_steps: 50
+eval_table_size: null
+evals_per_epoch: null
+flash_attention: true
+fp16: false
+fsdp: null
+fsdp_config: null
+gradient_accumulation_steps: 4
+gradient_checkpointing: true
+group_by_length: true
+hub_model_id: sn56z1/f2bce4f8-3b4e-443f-a5e5-aff8790bb9e2
+hub_repo: null
+hub_strategy: checkpoint
+hub_token: null
+learning_rate: 0.0001
+load_in_4bit: false
+load_in_8bit: false
+local_rank: null
+logging_steps: 1
+lora_alpha: 128
+lora_dropout: 0.05
+lora_fan_in_fan_out: null
+lora_model_dir: null
+lora_r: 64
+lora_target_linear: true
+lr_scheduler: cosine
+max_grad_norm: 1.0
+max_memory:
+  0: 75GB
+max_steps: 200
+micro_batch_size: 8
+mlflow_experiment_name: /tmp/59ac76d3ecfca12a_train_data.json
+model_type: AutoModelForCausalLM
+num_epochs: 3
+optim_args:
+  adam_beta1: 0.9
+  adam_beta2: 0.95
+  adam_epsilon: 1e-5
+optimizer: adamw_bnb_8bit
+output_dir: miner_id_24
+pad_to_sequence_len: true
+resume_from_checkpoint: null
+s2_attention: null
+sample_packing: false
+save_steps: 50
+saves_per_epoch: null
+sequence_len: 1024
+strict: false
+tf32: true
+tokenizer_type: AutoTokenizer
+train_on_inputs: false
+trust_remote_code: true
+val_set_size: 0.05
+wandb_entity: sn56-miner
+wandb_mode: disabled
+wandb_name: sn56z1/f2bce4f8
+wandb_project: god
+wandb_run: 7gfq
+wandb_runid: sn56z1/f2bce4f8
+warmup_steps: 10
+weight_decay: 0.0
+xformers_attention: null
+
+```
+
+</details><br>
+
+# 35807913-0500-4394-b20d-9e17a46f5b9e
+
+This model is a fine-tuned version of [lcw99/zephykor-ko-7b-chang](https://huggingface.co/lcw99/zephykor-ko-7b-chang) on the None dataset.
+It achieves the following results on the evaluation set:
+- Loss: 1.2798
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.0001
+- train_batch_size: 8
+- eval_batch_size: 4
+- seed: 42
+- gradient_accumulation_steps: 4
+- total_train_batch_size: 32
+- optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=adam_beta1=0.9,adam_beta2=0.95,adam_epsilon=1e-5
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_steps: 10
+- training_steps: 200
+
+### Training results
+
+| Training Loss | Epoch  | Step | Validation Loss |
+|:-------------:|:------:|:----:|:---------------:|
+| 10.2522       | 0.0006 | 1    | 3.3373          |
+| 3.9391        | 0.0324 | 50   | 1.5989          |
+| 4.3244        | 0.0648 | 100  | 1.4228          |
+| 4.0351        | 0.0972 | 150  | 1.3237          |
+| 3.2647        | 0.1296 | 200  | 1.2798          |
+
+
+### Framework versions
+
+- PEFT 0.13.2
+- Transformers 4.46.0
+- Pytorch 2.5.0+cu124
+- Datasets 3.0.1
+- Tokenizers 0.20.1

adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "lcw99/zephykor-ko-7b-chang",
+  "bias": "none",
+  "fan_in_fan_out": null,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "q_proj",
+    "down_proj",
+    "k_proj",
+    "o_proj",
+    "up_proj",
+    "gate_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:744fe20c816eb1f4717dc39482c2a9501342e58f499641bc85f1ccb8ee858c31
+size 671250634

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f5cb37845c856dcd26a7bbf37b81579e102da3e3cf5b0b29579ea39f27cba452
+size 671149168

config.json

@@ -0,0 +1,29 @@
+{
+  "_attn_implementation_autoset": true,
+  "_name_or_path": "lcw99/zephykor-ko-7b-chang",
+  "architectures": [
+    "MistralForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 32768,
+  "model_type": "mistral",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "pad_token_id": 2,
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 10000.0,
+  "sliding_window": 4096,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float16",
+  "transformers_version": "4.46.0",
+  "use_cache": false,
+  "vocab_size": 47772
+}

last-checkpoint/README.md

@@ -0,0 +1,202 @@
+---
+base_model: lcw99/zephykor-ko-7b-chang
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.2

last-checkpoint/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "lcw99/zephykor-ko-7b-chang",
+  "bias": "none",
+  "fan_in_fan_out": null,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "q_proj",
+    "down_proj",
+    "k_proj",
+    "o_proj",
+    "up_proj",
+    "gate_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

last-checkpoint/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f5cb37845c856dcd26a7bbf37b81579e102da3e3cf5b0b29579ea39f27cba452
+size 671149168

last-checkpoint/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:94d4d52e64edbbf475dad2ae8b0753944465f0255129ffec9279f4b2ed61c749
+size 341314196

last-checkpoint/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b18a58f0aa04210f3b214777be2a67b8dbd8b8e54caa8e293b2bba00cca87a4
+size 14244

last-checkpoint/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2d754412c61116546142914503e7369d0cc35d3c380a07e5218f595d76b6d96
+size 1064

last-checkpoint/special_tokens_map.json

@@ -0,0 +1,43 @@
+{
+  "additional_special_tokens": [
+    "<unk>",
+    "<s>",
+    "</s>",
+    "<0x00>"
+  ],
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "<0x00>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

last-checkpoint/tokenizer_config.json

@@ -0,0 +1,59 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<0x00>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<unk>",
+    "<s>",
+    "</s>",
+    "<0x00>"
+  ],
+  "bos_token": "<s>",
+  "chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "sep_token": "<0x00>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "truncation_side": "left",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": true
+}

last-checkpoint/trainer_state.json

@@ -0,0 +1,1482 @@
+{
+  "best_metric": 1.2797729969024658,
+  "best_model_checkpoint": "miner_id_24/checkpoint-200",
+  "epoch": 0.12961762799740764,
+  "eval_steps": 50,
+  "global_step": 200,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0006480881399870382,
+      "grad_norm": 32.77246856689453,
+      "learning_rate": 1e-05,
+      "loss": 10.2522,
+      "step": 1
+    },
+    {
+      "epoch": 0.0006480881399870382,
+      "eval_loss": 3.3373067378997803,
+      "eval_runtime": 367.421,
+      "eval_samples_per_second": 7.074,
+      "eval_steps_per_second": 1.769,
+      "step": 1
+    },
+    {
+      "epoch": 0.0012961762799740765,
+      "grad_norm": 34.492122650146484,
+      "learning_rate": 2e-05,
+      "loss": 10.4056,
+      "step": 2
+    },
+    {
+      "epoch": 0.0019442644199611147,
+      "grad_norm": 22.21146011352539,
+      "learning_rate": 3e-05,
+      "loss": 10.2703,
+      "step": 3
+    },
+    {
+      "epoch": 0.002592352559948153,
+      "grad_norm": 17.19241714477539,
+      "learning_rate": 4e-05,
+      "loss": 9.8782,
+      "step": 4
+    },
+    {
+      "epoch": 0.0032404406999351912,
+      "grad_norm": 20.113632202148438,
+      "learning_rate": 5e-05,
+      "loss": 10.3234,
+      "step": 5
+    },
+    {
+      "epoch": 0.0038885288399222295,
+      "grad_norm": 20.066024780273438,
+      "learning_rate": 6e-05,
+      "loss": 9.7434,
+      "step": 6
+    },
+    {
+      "epoch": 0.004536616979909268,
+      "grad_norm": 17.017101287841797,
+      "learning_rate": 7e-05,
+      "loss": 9.4544,
+      "step": 7
+    },
+    {
+      "epoch": 0.005184705119896306,
+      "grad_norm": 15.503304481506348,
+      "learning_rate": 8e-05,
+      "loss": 9.365,
+      "step": 8
+    },
+    {
+      "epoch": 0.005832793259883344,
+      "grad_norm": 15.54104232788086,
+      "learning_rate": 9e-05,
+      "loss": 8.8782,
+      "step": 9
+    },
+    {
+      "epoch": 0.0064808813998703824,
+      "grad_norm": 14.938676834106445,
+      "learning_rate": 0.0001,
+      "loss": 8.6603,
+      "step": 10
+    },
+    {
+      "epoch": 0.007128969539857421,
+      "grad_norm": 14.290221214294434,
+      "learning_rate": 9.999316524962345e-05,
+      "loss": 8.3563,
+      "step": 11
+    },
+    {
+      "epoch": 0.007777057679844459,
+      "grad_norm": 13.883099555969238,
+      "learning_rate": 9.997266286704631e-05,
+      "loss": 7.8561,
+      "step": 12
+    },
+    {
+      "epoch": 0.008425145819831496,
+      "grad_norm": 13.62983226776123,
+      "learning_rate": 9.993849845741524e-05,
+      "loss": 7.6208,
+      "step": 13
+    },
+    {
+      "epoch": 0.009073233959818535,
+      "grad_norm": 13.19063949584961,
+      "learning_rate": 9.989068136093873e-05,
+      "loss": 7.4653,
+      "step": 14
+    },
+    {
+      "epoch": 0.009721322099805573,
+      "grad_norm": 12.796319007873535,
+      "learning_rate": 9.98292246503335e-05,
+      "loss": 7.0079,
+      "step": 15
+    },
+    {
+      "epoch": 0.010369410239792612,
+      "grad_norm": 13.376047134399414,
+      "learning_rate": 9.975414512725057e-05,
+      "loss": 7.3178,
+      "step": 16
+    },
+    {
+      "epoch": 0.01101749837977965,
+      "grad_norm": 11.75101375579834,
+      "learning_rate": 9.966546331768191e-05,
+      "loss": 6.9637,
+      "step": 17
+    },
+    {
+      "epoch": 0.011665586519766688,
+      "grad_norm": 12.563492774963379,
+      "learning_rate": 9.956320346634876e-05,
+      "loss": 6.9917,
+      "step": 18
+    },
+    {
+      "epoch": 0.012313674659753726,
+      "grad_norm": 11.620759963989258,
+      "learning_rate": 9.944739353007344e-05,
+      "loss": 6.7609,
+      "step": 19
+    },
+    {
+      "epoch": 0.012961762799740765,
+      "grad_norm": 10.904376983642578,
+      "learning_rate": 9.931806517013612e-05,
+      "loss": 7.1235,
+      "step": 20
+    },
+    {
+      "epoch": 0.013609850939727802,
+      "grad_norm": 10.614338874816895,
+      "learning_rate": 9.917525374361912e-05,
+      "loss": 6.8806,
+      "step": 21
+    },
+    {
+      "epoch": 0.014257939079714841,
+      "grad_norm": 11.81243896484375,
+      "learning_rate": 9.901899829374047e-05,
+      "loss": 6.8972,
+      "step": 22
+    },
+    {
+      "epoch": 0.014906027219701879,
+      "grad_norm": 11.23525619506836,
+      "learning_rate": 9.884934153917997e-05,
+      "loss": 6.2781,
+      "step": 23
+    },
+    {
+      "epoch": 0.015554115359688918,
+      "grad_norm": 12.023797035217285,
+      "learning_rate": 9.86663298624003e-05,
+      "loss": 7.0459,
+      "step": 24
+    },
+    {
+      "epoch": 0.016202203499675955,
+      "grad_norm": 10.596454620361328,
+      "learning_rate": 9.847001329696653e-05,
+      "loss": 5.9495,
+      "step": 25
+    },
+    {
+      "epoch": 0.016850291639662993,
+      "grad_norm": 11.241633415222168,
+      "learning_rate": 9.826044551386744e-05,
+      "loss": 6.3805,
+      "step": 26
+    },
+    {
+      "epoch": 0.017498379779650033,
+      "grad_norm": 12.698907852172852,
+      "learning_rate": 9.803768380684242e-05,
+      "loss": 6.5493,
+      "step": 27
+    },
+    {
+      "epoch": 0.01814646791963707,
+      "grad_norm": 11.900014877319336,
+      "learning_rate": 9.780178907671789e-05,
+      "loss": 6.3326,
+      "step": 28
+    },
+    {
+      "epoch": 0.018794556059624108,
+      "grad_norm": 11.329556465148926,
+      "learning_rate": 9.755282581475769e-05,
+      "loss": 6.3898,
+      "step": 29
+    },
+    {
+      "epoch": 0.019442644199611146,
+      "grad_norm": 10.801814079284668,
+      "learning_rate": 9.729086208503174e-05,
+      "loss": 6.1835,
+      "step": 30
+    },
+    {
+      "epoch": 0.020090732339598186,
+      "grad_norm": 11.480464935302734,
+      "learning_rate": 9.701596950580806e-05,
+      "loss": 6.1471,
+      "step": 31
+    },
+    {
+      "epoch": 0.020738820479585224,
+      "grad_norm": 12.129923820495605,
+      "learning_rate": 9.672822322997305e-05,
+      "loss": 6.3976,
+      "step": 32
+    },
+    {
+      "epoch": 0.02138690861957226,
+      "grad_norm": 11.872269630432129,
+      "learning_rate": 9.642770192448536e-05,
+      "loss": 5.9649,
+      "step": 33
+    },
+    {
+      "epoch": 0.0220349967595593,
+      "grad_norm": 13.49704360961914,
+      "learning_rate": 9.611448774886924e-05,
+      "loss": 6.0253,
+      "step": 34
+    },
+    {
+      "epoch": 0.02268308489954634,
+      "grad_norm": 13.458520889282227,
+      "learning_rate": 9.578866633275288e-05,
+      "loss": 6.6214,
+      "step": 35
+    },
+    {
+      "epoch": 0.023331173039533377,
+      "grad_norm": 14.058696746826172,
+      "learning_rate": 9.545032675245813e-05,
+      "loss": 6.07,
+      "step": 36
+    },
+    {
+      "epoch": 0.023979261179520414,
+      "grad_norm": 13.415166854858398,
+      "learning_rate": 9.509956150664796e-05,
+      "loss": 6.1323,
+      "step": 37
+    },
+    {
+      "epoch": 0.02462734931950745,
+      "grad_norm": 14.339570045471191,
+      "learning_rate": 9.473646649103818e-05,
+      "loss": 6.3082,
+      "step": 38
+    },
+    {
+      "epoch": 0.025275437459494492,
+      "grad_norm": 16.123464584350586,
+      "learning_rate": 9.43611409721806e-05,
+      "loss": 6.6827,
+      "step": 39
+    },
+    {
+      "epoch": 0.02592352559948153,
+      "grad_norm": 14.134522438049316,
+      "learning_rate": 9.397368756032445e-05,
+      "loss": 6.4626,
+      "step": 40
+    },
+    {
+      "epoch": 0.026571613739468567,
+      "grad_norm": 17.929588317871094,
+      "learning_rate": 9.357421218136386e-05,
+      "loss": 7.7005,
+      "step": 41
+    },
+    {
+      "epoch": 0.027219701879455604,
+      "grad_norm": 18.088991165161133,
+      "learning_rate": 9.316282404787871e-05,
+      "loss": 7.5911,
+      "step": 42
+    },
+    {
+      "epoch": 0.027867790019442645,
+      "grad_norm": 18.61870002746582,
+      "learning_rate": 9.273963562927695e-05,
+      "loss": 6.8512,
+      "step": 43
+    },
+    {
+      "epoch": 0.028515878159429683,
+      "grad_norm": 32.17625427246094,
+      "learning_rate": 9.230476262104677e-05,
+      "loss": 9.7627,
+      "step": 44
+    },
+    {
+      "epoch": 0.02916396629941672,
+      "grad_norm": 17.418167114257812,
+      "learning_rate": 9.185832391312644e-05,
+      "loss": 5.6137,
+      "step": 45
+    },
+    {
+      "epoch": 0.029812054439403757,
+      "grad_norm": 13.998497009277344,
+      "learning_rate": 9.140044155740101e-05,
+      "loss": 4.9864,
+      "step": 46
+    },
+    {
+      "epoch": 0.0304601425793908,
+      "grad_norm": 10.171086311340332,
+      "learning_rate": 9.093124073433463e-05,
+      "loss": 4.6507,
+      "step": 47
+    },
+    {
+      "epoch": 0.031108230719377836,
+      "grad_norm": 15.122888565063477,
+      "learning_rate": 9.045084971874738e-05,
+      "loss": 4.7535,
+      "step": 48
+    },
+    {
+      "epoch": 0.031756318859364877,
+      "grad_norm": 13.40555477142334,
+      "learning_rate": 8.995939984474624e-05,
+      "loss": 4.6254,
+      "step": 49
+    },
+    {
+      "epoch": 0.03240440699935191,
+      "grad_norm": 9.962352752685547,
+      "learning_rate": 8.945702546981969e-05,
+      "loss": 3.9391,
+      "step": 50
+    },
+    {
+      "epoch": 0.03240440699935191,
+      "eval_loss": 1.598860502243042,
+      "eval_runtime": 370.4404,
+      "eval_samples_per_second": 7.016,
+      "eval_steps_per_second": 1.755,
+      "step": 50
+    },
+    {
+      "epoch": 0.03305249513933895,
+      "grad_norm": 6.8655853271484375,
+      "learning_rate": 8.894386393810563e-05,
+      "loss": 6.9079,
+      "step": 51
+    },
+    {
+      "epoch": 0.033700583279325985,
+      "grad_norm": 6.823023319244385,
+      "learning_rate": 8.842005554284296e-05,
+      "loss": 6.6039,
+      "step": 52
+    },
+    {
+      "epoch": 0.034348671419313026,
+      "grad_norm": 6.079746723175049,
+      "learning_rate": 8.788574348801675e-05,
+      "loss": 6.7465,
+      "step": 53
+    },
+    {
+      "epoch": 0.03499675955930007,
+      "grad_norm": 6.140050411224365,
+      "learning_rate": 8.73410738492077e-05,
+      "loss": 6.3824,
+      "step": 54
+    },
+    {
+      "epoch": 0.0356448476992871,
+      "grad_norm": 6.5793867111206055,
+      "learning_rate": 8.678619553365659e-05,
+      "loss": 6.5986,
+      "step": 55
+    },
+    {
+      "epoch": 0.03629293583927414,
+      "grad_norm": 6.7130842208862305,
+      "learning_rate": 8.622126023955446e-05,
+      "loss": 6.289,
+      "step": 56
+    },
+    {
+      "epoch": 0.03694102397926118,
+      "grad_norm": 6.729769706726074,
+      "learning_rate": 8.564642241456986e-05,
+      "loss": 6.1245,
+      "step": 57
+    },
+    {
+      "epoch": 0.037589112119248216,
+      "grad_norm": 7.293793201446533,
+      "learning_rate": 8.506183921362443e-05,
+      "loss": 6.2626,
+      "step": 58
+    },
+    {
+      "epoch": 0.03823720025923526,
+      "grad_norm": 7.541038990020752,
+      "learning_rate": 8.44676704559283e-05,
+      "loss": 5.8134,
+      "step": 59
+    },
+    {
+      "epoch": 0.03888528839922229,
+      "grad_norm": 7.872086048126221,
+      "learning_rate": 8.386407858128706e-05,
+      "loss": 5.9606,
+      "step": 60
+    },
+    {
+      "epoch": 0.03953337653920933,
+      "grad_norm": 7.872177600860596,
+      "learning_rate": 8.32512286056924e-05,
+      "loss": 5.771,
+      "step": 61
+    },
+    {
+      "epoch": 0.04018146467919637,
+      "grad_norm": 8.131582260131836,
+      "learning_rate": 8.262928807620843e-05,
+      "loss": 5.5896,
+      "step": 62
+    },
+    {
+      "epoch": 0.04082955281918341,
+      "grad_norm": 8.109619140625,
+      "learning_rate": 8.199842702516583e-05,
+      "loss": 6.0695,
+      "step": 63
+    },
+    {
+      "epoch": 0.04147764095917045,
+      "grad_norm": 7.7606987953186035,
+      "learning_rate": 8.135881792367686e-05,
+      "loss": 5.958,
+      "step": 64
+    },
+    {
+      "epoch": 0.04212572909915749,
+      "grad_norm": 8.114358901977539,
+      "learning_rate": 8.07106356344834e-05,
+      "loss": 5.2865,
+      "step": 65
+    },
+    {
+      "epoch": 0.04277381723914452,
+      "grad_norm": 8.963470458984375,
+      "learning_rate": 8.005405736415126e-05,
+      "loss": 5.3713,
+      "step": 66
+    },
+    {
+      "epoch": 0.04342190537913156,
+      "grad_norm": 8.013565063476562,
+      "learning_rate": 7.938926261462366e-05,
+      "loss": 5.5395,
+      "step": 67
+    },
+    {
+      "epoch": 0.0440699935191186,
+      "grad_norm": 8.421703338623047,
+      "learning_rate": 7.871643313414718e-05,
+      "loss": 6.0121,
+      "step": 68
+    },
+    {
+      "epoch": 0.04471808165910564,
+      "grad_norm": 8.579452514648438,
+      "learning_rate": 7.803575286758364e-05,
+      "loss": 5.5095,
+      "step": 69
+    },
+    {
+      "epoch": 0.04536616979909268,
+      "grad_norm": 8.891437530517578,
+      "learning_rate": 7.734740790612136e-05,
+      "loss": 5.6916,
+      "step": 70
+    },
+    {
+      "epoch": 0.04601425793907971,
+      "grad_norm": 8.683208465576172,
+      "learning_rate": 7.66515864363997e-05,
+      "loss": 5.2269,
+      "step": 71
+    },
+    {
+      "epoch": 0.046662346079066754,
+      "grad_norm": 9.19063949584961,
+      "learning_rate": 7.594847868906076e-05,
+      "loss": 5.8701,
+      "step": 72
+    },
+    {
+      "epoch": 0.047310434219053794,
+      "grad_norm": 8.987152099609375,
+      "learning_rate": 7.52382768867422e-05,
+      "loss": 6.0793,
+      "step": 73
+    },
+    {
+      "epoch": 0.04795852235904083,
+      "grad_norm": 9.380393981933594,
+      "learning_rate": 7.452117519152542e-05,
+      "loss": 5.7721,
+      "step": 74
+    },
+    {
+      "epoch": 0.04860661049902787,
+      "grad_norm": 8.99072551727295,
+      "learning_rate": 7.379736965185368e-05,
+      "loss": 5.7118,
+      "step": 75
+    },
+    {
+      "epoch": 0.0492546986390149,
+      "grad_norm": 9.631376266479492,
+      "learning_rate": 7.30670581489344e-05,
+      "loss": 5.6135,
+      "step": 76
+    },
+    {
+      "epoch": 0.049902786779001944,
+      "grad_norm": 9.81628131866455,
+      "learning_rate": 7.233044034264034e-05,
+      "loss": 5.4346,
+      "step": 77
+    },
+    {
+      "epoch": 0.050550874918988985,
+      "grad_norm": 9.82709789276123,
+      "learning_rate": 7.158771761692464e-05,
+      "loss": 5.0797,
+      "step": 78
+    },
+    {
+      "epoch": 0.05119896305897602,
+      "grad_norm": 9.465619087219238,
+      "learning_rate": 7.083909302476453e-05,
+      "loss": 5.2089,
+      "step": 79
+    },
+    {
+      "epoch": 0.05184705119896306,
+      "grad_norm": 9.651265144348145,
+      "learning_rate": 7.008477123264848e-05,
+      "loss": 5.292,
+      "step": 80
+    },
+    {
+      "epoch": 0.0524951393389501,
+      "grad_norm": 9.914762496948242,
+      "learning_rate": 6.932495846462261e-05,
+      "loss": 5.6542,
+      "step": 81
+    },
+    {
+      "epoch": 0.053143227478937134,
+      "grad_norm": 9.019952774047852,
+      "learning_rate": 6.855986244591104e-05,
+      "loss": 5.0893,
+      "step": 82
+    },
+    {
+      "epoch": 0.053791315618924175,
+      "grad_norm": 11.01401424407959,
+      "learning_rate": 6.778969234612584e-05,
+      "loss": 6.0897,
+      "step": 83
+    },
+    {
+      "epoch": 0.05443940375891121,
+      "grad_norm": 10.864527702331543,
+      "learning_rate": 6.701465872208216e-05,
+      "loss": 5.5917,
+      "step": 84
+    },
+    {
+      "epoch": 0.05508749189889825,
+      "grad_norm": 10.473554611206055,
+      "learning_rate": 6.623497346023418e-05,
+      "loss": 5.494,
+      "step": 85
+    },
+    {
+      "epoch": 0.05573558003888529,
+      "grad_norm": 11.461551666259766,
+      "learning_rate": 6.545084971874738e-05,
+      "loss": 5.6266,
+      "step": 86
+    },
+    {
+      "epoch": 0.056383668178872325,
+      "grad_norm": 11.780827522277832,
+      "learning_rate": 6.466250186922325e-05,
+      "loss": 5.6434,
+      "step": 87
+    },
+    {
+      "epoch": 0.057031756318859365,
+      "grad_norm": 12.216592788696289,
+      "learning_rate": 6.387014543809223e-05,
+      "loss": 6.0677,
+      "step": 88
+    },
+    {
+      "epoch": 0.057679844458846406,
+      "grad_norm": 12.341825485229492,
+      "learning_rate": 6.307399704769099e-05,
+      "loss": 5.9295,
+      "step": 89
+    },
+    {
+      "epoch": 0.05832793259883344,
+      "grad_norm": 11.894737243652344,
+      "learning_rate": 6.227427435703997e-05,
+      "loss": 5.9671,
+      "step": 90
+    },
+    {
+      "epoch": 0.05897602073882048,
+      "grad_norm": 13.942716598510742,
+      "learning_rate": 6.147119600233758e-05,
+      "loss": 5.7762,
+      "step": 91
+    },
+    {
+      "epoch": 0.059624108878807515,
+      "grad_norm": 14.185612678527832,
+      "learning_rate": 6.066498153718735e-05,
+      "loss": 6.1956,
+      "step": 92
+    },
+    {
+      "epoch": 0.060272197018794556,
+      "grad_norm": 16.345291137695312,
+      "learning_rate": 5.985585137257401e-05,
+      "loss": 6.9774,
+      "step": 93
+    },
+    {
+      "epoch": 0.0609202851587816,
+      "grad_norm": 15.189043998718262,
+      "learning_rate": 5.90440267166055e-05,
+      "loss": 5.8435,
+      "step": 94
+    },
+    {
+      "epoch": 0.06156837329876863,
+      "grad_norm": 11.216886520385742,
+      "learning_rate": 5.8229729514036705e-05,
+      "loss": 4.6006,
+      "step": 95
+    },
+    {
+      "epoch": 0.06221646143875567,
+      "grad_norm": 11.1788330078125,
+      "learning_rate": 5.74131823855921e-05,
+      "loss": 4.1444,
+      "step": 96
+    },
+    {
+      "epoch": 0.06286454957874271,
+      "grad_norm": 9.843339920043945,
+      "learning_rate": 5.6594608567103456e-05,
+      "loss": 4.9087,
+      "step": 97
+    },
+    {
+      "epoch": 0.06351263771872975,
+      "grad_norm": 7.888655662536621,
+      "learning_rate": 5.577423184847932e-05,
+      "loss": 4.3402,
+      "step": 98
+    },
+    {
+      "epoch": 0.06416072585871678,
+      "grad_norm": 9.263370513916016,
+      "learning_rate": 5.495227651252315e-05,
+      "loss": 4.621,
+      "step": 99
+    },
+    {
+      "epoch": 0.06480881399870382,
+      "grad_norm": 8.855140686035156,
+      "learning_rate": 5.4128967273616625e-05,
+      "loss": 4.3244,
+      "step": 100
+    },
+    {
+      "epoch": 0.06480881399870382,
+      "eval_loss": 1.4227622747421265,
+      "eval_runtime": 370.2885,
+      "eval_samples_per_second": 7.019,
+      "eval_steps_per_second": 1.755,
+      "step": 100
+    },
+    {
+      "epoch": 0.06545690213869086,
+      "grad_norm": 4.787423133850098,
+      "learning_rate": 5.330452921628497e-05,
+      "loss": 5.9553,
+      "step": 101
+    },
+    {
+      "epoch": 0.0661049902786779,
+      "grad_norm": 4.652994155883789,
+      "learning_rate": 5.247918773366112e-05,
+      "loss": 5.5491,
+      "step": 102
+    },
+    {
+      "epoch": 0.06675307841866494,
+      "grad_norm": 5.026247024536133,
+      "learning_rate": 5.165316846586541e-05,
+      "loss": 5.59,
+      "step": 103
+    },
+    {
+      "epoch": 0.06740116655865197,
+      "grad_norm": 5.296930313110352,
+      "learning_rate": 5.0826697238317935e-05,
+      "loss": 5.7854,
+      "step": 104
+    },
+    {
+      "epoch": 0.06804925469863901,
+      "grad_norm": 5.847901344299316,
+      "learning_rate": 5e-05,
+      "loss": 5.9658,
+      "step": 105
+    },
+    {
+      "epoch": 0.06869734283862605,
+      "grad_norm": 6.000091075897217,
+      "learning_rate": 4.917330276168208e-05,
+      "loss": 5.5761,
+      "step": 106
+    },
+    {
+      "epoch": 0.06934543097861309,
+      "grad_norm": 6.229143142700195,
+      "learning_rate": 4.834683153413459e-05,
+      "loss": 5.2815,
+      "step": 107
+    },
+    {
+      "epoch": 0.06999351911860013,
+      "grad_norm": 6.7049479484558105,
+      "learning_rate": 4.7520812266338885e-05,
+      "loss": 5.6108,
+      "step": 108
+    },
+    {
+      "epoch": 0.07064160725858717,
+      "grad_norm": 6.860906600952148,
+      "learning_rate": 4.669547078371504e-05,
+      "loss": 5.5159,
+      "step": 109
+    },
+    {
+      "epoch": 0.0712896953985742,
+      "grad_norm": 7.019985198974609,
+      "learning_rate": 4.5871032726383386e-05,
+      "loss": 5.6056,
+      "step": 110
+    },
+    {
+      "epoch": 0.07193778353856124,
+      "grad_norm": 7.05626916885376,
+      "learning_rate": 4.504772348747687e-05,
+      "loss": 5.4075,
+      "step": 111
+    },
+    {
+      "epoch": 0.07258587167854828,
+      "grad_norm": 6.539259910583496,
+      "learning_rate": 4.4225768151520694e-05,
+      "loss": 5.0558,
+      "step": 112
+    },
+    {
+      "epoch": 0.07323395981853532,
+      "grad_norm": 7.1625447273254395,
+      "learning_rate": 4.3405391432896555e-05,
+      "loss": 5.351,
+      "step": 113
+    },
+    {
+      "epoch": 0.07388204795852236,
+      "grad_norm": 7.557711601257324,
+      "learning_rate": 4.2586817614407895e-05,
+      "loss": 5.6869,
+      "step": 114
+    },
+    {
+      "epoch": 0.07453013609850939,
+      "grad_norm": 7.120979309082031,
+      "learning_rate": 4.17702704859633e-05,
+      "loss": 5.1624,
+      "step": 115
+    },
+    {
+      "epoch": 0.07517822423849643,
+      "grad_norm": 7.815059185028076,
+      "learning_rate": 4.095597328339452e-05,
+      "loss": 5.8814,
+      "step": 116
+    },
+    {
+      "epoch": 0.07582631237848347,
+      "grad_norm": 7.95869255065918,
+      "learning_rate": 4.0144148627425993e-05,
+      "loss": 5.2809,
+      "step": 117
+    },
+    {
+      "epoch": 0.07647440051847051,
+      "grad_norm": 8.076533317565918,
+      "learning_rate": 3.933501846281267e-05,
+      "loss": 5.6616,
+      "step": 118
+    },
+    {
+      "epoch": 0.07712248865845756,
+      "grad_norm": 7.682056903839111,
+      "learning_rate": 3.852880399766243e-05,
+      "loss": 5.1071,
+      "step": 119
+    },
+    {
+      "epoch": 0.07777057679844458,
+      "grad_norm": 8.244827270507812,
+      "learning_rate": 3.772572564296005e-05,
+      "loss": 5.5138,
+      "step": 120
+    },
+    {
+      "epoch": 0.07841866493843162,
+      "grad_norm": 8.22045612335205,
+      "learning_rate": 3.6926002952309016e-05,
+      "loss": 5.0651,
+      "step": 121
+    },
+    {
+      "epoch": 0.07906675307841866,
+      "grad_norm": 8.04766845703125,
+      "learning_rate": 3.612985456190778e-05,
+      "loss": 5.567,
+      "step": 122
+    },
+    {
+      "epoch": 0.0797148412184057,
+      "grad_norm": 8.576169967651367,
+      "learning_rate": 3.533749813077677e-05,
+      "loss": 5.454,
+      "step": 123
+    },
+    {
+      "epoch": 0.08036292935839275,
+      "grad_norm": 8.193389892578125,
+      "learning_rate": 3.4549150281252636e-05,
+      "loss": 5.1683,
+      "step": 124
+    },
+    {
+      "epoch": 0.08101101749837979,
+      "grad_norm": 8.050642967224121,
+      "learning_rate": 3.3765026539765834e-05,
+      "loss": 4.6637,
+      "step": 125
+    },
+    {
+      "epoch": 0.08165910563836681,
+      "grad_norm": 8.81507396697998,
+      "learning_rate": 3.298534127791785e-05,
+      "loss": 5.3693,
+      "step": 126
+    },
+    {
+      "epoch": 0.08230719377835385,
+      "grad_norm": 8.727811813354492,
+      "learning_rate": 3.221030765387417e-05,
+      "loss": 5.714,
+      "step": 127
+    },
+    {
+      "epoch": 0.0829552819183409,
+      "grad_norm": 8.717565536499023,
+      "learning_rate": 3.144013755408895e-05,
+      "loss": 5.7399,
+      "step": 128
+    },
+    {
+      "epoch": 0.08360337005832794,
+      "grad_norm": 8.756967544555664,
+      "learning_rate": 3.0675041535377405e-05,
+      "loss": 5.2997,
+      "step": 129
+    },
+    {
+      "epoch": 0.08425145819831498,
+      "grad_norm": 9.31334400177002,
+      "learning_rate": 2.991522876735154e-05,
+      "loss": 5.4564,
+      "step": 130
+    },
+    {
+      "epoch": 0.084899546338302,
+      "grad_norm": 8.90140151977539,
+      "learning_rate": 2.916090697523549e-05,
+      "loss": 5.0921,
+      "step": 131
+    },
+    {
+      "epoch": 0.08554763447828904,
+      "grad_norm": 9.364441871643066,
+      "learning_rate": 2.8412282383075363e-05,
+      "loss": 4.495,
+      "step": 132
+    },
+    {
+      "epoch": 0.08619572261827609,
+      "grad_norm": 10.136019706726074,
+      "learning_rate": 2.766955965735968e-05,
+      "loss": 5.7014,
+      "step": 133
+    },
+    {
+      "epoch": 0.08684381075826313,
+      "grad_norm": 9.147400856018066,
+      "learning_rate": 2.693294185106562e-05,
+      "loss": 4.7557,
+      "step": 134
+    },
+    {
+      "epoch": 0.08749189889825017,
+      "grad_norm": 9.371038436889648,
+      "learning_rate": 2.6202630348146324e-05,
+      "loss": 4.2169,
+      "step": 135
+    },
+    {
+      "epoch": 0.0881399870382372,
+      "grad_norm": 10.364588737487793,
+      "learning_rate": 2.547882480847461e-05,
+      "loss": 5.5328,
+      "step": 136
+    },
+    {
+      "epoch": 0.08878807517822424,
+      "grad_norm": 12.089287757873535,
+      "learning_rate": 2.476172311325783e-05,
+      "loss": 5.7587,
+      "step": 137
+    },
+    {
+      "epoch": 0.08943616331821128,
+      "grad_norm": 10.678025245666504,
+      "learning_rate": 2.405152131093926e-05,
+      "loss": 4.7723,
+      "step": 138
+    },
+    {
+      "epoch": 0.09008425145819832,
+      "grad_norm": 10.811649322509766,
+      "learning_rate": 2.3348413563600325e-05,
+      "loss": 4.6393,
+      "step": 139
+    },
+    {
+      "epoch": 0.09073233959818536,
+      "grad_norm": 13.51014518737793,
+      "learning_rate": 2.2652592093878666e-05,
+      "loss": 5.6152,
+      "step": 140
+    },
+    {
+      "epoch": 0.09138042773817238,
+      "grad_norm": 14.925034523010254,
+      "learning_rate": 2.196424713241637e-05,
+      "loss": 6.0521,
+      "step": 141
+    },
+    {
+      "epoch": 0.09202851587815943,
+      "grad_norm": 13.902508735656738,
+      "learning_rate": 2.128356686585282e-05,
+      "loss": 5.4761,
+      "step": 142
+    },
+    {
+      "epoch": 0.09267660401814647,
+      "grad_norm": 20.527427673339844,
+      "learning_rate": 2.061073738537635e-05,
+      "loss": 7.0354,
+      "step": 143
+    },
+    {
+      "epoch": 0.09332469215813351,
+      "grad_norm": 11.042694091796875,
+      "learning_rate": 1.9945942635848748e-05,
+      "loss": 5.1453,
+      "step": 144
+    },
+    {
+      "epoch": 0.09397278029812055,
+      "grad_norm": 9.60646915435791,
+      "learning_rate": 1.928936436551661e-05,
+      "loss": 4.696,
+      "step": 145
+    },
+    {
+      "epoch": 0.09462086843810759,
+      "grad_norm": 7.381638526916504,
+      "learning_rate": 1.8641182076323148e-05,
+      "loss": 3.5106,
+      "step": 146
+    },
+    {
+      "epoch": 0.09526895657809462,
+      "grad_norm": 8.033303260803223,
+      "learning_rate": 1.800157297483417e-05,
+      "loss": 3.8758,
+      "step": 147
+    },
+    {
+      "epoch": 0.09591704471808166,
+      "grad_norm": 8.180899620056152,
+      "learning_rate": 1.7370711923791567e-05,
+      "loss": 3.5253,
+      "step": 148
+    },
+    {
+      "epoch": 0.0965651328580687,
+      "grad_norm": 7.356280326843262,
+      "learning_rate": 1.6748771394307585e-05,
+      "loss": 3.2546,
+      "step": 149
+    },
+    {
+      "epoch": 0.09721322099805574,
+      "grad_norm": 9.28185749053955,
+      "learning_rate": 1.6135921418712956e-05,
+      "loss": 4.0351,
+      "step": 150
+    },
+    {
+      "epoch": 0.09721322099805574,
+      "eval_loss": 1.323728084564209,
+      "eval_runtime": 370.5316,
+      "eval_samples_per_second": 7.014,
+      "eval_steps_per_second": 1.754,
+      "step": 150
+    },
+    {
+      "epoch": 0.09786130913804278,
+      "grad_norm": 6.201549053192139,
+      "learning_rate": 1.553232954407171e-05,
+      "loss": 5.4942,
+      "step": 151
+    },
+    {
+      "epoch": 0.0985093972780298,
+      "grad_norm": 6.157083511352539,
+      "learning_rate": 1.4938160786375572e-05,
+      "loss": 5.6309,
+      "step": 152
+    },
+    {
+      "epoch": 0.09915748541801685,
+      "grad_norm": 5.9916791915893555,
+      "learning_rate": 1.435357758543015e-05,
+      "loss": 5.6718,
+      "step": 153
+    },
+    {
+      "epoch": 0.09980557355800389,
+      "grad_norm": 5.736362934112549,
+      "learning_rate": 1.3778739760445552e-05,
+      "loss": 5.4077,
+      "step": 154
+    },
+    {
+      "epoch": 0.10045366169799093,
+      "grad_norm": 5.472987651824951,
+      "learning_rate": 1.3213804466343421e-05,
+      "loss": 5.1076,
+      "step": 155
+    },
+    {
+      "epoch": 0.10110174983797797,
+      "grad_norm": 5.656830310821533,
+      "learning_rate": 1.2658926150792322e-05,
+      "loss": 5.1589,
+      "step": 156
+    },
+    {
+      "epoch": 0.101749837977965,
+      "grad_norm": 6.4442644119262695,
+      "learning_rate": 1.2114256511983274e-05,
+      "loss": 5.5476,
+      "step": 157
+    },
+    {
+      "epoch": 0.10239792611795204,
+      "grad_norm": 6.049374580383301,
+      "learning_rate": 1.157994445715706e-05,
+      "loss": 5.3783,
+      "step": 158
+    },
+    {
+      "epoch": 0.10304601425793908,
+      "grad_norm": 6.355889320373535,
+      "learning_rate": 1.1056136061894384e-05,
+      "loss": 5.5186,
+      "step": 159
+    },
+    {
+      "epoch": 0.10369410239792612,
+      "grad_norm": 6.612687587738037,
+      "learning_rate": 1.0542974530180327e-05,
+      "loss": 5.1498,
+      "step": 160
+    },
+    {
+      "epoch": 0.10434219053791316,
+      "grad_norm": 6.526288986206055,
+      "learning_rate": 1.0040600155253765e-05,
+      "loss": 5.0996,
+      "step": 161
+    },
+    {
+      "epoch": 0.1049902786779002,
+      "grad_norm": 7.491959571838379,
+      "learning_rate": 9.549150281252633e-06,
+      "loss": 4.9242,
+      "step": 162
+    },
+    {
+      "epoch": 0.10563836681788723,
+      "grad_norm": 6.922194957733154,
+      "learning_rate": 9.068759265665384e-06,
+      "loss": 5.3422,
+      "step": 163
+    },
+    {
+      "epoch": 0.10628645495787427,
+      "grad_norm": 6.840357303619385,
+      "learning_rate": 8.599558442598998e-06,
+      "loss": 4.7479,
+      "step": 164
+    },
+    {
+      "epoch": 0.10693454309786131,
+      "grad_norm": 7.014447212219238,
+      "learning_rate": 8.141676086873572e-06,
+      "loss": 4.8852,
+      "step": 165
+    },
+    {
+      "epoch": 0.10758263123784835,
+      "grad_norm": 7.619250297546387,
+      "learning_rate": 7.695237378953223e-06,
+      "loss": 5.3302,
+      "step": 166
+    },
+    {
+      "epoch": 0.10823071937783539,
+      "grad_norm": 7.709277629852295,
+      "learning_rate": 7.260364370723044e-06,
+      "loss": 5.6599,
+      "step": 167
+    },
+    {
+      "epoch": 0.10887880751782242,
+      "grad_norm": 7.755810260772705,
+      "learning_rate": 6.837175952121306e-06,
+      "loss": 5.6992,
+      "step": 168
+    },
+    {
+      "epoch": 0.10952689565780946,
+      "grad_norm": 7.319242000579834,
+      "learning_rate": 6.425787818636131e-06,
+      "loss": 4.9321,
+      "step": 169
+    },
+    {
+      "epoch": 0.1101749837977965,
+      "grad_norm": 8.059549331665039,
+      "learning_rate": 6.026312439675552e-06,
+      "loss": 5.3413,
+      "step": 170
+    },
+    {
+      "epoch": 0.11082307193778354,
+      "grad_norm": 8.143264770507812,
+      "learning_rate": 5.6388590278194096e-06,
+      "loss": 5.2662,
+      "step": 171
+    },
+    {
+      "epoch": 0.11147116007777058,
+      "grad_norm": 8.14770793914795,
+      "learning_rate": 5.263533508961827e-06,
+      "loss": 5.8498,
+      "step": 172
+    },
+    {
+      "epoch": 0.11211924821775761,
+      "grad_norm": 7.8053388595581055,
+      "learning_rate": 4.900438493352055e-06,
+      "loss": 4.9943,
+      "step": 173
+    },
+    {
+      "epoch": 0.11276733635774465,
+      "grad_norm": 7.764467239379883,
+      "learning_rate": 4.549673247541875e-06,
+      "loss": 4.6429,
+      "step": 174
+    },
+    {
+      "epoch": 0.11341542449773169,
+      "grad_norm": 8.759538650512695,
+      "learning_rate": 4.2113336672471245e-06,
+      "loss": 5.2754,
+      "step": 175
+    },
+    {
+      "epoch": 0.11406351263771873,
+      "grad_norm": 8.908333778381348,
+      "learning_rate": 3.885512251130763e-06,
+      "loss": 5.5783,
+      "step": 176
+    },
+    {
+      "epoch": 0.11471160077770577,
+      "grad_norm": 8.526325225830078,
+      "learning_rate": 3.5722980755146517e-06,
+      "loss": 5.172,
+      "step": 177
+    },
+    {
+      "epoch": 0.11535968891769281,
+      "grad_norm": 8.65754508972168,
+      "learning_rate": 3.271776770026963e-06,
+      "loss": 5.4467,
+      "step": 178
+    },
+    {
+      "epoch": 0.11600777705767984,
+      "grad_norm": 8.814269065856934,
+      "learning_rate": 2.9840304941919415e-06,
+      "loss": 5.1604,
+      "step": 179
+    },
+    {
+      "epoch": 0.11665586519766688,
+      "grad_norm": 9.831783294677734,
+      "learning_rate": 2.7091379149682685e-06,
+      "loss": 5.579,
+      "step": 180
+    },
+    {
+      "epoch": 0.11730395333765392,
+      "grad_norm": 8.610177993774414,
+      "learning_rate": 2.4471741852423237e-06,
+      "loss": 4.7613,
+      "step": 181
+    },
+    {
+      "epoch": 0.11795204147764096,
+      "grad_norm": 9.330480575561523,
+      "learning_rate": 2.1982109232821178e-06,
+      "loss": 5.5626,
+      "step": 182
+    },
+    {
+      "epoch": 0.118600129617628,
+      "grad_norm": 9.307762145996094,
+      "learning_rate": 1.962316193157593e-06,
+      "loss": 4.9636,
+      "step": 183
+    },
+    {
+      "epoch": 0.11924821775761503,
+      "grad_norm": 9.787566184997559,
+      "learning_rate": 1.7395544861325718e-06,
+      "loss": 5.4634,
+      "step": 184
+    },
+    {
+      "epoch": 0.11989630589760207,
+      "grad_norm": 9.763351440429688,
+      "learning_rate": 1.5299867030334814e-06,
+      "loss": 5.2881,
+      "step": 185
+    },
+    {
+      "epoch": 0.12054439403758911,
+      "grad_norm": 9.747420310974121,
+      "learning_rate": 1.333670137599713e-06,
+      "loss": 4.7346,
+      "step": 186
+    },
+    {
+      "epoch": 0.12119248217757615,
+      "grad_norm": 10.178956985473633,
+      "learning_rate": 1.1506584608200367e-06,
+      "loss": 5.2031,
+      "step": 187
+    },
+    {
+      "epoch": 0.1218405703175632,
+      "grad_norm": 11.203434944152832,
+      "learning_rate": 9.810017062595322e-07,
+      "loss": 4.9873,
+      "step": 188
+    },
+    {
+      "epoch": 0.12248865845755022,
+      "grad_norm": 9.708922386169434,
+      "learning_rate": 8.247462563808817e-07,
+      "loss": 4.6898,
+      "step": 189
+    },
+    {
+      "epoch": 0.12313674659753726,
+      "grad_norm": 12.202812194824219,
+      "learning_rate": 6.819348298638839e-07,
+      "loss": 5.7993,
+      "step": 190
+    },
+    {
+      "epoch": 0.1237848347375243,
+      "grad_norm": 13.126566886901855,
+      "learning_rate": 5.526064699265753e-07,
+      "loss": 5.4346,
+      "step": 191
+    },
+    {
+      "epoch": 0.12443292287751134,
+      "grad_norm": 12.742057800292969,
+      "learning_rate": 4.367965336512403e-07,
+      "loss": 5.5181,
+      "step": 192
+    },
+    {
+      "epoch": 0.12508101101749838,
+      "grad_norm": 16.250934600830078,
+      "learning_rate": 3.3453668231809286e-07,
+      "loss": 6.6257,
+      "step": 193
+    },
+    {
+      "epoch": 0.12572909915748542,
+      "grad_norm": 23.86930274963379,
+      "learning_rate": 2.458548727494292e-07,
+      "loss": 7.243,
+      "step": 194
+    },
+    {
+      "epoch": 0.12637718729747247,
+      "grad_norm": 24.675596237182617,
+      "learning_rate": 1.7077534966650766e-07,
+      "loss": 7.7352,
+      "step": 195
+    },
+    {
+      "epoch": 0.1270252754374595,
+      "grad_norm": 8.297137260437012,
+      "learning_rate": 1.0931863906127327e-07,
+      "loss": 3.8953,
+      "step": 196
+    },
+    {
+      "epoch": 0.12767336357744652,
+      "grad_norm": 6.812832355499268,
+      "learning_rate": 6.150154258476315e-08,
+      "loss": 3.4129,
+      "step": 197
+    },
+    {
+      "epoch": 0.12832145171743356,
+      "grad_norm": 8.038043975830078,
+      "learning_rate": 2.7337132953697554e-08,
+      "loss": 3.5709,
+      "step": 198
+    },
+    {
+      "epoch": 0.1289695398574206,
+      "grad_norm": 8.24399471282959,
+      "learning_rate": 6.834750376549792e-09,
+      "loss": 3.2227,
+      "step": 199
+    },
+    {
+      "epoch": 0.12961762799740764,
+      "grad_norm": 7.731753826141357,
+      "learning_rate": 0.0,
+      "loss": 3.2647,
+      "step": 200
+    },
+    {
+      "epoch": 0.12961762799740764,
+      "eval_loss": 1.2797729969024658,
+      "eval_runtime": 370.7386,
+      "eval_samples_per_second": 7.01,
+      "eval_steps_per_second": 1.753,
+      "step": 200
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 200,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 50,
+  "stateful_callbacks": {
+    "EarlyStoppingCallback": {
+      "args": {
+        "early_stopping_patience": 5,
+        "early_stopping_threshold": 0.0
+      },
+      "attributes": {
+        "early_stopping_patience_counter": 0
+      }
+    },
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 5.442746364859515e+17,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

last-checkpoint/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ef9796c20e529a670fb876ca629d0a395f3ca18360476b773e8b75ee15f882b9
+size 6840

special_tokens_map.json

@@ -0,0 +1,43 @@
+{
+  "additional_special_tokens": [
+    "<unk>",
+    "<s>",
+    "</s>",
+    "<0x00>"
+  ],
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "<0x00>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,59 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<0x00>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<unk>",
+    "<s>",
+    "</s>",
+    "<0x00>"
+  ],
+  "bos_token": "<s>",
+  "chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "sep_token": "<0x00>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "truncation_side": "left",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": true
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ef9796c20e529a670fb876ca629d0a395f3ca18360476b773e8b75ee15f882b9
+size 6840