Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,270 +1,73 @@
 ---
-language:
-- am
-- ar
-- bn
-- zh
-- cs
-- nl
-- en
-- fr
-- de
-- el
-- ha
-- he
-- hi
-- id
-- it
-- ja
-- jv
-- km
-- ko
-- lo
-- ms
-- mr
-- fa
-- pl
-- pt
-- ro
-- ru
-- es
-- sw
-- sv
-- tl
-- ta
-- te
-- th
-- tr
-- uk
-- ur
-- vi
-license: apache-2.0
-datasets:
-- lightblue/reasoning-multilingual-R1-Llama-70B-train
+library_name: transformers
+license: other
+base_model: deepseek-ai/DeepSeek-R1-Distill-Qwen-14B
 tags:
-- reasoning
+- llama-factory
+- full
+- generated_from_trainer
+model-index:
+- name: reasoning-multilingual-R1-Llama-70B-train
+  results: []
 ---
 
-# lightblue/DeepSeek-R1-Distill-Qwen-7B-Multilingual
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
 
-<div style="width: 100%; height: 160px; 
-            display: flex; align-items: center; 
-            justify-content: center; 
-            border: 8px solid black; 
-            font-size: 120px; font-weight: bold; 
-            text-align: center; 
-            color: #438db8,
-            font-family: 'Helvetica Neue', sans-serif;">
-  <span style="color: #438db8;">R1</span>
-  &nbsp;
-  <span style="color: blue;">m</span>
-  <span style="color: green;">u</span>
-  <span style="color: purple;">l</span>
-  <span style="color: yellow;">t</span>
-  <span style="color: pink;">i</span>
-  <span style="color: cyan;">l</span>
-  <span style="color: magenta;">i</span>
-  <span style="color: lime;">n</span>
-  <span style="color: teal;">g</span>
-</div>
+# reasoning-multilingual-R1-Llama-70B-train
 
-This is a Deepseek distill finetune trained on multilingual Chain-of-Thought (CoT).
-When this model is prompted in a language, it will both think and respond in that language, unlike the original R1 which will often think in either Chinese or English.
-This will make the outputs of these AIs more understandable and explainable to a wider audience.
-Hopefully this will be useful to the AI community, particularly those developing for languages aside from English and Chinese.
+This model is a fine-tuned version of [deepseek-ai/DeepSeek-R1-Distill-Qwen-14B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B) on the reasoning-multilingual-R1-Llama-70B-train dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.4441
 
-This model is a multilingual fine-tuned version of [deepseek-ai/DeepSeek-R1-Distill-Qwen-14B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B).
+## Model description
 
-Other fine-tuned versions of this model can be found in [our collection, here](https://huggingface.co/collections/lightblue/r1-multilingual-679c890166ac0a84e83e38fa).
+More information needed
 
-This model was trained was trained for ~10 minutes on the 8 x L20 instance ([ecs.gn8is-8x.32xlarge](https://www.alibabacloud.com/help/en/ecs/user-guide/gpu-accelerated-compute-optimized-and-vgpu-accelerated-instance-families-1)) on [Alibaba Cloud](https://www.alibabacloud.com/).
+## Intended uses & limitations
 
-# How to use
+More information needed
 
-When using these models, we recommend using a sampling temperature of between 0.5-0.7, [as per the original distilled R1 models](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B#usage-recommendations).
+## Training and evaluation data
 
-Additionally, we have observed that the model sometimes tends to repeat for more niche languages, so we also recommend setting `repetition_penalty` to 1.1, or higher if the model repeats itself when processing your prompts.
+More information needed
 
-We include scripts to use this model in vLLM:
+## Training procedure
 
-<ul>
-  <li><b>vLLM</b>
+### Training hyperparameters
 
-Install [vLLM](https://github.com/vllm-project/vllm/) using `pip install vllm`.
+The following hyperparameters were used during training:
+- learning_rate: 1e-05
+- train_batch_size: 1
+- eval_batch_size: 1
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 8
+- total_train_batch_size: 8
+- total_eval_batch_size: 8
+- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_ratio: 0.01
+- num_epochs: 1.0
 
-<details open>
-  <summary>Show vLLM code</summary>
-  
-```python
-from vllm import LLM, SamplingParams
+### Training results
 
-llm = LLM(
-    model="lightblue/DeepSeek-R1-Distill-Qwen-7B-Multilingual",
-    max_model_len=8_000
-)
+| Training Loss | Epoch  | Step | Validation Loss |
+|:-------------:|:------:|:----:|:---------------:|
+| 0.4962        | 0.1019 | 11   | 0.5000          |
+| 0.5313        | 0.2037 | 22   | 0.4791          |
+| 0.4692        | 0.3056 | 33   | 0.4685          |
+| 0.3876        | 0.4074 | 44   | 0.4595          |
+| 0.4768        | 0.5093 | 55   | 0.4542          |
+| 0.4985        | 0.6111 | 66   | 0.4496          |
+| 0.4687        | 0.7130 | 77   | 0.4465          |
+| 0.4484        | 0.8148 | 88   | 0.4449          |
+| 0.4809        | 0.9167 | 99   | 0.4442          |
 
-sampling_params = SamplingParams(
-    temperature=0.5, 
-    max_tokens=8_000
-)
 
-prompts = [
-    """学校には1クラスにつき20人の生徒がおり、クラスは合計3つあります。
-学校全体では男子と女子がそれぞれ50%ずついます。
-1つ目のクラスには女子が15人、2つ目のクラスには女子が12人います。
-3つ目のクラスには何人の男子がいますか？"""
-]
+### Framework versions
 
-conversations = [
-    [{"role": "user", "content": x}] for x in prompts
-]
-
-outputs = llm.chat(conversations, sampling_params=sampling_params)
-
-for output in outputs:
-    print(output.outputs[0].text)
-
-# <think>
-# まず、学校の総生徒数を算出します。各クラスに20人の生徒があり、クラスは3つあるため、総生徒数は60人です。
-
-# 次に、学校全体で男子と女子は同じ人数で分布しています。したがって、男子と女子各有30人。
-...
-# したがって、3つ目のクラスの男子数は20 - 3 = 17人です。
-# </think>
-
-# **解答：**
-
-# 学校の総生徒数を算出します。
-...
-# **最終的な答え：**
-# \[
-# \boxed{17}
-# \]
-```
-
-</details></li>
-</ul>	
-
-# Evaluation
-
-Through some quick evaluation of our own, we found this model can produce much correctly formatted and accurate results for higher resource languages, such as Japanese, English, German, than lower resource languages, such as Amharic or Lao.
-
-We did a **very** quick evaluation of 5 questions with each dataset (written by me and translated by GPT4o Mini) on the [lightblue/DeepSeek-R1-Distill-Qwen-7B-Multilingual](https://huggingface.co/lightblue/DeepSeek-R1-Distill-Qwen-7B-Multilingual) model, and we find that the model is able to fairly reliably output the correct answers and in the correct language for a large variety of languages:
-
-For this evaluation, a score of >=0.8 is good, as one of the questions was very hard. The language detection was done using [pycld2](https://pypi.org/project/pycld2/) so errors may occur with the correct language being mistaken for another one.
-
-| language        |   Has a correct think statement |   Has the think statement in the correct language |   Is the response in the correct language |   Is the answer correct |
-|:----------------|------------:|------------------------:|----------------------:|-------------:|
-| Amharic         |         0.2 |                     0   |                   0   |          0   |
-| Arabic          |         1   |                     0.8 |                   0.8 |          0.6 |
-| Bengali         |         1   |                     1   |                   1   |          0.2 |
-| Chinese         |         1   |                     1   |                   1   |          0.8 |
-| Czech           |         1   |                     1   |                   1   |          0.8 |
-| Dutch           |         1   |                     1   |                   1   |          0.8 |
-| English         |         1   |                     1   |                   1   |          0.8 |
-| French          |         1   |                     1   |                   1   |          0.8 |
-| German          |         1   |                     1   |                   1   |          0.8 |
-| Greek           |         1   |                     1   |                   1   |          0.6 |
-| Hausa           |         0.4 |                     0   |                   0   |          0   |
-| Hebrew          |         1   |                     0.8 |                   1   |          0.6 |
-| Hindi           |         1   |                     1   |                   1   |          0.8 |
-| Indonesian      |         1   |                     1   |                   1   |          0.8 |
-| Italian         |         1   |                     1   |                   1   |          0.8 |
-| Japanese        |         1   |                     1   |                   0.8 |          0.6 |
-| Javanese        |         0.8 |                     0.2 |                   0.2 |          0.6 |
-| Khmer           |         0.6 |                     0.6 |                   0.6 |          0   |
-| Korean          |         1   |                     1   |                   1   |          1   |
-| Lao             |         0.4 |                     0.4 |                   0.4 |          0   |
-| Malay           |         1   |                     0.4 |                   0.4 |          0.8 |
-| Marathi         |         0.6 |                     0.4 |                   0.6 |          0.2 |
-| Persian (Farsi) |         0.6 |                     None*   |                   None*   |          0.2 |
-| Polish          |         1   |                     1   |                   1   |          0.6 |
-| Portuguese      |         1   |                     1   |                   1   |          0.8 |
-| Romanian        |         1   |                     1   |                   1   |          0.8 |
-| Russian         |         1   |                     1   |                   1   |          0.8 |
-| Spanish         |         1   |                     1   |                   1   |          0.8 |
-| Swahili         |         0.4 |                     0.4 |                   0.4 |          0   |
-| Swedish         |         1   |                     1   |                   1   |          0.8 |
-| Tagalog         |         1   |                     1   |                   1   |          0.8 |
-| Tamil           |         0.8 |                     0.8 |                   0.8 |          0.2 |
-| Telugu          |         0.8 |                     0.6 |                   0.8 |          0   |
-| Thai            |         1   |                     1   |                   1   |          0.8 |
-| Turkish         |         1   |                     1   |                   1   |          0.8 |
-| Ukrainian       |         1   |                     1   |                   1   |          0.8 |
-| Urdu            |         1   |                     1   |                   1   |          0.6 |
-| Vietnamese      |         1   |                     1   |                   1   |          1   |
-
-* There was an error with Farsi detection (my own fault) so we do not report Farsi scores.
-
-The evaluation code for this can be found [here](https://drive.google.com/file/d/1P33GpqvKmHoZUsWqqBPXHTToN2W7MDRG/view?usp=sharing).
-
-# Training code
-
-```yaml
-### model
-model_name_or_path: deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
-
-### method
-stage: sft
-do_train: true
-finetuning_type: full
-deepspeed: /root/LLaMA-Factory/examples/deepspeed/ds_z2_config.json
-
-### dataset
-dataset: reasoning-multilingual-R1-Llama-70B-train
-template: qwen
-cutoff_len: 4500
-overwrite_cache: true
-preprocessing_num_workers: 16
-packing: true
-
-### output
-output_dir: /root/train_outputs/DeepSeek-R1-Distill-Qwen-7B/reasoning-multilingual-R1-Llama-70B-train
-logging_steps: 1
-save_steps: 0.99999
-plot_loss: true
-overwrite_output_dir: true
-
-### train
-per_device_train_batch_size: 1
-gradient_accumulation_steps: 1
-learning_rate: 1.0e-5
-num_train_epochs: 1.0
-lr_scheduler_type: cosine
-warmup_ratio: 0.01
-bf16: true
-ddp_timeout: 180000000
-
-### eval
-val_size: 0.01
-per_device_eval_batch_size: 1
-eval_strategy: steps
-eval_steps: 0.1
-```
-
-```bash
-echo '{
-  "reasoning-multilingual-R1-Llama-70B-train": {
-    "hf_hub_url": "lightblue/reasoning-multilingual-R1-Llama-70B-train",
-    "formatting": "sharegpt"
-  }
-}' > /root/LLaMA-Factory/data/dataset_info.json
-
-# 7B Qwen
-cd /root/LLaMA-Factory && llamafactory-cli train /root/reasoning_multilingual_train_7B.yaml
-rm -r /root/train_outputs/DeepSeek-R1-Distill-Qwen-7B/reasoning-multilingual-R1-Llama-70B-train/checkpoint*
-huggingface-cli upload lightblue/DeepSeek-R1-Distill-Qwen-7B-Multilingual /root/train_outputs/DeepSeek-R1-Distill-Qwen-7B/reasoning-multilingual-R1-Llama-70B-train
-
-```
-
-# License
-
-We share this model with the Apache 2.0 license.
-
-# Developed by
-
-<a href="https://www.lightblue-tech.com">
-<img src="https://www.lightblue-tech.com/wp-content/uploads/2023/08/color_%E6%A8%AA%E5%9E%8B-1536x469.png" alt="Lightblue technology logo" width="400"/>
-</a>
-
-This model was trained by Peter Devine ([ptrdvn](https://huggingface.co/ptrdvn)) for Lightblue
+- Transformers 4.48.1
+- Pytorch 2.5.1+cu124
+- Datasets 3.1.0
+- Tokenizers 0.21.0

all_results.json

@@ -0,0 +1,12 @@
+{
+    "epoch": 1.0,
+    "eval_loss": 0.444090336561203,
+    "eval_runtime": 7.0965,
+    "eval_samples_per_second": 1.268,
+    "eval_steps_per_second": 0.282,
+    "total_flos": 17851226259456.0,
+    "train_loss": 0.46821982275556634,
+    "train_runtime": 1249.8096,
+    "train_samples_per_second": 0.69,
+    "train_steps_per_second": 0.086
+}

config.json

@@ -0,0 +1,29 @@
+{
+  "_name_or_path": "deepseek-ai/DeepSeek-R1-Distill-Qwen-14B",
+  "architectures": [
+    "Qwen2ForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "eos_token_id": 151643,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 13824,
+  "max_position_embeddings": 131072,
+  "max_window_layers": 48,
+  "model_type": "qwen2",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 48,
+  "num_key_value_heads": 8,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 1000000.0,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.48.1",
+  "use_cache": false,
+  "use_sliding_window": false,
+  "vocab_size": 152064
+}

eval_results.json

@@ -0,0 +1,7 @@
+{
+    "epoch": 1.0,
+    "eval_loss": 0.444090336561203,
+    "eval_runtime": 7.0965,
+    "eval_samples_per_second": 1.268,
+    "eval_steps_per_second": 0.282
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 151646,
+  "do_sample": true,
+  "eos_token_id": 151643,
+  "temperature": 0.6,
+  "top_p": 0.95,
+  "transformers_version": "4.48.1"
+}

model-00001-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a4f31f8877c02b6b9caf33fd7c9abf1f729ce96f7a616f191b1a80e5a1b064ca
+size 4986211280

model-00002-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4ec30aac47515ce6acc0ea5acc9169523b8068cc5d752761ce7f7c7b1c32d988
+size 4954847344

model-00003-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bd6cb1be327f257531db935262a32ffe1d99bb37c42f6ab25e297fa4fef85f5f
+size 4954847392

model-00004-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d6662d6cff9fc0f583c84a05b8a42aef420e717a7273824cdac2be24b62ac33e
+size 4954847392

model-00005-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc5e2a0d2855e0066164fde079b7c42e5216c198c84095af739e7da33b4ab36a
+size 4954847392

model-00006-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ac3d4534506b0566250063309be8ea205b1d5dd988a9d51d9d872028bf93d3a
+size 4734533160

model.safetensors.index.json

@@ -0,0 +1,586 @@
+{
+  "metadata": {
+    "total_size": 29540067328
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00006-of-00006.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00006.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "model.layers.0.self_attn.k_proj.bias": "model-00001-of-00006.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.0.self_attn.q_proj.bias": "model-00001-of-00006.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.0.self_attn.v_proj.bias": "model-00001-of-00006.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "model.layers.1.self_attn.k_proj.bias": "model-00001-of-00006.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.1.self_attn.q_proj.bias": "model-00001-of-00006.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.1.self_attn.v_proj.bias": "model-00001-of-00006.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "model.layers.10.self_attn.k_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.10.self_attn.q_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.10.self_attn.v_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "model.layers.11.self_attn.k_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.11.self_attn.q_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.11.self_attn.v_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "model.layers.12.self_attn.k_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.12.self_attn.q_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.12.self_attn.v_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "model.layers.13.self_attn.k_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.13.self_attn.q_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.13.self_attn.v_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "model.layers.14.self_attn.k_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.14.self_attn.q_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.14.self_attn.v_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "model.layers.15.self_attn.k_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.15.self_attn.q_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.15.self_attn.v_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "model.layers.16.self_attn.k_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.16.self_attn.q_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.16.self_attn.v_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "model.layers.17.self_attn.k_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.17.self_attn.q_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.17.self_attn.v_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "model.layers.18.self_attn.k_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.18.self_attn.q_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.18.self_attn.v_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "model.layers.19.self_attn.k_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.19.self_attn.q_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.19.self_attn.v_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "model.layers.2.self_attn.k_proj.bias": "model-00001-of-00006.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.2.self_attn.q_proj.bias": "model-00001-of-00006.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.2.self_attn.v_proj.bias": "model-00001-of-00006.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "model.layers.20.self_attn.k_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.20.self_attn.q_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.20.self_attn.v_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "model.layers.21.self_attn.k_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.21.self_attn.q_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.21.self_attn.v_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "model.layers.22.self_attn.k_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.22.self_attn.q_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.22.self_attn.v_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "model.layers.23.self_attn.k_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.23.self_attn.q_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.23.self_attn.v_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "model.layers.24.self_attn.k_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.24.self_attn.q_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.24.self_attn.v_proj.bias": "model-00003-of-00006.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "model.layers.25.self_attn.k_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.25.self_attn.q_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.25.self_attn.v_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.26.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.26.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "model.layers.26.self_attn.k_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.26.self_attn.q_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.26.self_attn.v_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.27.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.27.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "model.layers.27.self_attn.k_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.27.self_attn.q_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.27.self_attn.v_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.28.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "model.layers.28.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.28.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.28.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "model.layers.28.self_attn.k_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.28.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.28.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.28.self_attn.q_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.28.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.28.self_attn.v_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.28.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.29.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "model.layers.29.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.29.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.29.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "model.layers.29.self_attn.k_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.29.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.29.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.29.self_attn.q_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.29.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.29.self_attn.v_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.29.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "model.layers.3.self_attn.k_proj.bias": "model-00001-of-00006.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.3.self_attn.q_proj.bias": "model-00001-of-00006.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.3.self_attn.v_proj.bias": "model-00001-of-00006.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.30.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "model.layers.30.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.30.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.30.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "model.layers.30.self_attn.k_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.30.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.30.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.30.self_attn.q_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.30.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.30.self_attn.v_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.30.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.31.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "model.layers.31.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.31.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.31.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "model.layers.31.self_attn.k_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.31.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.31.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.31.self_attn.q_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.31.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.31.self_attn.v_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.31.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.32.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "model.layers.32.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.32.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.32.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.32.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "model.layers.32.self_attn.k_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.32.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.32.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.32.self_attn.q_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.32.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.32.self_attn.v_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.32.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.33.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "model.layers.33.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.33.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.33.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.33.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "model.layers.33.self_attn.k_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.33.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.33.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.33.self_attn.q_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.33.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.33.self_attn.v_proj.bias": "model-00004-of-00006.safetensors",
+    "model.layers.33.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "model.layers.34.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "model.layers.34.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.34.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.34.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.34.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "model.layers.34.self_attn.k_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.34.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.34.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.34.self_attn.q_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.34.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.34.self_attn.v_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.34.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.35.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "model.layers.35.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.35.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.35.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.35.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "model.layers.35.self_attn.k_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.35.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.35.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.35.self_attn.q_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.35.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.35.self_attn.v_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.35.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.36.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "model.layers.36.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.36.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.36.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.36.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "model.layers.36.self_attn.k_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.36.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.36.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.36.self_attn.q_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.36.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.36.self_attn.v_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.36.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.37.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "model.layers.37.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.37.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.37.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.37.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "model.layers.37.self_attn.k_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.37.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.37.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.37.self_attn.q_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.37.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.37.self_attn.v_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.37.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.38.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "model.layers.38.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.38.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.38.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.38.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "model.layers.38.self_attn.k_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.38.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.38.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.38.self_attn.q_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.38.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.38.self_attn.v_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.38.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.39.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "model.layers.39.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.39.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.39.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.39.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "model.layers.39.self_attn.k_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.39.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.39.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.39.self_attn.q_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.39.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.39.self_attn.v_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.39.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "model.layers.4.self_attn.k_proj.bias": "model-00001-of-00006.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.4.self_attn.q_proj.bias": "model-00001-of-00006.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.4.self_attn.v_proj.bias": "model-00001-of-00006.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.40.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "model.layers.40.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.40.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.40.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.40.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "model.layers.40.self_attn.k_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.40.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.40.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.40.self_attn.q_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.40.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.40.self_attn.v_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.40.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.41.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "model.layers.41.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.41.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.41.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.41.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "model.layers.41.self_attn.k_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.41.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.41.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.41.self_attn.q_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.41.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.41.self_attn.v_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.41.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.42.input_layernorm.weight": "model-00006-of-00006.safetensors",
+    "model.layers.42.mlp.down_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.42.mlp.gate_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.42.mlp.up_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.42.post_attention_layernorm.weight": "model-00006-of-00006.safetensors",
+    "model.layers.42.self_attn.k_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.42.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.42.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.42.self_attn.q_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.42.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.42.self_attn.v_proj.bias": "model-00005-of-00006.safetensors",
+    "model.layers.42.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "model.layers.43.input_layernorm.weight": "model-00006-of-00006.safetensors",
+    "model.layers.43.mlp.down_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.43.mlp.gate_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.43.mlp.up_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.43.post_attention_layernorm.weight": "model-00006-of-00006.safetensors",
+    "model.layers.43.self_attn.k_proj.bias": "model-00006-of-00006.safetensors",
+    "model.layers.43.self_attn.k_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.43.self_attn.o_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.43.self_attn.q_proj.bias": "model-00006-of-00006.safetensors",
+    "model.layers.43.self_attn.q_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.43.self_attn.v_proj.bias": "model-00006-of-00006.safetensors",
+    "model.layers.43.self_attn.v_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.44.input_layernorm.weight": "model-00006-of-00006.safetensors",
+    "model.layers.44.mlp.down_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.44.mlp.gate_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.44.mlp.up_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.44.post_attention_layernorm.weight": "model-00006-of-00006.safetensors",
+    "model.layers.44.self_attn.k_proj.bias": "model-00006-of-00006.safetensors",
+    "model.layers.44.self_attn.k_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.44.self_attn.o_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.44.self_attn.q_proj.bias": "model-00006-of-00006.safetensors",
+    "model.layers.44.self_attn.q_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.44.self_attn.v_proj.bias": "model-00006-of-00006.safetensors",
+    "model.layers.44.self_attn.v_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.45.input_layernorm.weight": "model-00006-of-00006.safetensors",
+    "model.layers.45.mlp.down_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.45.mlp.gate_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.45.mlp.up_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.45.post_attention_layernorm.weight": "model-00006-of-00006.safetensors",
+    "model.layers.45.self_attn.k_proj.bias": "model-00006-of-00006.safetensors",
+    "model.layers.45.self_attn.k_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.45.self_attn.o_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.45.self_attn.q_proj.bias": "model-00006-of-00006.safetensors",
+    "model.layers.45.self_attn.q_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.45.self_attn.v_proj.bias": "model-00006-of-00006.safetensors",
+    "model.layers.45.self_attn.v_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.46.input_layernorm.weight": "model-00006-of-00006.safetensors",
+    "model.layers.46.mlp.down_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.46.mlp.gate_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.46.mlp.up_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.46.post_attention_layernorm.weight": "model-00006-of-00006.safetensors",
+    "model.layers.46.self_attn.k_proj.bias": "model-00006-of-00006.safetensors",
+    "model.layers.46.self_attn.k_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.46.self_attn.o_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.46.self_attn.q_proj.bias": "model-00006-of-00006.safetensors",
+    "model.layers.46.self_attn.q_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.46.self_attn.v_proj.bias": "model-00006-of-00006.safetensors",
+    "model.layers.46.self_attn.v_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.47.input_layernorm.weight": "model-00006-of-00006.safetensors",
+    "model.layers.47.mlp.down_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.47.mlp.gate_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.47.mlp.up_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.47.post_attention_layernorm.weight": "model-00006-of-00006.safetensors",
+    "model.layers.47.self_attn.k_proj.bias": "model-00006-of-00006.safetensors",
+    "model.layers.47.self_attn.k_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.47.self_attn.o_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.47.self_attn.q_proj.bias": "model-00006-of-00006.safetensors",
+    "model.layers.47.self_attn.q_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.47.self_attn.v_proj.bias": "model-00006-of-00006.safetensors",
+    "model.layers.47.self_attn.v_proj.weight": "model-00006-of-00006.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "model.layers.5.self_attn.k_proj.bias": "model-00001-of-00006.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.5.self_attn.q_proj.bias": "model-00001-of-00006.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.5.self_attn.v_proj.bias": "model-00001-of-00006.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "model.layers.6.self_attn.k_proj.bias": "model-00001-of-00006.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.6.self_attn.q_proj.bias": "model-00001-of-00006.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.6.self_attn.v_proj.bias": "model-00001-of-00006.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "model.layers.7.self_attn.k_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.7.self_attn.q_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.7.self_attn.v_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "model.layers.8.self_attn.k_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.8.self_attn.q_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.8.self_attn.v_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "model.layers.9.self_attn.k_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.9.self_attn.q_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "model.layers.9.self_attn.v_proj.bias": "model-00002-of-00006.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "model.norm.weight": "model-00006-of-00006.safetensors"
+  }
+}

special_tokens_map.json

@@ -0,0 +1,32 @@
+{
+  "additional_special_tokens": [
+    {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
+  ],
+  "bos_token": {
+    "content": "<｜begin▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<｜end▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<｜end▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02643f00207dfc5ed248992486bde04314c21dca556bf65ce520690962b8db63
+size 11422965

tokenizer_config.json

@@ -0,0 +1,208 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<｜end▁of▁sentence｜>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<｜User｜>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151645": {
+      "content": "<｜Assistant｜>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151646": {
+      "content": "<｜begin▁of▁sentence｜>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|EOT|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151648": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151649": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_end|>"
+  ],
+  "bos_token": "<｜begin▁of▁sentence｜>",
+  "chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<｜end▁of▁sentence｜>",
+  "extra_special_tokens": {},
+  "legacy": true,
+  "model_max_length": 4096,
+  "pad_token": "<｜end▁of▁sentence｜>",
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "split_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": null,
+  "use_default_system_prompt": false
+}

train_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 1.0,
+    "total_flos": 17851226259456.0,
+    "train_loss": 0.46821982275556634,
+    "train_runtime": 1249.8096,
+    "train_samples_per_second": 0.69,
+    "train_steps_per_second": 0.086
+}

trainer_log.jsonl

@@ -0,0 +1,118 @@
+{"current_steps": 1, "total_steps": 108, "loss": 0.5807, "lr": 5e-06, "epoch": 0.009259259259259259, "percentage": 0.93, "elapsed_time": "0:00:13", "remaining_time": "0:24:55"}
+{"current_steps": 2, "total_steps": 108, "loss": 0.534, "lr": 1e-05, "epoch": 0.018518518518518517, "percentage": 1.85, "elapsed_time": "0:00:24", "remaining_time": "0:21:27"}
+{"current_steps": 3, "total_steps": 108, "loss": 0.6069, "lr": 9.997804182543973e-06, "epoch": 0.027777777777777776, "percentage": 2.78, "elapsed_time": "0:00:34", "remaining_time": "0:20:09"}
+{"current_steps": 4, "total_steps": 108, "loss": 0.5863, "lr": 9.991218658821609e-06, "epoch": 0.037037037037037035, "percentage": 3.7, "elapsed_time": "0:00:44", "remaining_time": "0:19:22"}
+{"current_steps": 5, "total_steps": 108, "loss": 0.5952, "lr": 9.980249213076085e-06, "epoch": 0.046296296296296294, "percentage": 4.63, "elapsed_time": "0:00:54", "remaining_time": "0:18:50"}
+{"current_steps": 6, "total_steps": 108, "loss": 0.5384, "lr": 9.964905480067585e-06, "epoch": 0.05555555555555555, "percentage": 5.56, "elapsed_time": "0:01:05", "remaining_time": "0:18:27"}
+{"current_steps": 7, "total_steps": 108, "loss": 0.5779, "lr": 9.945200936610821e-06, "epoch": 0.06481481481481481, "percentage": 6.48, "elapsed_time": "0:01:15", "remaining_time": "0:18:07"}
+{"current_steps": 8, "total_steps": 108, "loss": 0.5487, "lr": 9.921152889737985e-06, "epoch": 0.07407407407407407, "percentage": 7.41, "elapsed_time": "0:01:25", "remaining_time": "0:17:49"}
+{"current_steps": 9, "total_steps": 108, "loss": 0.5162, "lr": 9.892782461497521e-06, "epoch": 0.08333333333333333, "percentage": 8.33, "elapsed_time": "0:01:35", "remaining_time": "0:17:33"}
+{"current_steps": 10, "total_steps": 108, "loss": 0.4825, "lr": 9.860114570402055e-06, "epoch": 0.09259259259259259, "percentage": 9.26, "elapsed_time": "0:01:45", "remaining_time": "0:17:18"}
+{"current_steps": 11, "total_steps": 108, "loss": 0.4962, "lr": 9.823177909541795e-06, "epoch": 0.10185185185185185, "percentage": 10.19, "elapsed_time": "0:01:56", "remaining_time": "0:17:04"}
+{"current_steps": 11, "total_steps": 108, "eval_loss": 0.5000379085540771, "epoch": 0.10185185185185185, "percentage": 10.19, "elapsed_time": "0:02:03", "remaining_time": "0:18:08"}
+{"current_steps": 12, "total_steps": 108, "loss": 0.4456, "lr": 9.782004921382612e-06, "epoch": 0.1111111111111111, "percentage": 11.11, "elapsed_time": "0:02:13", "remaining_time": "0:17:47"}
+{"current_steps": 13, "total_steps": 108, "loss": 0.4288, "lr": 9.736631769270958e-06, "epoch": 0.12037037037037036, "percentage": 12.04, "elapsed_time": "0:02:23", "remaining_time": "0:17:29"}
+{"current_steps": 14, "total_steps": 108, "loss": 0.5855, "lr": 9.687098305670606e-06, "epoch": 0.12962962962962962, "percentage": 12.96, "elapsed_time": "0:02:33", "remaining_time": "0:17:13"}
+{"current_steps": 15, "total_steps": 108, "loss": 0.5184, "lr": 9.633448037159167e-06, "epoch": 0.1388888888888889, "percentage": 13.89, "elapsed_time": "0:02:44", "remaining_time": "0:16:57"}
+{"current_steps": 16, "total_steps": 108, "loss": 0.5568, "lr": 9.575728086215093e-06, "epoch": 0.14814814814814814, "percentage": 14.81, "elapsed_time": "0:02:54", "remaining_time": "0:16:42"}
+{"current_steps": 17, "total_steps": 108, "loss": 0.4272, "lr": 9.513989149828718e-06, "epoch": 0.1574074074074074, "percentage": 15.74, "elapsed_time": "0:03:04", "remaining_time": "0:16:28"}
+{"current_steps": 18, "total_steps": 108, "loss": 0.487, "lr": 9.448285454973739e-06, "epoch": 0.16666666666666666, "percentage": 16.67, "elapsed_time": "0:03:14", "remaining_time": "0:16:14"}
+{"current_steps": 19, "total_steps": 108, "loss": 0.5321, "lr": 9.378674710978185e-06, "epoch": 0.17592592592592593, "percentage": 17.59, "elapsed_time": "0:03:25", "remaining_time": "0:16:00"}
+{"current_steps": 20, "total_steps": 108, "loss": 0.5396, "lr": 9.305218058836778e-06, "epoch": 0.18518518518518517, "percentage": 18.52, "elapsed_time": "0:03:35", "remaining_time": "0:15:47"}
+{"current_steps": 21, "total_steps": 108, "loss": 0.446, "lr": 9.22798001750913e-06, "epoch": 0.19444444444444445, "percentage": 19.44, "elapsed_time": "0:03:45", "remaining_time": "0:15:34"}
+{"current_steps": 22, "total_steps": 108, "loss": 0.5313, "lr": 9.14702842725101e-06, "epoch": 0.2037037037037037, "percentage": 20.37, "elapsed_time": "0:03:55", "remaining_time": "0:15:21"}
+{"current_steps": 22, "total_steps": 108, "eval_loss": 0.4791085124015808, "epoch": 0.2037037037037037, "percentage": 20.37, "elapsed_time": "0:04:02", "remaining_time": "0:15:49"}
+{"current_steps": 23, "total_steps": 108, "loss": 0.4983, "lr": 9.062434390028407e-06, "epoch": 0.21296296296296297, "percentage": 21.3, "elapsed_time": "0:04:13", "remaining_time": "0:15:36"}
+{"current_steps": 24, "total_steps": 108, "loss": 0.4642, "lr": 8.974272207066767e-06, "epoch": 0.2222222222222222, "percentage": 22.22, "elapsed_time": "0:04:23", "remaining_time": "0:15:23"}
+{"current_steps": 25, "total_steps": 108, "loss": 0.4832, "lr": 8.882619313590212e-06, "epoch": 0.23148148148148148, "percentage": 23.15, "elapsed_time": "0:04:34", "remaining_time": "0:15:10"}
+{"current_steps": 26, "total_steps": 108, "loss": 0.441, "lr": 8.787556210808101e-06, "epoch": 0.24074074074074073, "percentage": 24.07, "elapsed_time": "0:04:44", "remaining_time": "0:14:56"}
+{"current_steps": 27, "total_steps": 108, "loss": 0.4489, "lr": 8.689166395208638e-06, "epoch": 0.25, "percentage": 25.0, "elapsed_time": "0:04:54", "remaining_time": "0:14:43"}
+{"current_steps": 28, "total_steps": 108, "loss": 0.4971, "lr": 8.587536285221656e-06, "epoch": 0.25925925925925924, "percentage": 25.93, "elapsed_time": "0:05:04", "remaining_time": "0:14:31"}
+{"current_steps": 29, "total_steps": 108, "loss": 0.5289, "lr": 8.482755145314987e-06, "epoch": 0.26851851851851855, "percentage": 26.85, "elapsed_time": "0:05:15", "remaining_time": "0:14:18"}
+{"current_steps": 30, "total_steps": 108, "loss": 0.5496, "lr": 8.374915007591053e-06, "epoch": 0.2777777777777778, "percentage": 27.78, "elapsed_time": "0:05:25", "remaining_time": "0:14:06"}
+{"current_steps": 31, "total_steps": 108, "loss": 0.5261, "lr": 8.264110590952609e-06, "epoch": 0.28703703703703703, "percentage": 28.7, "elapsed_time": "0:05:35", "remaining_time": "0:13:53"}
+{"current_steps": 32, "total_steps": 108, "loss": 0.4814, "lr": 8.150439217908557e-06, "epoch": 0.2962962962962963, "percentage": 29.63, "elapsed_time": "0:05:45", "remaining_time": "0:13:41"}
+{"current_steps": 33, "total_steps": 108, "loss": 0.4692, "lr": 8.034000729092967e-06, "epoch": 0.3055555555555556, "percentage": 30.56, "elapsed_time": "0:05:56", "remaining_time": "0:13:29"}
+{"current_steps": 33, "total_steps": 108, "eval_loss": 0.46850860118865967, "epoch": 0.3055555555555556, "percentage": 30.56, "elapsed_time": "0:06:03", "remaining_time": "0:13:45"}
+{"current_steps": 34, "total_steps": 108, "loss": 0.4372, "lr": 7.914897395572362e-06, "epoch": 0.3148148148148148, "percentage": 31.48, "elapsed_time": "0:06:13", "remaining_time": "0:13:33"}
+{"current_steps": 35, "total_steps": 108, "loss": 0.5235, "lr": 7.793233829018263e-06, "epoch": 0.32407407407407407, "percentage": 32.41, "elapsed_time": "0:06:23", "remaining_time": "0:13:20"}
+{"current_steps": 36, "total_steps": 108, "loss": 0.4339, "lr": 7.669116889823955e-06, "epoch": 0.3333333333333333, "percentage": 33.33, "elapsed_time": "0:06:34", "remaining_time": "0:13:08"}
+{"current_steps": 37, "total_steps": 108, "loss": 0.4426, "lr": 7.542655593246103e-06, "epoch": 0.3425925925925926, "percentage": 34.26, "elapsed_time": "0:06:44", "remaining_time": "0:12:55"}
+{"current_steps": 38, "total_steps": 108, "loss": 0.5744, "lr": 7.413961013653725e-06, "epoch": 0.35185185185185186, "percentage": 35.19, "elapsed_time": "0:06:54", "remaining_time": "0:12:43"}
+{"current_steps": 39, "total_steps": 108, "loss": 0.3849, "lr": 7.283146186968566e-06, "epoch": 0.3611111111111111, "percentage": 36.11, "elapsed_time": "0:07:04", "remaining_time": "0:12:31"}
+{"current_steps": 40, "total_steps": 108, "loss": 0.4057, "lr": 7.1503260113826035e-06, "epoch": 0.37037037037037035, "percentage": 37.04, "elapsed_time": "0:07:15", "remaining_time": "0:12:19"}
+{"current_steps": 41, "total_steps": 108, "loss": 0.4149, "lr": 7.015617146439863e-06, "epoch": 0.37962962962962965, "percentage": 37.96, "elapsed_time": "0:07:25", "remaining_time": "0:12:07"}
+{"current_steps": 42, "total_steps": 108, "loss": 0.4557, "lr": 6.879137910571191e-06, "epoch": 0.3888888888888889, "percentage": 38.89, "elapsed_time": "0:07:35", "remaining_time": "0:11:55"}
+{"current_steps": 43, "total_steps": 108, "loss": 0.5052, "lr": 6.741008177171995e-06, "epoch": 0.39814814814814814, "percentage": 39.81, "elapsed_time": "0:07:45", "remaining_time": "0:11:43"}
+{"current_steps": 44, "total_steps": 108, "loss": 0.3876, "lr": 6.601349269314188e-06, "epoch": 0.4074074074074074, "percentage": 40.74, "elapsed_time": "0:07:55", "remaining_time": "0:11:32"}
+{"current_steps": 44, "total_steps": 108, "eval_loss": 0.4595443606376648, "epoch": 0.4074074074074074, "percentage": 40.74, "elapsed_time": "0:08:03", "remaining_time": "0:11:42"}
+{"current_steps": 45, "total_steps": 108, "loss": 0.4381, "lr": 6.46028385318488e-06, "epoch": 0.4166666666666667, "percentage": 41.67, "elapsed_time": "0:08:13", "remaining_time": "0:11:30"}
+{"current_steps": 46, "total_steps": 108, "loss": 0.433, "lr": 6.3179358303453386e-06, "epoch": 0.42592592592592593, "percentage": 42.59, "elapsed_time": "0:08:23", "remaining_time": "0:11:18"}
+{"current_steps": 47, "total_steps": 108, "loss": 0.4699, "lr": 6.17443022890492e-06, "epoch": 0.4351851851851852, "percentage": 43.52, "elapsed_time": "0:08:33", "remaining_time": "0:11:06"}
+{"current_steps": 48, "total_steps": 108, "loss": 0.4584, "lr": 6.029893093705492e-06, "epoch": 0.4444444444444444, "percentage": 44.44, "elapsed_time": "0:08:43", "remaining_time": "0:10:54"}
+{"current_steps": 49, "total_steps": 108, "loss": 0.5243, "lr": 5.884451375612865e-06, "epoch": 0.4537037037037037, "percentage": 45.37, "elapsed_time": "0:08:54", "remaining_time": "0:10:43"}
+{"current_steps": 50, "total_steps": 108, "loss": 0.4329, "lr": 5.738232820012407e-06, "epoch": 0.46296296296296297, "percentage": 46.3, "elapsed_time": "0:09:04", "remaining_time": "0:10:31"}
+{"current_steps": 51, "total_steps": 108, "loss": 0.5633, "lr": 5.591365854606829e-06, "epoch": 0.4722222222222222, "percentage": 47.22, "elapsed_time": "0:09:14", "remaining_time": "0:10:19"}
+{"current_steps": 52, "total_steps": 108, "loss": 0.445, "lr": 5.443979476614674e-06, "epoch": 0.48148148148148145, "percentage": 48.15, "elapsed_time": "0:09:24", "remaining_time": "0:10:08"}
+{"current_steps": 53, "total_steps": 108, "loss": 0.3809, "lr": 5.296203139468572e-06, "epoch": 0.49074074074074076, "percentage": 49.07, "elapsed_time": "0:09:35", "remaining_time": "0:09:56"}
+{"current_steps": 54, "total_steps": 108, "loss": 0.3964, "lr": 5.148166639112799e-06, "epoch": 0.5, "percentage": 50.0, "elapsed_time": "0:09:45", "remaining_time": "0:09:45"}
+{"current_steps": 55, "total_steps": 108, "loss": 0.4768, "lr": 5e-06, "epoch": 0.5092592592592593, "percentage": 50.93, "elapsed_time": "0:09:55", "remaining_time": "0:09:33"}
+{"current_steps": 55, "total_steps": 108, "eval_loss": 0.4542139768600464, "epoch": 0.5092592592592593, "percentage": 50.93, "elapsed_time": "0:10:02", "remaining_time": "0:09:40"}
+{"current_steps": 56, "total_steps": 108, "loss": 0.4238, "lr": 4.8518333608872015e-06, "epoch": 0.5185185185185185, "percentage": 51.85, "elapsed_time": "0:10:13", "remaining_time": "0:09:29"}
+{"current_steps": 57, "total_steps": 108, "loss": 0.5484, "lr": 4.703796860531429e-06, "epoch": 0.5277777777777778, "percentage": 52.78, "elapsed_time": "0:10:23", "remaining_time": "0:09:17"}
+{"current_steps": 58, "total_steps": 108, "loss": 0.397, "lr": 4.556020523385326e-06, "epoch": 0.5370370370370371, "percentage": 53.7, "elapsed_time": "0:10:33", "remaining_time": "0:09:06"}
+{"current_steps": 59, "total_steps": 108, "loss": 0.4452, "lr": 4.408634145393172e-06, "epoch": 0.5462962962962963, "percentage": 54.63, "elapsed_time": "0:10:44", "remaining_time": "0:08:54"}
+{"current_steps": 60, "total_steps": 108, "loss": 0.4896, "lr": 4.261767179987595e-06, "epoch": 0.5555555555555556, "percentage": 55.56, "elapsed_time": "0:10:54", "remaining_time": "0:08:43"}
+{"current_steps": 61, "total_steps": 108, "loss": 0.4887, "lr": 4.115548624387136e-06, "epoch": 0.5648148148148148, "percentage": 56.48, "elapsed_time": "0:11:04", "remaining_time": "0:08:31"}
+{"current_steps": 62, "total_steps": 108, "loss": 0.4997, "lr": 3.970106906294509e-06, "epoch": 0.5740740740740741, "percentage": 57.41, "elapsed_time": "0:11:14", "remaining_time": "0:08:20"}
+{"current_steps": 63, "total_steps": 108, "loss": 0.417, "lr": 3.825569771095082e-06, "epoch": 0.5833333333333334, "percentage": 58.33, "elapsed_time": "0:11:24", "remaining_time": "0:08:09"}
+{"current_steps": 64, "total_steps": 108, "loss": 0.4504, "lr": 3.682064169654663e-06, "epoch": 0.5925925925925926, "percentage": 59.26, "elapsed_time": "0:11:35", "remaining_time": "0:07:57"}
+{"current_steps": 65, "total_steps": 108, "loss": 0.4694, "lr": 3.539716146815122e-06, "epoch": 0.6018518518518519, "percentage": 60.19, "elapsed_time": "0:11:45", "remaining_time": "0:07:46"}
+{"current_steps": 66, "total_steps": 108, "loss": 0.4985, "lr": 3.398650730685813e-06, "epoch": 0.6111111111111112, "percentage": 61.11, "elapsed_time": "0:11:55", "remaining_time": "0:07:35"}
+{"current_steps": 66, "total_steps": 108, "eval_loss": 0.4495806097984314, "epoch": 0.6111111111111112, "percentage": 61.11, "elapsed_time": "0:12:02", "remaining_time": "0:07:40"}
+{"current_steps": 67, "total_steps": 108, "loss": 0.4104, "lr": 3.258991822828007e-06, "epoch": 0.6203703703703703, "percentage": 62.04, "elapsed_time": "0:12:13", "remaining_time": "0:07:28"}
+{"current_steps": 68, "total_steps": 108, "loss": 0.4926, "lr": 3.1208620894288105e-06, "epoch": 0.6296296296296297, "percentage": 62.96, "elapsed_time": "0:12:23", "remaining_time": "0:07:17"}
+{"current_steps": 69, "total_steps": 108, "loss": 0.4655, "lr": 2.98438285356014e-06, "epoch": 0.6388888888888888, "percentage": 63.89, "elapsed_time": "0:12:34", "remaining_time": "0:07:06"}
+{"current_steps": 70, "total_steps": 108, "loss": 0.4535, "lr": 2.8496739886173994e-06, "epoch": 0.6481481481481481, "percentage": 64.81, "elapsed_time": "0:12:44", "remaining_time": "0:06:54"}
+{"current_steps": 71, "total_steps": 108, "loss": 0.5186, "lr": 2.716853813031435e-06, "epoch": 0.6574074074074074, "percentage": 65.74, "elapsed_time": "0:12:54", "remaining_time": "0:06:43"}
+{"current_steps": 72, "total_steps": 108, "loss": 0.522, "lr": 2.5860389863462765e-06, "epoch": 0.6666666666666666, "percentage": 66.67, "elapsed_time": "0:13:04", "remaining_time": "0:06:32"}
+{"current_steps": 73, "total_steps": 108, "loss": 0.4245, "lr": 2.457344406753899e-06, "epoch": 0.6759259259259259, "percentage": 67.59, "elapsed_time": "0:13:14", "remaining_time": "0:06:21"}
+{"current_steps": 74, "total_steps": 108, "loss": 0.4806, "lr": 2.330883110176049e-06, "epoch": 0.6851851851851852, "percentage": 68.52, "elapsed_time": "0:13:25", "remaining_time": "0:06:09"}
+{"current_steps": 75, "total_steps": 108, "loss": 0.5149, "lr": 2.2067661709817384e-06, "epoch": 0.6944444444444444, "percentage": 69.44, "elapsed_time": "0:13:35", "remaining_time": "0:05:58"}
+{"current_steps": 76, "total_steps": 108, "loss": 0.4107, "lr": 2.0851026044276405e-06, "epoch": 0.7037037037037037, "percentage": 70.37, "elapsed_time": "0:13:45", "remaining_time": "0:05:47"}
+{"current_steps": 77, "total_steps": 108, "loss": 0.4687, "lr": 1.9659992709070346e-06, "epoch": 0.7129629629629629, "percentage": 71.3, "elapsed_time": "0:13:55", "remaining_time": "0:05:36"}
+{"current_steps": 77, "total_steps": 108, "eval_loss": 0.44653645157814026, "epoch": 0.7129629629629629, "percentage": 71.3, "elapsed_time": "0:14:03", "remaining_time": "0:05:39"}
+{"current_steps": 78, "total_steps": 108, "loss": 0.4197, "lr": 1.8495607820914451e-06, "epoch": 0.7222222222222222, "percentage": 72.22, "elapsed_time": "0:14:13", "remaining_time": "0:05:28"}
+{"current_steps": 79, "total_steps": 108, "loss": 0.3619, "lr": 1.7358894090473928e-06, "epoch": 0.7314814814814815, "percentage": 73.15, "elapsed_time": "0:14:23", "remaining_time": "0:05:17"}
+{"current_steps": 80, "total_steps": 108, "loss": 0.3587, "lr": 1.6250849924089485e-06, "epoch": 0.7407407407407407, "percentage": 74.07, "elapsed_time": "0:14:34", "remaining_time": "0:05:05"}
+{"current_steps": 81, "total_steps": 108, "loss": 0.3391, "lr": 1.5172448546850166e-06, "epoch": 0.75, "percentage": 75.0, "elapsed_time": "0:14:44", "remaining_time": "0:04:54"}
+{"current_steps": 82, "total_steps": 108, "loss": 0.5386, "lr": 1.4124637147783431e-06, "epoch": 0.7592592592592593, "percentage": 75.93, "elapsed_time": "0:14:54", "remaining_time": "0:04:43"}
+{"current_steps": 83, "total_steps": 108, "loss": 0.4069, "lr": 1.3108336047913633e-06, "epoch": 0.7685185185185185, "percentage": 76.85, "elapsed_time": "0:15:04", "remaining_time": "0:04:32"}
+{"current_steps": 84, "total_steps": 108, "loss": 0.4371, "lr": 1.2124437891918995e-06, "epoch": 0.7777777777777778, "percentage": 77.78, "elapsed_time": "0:15:15", "remaining_time": "0:04:21"}
+{"current_steps": 85, "total_steps": 108, "loss": 0.4374, "lr": 1.1173806864097885e-06, "epoch": 0.7870370370370371, "percentage": 78.7, "elapsed_time": "0:15:25", "remaining_time": "0:04:10"}
+{"current_steps": 86, "total_steps": 108, "loss": 0.3986, "lr": 1.0257277929332332e-06, "epoch": 0.7962962962962963, "percentage": 79.63, "elapsed_time": "0:15:35", "remaining_time": "0:03:59"}
+{"current_steps": 87, "total_steps": 108, "loss": 0.4473, "lr": 9.375656099715935e-07, "epoch": 0.8055555555555556, "percentage": 80.56, "elapsed_time": "0:15:45", "remaining_time": "0:03:48"}
+{"current_steps": 88, "total_steps": 108, "loss": 0.4484, "lr": 8.529715727489912e-07, "epoch": 0.8148148148148148, "percentage": 81.48, "elapsed_time": "0:15:56", "remaining_time": "0:03:37"}
+{"current_steps": 88, "total_steps": 108, "eval_loss": 0.4449349045753479, "epoch": 0.8148148148148148, "percentage": 81.48, "elapsed_time": "0:16:03", "remaining_time": "0:03:38"}
+{"current_steps": 89, "total_steps": 108, "loss": 0.3425, "lr": 7.720199824908692e-07, "epoch": 0.8240740740740741, "percentage": 82.41, "elapsed_time": "0:16:13", "remaining_time": "0:03:27"}
+{"current_steps": 90, "total_steps": 108, "loss": 0.4269, "lr": 6.947819411632223e-07, "epoch": 0.8333333333333334, "percentage": 83.33, "elapsed_time": "0:16:24", "remaining_time": "0:03:16"}
+{"current_steps": 91, "total_steps": 108, "loss": 0.5674, "lr": 6.213252890218163e-07, "epoch": 0.8425925925925926, "percentage": 84.26, "elapsed_time": "0:16:34", "remaining_time": "0:03:05"}
+{"current_steps": 92, "total_steps": 108, "loss": 0.454, "lr": 5.517145450262639e-07, "epoch": 0.8518518518518519, "percentage": 85.19, "elapsed_time": "0:16:44", "remaining_time": "0:02:54"}
+{"current_steps": 93, "total_steps": 108, "loss": 0.3412, "lr": 4.860108501712824e-07, "epoch": 0.8611111111111112, "percentage": 86.11, "elapsed_time": "0:16:54", "remaining_time": "0:02:43"}
+{"current_steps": 94, "total_steps": 108, "loss": 0.4292, "lr": 4.242719137849077e-07, "epoch": 0.8703703703703703, "percentage": 87.04, "elapsed_time": "0:17:05", "remaining_time": "0:02:32"}
+{"current_steps": 95, "total_steps": 108, "loss": 0.4765, "lr": 3.665519628408332e-07, "epoch": 0.8796296296296297, "percentage": 87.96, "elapsed_time": "0:17:15", "remaining_time": "0:02:21"}
+{"current_steps": 96, "total_steps": 108, "loss": 0.428, "lr": 3.1290169432939556e-07, "epoch": 0.8888888888888888, "percentage": 88.89, "elapsed_time": "0:17:25", "remaining_time": "0:02:10"}
+{"current_steps": 97, "total_steps": 108, "loss": 0.4488, "lr": 2.6336823072904305e-07, "epoch": 0.8981481481481481, "percentage": 89.81, "elapsed_time": "0:17:35", "remaining_time": "0:01:59"}
+{"current_steps": 98, "total_steps": 108, "loss": 0.3881, "lr": 2.179950786173879e-07, "epoch": 0.9074074074074074, "percentage": 90.74, "elapsed_time": "0:17:46", "remaining_time": "0:01:48"}
+{"current_steps": 99, "total_steps": 108, "loss": 0.4809, "lr": 1.7682209045820687e-07, "epoch": 0.9166666666666666, "percentage": 91.67, "elapsed_time": "0:17:56", "remaining_time": "0:01:37"}
+{"current_steps": 99, "total_steps": 108, "eval_loss": 0.4441840946674347, "epoch": 0.9166666666666666, "percentage": 91.67, "elapsed_time": "0:18:03", "remaining_time": "0:01:38"}
+{"current_steps": 100, "total_steps": 108, "loss": 0.4533, "lr": 1.3988542959794627e-07, "epoch": 0.9259259259259259, "percentage": 92.59, "elapsed_time": "0:18:13", "remaining_time": "0:01:27"}
+{"current_steps": 101, "total_steps": 108, "loss": 0.5236, "lr": 1.0721753850247984e-07, "epoch": 0.9351851851851852, "percentage": 93.52, "elapsed_time": "0:18:24", "remaining_time": "0:01:16"}
+{"current_steps": 102, "total_steps": 108, "loss": 0.3763, "lr": 7.884711026201586e-08, "epoch": 0.9444444444444444, "percentage": 94.44, "elapsed_time": "0:18:34", "remaining_time": "0:01:05"}
+{"current_steps": 103, "total_steps": 108, "loss": 0.4101, "lr": 5.479906338917984e-08, "epoch": 0.9537037037037037, "percentage": 95.37, "elapsed_time": "0:18:44", "remaining_time": "0:00:54"}
+{"current_steps": 104, "total_steps": 108, "loss": 0.4942, "lr": 3.5094519932415417e-08, "epoch": 0.9629629629629629, "percentage": 96.3, "elapsed_time": "0:18:54", "remaining_time": "0:00:43"}
+{"current_steps": 105, "total_steps": 108, "loss": 0.4778, "lr": 1.975078692391552e-08, "epoch": 0.9722222222222222, "percentage": 97.22, "elapsed_time": "0:19:05", "remaining_time": "0:00:32"}
+{"current_steps": 106, "total_steps": 108, "loss": 0.4537, "lr": 8.781341178393244e-09, "epoch": 0.9814814814814815, "percentage": 98.15, "elapsed_time": "0:19:15", "remaining_time": "0:00:21"}
+{"current_steps": 107, "total_steps": 108, "loss": 0.4192, "lr": 2.19581745602826e-09, "epoch": 0.9907407407407407, "percentage": 99.07, "elapsed_time": "0:19:25", "remaining_time": "0:00:10"}
+{"current_steps": 108, "total_steps": 108, "loss": 0.3589, "lr": 0.0, "epoch": 1.0, "percentage": 100.0, "elapsed_time": "0:19:35", "remaining_time": "0:00:00"}
+{"current_steps": 108, "total_steps": 108, "epoch": 1.0, "percentage": 100.0, "elapsed_time": "0:20:48", "remaining_time": "0:00:00"}

trainer_state.json

@@ -0,0 +1,870 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 11,
+  "global_step": 108,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.009259259259259259,
+      "grad_norm": 2.362648319863932,
+      "learning_rate": 5e-06,
+      "loss": 0.5807,
+      "step": 1
+    },
+    {
+      "epoch": 0.018518518518518517,
+      "grad_norm": 2.136281230840684,
+      "learning_rate": 1e-05,
+      "loss": 0.534,
+      "step": 2
+    },
+    {
+      "epoch": 0.027777777777777776,
+      "grad_norm": 2.2915803466254743,
+      "learning_rate": 9.997804182543973e-06,
+      "loss": 0.6069,
+      "step": 3
+    },
+    {
+      "epoch": 0.037037037037037035,
+      "grad_norm": 1.817521417636344,
+      "learning_rate": 9.991218658821609e-06,
+      "loss": 0.5863,
+      "step": 4
+    },
+    {
+      "epoch": 0.046296296296296294,
+      "grad_norm": 1.4406196993263543,
+      "learning_rate": 9.980249213076085e-06,
+      "loss": 0.5952,
+      "step": 5
+    },
+    {
+      "epoch": 0.05555555555555555,
+      "grad_norm": 1.098132624881383,
+      "learning_rate": 9.964905480067585e-06,
+      "loss": 0.5384,
+      "step": 6
+    },
+    {
+      "epoch": 0.06481481481481481,
+      "grad_norm": 1.2038829059152325,
+      "learning_rate": 9.945200936610821e-06,
+      "loss": 0.5779,
+      "step": 7
+    },
+    {
+      "epoch": 0.07407407407407407,
+      "grad_norm": 1.1568981182086557,
+      "learning_rate": 9.921152889737985e-06,
+      "loss": 0.5487,
+      "step": 8
+    },
+    {
+      "epoch": 0.08333333333333333,
+      "grad_norm": 1.009464783024193,
+      "learning_rate": 9.892782461497521e-06,
+      "loss": 0.5162,
+      "step": 9
+    },
+    {
+      "epoch": 0.09259259259259259,
+      "grad_norm": 1.0403262733706924,
+      "learning_rate": 9.860114570402055e-06,
+      "loss": 0.4825,
+      "step": 10
+    },
+    {
+      "epoch": 0.10185185185185185,
+      "grad_norm": 1.017871214392586,
+      "learning_rate": 9.823177909541795e-06,
+      "loss": 0.4962,
+      "step": 11
+    },
+    {
+      "epoch": 0.10185185185185185,
+      "eval_loss": 0.5000379085540771,
+      "eval_runtime": 7.2133,
+      "eval_samples_per_second": 1.248,
+      "eval_steps_per_second": 0.277,
+      "step": 11
+    },
+    {
+      "epoch": 0.1111111111111111,
+      "grad_norm": 1.0511034662384473,
+      "learning_rate": 9.782004921382612e-06,
+      "loss": 0.4456,
+      "step": 12
+    },
+    {
+      "epoch": 0.12037037037037036,
+      "grad_norm": 0.8657886515614241,
+      "learning_rate": 9.736631769270958e-06,
+      "loss": 0.4288,
+      "step": 13
+    },
+    {
+      "epoch": 0.12962962962962962,
+      "grad_norm": 1.040741797545574,
+      "learning_rate": 9.687098305670606e-06,
+      "loss": 0.5855,
+      "step": 14
+    },
+    {
+      "epoch": 0.1388888888888889,
+      "grad_norm": 1.3516228019504957,
+      "learning_rate": 9.633448037159167e-06,
+      "loss": 0.5184,
+      "step": 15
+    },
+    {
+      "epoch": 0.14814814814814814,
+      "grad_norm": 0.9843907475003576,
+      "learning_rate": 9.575728086215093e-06,
+      "loss": 0.5568,
+      "step": 16
+    },
+    {
+      "epoch": 0.1574074074074074,
+      "grad_norm": 0.8979836323930017,
+      "learning_rate": 9.513989149828718e-06,
+      "loss": 0.4272,
+      "step": 17
+    },
+    {
+      "epoch": 0.16666666666666666,
+      "grad_norm": 1.1188974861045773,
+      "learning_rate": 9.448285454973739e-06,
+      "loss": 0.487,
+      "step": 18
+    },
+    {
+      "epoch": 0.17592592592592593,
+      "grad_norm": 1.048669002845475,
+      "learning_rate": 9.378674710978185e-06,
+      "loss": 0.5321,
+      "step": 19
+    },
+    {
+      "epoch": 0.18518518518518517,
+      "grad_norm": 1.2366432725685659,
+      "learning_rate": 9.305218058836778e-06,
+      "loss": 0.5396,
+      "step": 20
+    },
+    {
+      "epoch": 0.19444444444444445,
+      "grad_norm": 1.0072812014692536,
+      "learning_rate": 9.22798001750913e-06,
+      "loss": 0.446,
+      "step": 21
+    },
+    {
+      "epoch": 0.2037037037037037,
+      "grad_norm": 1.0654455593583398,
+      "learning_rate": 9.14702842725101e-06,
+      "loss": 0.5313,
+      "step": 22
+    },
+    {
+      "epoch": 0.2037037037037037,
+      "eval_loss": 0.4791085124015808,
+      "eval_runtime": 7.1652,
+      "eval_samples_per_second": 1.256,
+      "eval_steps_per_second": 0.279,
+      "step": 22
+    },
+    {
+      "epoch": 0.21296296296296297,
+      "grad_norm": 1.0051504278398855,
+      "learning_rate": 9.062434390028407e-06,
+      "loss": 0.4983,
+      "step": 23
+    },
+    {
+      "epoch": 0.2222222222222222,
+      "grad_norm": 0.9621751996571424,
+      "learning_rate": 8.974272207066767e-06,
+      "loss": 0.4642,
+      "step": 24
+    },
+    {
+      "epoch": 0.23148148148148148,
+      "grad_norm": 0.9763151052182862,
+      "learning_rate": 8.882619313590212e-06,
+      "loss": 0.4832,
+      "step": 25
+    },
+    {
+      "epoch": 0.24074074074074073,
+      "grad_norm": 0.934861026969845,
+      "learning_rate": 8.787556210808101e-06,
+      "loss": 0.441,
+      "step": 26
+    },
+    {
+      "epoch": 0.25,
+      "grad_norm": 1.0362041830687405,
+      "learning_rate": 8.689166395208638e-06,
+      "loss": 0.4489,
+      "step": 27
+    },
+    {
+      "epoch": 0.25925925925925924,
+      "grad_norm": 0.9049359816184893,
+      "learning_rate": 8.587536285221656e-06,
+      "loss": 0.4971,
+      "step": 28
+    },
+    {
+      "epoch": 0.26851851851851855,
+      "grad_norm": 0.9982959368756156,
+      "learning_rate": 8.482755145314987e-06,
+      "loss": 0.5289,
+      "step": 29
+    },
+    {
+      "epoch": 0.2777777777777778,
+      "grad_norm": 2.050950638692456,
+      "learning_rate": 8.374915007591053e-06,
+      "loss": 0.5496,
+      "step": 30
+    },
+    {
+      "epoch": 0.28703703703703703,
+      "grad_norm": 1.0501945401080932,
+      "learning_rate": 8.264110590952609e-06,
+      "loss": 0.5261,
+      "step": 31
+    },
+    {
+      "epoch": 0.2962962962962963,
+      "grad_norm": 0.9229687907732845,
+      "learning_rate": 8.150439217908557e-06,
+      "loss": 0.4814,
+      "step": 32
+    },
+    {
+      "epoch": 0.3055555555555556,
+      "grad_norm": 1.4319914933917381,
+      "learning_rate": 8.034000729092967e-06,
+      "loss": 0.4692,
+      "step": 33
+    },
+    {
+      "epoch": 0.3055555555555556,
+      "eval_loss": 0.46850860118865967,
+      "eval_runtime": 7.148,
+      "eval_samples_per_second": 1.259,
+      "eval_steps_per_second": 0.28,
+      "step": 33
+    },
+    {
+      "epoch": 0.3148148148148148,
+      "grad_norm": 0.8881275342675574,
+      "learning_rate": 7.914897395572362e-06,
+      "loss": 0.4372,
+      "step": 34
+    },
+    {
+      "epoch": 0.32407407407407407,
+      "grad_norm": 0.9840343978467488,
+      "learning_rate": 7.793233829018263e-06,
+      "loss": 0.5235,
+      "step": 35
+    },
+    {
+      "epoch": 0.3333333333333333,
+      "grad_norm": 0.8470332819900018,
+      "learning_rate": 7.669116889823955e-06,
+      "loss": 0.4339,
+      "step": 36
+    },
+    {
+      "epoch": 0.3425925925925926,
+      "grad_norm": 0.8893330836178615,
+      "learning_rate": 7.542655593246103e-06,
+      "loss": 0.4426,
+      "step": 37
+    },
+    {
+      "epoch": 0.35185185185185186,
+      "grad_norm": 1.0550698950573756,
+      "learning_rate": 7.413961013653725e-06,
+      "loss": 0.5744,
+      "step": 38
+    },
+    {
+      "epoch": 0.3611111111111111,
+      "grad_norm": 0.8784333801568516,
+      "learning_rate": 7.283146186968566e-06,
+      "loss": 0.3849,
+      "step": 39
+    },
+    {
+      "epoch": 0.37037037037037035,
+      "grad_norm": 0.9142401659654863,
+      "learning_rate": 7.1503260113826035e-06,
+      "loss": 0.4057,
+      "step": 40
+    },
+    {
+      "epoch": 0.37962962962962965,
+      "grad_norm": 0.8743924562233322,
+      "learning_rate": 7.015617146439863e-06,
+      "loss": 0.4149,
+      "step": 41
+    },
+    {
+      "epoch": 0.3888888888888889,
+      "grad_norm": 0.9155700365107549,
+      "learning_rate": 6.879137910571191e-06,
+      "loss": 0.4557,
+      "step": 42
+    },
+    {
+      "epoch": 0.39814814814814814,
+      "grad_norm": 0.9844599172183693,
+      "learning_rate": 6.741008177171995e-06,
+      "loss": 0.5052,
+      "step": 43
+    },
+    {
+      "epoch": 0.4074074074074074,
+      "grad_norm": 0.8432119339426061,
+      "learning_rate": 6.601349269314188e-06,
+      "loss": 0.3876,
+      "step": 44
+    },
+    {
+      "epoch": 0.4074074074074074,
+      "eval_loss": 0.4595443606376648,
+      "eval_runtime": 7.1719,
+      "eval_samples_per_second": 1.255,
+      "eval_steps_per_second": 0.279,
+      "step": 44
+    },
+    {
+      "epoch": 0.4166666666666667,
+      "grad_norm": 0.9547733839048285,
+      "learning_rate": 6.46028385318488e-06,
+      "loss": 0.4381,
+      "step": 45
+    },
+    {
+      "epoch": 0.42592592592592593,
+      "grad_norm": 0.9165610947735227,
+      "learning_rate": 6.3179358303453386e-06,
+      "loss": 0.433,
+      "step": 46
+    },
+    {
+      "epoch": 0.4351851851851852,
+      "grad_norm": 0.9276788988821667,
+      "learning_rate": 6.17443022890492e-06,
+      "loss": 0.4699,
+      "step": 47
+    },
+    {
+      "epoch": 0.4444444444444444,
+      "grad_norm": 0.9408707549804078,
+      "learning_rate": 6.029893093705492e-06,
+      "loss": 0.4584,
+      "step": 48
+    },
+    {
+      "epoch": 0.4537037037037037,
+      "grad_norm": 0.9822969953152866,
+      "learning_rate": 5.884451375612865e-06,
+      "loss": 0.5243,
+      "step": 49
+    },
+    {
+      "epoch": 0.46296296296296297,
+      "grad_norm": 1.0689960747907674,
+      "learning_rate": 5.738232820012407e-06,
+      "loss": 0.4329,
+      "step": 50
+    },
+    {
+      "epoch": 0.4722222222222222,
+      "grad_norm": 1.0494906655322942,
+      "learning_rate": 5.591365854606829e-06,
+      "loss": 0.5633,
+      "step": 51
+    },
+    {
+      "epoch": 0.48148148148148145,
+      "grad_norm": 0.8987830176611722,
+      "learning_rate": 5.443979476614674e-06,
+      "loss": 0.445,
+      "step": 52
+    },
+    {
+      "epoch": 0.49074074074074076,
+      "grad_norm": 0.8746204581445627,
+      "learning_rate": 5.296203139468572e-06,
+      "loss": 0.3809,
+      "step": 53
+    },
+    {
+      "epoch": 0.5,
+      "grad_norm": 0.8125477508425051,
+      "learning_rate": 5.148166639112799e-06,
+      "loss": 0.3964,
+      "step": 54
+    },
+    {
+      "epoch": 0.5092592592592593,
+      "grad_norm": 0.907249706367073,
+      "learning_rate": 5e-06,
+      "loss": 0.4768,
+      "step": 55
+    },
+    {
+      "epoch": 0.5092592592592593,
+      "eval_loss": 0.4542139768600464,
+      "eval_runtime": 7.1716,
+      "eval_samples_per_second": 1.255,
+      "eval_steps_per_second": 0.279,
+      "step": 55
+    },
+    {
+      "epoch": 0.5185185185185185,
+      "grad_norm": 0.8641393129712072,
+      "learning_rate": 4.8518333608872015e-06,
+      "loss": 0.4238,
+      "step": 56
+    },
+    {
+      "epoch": 0.5277777777777778,
+      "grad_norm": 1.3038267949545501,
+      "learning_rate": 4.703796860531429e-06,
+      "loss": 0.5484,
+      "step": 57
+    },
+    {
+      "epoch": 0.5370370370370371,
+      "grad_norm": 1.0286161889066556,
+      "learning_rate": 4.556020523385326e-06,
+      "loss": 0.397,
+      "step": 58
+    },
+    {
+      "epoch": 0.5462962962962963,
+      "grad_norm": 0.8754084053422794,
+      "learning_rate": 4.408634145393172e-06,
+      "loss": 0.4452,
+      "step": 59
+    },
+    {
+      "epoch": 0.5555555555555556,
+      "grad_norm": 0.935284958978312,
+      "learning_rate": 4.261767179987595e-06,
+      "loss": 0.4896,
+      "step": 60
+    },
+    {
+      "epoch": 0.5648148148148148,
+      "grad_norm": 0.9674393986889579,
+      "learning_rate": 4.115548624387136e-06,
+      "loss": 0.4887,
+      "step": 61
+    },
+    {
+      "epoch": 0.5740740740740741,
+      "grad_norm": 0.9724892358854195,
+      "learning_rate": 3.970106906294509e-06,
+      "loss": 0.4997,
+      "step": 62
+    },
+    {
+      "epoch": 0.5833333333333334,
+      "grad_norm": 0.8794971763479272,
+      "learning_rate": 3.825569771095082e-06,
+      "loss": 0.417,
+      "step": 63
+    },
+    {
+      "epoch": 0.5925925925925926,
+      "grad_norm": 0.8881388944091057,
+      "learning_rate": 3.682064169654663e-06,
+      "loss": 0.4504,
+      "step": 64
+    },
+    {
+      "epoch": 0.6018518518518519,
+      "grad_norm": 0.9193120996402042,
+      "learning_rate": 3.539716146815122e-06,
+      "loss": 0.4694,
+      "step": 65
+    },
+    {
+      "epoch": 0.6111111111111112,
+      "grad_norm": 1.0073500011381085,
+      "learning_rate": 3.398650730685813e-06,
+      "loss": 0.4985,
+      "step": 66
+    },
+    {
+      "epoch": 0.6111111111111112,
+      "eval_loss": 0.4495806097984314,
+      "eval_runtime": 7.1798,
+      "eval_samples_per_second": 1.254,
+      "eval_steps_per_second": 0.279,
+      "step": 66
+    },
+    {
+      "epoch": 0.6203703703703703,
+      "grad_norm": 0.824519734053682,
+      "learning_rate": 3.258991822828007e-06,
+      "loss": 0.4104,
+      "step": 67
+    },
+    {
+      "epoch": 0.6296296296296297,
+      "grad_norm": 0.9368830049136432,
+      "learning_rate": 3.1208620894288105e-06,
+      "loss": 0.4926,
+      "step": 68
+    },
+    {
+      "epoch": 0.6388888888888888,
+      "grad_norm": 1.1454803265014575,
+      "learning_rate": 2.98438285356014e-06,
+      "loss": 0.4655,
+      "step": 69
+    },
+    {
+      "epoch": 0.6481481481481481,
+      "grad_norm": 0.8964782710358511,
+      "learning_rate": 2.8496739886173994e-06,
+      "loss": 0.4535,
+      "step": 70
+    },
+    {
+      "epoch": 0.6574074074074074,
+      "grad_norm": 0.9529134597913587,
+      "learning_rate": 2.716853813031435e-06,
+      "loss": 0.5186,
+      "step": 71
+    },
+    {
+      "epoch": 0.6666666666666666,
+      "grad_norm": 0.963788290306135,
+      "learning_rate": 2.5860389863462765e-06,
+      "loss": 0.522,
+      "step": 72
+    },
+    {
+      "epoch": 0.6759259259259259,
+      "grad_norm": 0.8411162085924286,
+      "learning_rate": 2.457344406753899e-06,
+      "loss": 0.4245,
+      "step": 73
+    },
+    {
+      "epoch": 0.6851851851851852,
+      "grad_norm": 0.8871221033397801,
+      "learning_rate": 2.330883110176049e-06,
+      "loss": 0.4806,
+      "step": 74
+    },
+    {
+      "epoch": 0.6944444444444444,
+      "grad_norm": 0.9550800145418555,
+      "learning_rate": 2.2067661709817384e-06,
+      "loss": 0.5149,
+      "step": 75
+    },
+    {
+      "epoch": 0.7037037037037037,
+      "grad_norm": 0.8501908519121222,
+      "learning_rate": 2.0851026044276405e-06,
+      "loss": 0.4107,
+      "step": 76
+    },
+    {
+      "epoch": 0.7129629629629629,
+      "grad_norm": 0.8678719046436287,
+      "learning_rate": 1.9659992709070346e-06,
+      "loss": 0.4687,
+      "step": 77
+    },
+    {
+      "epoch": 0.7129629629629629,
+      "eval_loss": 0.44653645157814026,
+      "eval_runtime": 7.1486,
+      "eval_samples_per_second": 1.259,
+      "eval_steps_per_second": 0.28,
+      "step": 77
+    },
+    {
+      "epoch": 0.7222222222222222,
+      "grad_norm": 0.876245605736801,
+      "learning_rate": 1.8495607820914451e-06,
+      "loss": 0.4197,
+      "step": 78
+    },
+    {
+      "epoch": 0.7314814814814815,
+      "grad_norm": 0.7970757955872202,
+      "learning_rate": 1.7358894090473928e-06,
+      "loss": 0.3619,
+      "step": 79
+    },
+    {
+      "epoch": 0.7407407407407407,
+      "grad_norm": 0.7955115388136204,
+      "learning_rate": 1.6250849924089485e-06,
+      "loss": 0.3587,
+      "step": 80
+    },
+    {
+      "epoch": 0.75,
+      "grad_norm": 0.7782242902212891,
+      "learning_rate": 1.5172448546850166e-06,
+      "loss": 0.3391,
+      "step": 81
+    },
+    {
+      "epoch": 0.7592592592592593,
+      "grad_norm": 0.9995473034082181,
+      "learning_rate": 1.4124637147783431e-06,
+      "loss": 0.5386,
+      "step": 82
+    },
+    {
+      "epoch": 0.7685185185185185,
+      "grad_norm": 0.8444693019666363,
+      "learning_rate": 1.3108336047913633e-06,
+      "loss": 0.4069,
+      "step": 83
+    },
+    {
+      "epoch": 0.7777777777777778,
+      "grad_norm": 0.8808187390006551,
+      "learning_rate": 1.2124437891918995e-06,
+      "loss": 0.4371,
+      "step": 84
+    },
+    {
+      "epoch": 0.7870370370370371,
+      "grad_norm": 0.9813077333772956,
+      "learning_rate": 1.1173806864097885e-06,
+      "loss": 0.4374,
+      "step": 85
+    },
+    {
+      "epoch": 0.7962962962962963,
+      "grad_norm": 0.8064326056384253,
+      "learning_rate": 1.0257277929332332e-06,
+      "loss": 0.3986,
+      "step": 86
+    },
+    {
+      "epoch": 0.8055555555555556,
+      "grad_norm": 0.8710631137320209,
+      "learning_rate": 9.375656099715935e-07,
+      "loss": 0.4473,
+      "step": 87
+    },
+    {
+      "epoch": 0.8148148148148148,
+      "grad_norm": 0.887311644170428,
+      "learning_rate": 8.529715727489912e-07,
+      "loss": 0.4484,
+      "step": 88
+    },
+    {
+      "epoch": 0.8148148148148148,
+      "eval_loss": 0.4449349045753479,
+      "eval_runtime": 7.1634,
+      "eval_samples_per_second": 1.256,
+      "eval_steps_per_second": 0.279,
+      "step": 88
+    },
+    {
+      "epoch": 0.8240740740740741,
+      "grad_norm": 0.7647806015755853,
+      "learning_rate": 7.720199824908692e-07,
+      "loss": 0.3425,
+      "step": 89
+    },
+    {
+      "epoch": 0.8333333333333334,
+      "grad_norm": 0.8976917694082569,
+      "learning_rate": 6.947819411632223e-07,
+      "loss": 0.4269,
+      "step": 90
+    },
+    {
+      "epoch": 0.8425925925925926,
+      "grad_norm": 0.9316801386853545,
+      "learning_rate": 6.213252890218163e-07,
+      "loss": 0.5674,
+      "step": 91
+    },
+    {
+      "epoch": 0.8518518518518519,
+      "grad_norm": 0.8870482831956321,
+      "learning_rate": 5.517145450262639e-07,
+      "loss": 0.454,
+      "step": 92
+    },
+    {
+      "epoch": 0.8611111111111112,
+      "grad_norm": 0.7768323848063172,
+      "learning_rate": 4.860108501712824e-07,
+      "loss": 0.3412,
+      "step": 93
+    },
+    {
+      "epoch": 0.8703703703703703,
+      "grad_norm": 0.8658069877384894,
+      "learning_rate": 4.242719137849077e-07,
+      "loss": 0.4292,
+      "step": 94
+    },
+    {
+      "epoch": 0.8796296296296297,
+      "grad_norm": 0.9112502068381936,
+      "learning_rate": 3.665519628408332e-07,
+      "loss": 0.4765,
+      "step": 95
+    },
+    {
+      "epoch": 0.8888888888888888,
+      "grad_norm": 0.8505617937603335,
+      "learning_rate": 3.1290169432939556e-07,
+      "loss": 0.428,
+      "step": 96
+    },
+    {
+      "epoch": 0.8981481481481481,
+      "grad_norm": 0.8868085158482631,
+      "learning_rate": 2.6336823072904305e-07,
+      "loss": 0.4488,
+      "step": 97
+    },
+    {
+      "epoch": 0.9074074074074074,
+      "grad_norm": 0.8291365248164002,
+      "learning_rate": 2.179950786173879e-07,
+      "loss": 0.3881,
+      "step": 98
+    },
+    {
+      "epoch": 0.9166666666666666,
+      "grad_norm": 0.9100636810782011,
+      "learning_rate": 1.7682209045820687e-07,
+      "loss": 0.4809,
+      "step": 99
+    },
+    {
+      "epoch": 0.9166666666666666,
+      "eval_loss": 0.4441840946674347,
+      "eval_runtime": 7.1692,
+      "eval_samples_per_second": 1.255,
+      "eval_steps_per_second": 0.279,
+      "step": 99
+    },
+    {
+      "epoch": 0.9259259259259259,
+      "grad_norm": 0.8985637565061618,
+      "learning_rate": 1.3988542959794627e-07,
+      "loss": 0.4533,
+      "step": 100
+    },
+    {
+      "epoch": 0.9351851851851852,
+      "grad_norm": 0.9612580974302548,
+      "learning_rate": 1.0721753850247984e-07,
+      "loss": 0.5236,
+      "step": 101
+    },
+    {
+      "epoch": 0.9444444444444444,
+      "grad_norm": 0.8120796428633341,
+      "learning_rate": 7.884711026201586e-08,
+      "loss": 0.3763,
+      "step": 102
+    },
+    {
+      "epoch": 0.9537037037037037,
+      "grad_norm": 0.8644849276991213,
+      "learning_rate": 5.479906338917984e-08,
+      "loss": 0.4101,
+      "step": 103
+    },
+    {
+      "epoch": 0.9629629629629629,
+      "grad_norm": 0.8946182268462048,
+      "learning_rate": 3.5094519932415417e-08,
+      "loss": 0.4942,
+      "step": 104
+    },
+    {
+      "epoch": 0.9722222222222222,
+      "grad_norm": 0.9166159846522884,
+      "learning_rate": 1.975078692391552e-08,
+      "loss": 0.4778,
+      "step": 105
+    },
+    {
+      "epoch": 0.9814814814814815,
+      "grad_norm": 0.9473852601201989,
+      "learning_rate": 8.781341178393244e-09,
+      "loss": 0.4537,
+      "step": 106
+    },
+    {
+      "epoch": 0.9907407407407407,
+      "grad_norm": 0.8895854364679423,
+      "learning_rate": 2.19581745602826e-09,
+      "loss": 0.4192,
+      "step": 107
+    },
+    {
+      "epoch": 1.0,
+      "grad_norm": 0.7849733548858487,
+      "learning_rate": 0.0,
+      "loss": 0.3589,
+      "step": 108
+    },
+    {
+      "epoch": 1.0,
+      "step": 108,
+      "total_flos": 17851226259456.0,
+      "train_loss": 0.46821982275556634,
+      "train_runtime": 1249.8096,
+      "train_samples_per_second": 0.69,
+      "train_steps_per_second": 0.086
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 108,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 108,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 17851226259456.0,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c0f5d792c52ef3a9ba9189020a5cf49fda646d508b5c06f435c64a9711065aef
+size 7608