Hugging Face's logo

fay-ong
/
llama-3-8b-finetuned

PEFT
TensorBoard
Safetensors
llama
llama-factory
lora
unsloth
Generated from Trainer
4-bit precision
bitsandbytes
Model card Files Metrics Community
fay-ong commited on
Commit
6ed31a9
•
1 Parent(s): 0be4976

Model save

Browse files
Files changed (6) hide show
  1. README.md +5 -6
  2. adapter_config.json +5 -5
  3. adapter_model.safetensors +1 -1
  4. runs/Jun04_12-44-50_5e9a4843b991/events.out.tfevents.1717505161.5e9a4843b991.4123.0 +3 -0
  5. trainer_log.jsonl +13 -26
  6. training_args.bin +1 -1
README.md CHANGED
@@ -2,9 +2,8 @@
2
  license: apache-2.0
3
  library_name: peft
4
  tags:
5
- - llama-factory
6
- - lora
7
  - unsloth
 
8
  - generated_from_trainer
9
  base_model: unsloth/llama-3-8b-Instruct-bnb-4bit
10
  model-index:
@@ -17,7 +16,7 @@ should probably proofread and complete it, then remove this comment. -->
17
 
18
  # llama-3-8b-finetuned
19
 
20
- This model is a fine-tuned version of [unsloth/llama-3-8b-Instruct-bnb-4bit](https://huggingface.co/unsloth/llama-3-8b-Instruct-bnb-4bit) on the formatted_data dataset.
21
 
22
  ## Model description
23
 
@@ -45,7 +44,7 @@ The following hyperparameters were used during training:
45
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
46
  - lr_scheduler_type: cosine
47
  - lr_scheduler_warmup_ratio: 0.1
48
- - num_epochs: 2.0
49
  - mixed_precision_training: Native AMP
50
 
51
  ### Training results
@@ -55,7 +54,7 @@ The following hyperparameters were used during training:
55
  ### Framework versions
56
 
57
  - PEFT 0.11.1
58
- - Transformers 4.41.1
59
  - Pytorch 2.3.0+cu121
60
- - Datasets 2.19.1
61
  - Tokenizers 0.19.1
 
2
  license: apache-2.0
3
  library_name: peft
4
  tags:
 
 
5
  - unsloth
6
+ - llama-factory
7
  - generated_from_trainer
8
  base_model: unsloth/llama-3-8b-Instruct-bnb-4bit
9
  model-index:
 
16
 
17
  # llama-3-8b-finetuned
18
 
19
+ This model is a fine-tuned version of [unsloth/llama-3-8b-Instruct-bnb-4bit](https://huggingface.co/unsloth/llama-3-8b-Instruct-bnb-4bit) on an unknown dataset.
20
 
21
  ## Model description
22
 
 
44
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
45
  - lr_scheduler_type: cosine
46
  - lr_scheduler_warmup_ratio: 0.1
47
+ - num_epochs: 1.0
48
  - mixed_precision_training: Native AMP
49
 
50
  ### Training results
 
54
  ### Framework versions
55
 
56
  - PEFT 0.11.1
57
+ - Transformers 4.41.2
58
  - Pytorch 2.3.0+cu121
59
+ - Datasets 2.19.2
60
  - Tokenizers 0.19.1
adapter_config.json CHANGED
@@ -20,13 +20,13 @@
20
  "rank_pattern": {},
21
  "revision": "unsloth",
22
  "target_modules": [
23
- "v_proj",
24
- "up_proj",
25
  "o_proj",
26
- "gate_proj",
27
- "down_proj",
28
  "q_proj",
29
- "k_proj"
 
 
 
30
  ],
31
  "task_type": "CAUSAL_LM",
32
  "use_dora": false,
 
20
  "rank_pattern": {},
21
  "revision": "unsloth",
22
  "target_modules": [
 
 
23
  "o_proj",
24
+ "up_proj",
 
25
  "q_proj",
26
+ "down_proj",
27
+ "v_proj",
28
+ "k_proj",
29
+ "gate_proj"
30
  ],
31
  "task_type": "CAUSAL_LM",
32
  "use_dora": false,
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:4990c61376368ec7f611b05e97262222563a2dec687b95cbe52b3ba3bf67ef01
3
  size 83945296
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:274ecca0c1ecb79ba01d0ce4b4e8f63918fe12f7b715537f3f0566c100c1f57b
3
  size 83945296
runs/Jun04_12-44-50_5e9a4843b991/events.out.tfevents.1717505161.5e9a4843b991.4123.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8c0a1e0827dcafdcfb72b2e4d69a9585c9a7c7474c988f92de55d6584fbecc45
3
+ size 8301
trainer_log.jsonl CHANGED
@@ -1,26 +1,13 @@
1
- {"current_steps": 10, "total_steps": 250, "loss": 1.545, "learning_rate": 6.4000000000000006e-06, "epoch": 0.08, "percentage": 4.0, "elapsed_time": "0:01:04", "remaining_time": "0:25:45"}
2
- {"current_steps": 20, "total_steps": 250, "loss": 0.6012, "learning_rate": 1.4400000000000001e-05, "epoch": 0.16, "percentage": 8.0, "elapsed_time": "0:02:01", "remaining_time": "0:23:14"}
3
- {"current_steps": 30, "total_steps": 250, "loss": 0.3138, "learning_rate": 1.9991228300988586e-05, "epoch": 0.24, "percentage": 12.0, "elapsed_time": "0:02:58", "remaining_time": "0:21:45"}
4
- {"current_steps": 40, "total_steps": 250, "loss": 0.2683, "learning_rate": 1.983571470813386e-05, "epoch": 0.32, "percentage": 16.0, "elapsed_time": "0:03:54", "remaining_time": "0:20:30"}
5
- {"current_steps": 50, "total_steps": 250, "loss": 0.0889, "learning_rate": 1.9488760116444966e-05, "epoch": 0.4, "percentage": 20.0, "elapsed_time": "0:04:50", "remaining_time": "0:19:22"}
6
- {"current_steps": 60, "total_steps": 250, "loss": 0.202, "learning_rate": 1.895711760239413e-05, "epoch": 0.48, "percentage": 24.0, "elapsed_time": "0:05:47", "remaining_time": "0:18:19"}
7
- {"current_steps": 70, "total_steps": 250, "loss": 0.1592, "learning_rate": 1.8251134982782952e-05, "epoch": 0.56, "percentage": 28.0, "elapsed_time": "0:06:43", "remaining_time": "0:17:18"}
8
- {"current_steps": 80, "total_steps": 250, "loss": 0.1, "learning_rate": 1.7384553406258842e-05, "epoch": 0.64, "percentage": 32.0, "elapsed_time": "0:07:40", "remaining_time": "0:16:19"}
9
- {"current_steps": 90, "total_steps": 250, "loss": 0.1889, "learning_rate": 1.63742398974869e-05, "epoch": 0.72, "percentage": 36.0, "elapsed_time": "0:08:38", "remaining_time": "0:15:21"}
10
- {"current_steps": 100, "total_steps": 250, "loss": 0.1133, "learning_rate": 1.5239859059700794e-05, "epoch": 0.8, "percentage": 40.0, "elapsed_time": "0:09:34", "remaining_time": "0:14:21"}
11
- {"current_steps": 110, "total_steps": 250, "loss": 0.0466, "learning_rate": 1.4003490325568953e-05, "epoch": 0.88, "percentage": 44.0, "elapsed_time": "0:10:31", "remaining_time": "0:13:23"}
12
- {"current_steps": 120, "total_steps": 250, "loss": 0.1327, "learning_rate": 1.2689198206152657e-05, "epoch": 0.96, "percentage": 48.0, "elapsed_time": "0:11:27", "remaining_time": "0:12:25"}
13
- {"current_steps": 130, "total_steps": 250, "loss": 0.0181, "learning_rate": 1.1322563902571227e-05, "epoch": 1.04, "percentage": 52.0, "elapsed_time": "0:12:24", "remaining_time": "0:11:27"}
14
- {"current_steps": 140, "total_steps": 250, "loss": 0.0681, "learning_rate": 9.930187397020385e-06, "epoch": 1.12, "percentage": 56.0, "elapsed_time": "0:13:20", "remaining_time": "0:10:29"}
15
- {"current_steps": 150, "total_steps": 250, "loss": 0.0355, "learning_rate": 8.539169714375885e-06, "epoch": 1.2, "percentage": 60.0, "elapsed_time": "0:14:17", "remaining_time": "0:09:31"}
16
- {"current_steps": 160, "total_steps": 250, "loss": 0.1156, "learning_rate": 7.176585431571235e-06, "epoch": 1.28, "percentage": 64.0, "elapsed_time": "0:15:14", "remaining_time": "0:08:34"}
17
- {"current_steps": 170, "total_steps": 250, "loss": 0.0125, "learning_rate": 5.868955701754584e-06, "epoch": 1.3599999999999999, "percentage": 68.0, "elapsed_time": "0:16:11", "remaining_time": "0:07:37"}
18
- {"current_steps": 180, "total_steps": 250, "loss": 0.0465, "learning_rate": 4.641732050210032e-06, "epoch": 1.44, "percentage": 72.0, "elapsed_time": "0:17:07", "remaining_time": "0:06:39"}
19
- {"current_steps": 190, "total_steps": 250, "loss": 0.0085, "learning_rate": 3.5188009893686916e-06, "epoch": 1.52, "percentage": 76.0, "elapsed_time": "0:18:04", "remaining_time": "0:05:42"}
20
- {"current_steps": 200, "total_steps": 250, "loss": 0.0382, "learning_rate": 2.522019095014683e-06, "epoch": 1.6, "percentage": 80.0, "elapsed_time": "0:19:00", "remaining_time": "0:04:45"}
21
- {"current_steps": 210, "total_steps": 250, "loss": 0.0105, "learning_rate": 1.6707875928990059e-06, "epoch": 1.6800000000000002, "percentage": 84.0, "elapsed_time": "0:19:57", "remaining_time": "0:03:48"}
22
- {"current_steps": 220, "total_steps": 250, "loss": 0.0363, "learning_rate": 9.816747359488632e-07, "epoch": 1.76, "percentage": 88.0, "elapsed_time": "0:20:53", "remaining_time": "0:02:50"}
23
- {"current_steps": 230, "total_steps": 250, "loss": 0.0317, "learning_rate": 4.6809332207053083e-07, "epoch": 1.8399999999999999, "percentage": 92.0, "elapsed_time": "0:21:50", "remaining_time": "0:01:53"}
24
- {"current_steps": 240, "total_steps": 250, "loss": 0.0262, "learning_rate": 1.400396292949513e-07, "epoch": 1.92, "percentage": 96.0, "elapsed_time": "0:22:47", "remaining_time": "0:00:56"}
25
- {"current_steps": 250, "total_steps": 250, "loss": 0.0135, "learning_rate": 3.898849596456477e-09, "epoch": 2.0, "percentage": 100.0, "elapsed_time": "0:23:44", "remaining_time": "0:00:00"}
26
- {"current_steps": 250, "total_steps": 250, "epoch": 2.0, "percentage": 100.0, "elapsed_time": "0:23:44", "remaining_time": "0:00:00"}
 
1
+ {"current_steps": 10, "total_steps": 125, "loss": 1.5694, "learning_rate": 1.3846153846153847e-05, "epoch": 0.08, "percentage": 8.0, "elapsed_time": "0:01:14", "remaining_time": "0:14:13"}
2
+ {"current_steps": 20, "total_steps": 125, "loss": 0.3644, "learning_rate": 1.985871018518236e-05, "epoch": 0.16, "percentage": 16.0, "elapsed_time": "0:02:15", "remaining_time": "0:11:52"}
3
+ {"current_steps": 30, "total_steps": 125, "loss": 0.374, "learning_rate": 1.912783265061319e-05, "epoch": 0.24, "percentage": 24.0, "elapsed_time": "0:03:14", "remaining_time": "0:10:15"}
4
+ {"current_steps": 40, "total_steps": 125, "loss": 0.1797, "learning_rate": 1.7640373758216075e-05, "epoch": 0.32, "percentage": 32.0, "elapsed_time": "0:04:14", "remaining_time": "0:09:00"}
5
+ {"current_steps": 50, "total_steps": 125, "loss": 0.0685, "learning_rate": 1.5555702330196024e-05, "epoch": 0.4, "percentage": 40.0, "elapsed_time": "0:05:13", "remaining_time": "0:07:49"}
6
+ {"current_steps": 60, "total_steps": 125, "loss": 0.1244, "learning_rate": 1.3036767451096148e-05, "epoch": 0.48, "percentage": 48.0, "elapsed_time": "0:06:11", "remaining_time": "0:06:42"}
7
+ {"current_steps": 70, "total_steps": 125, "loss": 0.1301, "learning_rate": 1.028046256275869e-05, "epoch": 0.56, "percentage": 56.0, "elapsed_time": "0:07:10", "remaining_time": "0:05:38"}
8
+ {"current_steps": 80, "total_steps": 125, "loss": 0.2033, "learning_rate": 7.50223521832773e-06, "epoch": 0.64, "percentage": 64.0, "elapsed_time": "0:08:10", "remaining_time": "0:04:35"}
9
+ {"current_steps": 90, "total_steps": 125, "loss": 0.1496, "learning_rate": 4.919246547534709e-06, "epoch": 0.72, "percentage": 72.0, "elapsed_time": "0:09:08", "remaining_time": "0:03:33"}
10
+ {"current_steps": 100, "total_steps": 125, "loss": 0.0447, "learning_rate": 2.7333967796597317e-06, "epoch": 0.8, "percentage": 80.0, "elapsed_time": "0:10:07", "remaining_time": "0:02:31"}
11
+ {"current_steps": 110, "total_steps": 125, "loss": 0.1065, "learning_rate": 1.1155436402112785e-06, "epoch": 0.88, "percentage": 88.0, "elapsed_time": "0:11:05", "remaining_time": "0:01:30"}
12
+ {"current_steps": 120, "total_steps": 125, "loss": 0.1084, "learning_rate": 1.921471959676957e-07, "epoch": 0.96, "percentage": 96.0, "elapsed_time": "0:12:02", "remaining_time": "0:00:30"}
13
+ {"current_steps": 125, "total_steps": 125, "epoch": 1.0, "percentage": 100.0, "elapsed_time": "0:12:31", "remaining_time": "0:00:00"}
 
 
 
 
 
 
 
 
 
 
 
 
 
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:1df3d3df2e051b0909b1c2e589fbef3f5dfb2f78c0aaab77cae7a9c42162ebbb
3
  size 5304
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:458f4f236202372b0f1f2aea8bc620ca6c31ffcbc03c13fdbd44790a89044910
3
  size 5304