Hugging Face's logo

nrishabh
/
llama3-8b-instruct-qlora-mini

PEFT
Safetensors
trl
sft
Generated from Trainer
Model card Files Community
nrishabh commited on
Commit
1564fca
•
1 Parent(s): 1a715df

End of training

Browse files
Files changed (5) hide show
  1. README.md +32 -12
  2. adapter_config.json +2 -2
  3. adapter_model.safetensors +1 -1
  4. tokenizer.json +1 -6
  5. training_args.bin +1 -1
README.md CHANGED
@@ -18,7 +18,7 @@ should probably proofread and complete it, then remove this comment. -->
18
 
19
  This model is a fine-tuned version of [LoftQ/Meta-Llama-3-8B-Instruct-4bit-64rank](https://huggingface.co/LoftQ/Meta-Llama-3-8B-Instruct-4bit-64rank) on an unknown dataset.
20
  It achieves the following results on the evaluation set:
21
- - Loss: 1.0052
22
 
23
  ## Model description
24
 
@@ -43,22 +43,42 @@ The following hyperparameters were used during training:
43
  - seed: 42
44
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
45
  - lr_scheduler_type: cosine
46
- - num_epochs: 10
47
 
48
  ### Training results
49
 
50
  | Training Loss | Epoch | Step | Validation Loss |
51
  |:-------------:|:-----:|:----:|:---------------:|
52
- | 2.3911 | 1.0 | 53 | 1.5495 |
53
- | 2.1963 | 2.0 | 106 | 1.2801 |
54
- | 2.1026 | 3.0 | 159 | 1.1126 |
55
- | 2.0158 | 4.0 | 212 | 1.0529 |
56
- | 1.9591 | 5.0 | 265 | 1.0237 |
57
- | 1.9115 | 6.0 | 318 | 1.0124 |
58
- | 1.8671 | 7.0 | 371 | 1.0084 |
59
- | 1.8498 | 8.0 | 424 | 1.0052 |
60
- | 1.831 | 9.0 | 477 | 1.0056 |
61
- | 1.8301 | 10.0 | 530 | 1.0052 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
62
 
63
 
64
  ### Framework versions
 
18
 
19
  This model is a fine-tuned version of [LoftQ/Meta-Llama-3-8B-Instruct-4bit-64rank](https://huggingface.co/LoftQ/Meta-Llama-3-8B-Instruct-4bit-64rank) on an unknown dataset.
20
  It achieves the following results on the evaluation set:
21
+ - Loss: 0.8668
22
 
23
  ## Model description
24
 
 
43
  - seed: 42
44
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
45
  - lr_scheduler_type: cosine
46
+ - num_epochs: 30
47
 
48
  ### Training results
49
 
50
  | Training Loss | Epoch | Step | Validation Loss |
51
  |:-------------:|:-----:|:----:|:---------------:|
52
+ | 2.3909 | 1.0 | 53 | 1.5473 |
53
+ | 2.1937 | 2.0 | 106 | 1.2690 |
54
+ | 2.0915 | 3.0 | 159 | 1.0977 |
55
+ | 1.9927 | 4.0 | 212 | 1.0320 |
56
+ | 1.9058 | 5.0 | 265 | 1.0046 |
57
+ | 1.8032 | 6.0 | 318 | 0.9885 |
58
+ | 1.6688 | 7.0 | 371 | 0.9754 |
59
+ | 1.5215 | 8.0 | 424 | 0.9745 |
60
+ | 1.3617 | 9.0 | 477 | 0.9640 |
61
+ | 1.2074 | 10.0 | 530 | 0.9579 |
62
+ | 1.0429 | 11.0 | 583 | 0.9441 |
63
+ | 0.9013 | 12.0 | 636 | 0.9355 |
64
+ | 0.7969 | 13.0 | 689 | 0.9278 |
65
+ | 0.7092 | 14.0 | 742 | 0.9171 |
66
+ | 0.6272 | 15.0 | 795 | 0.9070 |
67
+ | 0.5688 | 16.0 | 848 | 0.9052 |
68
+ | 0.5128 | 17.0 | 901 | 0.8942 |
69
+ | 0.469 | 18.0 | 954 | 0.8894 |
70
+ | 0.4294 | 19.0 | 1007 | 0.8871 |
71
+ | 0.3953 | 20.0 | 1060 | 0.8807 |
72
+ | 0.371 | 21.0 | 1113 | 0.8756 |
73
+ | 0.3533 | 22.0 | 1166 | 0.8750 |
74
+ | 0.3335 | 23.0 | 1219 | 0.8730 |
75
+ | 0.3212 | 24.0 | 1272 | 0.8699 |
76
+ | 0.3108 | 25.0 | 1325 | 0.8687 |
77
+ | 0.3089 | 26.0 | 1378 | 0.8676 |
78
+ | 0.3031 | 27.0 | 1431 | 0.8678 |
79
+ | 0.3014 | 28.0 | 1484 | 0.8675 |
80
+ | 0.3013 | 29.0 | 1537 | 0.8666 |
81
+ | 0.2978 | 30.0 | 1590 | 0.8668 |
82
 
83
 
84
  ### Framework versions
adapter_config.json CHANGED
@@ -20,11 +20,11 @@
20
  "rank_pattern": {},
21
  "revision": null,
22
  "target_modules": [
 
23
  "all-linear",
24
  "q_proj",
25
  "v_proj",
26
- "k_proj",
27
- "o_proj"
28
  ],
29
  "task_type": "CAUSAL_LM",
30
  "use_dora": false,
 
20
  "rank_pattern": {},
21
  "revision": null,
22
  "target_modules": [
23
+ "o_proj",
24
  "all-linear",
25
  "q_proj",
26
  "v_proj",
27
+ "k_proj"
 
28
  ],
29
  "task_type": "CAUSAL_LM",
30
  "use_dora": false,
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:7c05061ab3d9f3f6e9aad31c1a0b96ae2e3da5468a73a43d2946b36d3f003082
3
  size 218138576
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6845c582b4cee8404cbc4538ef581d423cd0c2b23006af1e8ffb3735ff88fb8f
3
  size 218138576
tokenizer.json CHANGED
@@ -1,11 +1,6 @@
1
  {
2
  "version": "1.0",
3
- "truncation": {
4
- "direction": "Right",
5
- "max_length": 1024,
6
- "strategy": "LongestFirst",
7
- "stride": 0
8
- },
9
  "padding": null,
10
  "added_tokens": [
11
  {
 
1
  {
2
  "version": "1.0",
3
+ "truncation": null,
 
 
 
 
 
4
  "padding": null,
5
  "added_tokens": [
6
  {
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:508ef04d248d3ca361ccf90aea9f309f1084f6c4206885e043ab9e69ef1d759f
3
  size 5048
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:144e158846b8b07db045419a88ebc514d6630d59c4798dfa0e4b5643632a50c1
3
  size 5048