QLoRA applied #2

README.md

@@ -2,8 +2,6 @@
 base_model: ybelkada/falcon-7b-sharded-bf16
 tags:
 - generated_from_trainer
-metrics:
-- f1
 model-index:
 - name: falcon-7b-sharded-2
   results: []
@@ -15,9 +13,6 @@ should probably proofread and complete it, then remove this comment. -->
 # falcon-7b-sharded-2
 
 This model is a fine-tuned version of [ybelkada/falcon-7b-sharded-bf16](https://huggingface.co/ybelkada/falcon-7b-sharded-bf16) on an unknown dataset.
-It achieves the following results on the evaluation set:
-- Loss: nan
-- F1: 0.0337
 
 ## Model description
 
@@ -45,14 +40,6 @@ The following hyperparameters were used during training:
 - lr_scheduler_warmup_ratio: 0.03
 - training_steps: 500
 
-### Training results
-
-| Training Loss | Epoch | Step | Validation Loss | F1     |
-|:-------------:|:-----:|:----:|:---------------:|:------:|
-| 7.6119        | 1.0   | 442  | nan             | 0.0337 |
-| 6.8711        | 1.13  | 500  | nan             | 0.0337 |
-
-
 ### Framework versions
 
 - Transformers 4.34.0

adapter_config.json

@@ -1,19 +1,23 @@
 {
+  "alpha_pattern": {},
   "auto_mapping": null,
   "base_model_name_or_path": "ybelkada/falcon-7b-sharded-bf16",
+  "bias": "none",
   "fan_in_fan_out": false,
-  "feedforward_modules": [
-    "dense_4h_to_h",
-    "dense_h_to_4h"
-  ],
   "inference_mode": true,
-  "init_ia3_weights": true,
+  "init_lora_weights": true,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "lora_alpha": 16,
+  "lora_dropout": 0.2,
   "modules_to_save": null,
-  "peft_type": "IA3",
+  "peft_type": "LORA",
+  "r": 32,
+  "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "query_key_value",
     "dense",
+    "query_key_value",
     "dense_4h_to_h",
     "dense_h_to_4h"
   ],

adapter_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:20c2c7dd065f99eab629a983caa759bb943b48e1e0572a30aee8f5332a9a10bd
-size 4170325
+oid sha256:9ea1eb39d8ceff97a117811fe16774356e662db177421214593ba8696ca1577e
+size 261227285

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3029d2fc1e20eadaf5b42a829fedb51d99fc527bbdcf3e8bd5112f5bb38a3e62
+oid sha256:cc96c2a3184d35d7403313432c71e978ee6768981e0bfb651e41e3e73a264c21
 size 4091