End of training

README.md

@@ -14,7 +14,9 @@ should probably proofread and complete it, then remove this comment. -->
 
 # arabic-nano-gpt
 
-This model is a fine-tuned version of [openai-community/gpt2](https://huggingface.co/openai-community/gpt2) on the None dataset.
+This model is a fine-tuned version of [openai-community/gpt2](https://huggingface.co/openai-community/gpt2) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 6.9452
 
 ## Model description
 
@@ -33,23 +35,29 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- learning_rate: 0.001
-- train_batch_size: 512
-- eval_batch_size: 512
+- learning_rate: 0.0001
+- train_batch_size: 32
+- eval_batch_size: 32
 - seed: 42
 - gradient_accumulation_steps: 32
-- total_train_batch_size: 16384
+- total_train_batch_size: 1024
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
 - lr_scheduler_type: linear
 - num_epochs: 5
 
 ### Training results
 
+| Training Loss | Epoch  | Step | Validation Loss |
+|:-------------:|:------:|:----:|:---------------:|
+| No log        | 1.1376 | 100  | 7.6593          |
+| No log        | 2.2752 | 200  | 7.3363          |
+| No log        | 3.4127 | 300  | 7.0512          |
+| No log        | 4.5503 | 400  | 6.9452          |
 
 
 ### Framework versions
 
-- Transformers 4.44.2
-- Pytorch 2.4.0
+- Transformers 4.45.2
+- Pytorch 2.5.0
 - Datasets 3.0.1
-- Tokenizers 0.19.1
+- Tokenizers 0.20.1

config.json

@@ -11,12 +11,12 @@
   "initializer_range": 0.02,
   "layer_norm_epsilon": 1e-05,
   "model_type": "gpt2",
-  "n_ctx": 128,
-  "n_embd": 128,
-  "n_head": 2,
+  "n_ctx": 1024,
+  "n_embd": 256,
+  "n_head": 4,
   "n_inner": null,
-  "n_layer": 2,
-  "n_positions": 128,
+  "n_layer": 4,
+  "n_positions": 1024,
   "reorder_and_upcast_attn": false,
   "resid_pdrop": 0.1,
   "scale_attn_by_inverse_layer_idx": false,
@@ -33,7 +33,7 @@
     }
   },
   "torch_dtype": "float32",
-  "transformers_version": "4.44.2",
+  "transformers_version": "4.45.2",
   "use_cache": true,
-  "vocab_size": 4096
+  "vocab_size": 8192
 }

generation_config.json

@@ -2,5 +2,5 @@
   "_from_model_config": true,
   "bos_token_id": 0,
   "eos_token_id": 0,
-  "transformers_version": "4.44.2"
+  "transformers_version": "4.45.2"
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:15a0159888cf8457308b1ed3b5ea9a5b487452f3f9395358ae5b45d9b9023e71
-size 3752592
+oid sha256:a787584f52254755b1c755a34770bd19ad6cb33948b047cc0e688a5d3a519851
+size 22080496

tokenizer_config.json

@@ -11,9 +11,9 @@
     }
   },
   "bos_token": "<|endoftext|>",
-  "clean_up_tokenization_spaces": true,
+  "clean_up_tokenization_spaces": false,
   "eos_token": "<|endoftext|>",
-  "model_max_length": 128,
+  "model_max_length": 1024,
   "pad_token": "<|endoftext|>",
   "tokenizer_class": "GPT2Tokenizer",
   "unk_token": "<|endoftext|>"

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:359cf6a01bd9be870b09352a55800ee50c76f059d9246e9455ed13c4ea715bb8
+oid sha256:60c2c4caa72b7bf4531d43d3b3f8acd872a5b1fc692746b6fbda3b84afe9437b
 size 5240