myBit-Llama2-jp-127M-test-2

This model is a fine-tuned version of TinyLlama/TinyLlama-1.1B-Chat-v1.0 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 10.6538

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.00024
• train_batch_size: 96
• eval_batch_size: 96
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: polynomial
• lr_scheduler_warmup_steps: 250
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
9.1311	0.04	100	7.5382
6.9064	0.07	200	6.6416
6.3625	0.11	300	6.1490
5.8774	0.15	400	5.6766
5.6517	0.18	500	6.1719
8.8804	0.22	600	10.4020
10.5191	0.26	700	10.5768
10.5934	0.29	800	10.6068
10.613	0.33	900	10.6190
10.622	0.37	1000	10.6256
10.6276	0.41	1100	10.6300
10.6315	0.44	1200	10.6332
10.6343	0.48	1300	10.6361
10.6368	0.52	1400	10.6383
10.6389	0.55	1500	10.6402
10.6408	0.59	1600	10.6419
10.6424	0.63	1700	10.6434
10.6439	0.66	1800	10.6447
10.6453	0.7	1900	10.6460
10.6465	0.74	2000	10.6473
10.6476	0.77	2100	10.6483
10.6487	0.81	2200	10.6494
10.6497	0.85	2300	10.6503
10.6505	0.88	2400	10.6511
10.6514	0.92	2500	10.6521
10.6521	0.96	2600	10.6527
10.6532	0.99	2700	10.6538

Framework versions

• Transformers 4.38.2
• Pytorch 2.1.0+cu121
• Datasets 2.18.0
• Tokenizers 0.15.2