medusa-microllama_305M_stage2

This model is a fine-tuned version of keeeeenw/MicroLlama on the None dataset. It achieves the following results on the evaluation set:

• Loss: 3.5262

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 40
• num_epochs: 2

Training results

Training Loss	Epoch	Step	Validation Loss
4.6913	0.0244	40	4.7578
4.8782	0.0489	80	4.8017
4.642	0.0733	120	4.7973
4.4601	0.0978	160	4.7589
4.4806	0.1222	200	4.6955
4.4856	0.1467	240	4.6196
4.4671	0.1711	280	4.5750
4.3228	0.1955	320	4.5563
4.1184	0.2200	360	4.5274
3.9986	0.2444	400	4.5031
4.2603	0.2689	440	4.4637
4.1344	0.2933	480	4.4349
4.1973	0.3178	520	4.4106
4.3961	0.3422	560	4.4202
4.1814	0.3666	600	4.3732
4.1685	0.3911	640	4.3877
4.3108	0.4155	680	4.3262
4.6294	0.4400	720	4.3108
4.3653	0.4644	760	4.2880
4.1505	0.4888	800	4.2835
3.8278	0.5133	840	4.2623
4.3567	0.5377	880	4.2253
4.2782	0.5622	920	4.1919
4.1025	0.5866	960	4.1846
4.2819	0.6111	1000	4.1637
3.9919	0.6355	1040	4.1323
4.1932	0.6599	1080	4.1017
4.0949	0.6844	1120	4.1085
3.7266	0.7088	1160	4.0668
4.1255	0.7333	1200	4.0500
4.3707	0.7577	1240	4.0207
4.1965	0.7822	1280	4.0065
3.4585	0.8066	1320	3.9363
3.7242	0.8310	1360	3.8893
3.9228	0.8555	1400	3.8569
4.2051	0.8799	1440	3.8412
3.6795	0.9044	1480	3.8245
3.2453	0.9288	1520	3.8132
3.5941	0.9533	1560	3.7907
3.6246	0.9777	1600	3.7573
2.8637	1.0021	1640	3.7530
2.8495	1.0266	1680	3.7741
3.0246	1.0510	1720	3.7690
2.99	1.0755	1760	3.7464
3.1902	1.0999	1800	3.7347
2.8099	1.1244	1840	3.7278
2.7652	1.1488	1880	3.7245
2.6362	1.1732	1920	3.7034
2.8562	1.1977	1960	3.6871
3.1712	1.2221	2000	3.6786
2.7405	1.2466	2040	3.6709
2.734	1.2710	2080	3.6404
3.1788	1.2954	2120	3.6310
2.9609	1.3199	2160	3.6176
3.0737	1.3443	2200	3.6136
2.751	1.3688	2240	3.5960
2.7105	1.3932	2280	3.5872
2.8158	1.4177	2320	3.5848
3.03	1.4421	2360	3.5679
2.8122	1.4665	2400	3.5718
2.5581	1.4910	2440	3.5568
2.9845	1.5154	2480	3.5496
2.83	1.5399	2520	3.5440
2.7004	1.5643	2560	3.5402
2.8271	1.5888	2600	3.5406
2.5315	1.6132	2640	3.5316
2.6001	1.6376	2680	3.5346
2.4959	1.6621	2720	3.5298
2.9174	1.6865	2760	3.5304
2.7219	1.7110	2800	3.5286
2.5395	1.7354	2840	3.5279
2.7464	1.7599	2880	3.5284
2.7532	1.7843	2920	3.5274
2.6472	1.8087	2960	3.5270
2.8263	1.8332	3000	3.5268
2.916	1.8576	3040	3.5263
3.0202	1.8821	3080	3.5262
2.7152	1.9065	3120	3.5261
2.7628	1.9310	3160	3.5261
2.783	1.9554	3200	3.5263
3.2587	1.9798	3240	3.5262

Framework versions

• Transformers 4.43.0
• Pytorch 2.3.1
• Datasets 2.15.0
• Tokenizers 0.19.1