opt-babylm2-rewritten-clean-spacy_random-removal-num-adj-earlystop-bpe_seed-1024_1e-3

This model was trained from scratch on the kanishka/babylm2-rewritten-clean-spacy_random-removal-num-adj dataset. It achieves the following results on the evaluation set:

• Loss: 2.6847
• Accuracy: 0.4786

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 32
• eval_batch_size: 64
• seed: 1024
• gradient_accumulation_steps: 8
• total_train_batch_size: 256
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 32000
• num_epochs: 20.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
4.0912	0.9997	2240	3.8117	0.3613
3.4432	1.9997	4480	3.2982	0.4102
3.1317	2.9997	6720	3.0935	0.4303
2.9687	3.9997	8960	2.9876	0.4411
2.8476	4.9997	11200	2.9233	0.4474
2.7908	5.9997	13440	2.8864	0.4513
2.7475	6.9997	15680	2.8587	0.4544
2.7106	7.9997	17920	2.8397	0.4567
2.6915	8.9997	20160	2.8251	0.4583
2.6701	9.9997	22400	2.8133	0.4597
2.6549	10.9997	24640	2.8065	0.4606
2.6374	11.9997	26880	2.7963	0.4614
2.6417	12.9997	29120	2.7917	0.4621
2.6321	13.9997	31360	2.7886	0.4626
2.6147	14.9997	33600	2.7674	0.4653
2.571	15.9997	35840	2.7426	0.4684
2.5232	16.9997	38080	2.7212	0.4715
2.4666	17.9997	40320	2.7036	0.4742
2.4024	18.9997	42560	2.6877	0.4770
2.3301	19.9997	44800	2.6847	0.4786

Framework versions

• Transformers 4.48.0
• Pytorch 2.5.1
• Datasets 3.2.0
• Tokenizers 0.21.0