kanishka/smolm-autoreg-bpe-counterfactual_babylm_aann_low_variability_noun-seed_211-1e-3

smolm-autoreg-bpe-counterfactual_babylm_aann_low_variability_noun_211-1e-3

This model was trained from scratch on the kanishka/counterfactual_babylm_aann_low_variability_noun dataset. It achieves the following results on the evaluation set:

• Loss: 3.4073
• Accuracy: 0.4102

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 32
• eval_batch_size: 64
• seed: 211
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 32000
• num_epochs: 20.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
3.6045	1.0	18595	3.8020	0.3570
3.3841	2.0	37190	3.5977	0.3801
3.2569	3.0	55785	3.4644	0.3932
3.1775	4.0	74380	3.4314	0.3981
3.1198	5.0	92975	3.4138	0.4017
3.0787	6.0	111570	3.3922	0.4036
3.0411	7.0	130165	3.3671	0.4059
3.0125	8.0	148760	3.3583	0.4072
2.9812	9.0	167355	3.3392	0.4088
2.9592	10.0	185950	3.3605	0.4087
2.9333	11.0	204545	3.3525	0.4092
2.9129	12.0	223140	3.3609	0.4097
2.8907	13.0	241735	3.3751	0.4097
2.8737	14.0	260330	3.3661	0.4099
2.8533	15.0	278925	3.3723	0.4102
2.8283	16.0	297520	3.3770	0.4106
2.817	17.0	316115	3.3919	0.4102
2.7925	18.0	334710	3.3901	0.4106
2.7778	19.0	353305	3.4038	0.4102
2.7587	20.0	371900	3.4073	0.4102

Framework versions

• Transformers 4.38.0
• Pytorch 2.3.1+cu121
• Datasets 2.16.1
• Tokenizers 0.15.2