esm2_t130_150M-lora-classifier_2024-04-25_23-13-58

This model is a fine-tuned version of facebook/esm2_t30_150M_UR50D on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.4754
• Accuracy: 0.8809

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005701568055793089
• train_batch_size: 12
• eval_batch_size: 12
• seed: 8893
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• num_epochs: 30
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
0.587	1.0	128	0.6443	0.5957
0.4373	2.0	256	0.6115	0.6699
0.3057	3.0	384	0.4991	0.7812
0.2758	4.0	512	0.4353	0.8242
0.4801	5.0	640	0.3155	0.8691
0.2161	6.0	768	0.3821	0.8301
0.178	7.0	896	0.2889	0.875
0.3202	8.0	1024	0.2716	0.8945
0.192	9.0	1152	0.3002	0.8848
0.0997	10.0	1280	0.3142	0.8828
0.0146	11.0	1408	0.3388	0.8965
0.0777	12.0	1536	0.4100	0.8711
0.0337	13.0	1664	0.3152	0.8848
0.4337	14.0	1792	0.4699	0.8848
0.2544	15.0	1920	0.3347	0.8867
0.0166	16.0	2048	0.4547	0.8770
0.0084	17.0	2176	0.3627	0.8867
0.3829	18.0	2304	0.3663	0.8887
0.096	19.0	2432	0.3994	0.8848
0.017	20.0	2560	0.4222	0.8867
0.0093	21.0	2688	0.4519	0.8906
0.0035	22.0	2816	0.4575	0.8828
0.0072	23.0	2944	0.4675	0.8828
0.0306	24.0	3072	0.4675	0.8867
0.1433	25.0	3200	0.4795	0.8828
0.0073	26.0	3328	0.4755	0.8789
0.3764	27.0	3456	0.4759	0.8809
0.02	28.0	3584	0.4723	0.8828
0.0061	29.0	3712	0.4736	0.8809
0.0042	30.0	3840	0.4754	0.8809

Framework versions

• PEFT 0.10.0
• Transformers 4.39.3
• Pytorch 2.2.1
• Datasets 2.16.1
• Tokenizers 0.15.2