distil-bert-tweaking

This model is a fine-tuned version of pilotj/distilbert-base-uncased-fibe-full-finetuned on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.4871

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-05
• train_batch_size: 32
• eval_batch_size: 128
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.5
• num_epochs: 2

Training results

Training Loss	Epoch	Step	Validation Loss
0.3542	0.1905	500	0.4708
0.418	0.3810	1000	0.4670
0.4303	0.5714	1500	0.4833
0.4404	0.7619	2000	0.5177
0.4589	0.9524	2500	0.5050
0.3911	1.1429	3000	0.5887
0.3654	1.3333	3500	0.5227
0.3415	1.5238	4000	0.5158
0.3284	1.7143	4500	0.5228
0.3178	1.9048	5000	0.4871

Framework versions

• Transformers 4.44.2
• Pytorch 2.4.0
• Datasets 3.0.0
• Tokenizers 0.19.1