openhermes-mistral-dpo-gptq

This model is a fine-tuned version of TheBloke/OpenHermes-2-Mistral-7B-GPTQ on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.5935
• Rewards/chosen: 0.0821
• Rewards/rejected: -0.1240
• Rewards/accuracies: 0.875
• Rewards/margins: 0.2061
• Logps/rejected: -193.3659
• Logps/chosen: -263.6407
• Logits/rejected: -2.0827
• Logits/chosen: -2.0590

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 1
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 2
• training_steps: 50
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6737	0.01	10	0.6325	0.1116	-0.0125	0.8125	0.1241	-192.2511	-263.3457	-2.0830	-2.0711
0.7088	0.01	20	0.5983	0.1753	-0.0556	1.0	0.2309	-192.6819	-262.7091	-2.0822	-2.0681
0.7333	0.01	30	0.5994	0.1257	-0.0786	0.875	0.2043	-192.9118	-263.2048	-2.0841	-2.0637
0.7023	0.02	40	0.5838	0.1417	-0.1014	0.8125	0.2431	-193.1394	-263.0449	-2.0813	-2.0600
0.6754	0.03	50	0.5935	0.0821	-0.1240	0.875	0.2061	-193.3659	-263.6407	-2.0827	-2.0590

Framework versions

• PEFT 0.9.0
• Transformers 4.38.2
• Pytorch 2.0.1+cu117
• Datasets 2.18.0
• Tokenizers 0.15.2