---
license: apache-2.0
library_name: peft
tags:
- trl
- dpo
- generated_from_trainer
base_model: alignment-handbook/zephyr-7b-sft-full
model-index:
- name: zephyr-dpo-qlora-gpt4-5e-7-epoch3
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# zephyr-dpo-qlora-gpt4-5e-7-epoch3

This model is a fine-tuned version of [alignment-handbook/zephyr-7b-sft-full](https://huggingface.co/alignment-handbook/zephyr-7b-sft-full) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.6607
- Rewards/chosen: -0.3036
- Rewards/rejected: -0.4008
- Rewards/accuracies: 0.6032
- Rewards/margins: 0.0972
- Rewards/margins Max: 0.5193
- Rewards/margins Min: -0.3338
- Rewards/margins Std: 0.3807
- Logps/rejected: -299.2639
- Logps/chosen: -315.5808
- Logits/rejected: -2.6356
- Logits/chosen: -2.6727

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-07
- train_batch_size: 2
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- total_train_batch_size: 16
- total_eval_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Rewards/margins Max | Rewards/margins Min | Rewards/margins Std | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:-------------------:|:-------------------:|:-------------------:|:--------------:|:------------:|:---------------:|:-------------:|
| 0.6815        | 0.28  | 100  | 0.6918          | -0.0019        | -0.0055          | 0.5516             | 0.0037          | 0.0181              | -0.0087             | 0.0120              | -259.7351      | -285.4075    | -2.8079         | -2.8531       |
| 0.6235        | 0.56  | 200  | 0.6873          | -0.0383        | -0.0542          | 0.5873             | 0.0160          | 0.0859              | -0.0499             | 0.0601              | -264.6065      | -289.0478    | -2.7712         | -2.8159       |
| 0.5521        | 0.85  | 300  | 0.6808          | -0.1327        | -0.1683          | 0.5952             | 0.0356          | 0.1823              | -0.1064             | 0.1266              | -276.0095      | -298.4897    | -2.7261         | -2.7701       |
| 0.4853        | 1.13  | 400  | 0.6749          | -0.2053        | -0.2614          | 0.6032             | 0.0561          | 0.2952              | -0.1704             | 0.2056              | -285.3263      | -305.7520    | -2.6873         | -2.7295       |
| 0.4561        | 1.41  | 500  | 0.6651          | -0.1807        | -0.2628          | 0.5913             | 0.0821          | 0.4091              | -0.2388             | 0.2874              | -285.4612      | -303.2937    | -2.6622         | -2.7037       |
| 0.4337        | 1.69  | 600  | 0.6630          | -0.2648        | -0.3479          | 0.6111             | 0.0831          | 0.4556              | -0.2917             | 0.3299              | -293.9761      | -311.7008    | -2.6522         | -2.6912       |
| 0.4052        | 1.97  | 700  | 0.6606          | -0.2499        | -0.3494          | 0.6151             | 0.0995          | 0.5023              | -0.3041             | 0.3604              | -294.1273      | -310.2143    | -2.6437         | -2.6819       |
| 0.3797        | 2.25  | 800  | 0.6601          | -0.2711        | -0.3716          | 0.6151             | 0.1005          | 0.5194              | -0.3194             | 0.3750              | -296.3420      | -312.3301    | -2.6373         | -2.6750       |
| 0.3692        | 2.54  | 900  | 0.6601          | -0.2914        | -0.3911          | 0.6032             | 0.0997          | 0.5207              | -0.3303             | 0.3804              | -298.2907      | -314.3626    | -2.6357         | -2.6730       |
| 0.3953        | 2.82  | 1000 | 0.6607          | -0.3036        | -0.4008          | 0.6032             | 0.0972          | 0.5193              | -0.3338             | 0.3807              | -299.2639      | -315.5808    | -2.6356         | -2.6727       |


### Framework versions

- PEFT 0.7.1
- Transformers 4.39.0.dev0
- Pytorch 2.1.2+cu121
- Datasets 2.14.6
- Tokenizers 0.15.2