File size: 2,351 Bytes
5c179bb 0e3c5fc 5c179bb 0e3c5fc 5c179bb 85fc806 5c179bb 0e3c5fc 5c179bb b29ef86 0e3c5fc 5c179bb 0e3c5fc 5c179bb 4c7b94c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 |
---
library_name: transformers
tags: []
---
This is the SFT checkpoint used for the project [RLHFlow/Online-RLHF](https://github.com/RLHFlow/Online-RLHF)
* **Paper**: [RLHF Workflow: From Reward Modeling to Online RLHF](https://arxiv.org/pdf/2405.07863) (Published in TMLR, 2024)
* **Authors**: Hanze Dong*, Wei Xiong*, Bo Pang*, Haoxiang Wang*, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, Tong Zhang
* **Code**: https://github.com/RLHFlow/Online-RLHF
The model is trained from [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) on [RLHFlow/RLHFlow-SFT-Dataset-ver2](https://huggingface.co/datasets/RLHFlow/RLHFlow-SFT-Dataset-ver2) for 2 epochs. We use a global batch size of 128 and a learning rate of 2e-5, where we pack the samples and split them into chunks of 8192 token. See more training details at https://github.com/RLHFlow/Online-RLHF/blob/main/sft/llama3-8b-it.yaml .
## Academic Benchmarks
We use ToRA script to evaluate GSM8K and MATH, Evalplut for HumanEval, and lm-evaluation-harness for other benchmarks. The model is evaluated in zero-shot setting.
| **Model** | **Size** | **Method** | **LC AlpacaEval** | **MT-Bench** | **GSM-8K** | **MATH** | **MMLU** | **HumanEval** | **TruthfulQA** | **ARC** |
|----------------------------|----------|-----------------|------------|------------|------------|----------|---------------|----------------|---------|----------|
| LLaMA-3-8B-it | 8B | RS+DPO+PPO |22.9|8.16| 79.6 | 26.3 | 66.0 | 61.6 | 43.9 | 59.5 |
| RLHFlow/LLaMA3-SFT | 8B | SFT |10.2|7.69| 74.2 | 30.0 | 64.6 | 63.4 | 53.5 | 58.6 |
| RLHFlow/LLaMA3-SFT-v2 | 8B | SFT |12.66|-| 83.4 | 41.1 | 64.8 | 66.5 | 53.9 | 60.0 |
## Citation
Please cite our techical report if you find our model is useful for your research or product.
```
@misc{dong2024rlhf,
title={RLHF Workflow: From Reward Modeling to Online RLHF},
author={Hanze Dong and Wei Xiong and Bo Pang and Haoxiang Wang and Han Zhao and Yingbo Zhou and Nan Jiang and Doyen Sahoo and Caiming Xiong and Tong Zhang},
year={2024},
eprint={2405.07863},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
```
|