RLHFlow/LLaMA3-SFT

This is the SFT checkpoint used for the project RLHFlow/Online-RLHF

• Paper: RLHF Workflow: From Reward Modeling to Online RLHF (Published in TMLR, 2024)
• Authors: Hanze Dong*, Wei Xiong*, Bo Pang*, Haoxiang Wang*, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, Tong Zhang
• Code: https://github.com/RLHFlow/Online-RLHF

The model is trained from meta-llama/Meta-Llama-3-8B on a mixture of diverse open-source high-quality data for 1 epoch with detailed parameters in the report. It has not been trained by RLHF and can serve as a good starting point for the RLHF research.

Academic Benchmarks

We use ToRA script to evaluate GSM8K and MATH, Evalplut for HumanEval, and lm-evaluation-harness for other benchmarks. The model is evaluated in zero-shot setting so the results here may be slightly different from that reported in the technical report.

Model	Size	Method	LC AlpacaEval	MT-Bench	GSM-8K	MMLU	HumanEval	TruthfulQA	ARC	MBPP
LLaMA-3-8B-it	8B	RS+DPO+PPO	22.9	8.16	79.6	66.0	61.6	43.9	59.5	61.1
Ours (SFT baseline)	8B	SFT	10.2	7.69	74.2	30.0	64.6	63.4	53.5	58.6
Ours (Iterative RLHF)	8B	Iterative DPO	37.2	8.46	80.7	65.3	64.6	60.4	64.3	60.8

Citation

Please cite our techical report if you find our model is useful for your research or product.

@misc{dong2024rlhf,
      title={RLHF Workflow: From Reward Modeling to Online RLHF}, 
      author={Hanze Dong and Wei Xiong and Bo Pang and Haoxiang Wang and Han Zhao and Yingbo Zhou and Nan Jiang and Doyen Sahoo and Caiming Xiong and Tong Zhang},
      year={2024},
      eprint={2405.07863},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}