README.md

---
datasets:
- instruction-pretrain/medicine-instruction-augmented-corpora
- Open-Orca/OpenOrca
- EleutherAI/pile
- GAIR/lima
- WizardLM/WizardLM_evol_instruct_V2_196k
language:
- en
license: llama3
tags:
- biology
- medical
---
# Instruction Pre-Training: Language Models are Supervised Multitask Learners (EMNLP 2024)
This repo contains the **biomedicine model developed from Llama3-8B** in our paper [Instruction Pre-Training: Language Models are Supervised Multitask Learners](https://huggingface.co/papers/2406.14491).

We explore supervised multitask pre-training by proposing ***Instruction Pre-Training***, a framework that scalably augments massive raw corpora with instruction-response pairs to pre-train language models. The instruction-response pairs are generated by an efficient instruction synthesizer built on open-source models. ***Instruction Pre-Training* outperforms *Vanilla Pre-training* in both general pre-training from scratch and domain-adaptive continual pre-training.** In pre-training from scratch, *Instruction Pre-Training* not only improves pre-trained base models but also benefits more from further instruction tuning. **In continual pre-training, *Instruction Pre-Training* enables Llama3-8B to be comparable to or even outperform Llama3-70B.**

<p align='center'>
    <img src="https://cdn-uploads.huggingface.co/production/uploads/66711d2ee12fa6cc5f5dfc89/vRdsFIVQptbNaGiZ18Lih.png" width="400">
</p>

**************************** **Updates** ****************************
* 2024/9/20: Our paper has been accepted by EMNLP 2024 main conference🎉
* 2024/9/11: Updated [FAQ on continual pre-training from Llama3](https://huggingface.co/instruction-pretrain/instruction-synthesizer)
* 2024/8/29: Updated [guidelines](https://huggingface.co/instruction-pretrain/medicine-Llama3-8B) on evaluating any 🤗Huggingface models on the domain-specific tasks
* 2024/7/31: Updated pre-training suggestions in the `Advanced Usage` section of [instruction-synthesizer](https://huggingface.co/instruction-pretrain/instruction-synthesizer)
* 2024/7/15: We scaled up the pre-trained tokens from 100B to 250B, with the number of synthesized instruction-response pairs reaching 500M. The performance trend on downstream tasks throughout the pre-training process:
<p align='left'>
    <img src="https://cdn-uploads.huggingface.co/production/uploads/66711d2ee12fa6cc5f5dfc89/0okCfRkC6uALTfuNxt0Fa.png" width="500">
</p>
* 2024/6/21: Released the [paper](https://huggingface.co/papers/2406.14491), [code](https://github.com/microsoft/LMOps), and [resources](https://huggingface.co/instruction-pretrain)

## Resources
**🤗 We share our data and models with example usages, feel free to open any discussions at [this page](https://huggingface.co/papers/2406.14491)! 🤗**

- Thanks to the demo [davanstrien/instruction-synthesizer](https://huggingface.co/spaces/davanstrien/instruction-synthesizer) for implementing our approach
- Context-Based Instruction Synthesizer: [instruction-synthesizer](https://huggingface.co/instruction-pretrain/instruction-synthesizer)
- Fine-Tuning Data for the Synthesizer: [ft-instruction-synthesizer-collection](https://huggingface.co/datasets/instruction-pretrain/ft-instruction-synthesizer-collection)
- General Models Pre-Trained from Scratch (on 100B tokes):
  - [InstructLM-500M](https://huggingface.co/instruction-pretrain/InstructLM-500M)
  - [InstructLM-1.3B](https://huggingface.co/instruction-pretrain/InstructLM-1.3B)
- Domain-Specific Models Pre-Trained from Llama3-8B:
  - [Finance-Llama3-8B](https://huggingface.co/instruction-pretrain/finance-Llama3-8B)
  - [Biomedicine-Llama3-8B](https://huggingface.co/instruction-pretrain/medicine-Llama3-8B)
- General Instruction-Augmented Corpora: [general-instruction-augmented-corpora](https://huggingface.co/datasets/instruction-pretrain/general-instruction-augmented-corpora)
- Domain-Specific Instruction-Augmented Corpora (no finance data to avoid ethical issues): [medicine-instruction-augmented-corpora](https://huggingface.co/datasets/instruction-pretrain/medicine-instruction-augmented-corpora)

## Domain-Adaptive Continued Pre-Training
Following [AdaptLLM](https://huggingface.co/AdaptLLM/medicine-chat), we augment the domain-specific raw corpora with instruction-response pairs generated by our [context-based instruction synthesizer](https://huggingface.co/instruction-pretrain/instruction-synthesizer).

### 1. To chat with the biomedicine-Llama3-8B model:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("instruction-pretrain/medicine-Llama3-8B")
tokenizer = AutoTokenizer.from_pretrained("instruction-pretrain/medicine-Llama3-8B")

# Put your input here, NO prompt template is required
user_input = '''Question: Which of the following is an example of monosomy?
Options:
- 46,XX
- 47,XXX
- 69,XYY
- 45,X

Please provide your choice first and then provide explanations if possible.'''

inputs = tokenizer(user_input, return_tensors="pt", add_special_tokens=True).input_ids.to(model.device)
outputs = model.generate(input_ids=inputs, max_new_tokens=400)[0]

answer_start = int(inputs.shape[-1])
pred = tokenizer.decode(outputs[answer_start:], skip_special_tokens=True)

print(pred)
```

### 2. To evaluate any Huggingface LMs on domain-specific tasks (💡New!)
You can use the following script to reproduce our results and evaluate any other Huggingface models on domain-specific tasks. Note that the script is NOT applicable to models that require specific prompt templates (e.g., Llama2-chat, Llama3-Instruct).

1). Set Up Dependencies
   ```bash
   git clone https://github.com/microsoft/LMOps
   cd LMOps/adaptllm
   pip install -r requirements.txt
   ```

2). Evaluate the Model
   ```bash
   # Select the domain from ['biomedicine', 'finance']
   DOMAIN='biomedicine'
  
   # Specify any Huggingface LM name (Not applicable to models requiring specific prompt templates)
   MODEL='instruction-pretrain/medicine-Llama3-8B'
  
   # Model parallelization:
   # - Set MODEL_PARALLEL=False if the model fits on a single GPU. 
   #   We observe that LMs smaller than 10B always meet this requirement.
   # - Set MODEL_PARALLEL=True if the model is too large and encounters OOM on a single GPU.
   MODEL_PARALLEL=False
  
   # Choose the number of GPUs from [1, 2, 4, 8]
   N_GPU=1
  
   # Whether to add a BOS token at the beginning of the prompt input:
   # - Set to False for AdaptLLM.
   # - Set to True for instruction-pretrain models.
   # If unsure, we recommend setting it to False, as this is suitable for most LMs.
   add_bos_token=True

   # Run the evaluation script
   bash scripts/inference.sh ${DOMAIN} ${MODEL} ${add_bos_token} ${MODEL_PARALLEL} ${N_GPU}
   ```

## FAQ on Continual Pre-Training from LLama3
**Q1: Do you use the official Llama3 instruction prompt for pre-training?**

No, the provided Llama3 instruction prompt is designed for the [instruction-tuned model](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct), but our continual pre-training is conducted on the [pre-trained base model](https://huggingface.co/meta-llama/Meta-Llama-3-8B) where only BOS (`<|begin_of_text|>`) and EOS (`<|end_of_text|>`) tokens are required.

**Q2: For the general instructions from OpenOrca, do you concatenate each instruction with its output using '\n'?**

No, as mentioned in the pre-training suggestions, we use a simple whitespace to concatenate each question with its response for the general instruction data from OpenOrca. This is because OpenOrca's data is already templated with diverse natural languge templates (such as those with `\n`), so a whitespace is sufficient to formulate the data.

Note that when using our templated instruction-augmented texts, you don't need to add any concatenations.

**Q3: What about those system prompts in OpenOrca?**

We simply discard the system prompts.

**To put it all together, the text before tokenization looks like this:**

```python
general_instruction_response_text = "<|begin_of_text|>{question} {response}<|end_of_text|>"

instruction_augmented_text = "<|begin_of_text|>{instruction augmented text}<|end_of_text|>"
```
Then, for tokenization, you don't need to add BOS and EOS token ids. The tokenization code looks like this:
```python
text_ids = tokenizer(text, add_special_tokens=False, **kwargs).input_ids
```

## Citation
If you find our work helpful, please cite us:

[Instruction Pre-Training](https://huggingface.co/papers/2406.14491) (EMNLP 2024)
```bibtex
@article{cheng2024instruction,
  title={Instruction Pre-Training: Language Models are Supervised Multitask Learners},
  author={Cheng, Daixuan and Gu, Yuxian and Huang, Shaohan and Bi, Junyu and Huang, Minlie and Wei, Furu},
  journal={arXiv preprint arXiv:2406.14491},
  year={2024}
}
```

[Adapt LLM to Domains](https://huggingface.co/papers/2309.09530)（ICLR 2024）
```bibtex
@inproceedings{
cheng2024adapting,
title={Adapting Large Language Models via Reading Comprehension},
author={Daixuan Cheng and Shaohan Huang and Furu Wei},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=y886UXPEZ0}
}
```