|
--- |
|
license: apache-2.0 |
|
tags: |
|
- medical |
|
datasets: |
|
- biomed |
|
--- |
|
# BioMedGPT-LM-7B |
|
|
|
**BioMedGPT-LM-7B** is the first large generative language model based on Llama2 in the biomedical domain. |
|
It was fine-tuned from the Llama2-7B-Chat with millions of biomedical papers from the [S2ORC corpus](https://github.com/allenai/s2orc/blob/master/README.md). |
|
Through further fine-tuning, BioMedGPT-LM-7B outperforms or is on par with human and significantly larger general-purpose foundation models on several biomedical QA benchmarks. |
|
|
|
|
|
### Training Details |
|
|
|
|
|
|
|
The model was trained with the following hyperparameters: |
|
|
|
* Epochs: 5 |
|
* Batch size: 192 |
|
* Cutoff length: 2048 |
|
* Learning rate: 2e-5 |
|
|
|
BioMedGPT-LM-7B is finetuned on over 26 billion tokens highly pertinent to the field of biomedicine. |
|
The fine-tuning data are extracted from 5.5 million biomedical papers in S2ORC data using PubMed Central (PMC)-ID and PubMed ID as criteria. |
|
|
|
|
|
### Model Developers |
|
PharMolix |
|
|
|
### How to Use |
|
|
|
BioMedGPT-LM-7B is the generative language model of **[BioMedGPT-10B](https://github.com/BioFM/OpenBioMed)**, an open-source version of BioMedGPT. |
|
BioMedGPT is an open multimodal generative pre-trained transformer (GPT) for biomedicine, which bridges the natural language modality and diverse biomedical data modalities via large generative language models. |
|
|
|
More technical details of BioMedGPT-LM-7B, BioMedGPT-10B, and BioMedGPT can be found in the [technical report](https://pan.baidu.com/s/1iAMBkuoZnNAylhopP5OgEg?pwd=7a6b). |
|
|
|
 |
|
|
|
|
|
**Intended Use Cases** |
|
|
|
| **Method** | Parameters (B) | Setting | MedMCQA(\%) | PubMedQA(\%) | |
|
|------------------------|----------------|-----------|-------------|--------------| |
|
| Human (pass)* | - | Manual | - | 60.0 | |
|
| Human (expert)* | - | Manual | 90 | 78.0 | |
|
|------------------------|----------------|-----------|-------------|--------------| |
|
| InstructGPT* | 175 | zero-shot | 44.0 | 73.2 | |
|
| ChatGPT* | - | zero-shot | 44.7 | 63.9 | |
|
| Llama* | 7 | zero-shot | 24.3 | 5.2 | |
|
| Llama2 | 7 | zero-shot | 30.6 | 3.7 | |
|
| Llama2-Chat | 7 | zero-shot | 35.5 | 21.9 | |
|
|------------------------|----------------| --------- |-------------|--------------| |
|
| Llama | 7 |Fine-tuing | 48.2 | 73.4 | |
|
| Llama2-Chat | 7 |Fine-tuing | 48.3 | 75.5 | |
|
| PMC-Llama | 7 |Fine-tuing | 50.5 | 69.5 | |
|
|------------------------|----------------|-----------|-------------|--------------| |
|
| **BioMedGPT-LM-7B** | 7 |Fine-tuing | **51.4** | **76.1** | |
|
|
|
**Out-of-scope Uses** |
|
|
|
|
|
### Technical Report |
|
"BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine" |
|
|
|
|
|
### github |
|
[https://github.com/BioFM/OpenBioMed](https://github.com/BioFM/OpenBioMed) |
|
|
|
|
|
### Limitations |
|
|
|
[Highlight any limitations or potential issues of your model.] |
|
|
|
|
|
|
