| {} | |
| # ModernBioBERT | |
| A modern variant of BioBERT based on ModernBERT. | |
| We continued the masked language modeling pre-training task for 1.000.000 steps on PubMed abstracts. | |
| ## Pre-Training Details | |
| ``` | |
| Batchsize: 512 | |
| Learningrate: 1e-4 | |
| Warmupsteps: 500 | |
| Learning Rate Scheduler: Cosine Schedule | |
| Max. Sequence Length: 512 | |
| Precision: bfloat16 | |
| ``` | |
| --- | |
| datasets: | |
| - ncbi/pubmed | |
| language: | |
| - en | |
| base_model: | |
| - answerdotai/ModernBERT-base | |
| --- |