---
{}
---
# ModernBioBERT

A modern variant of BioBERT based on ModernBERT.
We continued the masked language modeling pre-training task for 1.000.000 steps on PubMed abstracts.

## Pre-Training Details
```
Batchsize: 512
Learningrate: 1e-4
Warmupsteps: 500
Learning Rate Scheduler: Cosine Schedule
Max. Sequence Length: 512
Precision: bfloat16
```

---
datasets:
- ncbi/pubmed
language:
- en
base_model:
- answerdotai/ModernBERT-base
---