metadata
library_name: tokenizers
tags:
- Danish
- Morphological Tokenization
- CerebrasGPT
DA-MORPH-CEREBRAS-TOKEN
This morphological tokenizer is designed for the CerebrasGPT architecture and focuses on segmenting Danish text based on linguistic principles, enabling more meaningful subword tokenization.