|
--- |
|
library_name: tokenizers |
|
tags: [Danish, Mixed Tokenization, LLaMA] |
|
--- |
|
``` |
|
_______ ___ .___ ___. ______ .______ .______ __ __ |
|
| \ / \ | \/ | / __ \ | _ \ | _ \ | | | | |
|
| .--. | / ^ \ | \ / | | | | | | |_) | | |_) | | |__| | |
|
| | | | / /_\ \ | |\/| | | | | | | / | ___/ | __ | |
|
| '--' | / _____ \ | | | | | `--' | | |\ \----.| | | | | | |
|
|_______/ /__/ \__\ |__| |__| \______/ | _| `._____|| _| |__| |__| |
|
|
|
``` |
|
### DA-MIXED-LLAMA3.2-TOKEN |
|
|
|
A mixed tokenization approach combining morphology and BPE strategies, developed for the LLaMA architecture to enhance Danish text processing. |
