tangled-alpha-0.3-core / special_tokens_map.json
mtasic85's picture
tokenizer: fixed phi 4 tokenizer based on unsloth suggestions
3b835a1
raw
history blame
116 Bytes
{
"bos_token": "<|endoftext|>",
"eos_token": "<|im_end|>",
"pad_token": "<|pad|>",
"unk_token": "<|unk|>"
}