dvruette's picture
Upload tokenizer
f034951
raw
history blame contribute delete
133 Bytes
{
"<|assistant|>": 32004,
"<|prefix_begin|>": 32003,
"<|prefix_end|>": 32000,
"<|prompter|>": 32002,
"<|system|>": 32001
}