musiclang-4k / tokenizer_config.json
floriangardin's picture
Update tokenizer_config.json
8753368
raw
history blame
217 Bytes
{
"model_max_len": 4096,
"pad_token": "WILL_END",
"pad_token_id": 65,
"eos_token": "WILL_END",
"eos_token_id": 65,
"add_prefix_space": false,
"tokenizer_class": "PreTrainedTokenizer",
"type": "Split"
}