mxersion
/

Inteligent_ai

Text Generation

Inference Endpoints

Model card Files Files and versions Community

Inteligent_ai / tokenizer_config.json

Mxytyu's picture

Create tokenizer_config.json

cc40c19 verified 8 months ago

157 Bytes

	{
	"vocab_size": 42,
	"special_tokens": ["<banana>", "<potato>", "<squirrel>"],
	"unk_token": "<huh?>",
	"bos_token": "<go!>",
	"eos_token": "<whew>"
	}