Why don't the models of Yi-*-Chat have the "tokenizer.json" file?

#21

by joeking11829 - opened Apr 2

Apr 2

I want to perform TGI benchmark tests on the Yi model's token throughput, but I need the "tokenizer.json" file for the model.

Any Idea ?
Thanks.

YShow

Apr 30

You can load the tokenizer and save the information to a JSON file on your own

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment