Update tokenizer files
#2
by
Xenova
HF staff
- opened
No description provided.
Sure thing! This is what I got when I loaded the tokenizer with .from_pretrained followed by .save_pretrained directly afterwards. I got a warning saying that it needed to re-convert the tokenizer.
That is an expected warning based on how HF has that implemented currently.
psinger
changed pull request status to
closed