Tokenizer difference
#1
by
dranger003
- opened
Thanks for this great model! It is also great to see support in llama.cpp for it.
At this point however there seems to be an issue with the tokenizer - more details here:
https://github.com/ggerganov/llama.cpp/issues/5496
Edit: I just realised Cebtenzzre is part of Nomic, so I guess you already know about this then :)
great, I just experience this issue