Add Google Gemini Tokenizers
#11
by
CortexPE
- opened
If I remember correctly, the same tokenizers are used by the open-source Gemma models, so you should be able to use the "Gemma" option from the dropdown.
If I remember correctly, the same tokenizers are used by the open-source Gemma models, so you should be able to use the "Gemma" option from the dropdown.