Tokenizer needs to be fixed for BOS handling
#18
by
dzhulgakov
- opened
Same issues as https://huggingface.co/nltpt/Llama-3.2-1B-Instruct/discussions/8 - raw encode() doesn't prepend BOS
Same issues as https://huggingface.co/nltpt/Llama-3.2-1B-Instruct/discussions/8 - raw encode() doesn't prepend BOS