Tokenizer needs to be fixed for BOS handling

#18
by dzhulgakov - opened

Same issues as https://huggingface.co/nltpt/Llama-3.2-1B-Instruct/discussions/8 - raw encode() doesn't prepend BOS

Sign up or log in to comment