BOS token in front of every text chunk
#22
by
websterbei
- opened
tokenizer will automatically add BOS token to the front of each text chunk, just checking if this is the intended behavior? (i.e. how it was trained)