BOS token

by 1AH - opened Jun 19

1AH

Jun 19

 "eos_token_id": 151645,
  "bos_token_id": 151645,

is the bos token really "<|im_end|>"?

Jun 19

You're right. That's a mistake, it should be token ID 151643. Thanks for catching.

alpindale changed discussion status to closed Jun 19

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment