What is the id of eos_token?
#15
by
Xipotzzz
- opened
I found that tokenizer.eos_id == 20002 but model.config.eos_token_id == 150005.
I want to finetune chatglm but I wonder which value is correct for eos_token_id?
it seems to be 150005, i.e., id for eop_token. Although I don't know why the authors implement it this way.