KeyError: '<|endoftext|>' when using the tokenizer

#3
by YellowDoge - opened

Failed to initialize the tokenizer. Is it related to the version?

I also counter the same problem, have you solved it ?

I have the same problem。
图片.png

value_layer = value_layer.view(output_size[0] * output_size[1], value_layer.size(2), -1)
这个错误如何解决?
image.png

Same problem here. Hopefully anyone can fix it

Same problem +1

更新下transformers

Knowledge Engineering Group (KEG) & Data Mining at Tsinghua University org

按照依赖安装,截图中的问题已经解决,更新一下模型实现的文件

pip install --upgrade transformers solved my problem

pip install transformers==4.40.0

zRzRzRzRzRzRzR changed discussion status to closed

It seems this solution cannot work after installing 4.40.0. Use

pip install transformers --upgrade

can help.

Sign up or log in to comment