return_offsets_mapping in tokenizer does not meet expectations

#41
by cyue2436 - opened

image.png

image.png

Why is the output like this? I think this is not what I expected. Offsets_mapping should be an interval to express the corresponding character position?

I have tried several transformer versions and it doesn't seem to fix the problem.

Sign up or log in to comment