xgen-7b-8k-inst / tokenization_xgen.py

Commit History

Update tokenization_xgen.py (#16)
943f44c

tianxie-sf commited on

Update tokenization_xgen.py
d008fe8

rooa commited on

fix issue in get vocab (#14)
68f77de

tianxie-sf commited on

update _convert_id_to_token (#11)
5fe0f1b

tianxie-sf commited on

add pad token and default eos token (#7)
76a8dd5

rooa tianxie-sf commited on

.
b2d761b

rooa commited on