xgen-7b-4k-base / tokenization_xgen.py

Commit History

update _convert_id_to_token
4bb73d6

tianxie-sf commited on

add pad token and default eos token (#6)
3ee890e

rooa tianxie-sf commited on

.
f70089e

rooa commited on

.
d5c0667

rooa commited on

.
4fb6c92

rooa commited on