ehartford Praful932 commited on
Commit
dcab098
1 Parent(s): 51a49ee

delete extra tokens which caused error while doing inference (#19)

Browse files

- delete extra tokens which caused error while doing inference (d3d687e487658c94da5cfc5a9ed5adb3a7c0a7a0)


Co-authored-by: Praful Mohanan <[email protected]>

Files changed (1) hide show
  1. added_tokens.json +0 -3
added_tokens.json CHANGED
@@ -1,7 +1,4 @@
1
  {
2
- "</s>": 2,
3
- "<s>": 1,
4
- "<unk>": 0,
5
  "<|im_end|>": 32000,
6
  "<|im_start|>": 32001
7
  }
 
1
  {
 
 
 
2
  "<|im_end|>": 32000,
3
  "<|im_start|>": 32001
4
  }