Hugging Face's logo

pietrolesci
/
raw_tokenisers

Model card Files Community
pietrolesci commited on
Commit
889c570
·
verified ·
1 Parent(s): 72714a5

Upload folder using huggingface_hub

Browse files
Files changed (5) hide show
  1. .gitattributes +1 -0
  2. wordpiece_minipile_2025-02-03T10-57-36/all_merges.jsonl +0 -0
  3. wordpiece_minipile_2025-02-03T10-57-36/implemented_merges.jsonl +3 -0
  4. wordpiece_minipile_2025-02-03T10-57-36/metadata.yaml +3 -0
  5. wordpiece_minipile_2025-02-03T10-57-36/tokenizer.json +0 -0
.gitattributes CHANGED
@@ -41,3 +41,4 @@ bpe_finewebedu_2024-10-11T20-50-21/implemented_merges.jsonl filter=lfs diff=lfs
41
  bpe_finewebedu_2024-10-11T20-50-21/tokenizer.json filter=lfs diff=lfs merge=lfs -text
42
  bpe_minipile_2024-10-11T11-23-27/implemented_merges.jsonl filter=lfs diff=lfs merge=lfs -text
43
  bpe_minipile_2024-10-11T11-23-27/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 
 
41
  bpe_finewebedu_2024-10-11T20-50-21/tokenizer.json filter=lfs diff=lfs merge=lfs -text
42
  bpe_minipile_2024-10-11T11-23-27/implemented_merges.jsonl filter=lfs diff=lfs merge=lfs -text
43
  bpe_minipile_2024-10-11T11-23-27/tokenizer.json filter=lfs diff=lfs merge=lfs -text
44
+ wordpiece_minipile_2025-02-03T10-57-36/implemented_merges.jsonl filter=lfs diff=lfs merge=lfs -text
wordpiece_minipile_2025-02-03T10-57-36/all_merges.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
wordpiece_minipile_2025-02-03T10-57-36/implemented_merges.jsonl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:80145f35c26f997d9fb3d38e29d084141ba6c964b6ed78e2687bbc21568c3dba
3
+ size 36200113
wordpiece_minipile_2025-02-03T10-57-36/metadata.yaml ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ max_vocab_size: 320000
2
+ num_docs: 1000000
3
+ eos_token: <|endoftext|>
wordpiece_minipile_2025-02-03T10-57-36/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff