Hugging Face's logo

pietrolesci
/
raw_tokenisers

Model card Files Community
pietrolesci commited on
Commit
fe49495
·
verified ·
1 Parent(s): 889c570

Upload folder using huggingface_hub

Browse files
Files changed (5) hide show
  1. .gitattributes +1 -0
  2. wordpiece_minipile_2025-02-03T12-10-57/all_merges.jsonl +0 -0
  3. wordpiece_minipile_2025-02-03T12-10-57/implemented_merges.jsonl +3 -0
  4. wordpiece_minipile_2025-02-03T12-10-57/metadata.yaml +4 -0
  5. wordpiece_minipile_2025-02-03T12-10-57/tokenizer.json +0 -0
.gitattributes CHANGED
@@ -42,3 +42,4 @@ bpe_finewebedu_2024-10-11T20-50-21/tokenizer.json filter=lfs diff=lfs merge=lfs
42
  bpe_minipile_2024-10-11T11-23-27/implemented_merges.jsonl filter=lfs diff=lfs merge=lfs -text
43
  bpe_minipile_2024-10-11T11-23-27/tokenizer.json filter=lfs diff=lfs merge=lfs -text
44
  wordpiece_minipile_2025-02-03T10-57-36/implemented_merges.jsonl filter=lfs diff=lfs merge=lfs -text
 
 
42
  bpe_minipile_2024-10-11T11-23-27/implemented_merges.jsonl filter=lfs diff=lfs merge=lfs -text
43
  bpe_minipile_2024-10-11T11-23-27/tokenizer.json filter=lfs diff=lfs merge=lfs -text
44
  wordpiece_minipile_2025-02-03T10-57-36/implemented_merges.jsonl filter=lfs diff=lfs merge=lfs -text
45
+ wordpiece_minipile_2025-02-03T12-10-57/implemented_merges.jsonl filter=lfs diff=lfs merge=lfs -text
wordpiece_minipile_2025-02-03T12-10-57/all_merges.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
wordpiece_minipile_2025-02-03T12-10-57/implemented_merges.jsonl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:030d3efa446342bfd5dd49d244b3bcbc4f3f13a32b5c7bb864d313c20ed71c72
3
+ size 36200163
wordpiece_minipile_2025-02-03T12-10-57/metadata.yaml ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ max_vocab_size: 320000
2
+ num_docs: 1000000
3
+ eos_token: <|endoftext|>
4
+ unk_token: <|unk|>
wordpiece_minipile_2025-02-03T12-10-57/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff