Duplicate from BlinkDL/rwkv-5-world

Co-authored-by: BlinkDL <[email protected]>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,108 @@
+---
+language:
+- en
+- zh
+- fr
+- es
+- de
+- pt
+- ru
+- it
+- ja
+- ko
+- vi
+- ar
+tags:
+- pytorch
+- text-generation
+- causal-lm
+- rwkv
+license: apache-2.0
+datasets:
+- cerebras/SlimPajama-627B
+- EleutherAI/pile
+- bigcode/starcoderdata
+- oscar-corpus/OSCAR-2301
+---
+
+# RWKV-5 World
+
+Use rwkv pip package 0.8.22+ for RWKV-5 inference: https://pypi.org/project/rwkv/ (pipeline = PIPELINE(model, "rwkv_vocab_v20230424") for rwkv-world models)
+
+Online 7B Demo: https://huggingface.co/spaces/BlinkDL/RWKV-Gradio-2
+
+Online 1.5B Demo: https://huggingface.co/spaces/BlinkDL/RWKV-Gradio-1
+
+GUI: https://github.com/josStorer/RWKV-Runner (see Releases)
+
+Convert to HF formet: https://github.com/BBuf/RWKV-World-HF-Tokenizer
+
+For developer: https://github.com/BlinkDL/ChatRWKV/blob/main/API_DEMO_CHAT.py
+
+https://github.com/BlinkDL/ChatRWKV/blob/main/RWKV_v5_demo.py
+
+How it works: https://twitter.com/BlinkDL_AI/status/1685230712247795713
+
+https://www.rwkv.com/
+
+## Model Description
+
+RWKV-5 trained on 100+ world languages (70% English, 15% multilang, 15% code).
+
+World = Some_Pile + Some_SlimPajama + Some_StarCoder + Some_OSCAR + All_Wikipedia + All_ChatGPT_Data_I_can_find
+
+RWKV-5 training: set --my_testing "r2r4" in latest RWKV-LM v4neo: https://github.com/BlinkDL/RWKV-LM
+
+World v1 = 0.59T tokens
+
+World v2 = 1.12T tokens
+
+Imagine what happens when we use more data :)
+
+Recommended fine-tuning format (use \n for newlines):
+```
+User: xxxxxxxxxxxxxxx
+
+Assistant: xxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxx
+
+User: xxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxx
+
+Assistant: xxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxx
+```
+
+A good chat prompt (better replace \n\n in xxx to \n, such that there will be no newlines in xxx):
+```
+User: hi
+
+Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.
+
+User: xxx
+
+Assistant:
+```
+QA prompt (better replace \n\n in xxx to \n, such that there will be no newlines in xxx):
+```
+Question: xxx
+
+Answer:
+```
+and
+```
+Instruction: xxx
+
+Input: xxx
+
+Response:
+```
+
+!!! There should not be any space after your final ":" or you will upset the tokenizer and see non-English reponse !!!
+
+!!! There should not be any space after your final ":" or you will upset the tokenizer and see non-English reponse !!!
+
+!!! There should not be any space after your final ":" or you will upset the tokenizer and see non-English reponse !!!

RWKV-5-World-0.1B-v1-20230803-ctx4096.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c844a3ee05bcb9065848cb05b10c48a3f381f5ac1953aad89e156ecdf31d7703
+size 385598386

RWKV-5-World-0.4B-v2-20231113-ctx4096.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5a288c54c7f30b0e2d4af23991133fad2af2d5e59ec7ad850ffe78054a5e4f92
+size 923523954

RWKV-5-World-1B5-v2-20231025-ctx4096.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5a89f56be7f82ab9dd0835af9a6838f788477471616c02f7b041e3aea0c57435
+size 3155590194

RWKV-5-World-3B-v2-20231113-ctx4096.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a4bd430343c6fd138b85bbc68bb20262d3a2f053ea57dc4b41078269af68ff9c
+size 6126106674

RWKV-5-World-3B-v2-20231118-ctx16k.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:efa5178d1c824b94ef17c6c9a456674e5581a8be832becbda9aba4dc533f88c2
+size 6126106467

RWKV-5-World-7B-v2-20240128-ctx4096.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a88c7274184b211e5545c8f992f0b80d03c40a447980bbfcd0f6d5858982615a
+size 15036197526