ztyang196 commited on
Commit
15fd708
·
verified ·
1 Parent(s): b7c2f38

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +22 -53
README.md CHANGED
@@ -1,50 +1,25 @@
1
- ---
2
- frameworks:
3
- - Pytorch
4
- license: apache-2.0
5
- tasks:
6
- - motion-generation
7
-
8
- #model-type:
9
- ##如 gpt、phi、llama、chatglm、baichuan 等
10
- #- gpt
11
-
12
- #domain:
13
- ##如 nlp、cv、audio、multi-modal
14
- #- nlp
15
-
16
- #language:
17
- ##语言代码列表 https://help.aliyun.com/document_detail/215387.html?spm=a2c4g.11186623.0.0.9f8d7467kni6Aa
18
- #- cn
19
-
20
- #metrics:
21
- ##如 CIDEr、Blue、ROUGE 等
22
- #- CIDEr
23
-
24
- #tags:
25
- ##各种自定义,包括 pretrained、fine-tuned、instruction-tuned、RL-tuned 等训练方法和其他
26
- #- pretrained
27
-
28
- #tools:
29
- ##如 vllm、fastchat、llamacpp、AdaSeq 等
30
- #- vllm
31
- ---
32
  # The Matrix (Preview)
33
  <div align="center">
34
- <img src="https://huggingface.co/MatrixTeam/TheMatrix/blob/main/readme_src/white-logo.svg" width="50%"/>
35
  </div>
36
  <p align="center">
37
- Download The Matrix model weights at <a href="https://huggingface.co/spaces/THUDM/CogVideoX-5B" target="_blank"> 🤗 Huggingface Space</a> or <a href="https://modelscope.cn/studios/ZhipuAI/CogVideoX-5b-demo" target="_blank"> 🤖 ModelScope Space</a>
38
  </p>
39
  <p align="center">
40
- 📚 View the <a href="https://arxiv.org/abs/2408.06072" target="_blank">paper</a> and <a href="https://zhipu-ai.feishu.cn/wiki/DHCjw1TrJiTyeukfc9RceoSRnCh" target="_blank">Documentation</a>
41
  </p>
42
  <p align="center">
43
- 👋 See Hi to our team and members at <a href="resources/WECHAT.md" target="_blank">Matrix-Team</a>
44
  </p>
45
  <p align="center">
46
- 📍 (Coming Soon) Explore The Matrix playground online at <a href="https://chatglm.cn/video?lang=en?fr=osm_cogvideo">Journee</a> to experience real-time AI generated world.
47
  </p>
 
 
 
 
 
 
48
 
49
  ---
50
  ## What is The Matrix?
@@ -58,22 +33,12 @@ The Matrix is an advanced world model designed to generate **high-quality, infin
58
  At its core, The Matrix combines these elements to push the boundaries of interactive video generation, making real-time, high-quality, infinite-length content a reality.
59
 
60
  ## Documentation
61
- Comprehensive documentation is available in [English](). This includes detailed installation steps, tutorials, and training instructions. The [paper]() and [Project Page]() offer more details about the method.
62
 
63
 
64
  ## Model Weights
65
- Download using SDK
66
- ```bash
67
- pip install modelscope
68
- ```
69
- ```python
70
- from modelscope import snapshot_download
71
- model_dir = snapshot_download('AiurRuili/TheMatrix')
72
- ```
73
- Download using git
74
- ```
75
- git clone https://www.modelscope.cn/AiurRuili/TheMatrix.git
76
- ```
77
 
78
  ## Important Updates
79
 
@@ -81,15 +46,19 @@ According to a request from Alibaba Tongyi Lab, the previous version of The Matr
81
 
82
  As a result, the open release of our model has been delayed, and some components are still under development. These components will be released as soon as they are finished, including:
83
 
84
- [ ] Inference scripts for 8-GPU parallel inference of the DiT backbone, which will accelerate the inference speed by around 6-8 times.
85
- [ ] Training of the Stream Consistency Models, which will accelerate inference speed by around 7-10 times.
86
- [ ] Training on fused realistic and simulated data to acquire stronger generalization ability.
 
 
87
 
88
  ## Reimplementation contributions
89
  The successful release of The Matrix Project is built upon the collective efforts of our incredibly talented team members. We extend our heartfelt gratitude for their dedication, hard work, and invaluable contributions. Those members are:
90
 
91
  **Longxiang Tang, Zhicai Wang, Ruili Feng, Ruihang Chu, Han Zhang, and Zhantao Yang**
92
 
 
 
93
  ## Additional Notes
94
 
95
  There have been certain changes to the hyperparameter settings and training strategy compared to what is reported in the paper due to the re-implementation. Please be aware of these when reviewing the code.
@@ -111,4 +80,4 @@ If you find our work useful please consider citing:
111
  ## License
112
  The code in this repository is released under the Apache 2.0 License.
113
 
114
- The Matrix model (including its corresponding Transformers module and VAE module) is released under the Apache 2.0 License.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  # The Matrix (Preview)
2
  <div align="center">
3
+ <img src=readme_src/white-logo.svg width="50%"/>
4
  </div>
5
  <p align="center">
6
+ Download The Matrix model weights at <a href="https://huggingface.co/MatrixTeam/TheMatrix" target="_blank"> 🤗 Huggingface</a> or <a href="https://www.modelscope.cn/models/AiurRuili/TheMatrix" target="_blank"> 🤖 ModelScope</a>
7
  </p>
8
  <p align="center">
9
+ 📚 View the <a href="https://arxiv.org/abs/2412.03568" target="_blank">Paper</a>, <a href="https://matrixteam-ai.github.io/pages/TheMatrix/" target="_blank"> Website, and <a href="http://matrixteam-ai.github.io/docs/TheMatrixDocs" target="_blank">Documentation</a>
10
  </p>
11
  <p align="center">
12
+ 👋 Say Hi to our team and members at <a href="https://matrixteam-ai.github.io/" target="_blank">Matrix-Team</a>
13
  </p>
14
  <p align="center">
15
+ 📍 (Coming Soon) Explore The Matrix playground online at <a href="">Journee</a> to experience real-time AI generated world.
16
  </p>
17
+
18
+
19
+ <div align="center">
20
+ <img src=readme_src/font_30s.png width="36%"/>
21
+ </div>
22
+
23
 
24
  ---
25
  ## What is The Matrix?
 
33
  At its core, The Matrix combines these elements to push the boundaries of interactive video generation, making real-time, high-quality, infinite-length content a reality.
34
 
35
  ## Documentation
36
+ Comprehensive documentation is available in [English](). This includes detailed installation steps, tutorials, and training instructions. The [paper](https://arxiv.org/abs/2412.03568) and [Project Page](https://matrixteam-ai.github.io/pages/TheMatrix/) offer more details about the method.
37
 
38
 
39
  ## Model Weights
40
+ Model checkpoints can be found in [Huggingface](https://huggingface.co/MatrixTeam/TheMatrix) and [ModelScope](https://www.modelscope.cn/models/AiurRuili/TheMatrix). Please refer to the [Documentation](http://matrixteam-ai.github.io/docs/TheMatrixDocs) for how to load them for inferences.
41
+
 
 
 
 
 
 
 
 
 
 
42
 
43
  ## Important Updates
44
 
 
46
 
47
  As a result, the open release of our model has been delayed, and some components are still under development. These components will be released as soon as they are finished, including:
48
 
49
+ - [ ] Inference scripts for 8-GPU parallel inference of the DiT backbone, which will accelerate the inference speed by around 6-8 times.
50
+
51
+ - [ ] Training of the Stream Consistency Models, which will accelerate inference speed by around 7-10 times.
52
+
53
+ - [ ] Training on fused realistic and simulated data to acquire stronger generalization ability.
54
 
55
  ## Reimplementation contributions
56
  The successful release of The Matrix Project is built upon the collective efforts of our incredibly talented team members. We extend our heartfelt gratitude for their dedication, hard work, and invaluable contributions. Those members are:
57
 
58
  **Longxiang Tang, Zhicai Wang, Ruili Feng, Ruihang Chu, Han Zhang, and Zhantao Yang**
59
 
60
+ Special Thanks to **Longxiang** and **Zhicai** for their excellent contributions.
61
+
62
  ## Additional Notes
63
 
64
  There have been certain changes to the hyperparameter settings and training strategy compared to what is reported in the paper due to the re-implementation. Please be aware of these when reviewing the code.
 
80
  ## License
81
  The code in this repository is released under the Apache 2.0 License.
82
 
83
+ The Matrix model (including its corresponding Transformers module and VAE module) is released under the Apache 2.0 License.