Update README.md
Browse files
README.md
CHANGED
@@ -1,50 +1,25 @@
|
|
1 |
-
---
|
2 |
-
frameworks:
|
3 |
-
- Pytorch
|
4 |
-
license: apache-2.0
|
5 |
-
tasks:
|
6 |
-
- motion-generation
|
7 |
-
|
8 |
-
#model-type:
|
9 |
-
##如 gpt、phi、llama、chatglm、baichuan 等
|
10 |
-
#- gpt
|
11 |
-
|
12 |
-
#domain:
|
13 |
-
##如 nlp、cv、audio、multi-modal
|
14 |
-
#- nlp
|
15 |
-
|
16 |
-
#language:
|
17 |
-
##语言代码列表 https://help.aliyun.com/document_detail/215387.html?spm=a2c4g.11186623.0.0.9f8d7467kni6Aa
|
18 |
-
#- cn
|
19 |
-
|
20 |
-
#metrics:
|
21 |
-
##如 CIDEr、Blue、ROUGE 等
|
22 |
-
#- CIDEr
|
23 |
-
|
24 |
-
#tags:
|
25 |
-
##各种自定义,包括 pretrained、fine-tuned、instruction-tuned、RL-tuned 等训练方法和其他
|
26 |
-
#- pretrained
|
27 |
-
|
28 |
-
#tools:
|
29 |
-
##如 vllm、fastchat、llamacpp、AdaSeq 等
|
30 |
-
#- vllm
|
31 |
-
---
|
32 |
# The Matrix (Preview)
|
33 |
<div align="center">
|
34 |
-
<img src=
|
35 |
</div>
|
36 |
<p align="center">
|
37 |
-
Download The Matrix model weights at <a href="https://huggingface.co/
|
38 |
</p>
|
39 |
<p align="center">
|
40 |
-
📚 View the <a href="https://arxiv.org/abs/
|
41 |
</p>
|
42 |
<p align="center">
|
43 |
-
👋
|
44 |
</p>
|
45 |
<p align="center">
|
46 |
-
📍 (Coming Soon) Explore The Matrix playground online at <a href="
|
47 |
</p>
|
|
|
|
|
|
|
|
|
|
|
|
|
48 |
|
49 |
---
|
50 |
## What is The Matrix?
|
@@ -58,22 +33,12 @@ The Matrix is an advanced world model designed to generate **high-quality, infin
|
|
58 |
At its core, The Matrix combines these elements to push the boundaries of interactive video generation, making real-time, high-quality, infinite-length content a reality.
|
59 |
|
60 |
## Documentation
|
61 |
-
Comprehensive documentation is available in [English](). This includes detailed installation steps, tutorials, and training instructions. The [paper]() and [Project Page]() offer more details about the method.
|
62 |
|
63 |
|
64 |
## Model Weights
|
65 |
-
|
66 |
-
|
67 |
-
pip install modelscope
|
68 |
-
```
|
69 |
-
```python
|
70 |
-
from modelscope import snapshot_download
|
71 |
-
model_dir = snapshot_download('AiurRuili/TheMatrix')
|
72 |
-
```
|
73 |
-
Download using git
|
74 |
-
```
|
75 |
-
git clone https://www.modelscope.cn/AiurRuili/TheMatrix.git
|
76 |
-
```
|
77 |
|
78 |
## Important Updates
|
79 |
|
@@ -81,15 +46,19 @@ According to a request from Alibaba Tongyi Lab, the previous version of The Matr
|
|
81 |
|
82 |
As a result, the open release of our model has been delayed, and some components are still under development. These components will be released as soon as they are finished, including:
|
83 |
|
84 |
-
[ ] Inference scripts for 8-GPU parallel inference of the DiT backbone, which will accelerate the inference speed by around 6-8 times.
|
85 |
-
|
86 |
-
[ ] Training
|
|
|
|
|
87 |
|
88 |
## Reimplementation contributions
|
89 |
The successful release of The Matrix Project is built upon the collective efforts of our incredibly talented team members. We extend our heartfelt gratitude for their dedication, hard work, and invaluable contributions. Those members are:
|
90 |
|
91 |
**Longxiang Tang, Zhicai Wang, Ruili Feng, Ruihang Chu, Han Zhang, and Zhantao Yang**
|
92 |
|
|
|
|
|
93 |
## Additional Notes
|
94 |
|
95 |
There have been certain changes to the hyperparameter settings and training strategy compared to what is reported in the paper due to the re-implementation. Please be aware of these when reviewing the code.
|
@@ -111,4 +80,4 @@ If you find our work useful please consider citing:
|
|
111 |
## License
|
112 |
The code in this repository is released under the Apache 2.0 License.
|
113 |
|
114 |
-
The Matrix model (including its corresponding Transformers module and VAE module) is released under the Apache 2.0 License.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
# The Matrix (Preview)
|
2 |
<div align="center">
|
3 |
+
<img src=readme_src/white-logo.svg width="50%"/>
|
4 |
</div>
|
5 |
<p align="center">
|
6 |
+
Download The Matrix model weights at <a href="https://huggingface.co/MatrixTeam/TheMatrix" target="_blank"> 🤗 Huggingface</a> or <a href="https://www.modelscope.cn/models/AiurRuili/TheMatrix" target="_blank"> 🤖 ModelScope</a>
|
7 |
</p>
|
8 |
<p align="center">
|
9 |
+
📚 View the <a href="https://arxiv.org/abs/2412.03568" target="_blank">Paper</a>, <a href="https://matrixteam-ai.github.io/pages/TheMatrix/" target="_blank"> Website, and <a href="http://matrixteam-ai.github.io/docs/TheMatrixDocs" target="_blank">Documentation</a>
|
10 |
</p>
|
11 |
<p align="center">
|
12 |
+
👋 Say Hi to our team and members at <a href="https://matrixteam-ai.github.io/" target="_blank">Matrix-Team</a>
|
13 |
</p>
|
14 |
<p align="center">
|
15 |
+
📍 (Coming Soon) Explore The Matrix playground online at <a href="">Journee</a> to experience real-time AI generated world.
|
16 |
</p>
|
17 |
+
|
18 |
+
|
19 |
+
<div align="center">
|
20 |
+
<img src=readme_src/font_30s.png width="36%"/>
|
21 |
+
</div>
|
22 |
+
|
23 |
|
24 |
---
|
25 |
## What is The Matrix?
|
|
|
33 |
At its core, The Matrix combines these elements to push the boundaries of interactive video generation, making real-time, high-quality, infinite-length content a reality.
|
34 |
|
35 |
## Documentation
|
36 |
+
Comprehensive documentation is available in [English](). This includes detailed installation steps, tutorials, and training instructions. The [paper](https://arxiv.org/abs/2412.03568) and [Project Page](https://matrixteam-ai.github.io/pages/TheMatrix/) offer more details about the method.
|
37 |
|
38 |
|
39 |
## Model Weights
|
40 |
+
Model checkpoints can be found in [Huggingface](https://huggingface.co/MatrixTeam/TheMatrix) and [ModelScope](https://www.modelscope.cn/models/AiurRuili/TheMatrix). Please refer to the [Documentation](http://matrixteam-ai.github.io/docs/TheMatrixDocs) for how to load them for inferences.
|
41 |
+
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
42 |
|
43 |
## Important Updates
|
44 |
|
|
|
46 |
|
47 |
As a result, the open release of our model has been delayed, and some components are still under development. These components will be released as soon as they are finished, including:
|
48 |
|
49 |
+
- [ ] Inference scripts for 8-GPU parallel inference of the DiT backbone, which will accelerate the inference speed by around 6-8 times.
|
50 |
+
|
51 |
+
- [ ] Training of the Stream Consistency Models, which will accelerate inference speed by around 7-10 times.
|
52 |
+
|
53 |
+
- [ ] Training on fused realistic and simulated data to acquire stronger generalization ability.
|
54 |
|
55 |
## Reimplementation contributions
|
56 |
The successful release of The Matrix Project is built upon the collective efforts of our incredibly talented team members. We extend our heartfelt gratitude for their dedication, hard work, and invaluable contributions. Those members are:
|
57 |
|
58 |
**Longxiang Tang, Zhicai Wang, Ruili Feng, Ruihang Chu, Han Zhang, and Zhantao Yang**
|
59 |
|
60 |
+
Special Thanks to **Longxiang** and **Zhicai** for their excellent contributions.
|
61 |
+
|
62 |
## Additional Notes
|
63 |
|
64 |
There have been certain changes to the hyperparameter settings and training strategy compared to what is reported in the paper due to the re-implementation. Please be aware of these when reviewing the code.
|
|
|
80 |
## License
|
81 |
The code in this repository is released under the Apache 2.0 License.
|
82 |
|
83 |
+
The Matrix model (including its corresponding Transformers module and VAE module) is released under the Apache 2.0 License.
|