license: openrail++
CascadeV | An Implemention of Würstchen architecture for High-Resolution Video Generation
News
[2024.07.17] We release the code and pretrained weights of a DiT-based video VAE, which supports video reconstruction with a high compression factor (1x32x32=1024). The T2V model is still on the way.
Introduction
CascadeV is a video generation pipeline built upon the Würstchen architecture. By using a highly compressed latent representation, we can generate longer videos with higher resolution.
Video VAE
Comparison of Our Cascade Approach with Other VAEs (on Latent Space of Shape 8x32x32)
Video Recontruction: Original (left) vs. Reconstructed (right) | Click to view the videos
 |
 |
 |
 |
1. Model Architecture
1.1 DiT
We use PixArt-Σ as our base model with the following modifications:
- Replace the original VAE (of SDXL) with the one from Stable Video Diffusion.
- Use sematic compressor from StableCascade to provide the low-resolution latent input.
- Remove text encoder and all multi-head cross-attention layers since we are not using text condition.
- Replace all 2D attention layers to 3D. We find that 3D attention outperforms 2+1D (i.e. alternative spatial and temporal attention), especially in temporal consistency.
Comparison of 2+1D Attention (left) vs. 3D Attention (right)
1.2. Grid Attention
Using 3D attention requires much more computational resources than 2D/2+1D, especially with higher resolution. As a compromise solution, we replace some 3D attention layers with alternative spatial and temporal grid attention.
2. Evaluation
Dataset: We perform qualitative comparison with other baselines on the dataset Inter4K, by sampling the first 200 videos from the Inter4K to create a video dataset with a resolution of 1024x1024 and 30 FPS.
Metrics: We use PSNR, SSIM and LPIPS to evaluate the per-frame quality (and the similarity between original and reconstructed video) and VBench to evaluate the video quality independently.
2.1 PSNR/SSIM/LPIPS
Diffusion-based VAEs (like StableCascade and our model) performs poorly in reconstruction metrics, due to their ability to produce videos with more fine-grained details but less similiar to the original ones.
| Model/Compression Factor | PSNR↑ | SSIM↑ | LPIPS↓ |
|---|---|---|---|
| Open-Sora-Plan v1.1/4x8x8=256 | 25.7282 | 0.8000 | 0.1030 |
| EasyAnimate v3/4x8x8=256 | 28.8666 | 0.8505 | 0.0818 |
| StableCascade/1x32x32=1024 | 24.3336 | 0.6896 | 0.1395 |
| Ours/1x32x32=1024 | 23.7320 | 0.6742 | 0.1786 |
2.2 VBench
Our approach has comparable performance to the previous VAEs in both frame-wise and temporal quality even with much larger compression factor.
| Model/Compression Factor | Subject Consistency | Background Consistency | Temporal Flickering | Motion Smoothness | Imaging Quality | Aesthetic Quality |
|---|---|---|---|---|---|---|
| Open-Sora-Plan v1.1/4x8x8=256 | 0.9519 | 0.9618 | 0.9573 | 0.9789 | 0.6791 | 0.5450 |
| EasyAnimate v3/4x8x8=256 | 0.9578 | 0.9695 | 0.9615 | 0.9845 | 0.6735 | 0.5535 |
| StableCascade/1x32x32=1024 | 0.9490 | 0.9517 | 0.9430 | 0.9639 | 0.6811 | 0.5675 |
| Ours/1x32x32=1024 | 0.9601 | 0.9679 | 0.9626 | 0.9837 | 0.6747 | 0.5579 |
3. Usage
3.1 Installation
Recommend to use Conda
conda create -n cascadev python==3.9.0
conda activate cascadev
Install PixArt-Σ
bash install.sh
3.2 Download Pretrained Weights
bash pretrained/download.sh
3.3 Video Reconstruction
A sample script for video reconstruction with compression factor of 32
bash recon.sh
Results of Video Reconstruction: w/o LDM (left) vs. w/ LDM (right)
It takes almost 1 minutes to reconstruct a video of shape 8x1024x1024 with one NVIDIA-A800
3.4 Train VAE
- Replace "video_list" in configs/s1024.effn-f32.py with your own video datasets
- Then run
bash train_vae.sh
Acknowledgement
- PixArt-Σ: The main codebase we built upon.
- StableCascade: Würstchen architecture we used.
- Thanks Stable Video Diffusion for its amazing Video VAE.