metadata

license: other
license_link: https://huggingface.co/THUDM/CogVideoX-5b/blob/main/LICENSE
language:
  - en
tags:
  - cogvideox
  - video-generation
  - thudm
  - text-to-video
inference: false

CogVideoX-5B

📄 中文阅读 | 🌐 Github | 📜 arxiv

Demo Show

Model Introduction

CogVideoX is an open-source video generation model that shares the same origins as 清影. The table below provides a list of the video generation models we currently offer, along with their basic information.

Model Name	CogVideoX-2B	CogVideoX-5B (Current Repository)
Model Description	Entry-level model, balancing compatibility, operation, and low cost of secondary development.	A larger model that generates higher-quality videos with better visual effects.
Inference Precision	FP16, FP32, does not support BF16. Can run on mainstream NVIDIA GPUs.	BF16, FP32, does not support FP16. Requires NVIDIA GPUs with Ampere architecture or higher (e.g., A100, H100).
Inference Speed (Single A100, Step = 50)	FP16: ~90 s	BF16: ~200 s
Single GPU Inference Memory Usage	18GB using SAT 12GB (with tied VAE) using diffusers 24GB (without tied VAE) using diffusers	26GB using SAT 21GB (with tied VAE) using diffusers 41GB (without tied VAE) using diffusers
Multi-GPU Inference Memory Usage	20GB minimum per GPU using diffusers
Fine-tuning Memory Usage (per GPU)	47 GB (bs=1, LORA) 61 GB (bs=2, LORA) 62GB (bs=1, SFT)	63 GB (bs=1, LORA) 80 GB (bs=2, LORA) 75GB (bs=1, SFT)
Prompt Language	English*
Maximum Prompt Length	226 Tokens
Video Length	6 seconds
Frame Rate	8 frames/second
Video Resolution	720 x 480, does not support other resolutions (including fine-tuning)

Note Using SAT for inference and fine-tuning of SAT version models. Feel free to visit our GitHub for more information.

Quick Start 🤗

This model supports deployment using the huggingface diffusers library. You can deploy it by following these steps.

We recommend that you visit our GitHub and check out the relevant prompt optimizations and conversions to get a better experience.

Install the required dependencies

pip install --upgrade opencv-python transformers diffusers # Must using diffusers>=0.30.0

Run the code

import gc
import torch
from diffusers import CogVideoXPipeline
from diffusers.utils import export_to_video

prompt = "A panda, dressed in a small, red jacket and a tiny hat, sits on a wooden stool in a serene bamboo forest. The panda's fluffy paws strum a miniature acoustic guitar, producing soft, melodic tunes. Nearby, a few other pandas gather, watching curiously and some clapping in rhythm. Sunlight filters through the tall bamboo, casting a gentle glow on the scene. The panda's face is expressive, showing concentration and joy as it plays. The background includes a small, flowing stream and vibrant green foliage, enhancing the peaceful and magical atmosphere of this unique musical performance."

pipe = CogVideoXPipeline.from_pretrained(
    "THUDM/CogVideoX-5b",
    torch_dtype=torch.bfloat16
)

pipe.enable_model_cpu_offload()

gc.collect()
torch.cuda.empty_cache()
torch.cuda.reset_accumulated_memory_stats()
torch.cuda.reset_peak_memory_stats()
pipe.vae.enable_tiling()

video = pipe(
    prompt=prompt,
    num_videos_per_prompt=1,
    num_inference_steps=50,
    num_frames=49,
    guidance_scale=6,
    generator=torch.Generator(device="cuda").manual_seed(42),
).frames[0]

export_to_video(video, "output.mp4", fps=8)

Using a single A100 GPU, generating a video with the above configuration takes approximately 200 seconds

If the generated model appears “all green” and not viewable in the default MAC player, it is a normal phenomenon (due to OpenCV saving video issues). Simply use a different player to view the video.

Explore the Model

Welcome to our github, where you will find:

More detailed technical details and code explanation.
Optimization and conversion of prompt words.
Reasoning and fine-tuning of SAT version models, and even pre-release.
Project update log dynamics, more interactive opportunities.
CogVideoX toolchain to help you better use the model.

Model License

This model is released under the CogVideoX LICENSE.

Citation

@article{yang2024cogvideox,
      title={CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer}, 
      author={Zhuoyi Yang and Jiayan Teng and Wendi Zheng and Ming Ding and Shiyu Huang and JiaZheng Xu and Yuanming Yang and Xiaohan Zhang and Xiaotao Gu and Guanyu Feng and Da Yin and Wenyi Hong and Weihan Wang and Yean Cheng and Yuxuan Zhang and Ting Liu and Bin Xu and Yuxiao Dong and Jie Tang},
      year={2024},
}