InstructPix2Pix for AnimateDiff

This model is a fine-tuned version of the AnimateDiff V2 model using the InstructPix2Pix as the T2I model.

Inference

You need use a pipeline built for InstructPix2Pix and AnimateDiff inference. And example is here

Using the pipeline, InstructPix2Pix T2I model and this motion module, you can generate animations that use an init image faithfully.

Init Image

Make her smile

Make her on fire

Training

The training procedure was modified to pass the first frame of all videos to the extra channels of the InstructPix2Pix model. The model was trained 4500 steps with a learning rate of 5e-7 and 5500 steps at 1e-7, with a batch size of 1.

LICENSE

Creative OpenRAIL License