pixtral-12b video support

#30
by NmTamil2 - opened

I have a doubt regarding the mistral-community/pixtral-12b model. Does it support taking videos (sequence of frames) + text as input and generating a text response, similar to the video-llava-7b-hf model?

Pixtral-12b works well with images, but when I send 8 frames from a 40-second video to the model, I get a CUDA OOM (Out of Memory) error. Even when I try sending 5 frames, the model hits the token limit. However, when I send just 1 frame for the 40-second video, it gives a response, but the quality isn’t as good.

Can anyone clarify if this behavior is expected with Pixtral-12b, and how can I improve it?

Sign up or log in to comment