|
---
|
|
license: mit
|
|
---
|
|
|
|
## Model Summary
|
|
|
|
Video-CCAM-14B-v1.1 is a lightweight Video-MLLM developed by TencentQQ Multimedia Research Team.
|
|
|
|
## Usage
|
|
|
|
Inference using Huggingface transformers on NVIDIA GPUs. Requirements tested on python 3.9/3.10.
|
|
```
|
|
pip install -U pip torch transformers peft decord pysubs2 imageio
|
|
```
|
|
|
|
## Inference
|
|
|
|
```
|
|
import os
|
|
import torch
|
|
from PIL import Image
|
|
from transformers import AutoModel
|
|
|
|
from eval import load_decord
|
|
|
|
os.environ['TOKENIZERS_PARALLELISM'] = 'false'
|
|
|
|
videoccam = AutoModel.from_pretrained(
|
|
'<your_local_path_1>',
|
|
trust_remote_code=True,
|
|
torch_dtype=torch.bfloat16,
|
|
device_map='auto',
|
|
_attn_implementation='flash_attention_2',
|
|
# llm_name_or_path='<your_local_llm_path>',
|
|
# vision_encoder_name_or_path='<your_local_vision_encoder_path>'
|
|
)
|
|
|
|
|
|
messages = [
|
|
[
|
|
{
|
|
'role': 'user',
|
|
'content': '<image>\nDescribe this image in detail.'
|
|
}
|
|
], [
|
|
{
|
|
'role': 'user',
|
|
'content': '<video>\nDescribe this video in detail.'
|
|
}
|
|
]
|
|
]
|
|
|
|
images = [
|
|
Image.open('assets/example_image.jpg').convert('RGB'),
|
|
load_decord('assets/example_video.mp4', sample_type='uniform', num_frames=32)
|
|
]
|
|
|
|
response = videoccam.chat(messages, images, max_new_tokens=512, do_sample=False)
|
|
|
|
print(response)
|
|
```
|
|
|
|
Please refer to [Video-CCAM](https://github.com/QQ-MM/Video-CCAM) for more details.
|
|
|
|
### Benchmarks
|
|
|
|
|Benchmark|Video-CCAM-14B|Video-CCAM-14B-v1.1|
|
|
|:-:|:-:|:-:|
|
|
|MVBench (32 frames)|61.88|63.08|
|
|
|MSVD-QA (32 frames)|76.3/4.1|78.6/4.2|
|
|
|MSRVTT-QA (32 frames)|59.0/3.5|66.3/3.8|
|
|
|ActivityNet-QA (32 frames)|58.3/3.7|60.4/3.8|
|
|
|TGIF-QA (32 frames)|84.1/4.5|84.4/4.5|
|
|
|Video-MME (w/o sub, 96 frames)|53.2|53.9|
|
|
|Video-MME (w sub, 96 frames)|57.2|56.1|
|
|
|MLVU (M-Avg, 96 frames)|60.2|63.1|
|
|
|MLVU (G-Avg, 96 frames)|4.11|4.01|
|
|
|VideoVista (96 frames)|68.43|76.55|
|
|
|
|
* The accuracies and scores of MSVD-QA,MSRVTT-QA,ActivityNet-QA,TGIF-QA are evaluated by `gpt-3.5-turbo-0125`.
|
|
|
|
## Acknowledgement
|
|
|
|
* [xtuner](https://github.com/InternLM/xtuner): Video-CCAM-14B is trained using the xtuner framework. Thanks for their excellent works!
|
|
* [Phi-3-medium-4k-instruct](https://huggingface.co/microsoft/Phi-3-medium-4k-instruct): Powerful language models developed by Microsoft.
|
|
* [SigLIP SO400M](https://huggingface.co/google/siglip-so400m-patch14-384): Outstanding vision encoder developed by Google.
|
|
|
|
## License
|
|
The model is licensed under the MIT license.
|
|
|
