Update README.md

README.md

@@ -2,57 +2,106 @@
 license: cc-by-nc-4.0
 base_model: adept/fuyu-8b
 tags:
-- generated_from_trainer
+- multimodal
+- fuyu
+- mfuyu
+- mantis
+- lmm
+- vlm
 model-index:
-- name: mfuyu_1.5_8b_8192_720p
+- name: Mantis-8B-Fuyu
   results: []
+datasets:
+- TIGER-Lab/Mantis-Instruct
+language:
+- en
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
 
-# mfuyu_1.5_8b_8192_720p
+# 🔥 Mantis
 
-This model is a fine-tuned version of [adept/fuyu-8b](https://huggingface.co/adept/fuyu-8b) on an unknown dataset.
+[Paper](https://arxiv.org/abs/2405.01483) | [Website](https://tiger-ai-lab.github.io/Mantis/) | [Github](https://github.com/TIGER-AI-Lab/Mantis) | [Models](https://huggingface.co/collections/TIGER-Lab/mantis-6619b0834594c878cdb1d6e4) | [Demo](https://huggingface.co/spaces/TIGER-Lab/Mantis)
 
-## Model description
+![Mantis](https://tiger-ai-lab.github.io/Mantis/images/radar_chart.png)
 
-More information needed
 
-## Intended uses & limitations
+## Summary
 
-More information needed
+- Mantis is an LLaMA-3 based LMM with **interleaved text and image as inputs**, train on Mantis-Instruct under academic-level resources (i.e. 36 hours on 16xA100-40G).
+- Mantis is trained to have multi-image skills including co-reference, reasoning, comparing, temporal understanding.
+- Mantis reaches the state-of-the-art performance on five multi-image benchmarks (NLVR2, Q-Bench, BLINK, MVBench, Mantis-Eval), and also maintain a strong single-image performance on par with CogVLM and Emu2.
 
-## Training and evaluation data
+## Multi-Image Performance
 
-More information needed
+![SoTA performance on 5 multi-image benchmarks](https://tiger-ai-lab.github.io/Mantis/images/many_image_vqa.png)
 
-## Training procedure
+## Single-Image Performance
 
-### Training hyperparameters
+![Strong performance on single image results](https://tiger-ai-lab.github.io/Mantis/images/single_image_vqa.png)
 
-The following hyperparameters were used during training:
-- learning_rate: 1e-05
-- train_batch_size: 1
-- eval_batch_size: 1
-- seed: 42
-- distributed_type: multi-GPU
-- num_devices: 16
-- gradient_accumulation_steps: 8
-- total_train_batch_size: 128
-- total_eval_batch_size: 16
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: cosine
-- lr_scheduler_warmup_ratio: 0.03
-- num_epochs: 1.0
+## How to use
 
-### Training results
+### Installation
+```bash
+pip install git+https://github.com/TIGER-AI-Lab/Mantis.git
+```
 
+### Run example inference:
+```python
+from mantis.models.mllava import chat_mllava
+from PIL import Image
+import torch
 
 
-### Framework versions
+image1 = "image1.jpg"
+image2 = "image2.jpg"
+images = [Image.open(image1), Image.open(image2)]
 
-- Transformers 4.39.2
-- Pytorch 2.2.1
-- Datasets 2.18.0
-- Tokenizers 0.15.2
+# load processor and model
+from mantis.models.mllava import MLlavaProcessor, LlavaForConditionalGeneration
+processor = MLlavaProcessor.from_pretrained("TIGER-Lab/Mantis-8B-siglip-llama3")
+attn_implementation = None # or "flash_attention_2"
+model = LlavaForConditionalGeneration.from_pretrained("TIGER-Lab/Mantis-8B-siglip-llama3", device_map="cuda", torch_dtype=torch.bfloat16, attn_implementation=attn_implementation)
+
+generation_kwargs = {
+    "max_new_tokens": 1024,
+    "num_beams": 1,
+    "do_sample": False
+}
+
+# chat
+text = "Describe the difference of <image> and <image> as much as you can."
+response, history = chat_mllava(text, images, model, processor, **generation_kwargs)
+
+print("USER: ", text)
+print("ASSISTANT: ", response)
+
+text = "How many wallets are there in image 1 and image 2 respectively?"
+response, history = chat_mllava(text, images, model, processor, history=history, **generation_kwargs)
+
+print("USER: ", text)
+print("ASSISTANT: ", response)
+
+"""
+USER:  Describe the difference of <image> and <image> as much as you can.
+ASSISTANT:  The second image has more variety in terms of colors and designs. While the first image only shows two brown leather pouches, the second image features four different pouches in various colors and designs, including a purple one with a gold coin, a red one with a gold coin, a black one with a gold coin, and a brown one with a gold coin. This variety makes the second image more visually interesting and dynamic.
+USER:  How many wallets are there in image 1 and image 2 respectively?
+ASSISTANT:  There are two wallets in image 1, and four wallets in image 2.
+"""
+```
+
+### Training
+See [mantis/train](https://github.com/TIGER-AI-Lab/Mantis/tree/main/mantis/train) for details
+
+### Evaluation
+See [mantis/benchmark](https://github.com/TIGER-AI-Lab/Mantis/tree/main/mantis/benchmark) for details
+
+## Citation
+```
+@inproceedings{Jiang2024MANTISIM,
+  title={MANTIS: Interleaved Multi-Image Instruction Tuning},
+  author={Dongfu Jiang and Xuan He and Huaye Zeng and Cong Wei and Max W.F. Ku and Qian Liu and Wenhu Chen},
+  publisher={arXiv2405.01483}
+  year={2024},
+}
+```