Update README.md

README.md

@@ -1,3 +1,337 @@
----
-license: mit
----
+---
+license: mit
+datasets:
+- laion/laion2B-en
+- laion/laion-coco
+- laion/laion2B-multi
+- kakaobrain/coyo-700m
+- conceptual_captions
+- wanng/wukong100m
+pipeline_tag: image-feature-extraction
+---
+
+# InternViT-6B-448px-V2_5
+
+[\[📂 GitHub\]](https://github.com/OpenGVLab/InternVL)  [\[🆕 Blog\]](https://internvl.github.io/blog/)  [\[📜 InternVL 1.0 Paper\]](https://arxiv.org/abs/2312.14238)  [\[📜 InternVL 1.5 Report\]](https://arxiv.org/abs/2404.16821) [\[📜 Mini-InternVL Report\]](https://arxiv.org/abs/2410.16261)
+
+[\[🗨️ Chat Demo\]](https://internvl.opengvlab.com/)  [\[🤗 HF Demo\]](https://huggingface.co/spaces/OpenGVLab/InternVL)  [\[🚀 Quick Start\]](#quick-start)  [\[📖 Documents\]](https://internvl.readthedocs.io/en/latest/)
+
+InternViT-6B has been updated
+We develop InternViT-6B-448px-V2_5 based on the pre-training of the strong foundation of InternViT-6B-448px-V1-5. Through ViT Incremental Learning with NTP loss, the vision encoder has gained a stronger ability to extract visual features, allowing it to capture more comprehensive information—especially for domains that are relatively underrepresented in web-scale datasets (e.g., LAION-5B), such as multilingual OCR data and mathematical charts, among others.
+
+<div align="center">
+
+|      Model Name         |                  HF Link                   |
+| :---------------------: | :----------------------------------------: |
+|InternViT-300M-448px-V2_5|[🤗 link](https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5)  |
+|InternViT-6B-448px-V2_5|[🤗 link](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V2_5)|
+
+</div>
+
+## Model Details
+
+- **Model Type:** vision foundation model, feature backbone
+- **Model Stats:**
+  - Params (M): 5540 (the last 3 blocks are discarded)
+  - Image size: 448 x 448, training with 1 - 12 tiles
+- **Pretrain Dataset:** LAION-en, LAION-zh, COYO, GRIT, COCO, TextCaps, Objects365, OpenImages, All-Seeing, Wukong-OCR, LaionCOCO-OCR, and other OCR-related datasets. 
+To enhance the OCR capability of the model, we have incorporated additional OCR data alongside the general caption datasets. Specifically, we utilized PaddleOCR to perform Chinese OCR on images from Wukong and English OCR on images from LAION-COCO.
+- **Note:** InternViT-6B originally had 48 blocks, and we found that using the output after the fourth-to-last block worked best for MLLM. For ease of use and to save GPU memory, we simply discarded the last 3 blocks. Now, the model has only 45 blocks and the number of parameters has been reduced from 5.9B to 5.5B. Therefore, if you want to build a MLLM based on this model, **please make use of the features from the last layer.**
+
+## Performance
+
+
+### Image Classification
+
+<table style="width: 100%; text-align: center; border-collapse: collapse;">
+  <!-- <caption><strong>Image classification performance across different versions of InternViT.</strong><br>
+    We use IN-1K for training and evaluate on the IN-1K validation set as well as multiple ImageNet variants, including IN-ReaL, IN-V2, IN-A, IN-R, and IN-Sketch. Results are reported for both linear probing and attention pooling probing methods, with average accuracy for each method. $\Delta$ represents the performance gap between attention pooling probing and linear probing, where a larger $\Delta$ suggests a shift from learning simple linear features to capturing more complex, nonlinear semantic representations.
+  </caption> -->
+  <thead>
+    <tr>
+      <th rowspan="2">Model Name</th>
+      <th rowspan="2">res.</th>
+      <th colspan="7">Linear Probing</th>
+      <th colspan="7">Attention Pooling Probing</th>
+      <th rowspan="2">Δ</th>
+    </tr>
+    <tr>
+      <th>IN-1K</th>
+      <th>IN-ReaL</th>
+      <th>IN-V2</th>
+      <th>IN-A</th>
+      <th>IN-R</th>
+      <th>IN-Ske</th>
+      <th>avg.</th>
+      <th>IN-1K</th>
+      <th>IN-ReaL</th>
+      <th>IN-V2</th>
+      <th>IN-A</th>
+      <th>IN-R</th>
+      <th>IN-Ske</th>
+      <th>avg.</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr style="color: gray;">
+      <td>InternViT-6B-224px</td>
+      <td>224</td>
+      <td>88.2</td>
+      <td>90.4</td>
+      <td>79.9</td>
+      <td>77.5</td>
+      <td>89.8</td>
+      <td>69.1</td>
+      <td style="background-color: #d3d3d3;">82.5</td>
+      <td>89.2</td>
+      <td>91.1</td>
+      <td>82.3</td>
+      <td>84.7</td>
+      <td>93.1</td>
+      <td>72.7</td>
+      <td style="background-color: #d3d3d3;">85.5</td>
+      <td>3.0</td>
+    </tr>
+    <tr>
+      <td>InternViT-6B-224px</td>
+      <td>448</td>
+      <td>87.8</td>
+      <td>90.2</td>
+      <td>79.8</td>
+      <td>77.2</td>
+      <td>87.1</td>
+      <td>65.8</td>
+      <td style="background-color: #d3d3d3;">81.3</td>
+      <td>88.8</td>
+      <td>91.0</td>
+      <td>82.0</td>
+      <td>85.4</td>
+      <td>91.3</td>
+      <td>70.5</td>
+      <td style="background-color: #d3d3d3;">84.8</td>
+      <td>3.5</td>
+    </tr>
+    <tr>
+      <td>InternViT-6B-448px-V1.0</td>
+      <td>448</td>
+      <td>87.0</td>
+      <td>90.0</td>
+      <td>78.8</td>
+      <td>77.2</td>
+      <td>85.5</td>
+      <td>65.1</td>
+      <td style="background-color: #ffcccc;">80.6</td>
+      <td>88.7</td>
+      <td>91.0</td>
+      <td>82.0</td>
+      <td>88.7</td>
+      <td>92.8</td>
+      <td>72.0</td>
+      <td style="background-color: #90ee90;">85.9</td>
+      <td>5.3</td>
+    </tr>
+    <tr>
+      <td>InternViT-6B-448px-V1.2</td>
+      <td>448</td>
+      <td>87.0</td>
+      <td>89.9</td>
+      <td>78.5</td>
+      <td>77.1</td>
+      <td>83.9</td>
+      <td>59.7</td>
+      <td style="background-color: #ffcccc;">79.4</td>
+      <td>88.6</td>
+      <td>91.1</td>
+      <td>82.0</td>
+      <td>88.7</td>
+      <td>92.7</td>
+      <td>71.6</td>
+      <td style="background-color: #90ee90;">85.8</td>
+      <td>6.4</td>
+    </tr>
+    <tr>
+      <td>InternViT-6B-448px-V1.5</td>
+      <td>448</td>
+      <td>86.5</td>
+      <td>89.9</td>
+      <td>78.1</td>
+      <td>69.8</td>
+      <td>82.9</td>
+      <td>60.1</td>
+      <td style="background-color: #ffcccc;">77.9</td>
+      <td>88.4</td>
+      <td>91.2</td>
+      <td>81.6</td>
+      <td>86.0</td>
+      <td>92.2</td>
+      <td>70.9</td>
+      <td style="background-color: #90ee90;">85.1</td>
+      <td>7.2</td>
+    </tr>
+    <tr>
+      <td>InternViT-6B-448px-V2.5</td>
+      <td>448</td>
+      <td>86.6</td>
+      <td>90.1</td>
+      <td>77.8</td>
+      <td>73.7</td>
+      <td>82.7</td>
+      <td>60.0</td>
+      <td style="background-color: #ffcccc;">78.5</td>
+      <td>88.3</td>
+      <td>91.2</td>
+      <td>81.3</td>
+      <td>86.9</td>
+      <td>92.4</td>
+      <td>70.8</td>
+      <td style="background-color: #90ee90;">85.2</td>
+      <td>6.7</td>
+    </tr>
+  </tbody>
+</table>
+
+**Note:** ∆ represents the performance gap between attention pooling probing and linear probing, where a larger ∆ suggests a shift from learning simple linear features to capture more complex, nonlinear semantic representations
+
+### Semantic Segmentation
+
+<table style="width:100%;  border-collapse: collapse; text-align: center;">
+  <thead>
+    <tr>
+      <th rowspan="2">Model Name</th>
+      <th colspan="3" >Linear Probing</th>
+      <th colspan="3" >Head Tuning (UperNet)</th>
+      <th colspan="3" >Full Tuning (UperNet)</th>
+      <th rowspan="2">Δ<sub>1</sub></th>
+      <th rowspan="2">Δ<sub>2</sub></th>
+    </tr>
+    <tr>
+      <th>ADE20K</th>
+      <th>COCO</th>
+      <th>avg.</th>
+      <th>ADE20K</th>
+      <th>COCO</th>
+      <th>avg.</th>
+      <th>ADE20K</th>
+      <th>COCO</th>
+      <th>avg.</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>InternViT-6B-224px</td>
+      <td>47.2</td>
+      <td>42.8</td>
+      <td >45.0</td>
+      <td>54.9</td>
+      <td>48.9</td>
+      <td >51.9</td>
+      <td>58.9</td>
+      <td>51.6</td>
+      <td>55.3</td>
+      <td>6.9</td>
+      <td>10.2</td>
+    </tr>
+    <tr>
+      <td>InternViT-6B-448px-V1.0</td>
+      <td>43.6</td>
+      <td>38.5</td>
+      <td >41.0</td>
+      <td>55.4</td>
+      <td>49.4</td>
+      <td>52.4</td>
+      <td>58.1</td>
+      <td>51.7</td>
+      <td>54.9</td>
+      <td>11.3</td>
+      <td>13.9</td>
+    </tr>
+    <tr>
+      <td>InternViT-6B-448px-V1.2</td>
+      <td>40.7</td>
+      <td>36.1</td>
+      <td >38.4</td>
+      <td>55.2</td>
+      <td>48.8</td>
+      <td>52.0</td>
+      <td>58.8</td>
+      <td>51.7</td>
+      <td>55.2</td>
+      <td>13.6</td>
+      <td>16.8</td>
+    </tr>
+    <tr>
+      <td>InternViT-6B-448px-V1.5</td>
+      <td>40.9</td>
+      <td>36.3</td>
+      <td >38.6</td>
+      <td>55.0</td>
+      <td>49.1</td>
+      <td>52.0</td>
+      <td>58.8</td>
+      <td>51.5</td>
+      <td>55.2</td>
+      <td>13.4</td>
+      <td>16.6</td>
+    </tr>
+    <tr>
+      <td>InternViT-6B-448px-V2.5</td>
+      <td>39.4</td>
+      <td>35.6</td>
+      <td >37.5</td>
+      <td>55.4</td>
+      <td>49.7</td>
+      <td>52.6</td>
+      <td>58.6</td>
+      <td>51.8</td>
+      <td>55.2</td>
+      <td>15.1</td>
+      <td>17.7</td>
+    </tr>
+  </tbody>
+</table>
+
+**Note:** Δ<sub>1</sub> represents the gap between head tuning and linear probing, while Δ<sub>2</sub> shows the gap between full tuning and linear probing. A larger Δ value indicates a shift from simple linear features to more complex, nonlinear representations.
+
+## Model Usage (Image Embeddings)
+
+```python
+import torch
+from PIL import Image
+from transformers import AutoModel, CLIPImageProcessor
+
+model = AutoModel.from_pretrained(
+    'OpenGVLab/InternViT-6B-448px-V2_5',
+    torch_dtype=torch.bfloat16,
+    low_cpu_mem_usage=True,
+    trust_remote_code=True).cuda().eval()
+
+image = Image.open('./examples/image1.jpg').convert('RGB')
+
+image_processor = CLIPImageProcessor.from_pretrained('OpenGVLab/InternViT-6B-448px-V2_5')
+
+pixel_values = image_processor(images=image, return_tensors='pt').pixel_values
+pixel_values = pixel_values.to(torch.bfloat16).cuda()
+
+outputs = model(pixel_values)
+```
+
+## Citation
+
+If you find this project useful in your research, please consider citing:
+
+```BibTeX
+
+@article{chen2023internvl,
+  title={InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks},
+  author={Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and Zhong, Muyan and Zhang, Qinglong and Zhu, Xizhou and Lu, Lewei and Li, Bin and Luo, Ping and Lu, Tong and Qiao, Yu and Dai, Jifeng},
+  journal={arXiv preprint arXiv:2312.14238},
+  year={2023}
+}
+@article{chen2024far,
+  title={How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites},
+  author={Chen, Zhe and Wang, Weiyun and Tian, Hao and Ye, Shenglong and Gao, Zhangwei and Cui, Erfei and Tong, Wenwen and Hu, Kongzhi and Luo, Jiapeng and Ma, Zheng and others},
+  journal={arXiv preprint arXiv:2404.16821},
+  year={2024}
+}
+
+```