Update README.md

README.md

@@ -33,7 +33,7 @@ Pathumma-llm-vision-1.0.0 is designed to perform multi-modal tasks by integratin
 
 ## Training Data
 The model was fine-tuned on several datasets:
-- **Image Caption Competition (Kaggle)**: Data sourced from image captioning competitions on Kaggle.
+- **Thai Image Caption**: Data sourced from image captioning competitions on Kaggle.
 - **Thai Shorthand Dataset**: Data related to the Thai language.
 - **ShareGPT-4o (translated into Thai)**: Data translated from GPT-4o-mini outputs into Thai.
 - **Small-Thai-Wikipedia-location**: Articles in Thai from Wikipedia about geographic locations.
@@ -53,11 +53,11 @@ The model was fine-tuned on several datasets:
 
 ## Evaluation Results
 
-| Type                                  | Encoder                            | Decoder                        | Learning Rate | Sentence SacreBLEU | Unique Tokens |
-|---------------------------------------|------------------------------------|--------------------------------|---------------|--------------------|---------------|
-| Idefic3-8B-Llama3                     | siglip-so400m-patch14-384          | Meta-Llama-3.1-8B-Instruct     | -             | 0.02657            | 12990         |
-| Pathumma-llm-vision-beta-0.0.0        | siglip-so400m-patch14-384          | Meta-Llama-3.1-8B-Instruct     | 1e-4          | 13.45412           | 1148          |
-| Pathumma-llm-vision-1.0.0             | siglip-so400m-patch14-384          | Meta-Llama-3.1-8B-Instruct     | 1e-4          | 17.66370           | 1312          |
+| Type                                  | Encoder                            | Decoder                        | Sentence SacreBLEU <br>(test) | Unique Tokens |
+|---------------------------------------|------------------------------------|--------------------------------|-------------------------------|---------------|
+| Idefic3-8B-Llama3                     | siglip-so400m-patch14-384          | Meta-Llama-3.1-8B-Instruct     | 0.02657                       | 12990         |
+| Pathumma-llm-vision-beta-0.0.0        | siglip-so400m-patch14-384          | Meta-Llama-3.1-8B-Instruct     | 13.45412                      | 1148          |
+| Pathumma-llm-vision-1.0.0             | siglip-so400m-patch14-384          | Meta-Llama-3.1-8B-Instruct     | 17.66370                      | 1312          |
 
 
 - **Accuracy on Manual-VQA Tasks**: 30.34%
@@ -71,10 +71,34 @@ pip install git+https://github.com/andimarafioti/transformers.git@idefics3
 ```
 
 ## Usage
+We provide a [inference tutorial](https://colab.research.google.com/drive/1TakNg4v6hHFXLih-SFcibxzYBTs2-EFn?usp=sharing).
 To use the model with the Hugging Face `transformers` library:
 
 ```python
-from transformers import AutoProcessor, Idefics3ForConditionalGeneration
+import io
+import os
+import time
+import random
+import requests
+import shutil
+from IPython.display import display, Markdown
+from IPython.display import clear_output as cls
+
+import numpy as np
+import pandas as pd
+from PIL import Image
+
+import torch
+
+import transformers
+from transformers import (
+    Idefics3ForConditionalGeneration,
+    AutoProcessor,
+    BitsAndBytesConfig,
+)
+```
+
+```python
 
 DEVICE = f"cuda" if torch.cuda.is_available() else 'cpu' if torch.cpu.is_available() else 'mps'
 print(DEVICE)
@@ -82,8 +106,10 @@ if DEVICE == 'cuda': display(torch.cuda.device_count())
 
 N = 5
 
+revision = "quantized8bit"
 processor = AutoProcessor.from_pretrained(
     "nectec/Pathumma-llm-vision-1.0.0",
+    revision=revision,                         # Optional
     do_image_splitting=False,
     # size={"longest_edge": N*364},            # Optional
     # size={"height": N*364, "width": N*364},  # Optional
@@ -91,6 +117,7 @@ processor = AutoProcessor.from_pretrained(
 
 model = Idefics3ForConditionalGeneration.from_pretrained(
         "nectec/Pathumma-llm-vision-1.0.0",
+        revision=revision,                     # Optional
         torch_dtype=torch.float16,
         device_map=DEVICE
     )
@@ -152,7 +179,11 @@ answer_prompt = generated_text.split('Assistant:')[1].strip()
 
 # Output processing (depends on task requirements)
 print(answer_prompt)
-print(latency_time)
+print(f"latency_time: {latency_time:.3f} sec.")
+
+# >>> output:
+# >>> ลูกฮิปโปแคระกำลังยืนอยู่ข้างแม่ฮิปโปแคระที่กำลังอาบน้ำ
+# >>> latency_time: 7.642 sec.
 ```
 
 ## Limitations and Biases
@@ -168,13 +199,24 @@ If you use this model, please cite it as follows:
 
 ```bibtex
 @misc{PathummaVision,
-  author = {NECTEC Team},
+  author = {Thirawarit Pitiphiphat and NECTEC Team},
   title = {nectec/Pathumma-llm-vision-1.0.0},
   year = {2024},
   url = {https://huggingface.co/nectec/Pathumma-llm-vision-1.0.0}
 }
 ```
 
+```bibtex
+@misc{laurençon2024building,
+      title={Building and better understanding vision-language models: insights and future directions.}, 
+      author={Hugo Laurençon and Andrés Marafioti and Victor Sanh and Léo Tronchon},
+      year={2024},
+      eprint={2408.12637},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+
 ## Contact
 For questions or support, please contact **https://discord.gg/3WJwJjZt7r**.