Upload 2 files

README.md

@@ -1,3 +1,104 @@
 ---
 license: apache-2.0
+tags:
+- image-captioning
+languages:
+- en
+datasets:
+- michelecafagna26/hl-narratives
+language:
+- en
+metrics:
+- sacrebleu
+- rouge
+library_name: transformers
 ---
+## ClipCap fine-tuned for Narrative Image Captioning
+
+[ClipCap](https://arxiv.org/abs/2111.09734) base trained on the [HL Dataset](https://huggingface.co/datasets/michelecafagna26/hl) for **high-level action descriptions generation**
+
+## Model fine-tuning 🏋️‍
+
+We fine-tune LM + Mapping Network starting grom the model pretrained on COCO 
+
+- Trained for a 10 epochs
+- lr:  5e−5
+- Adam optimizer
+- half-precision (fp16)
+
+## Test set metrics 🧾
+
+    | Cider   | SacreBLEU  | Rouge-L|
+    |---------|------------|--------|
+    | 176.54  |   27.37    |  39.15 |
+
+## Demo
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1xcaJOxaAp8TRd8a6x1XnAptVjHQRv3Zj?usp=sharing)
+
+## Installation
+
+```bash
+pip install git+https://github.com/michelecafagna26/CLIPCap.git
+```
+
+## Download the model
+
+```bash
+git lfs install # if not installed
+git clone https://huggingface.co/michelecafagna26/clipcap-base-captioning-ft-hl-actions
+```
+
+## Model in Action 🚀
+
+
+```python
+from clipcap import ClipCaptionModel
+import torch
+from transformers import (
+    GPT2Tokenizer,
+    GPT2LMHeadModel,
+)
+import torch
+import clip
+import requests
+from PIL import Image
+
+model_path = "clipcap-base-captioning-ft-hl-actions/pytorch_model.pt" # change accordingly
+
+# load clip
+device = "cuda" if torch.cuda.is_available() else "cpu"
+clip_model, preprocess = clip.load("ViT-B/32", device=device, jit=False)
+tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
+prefix_length = 10
+
+# load ClipCap
+model = ClipCaptionModel(prefix_length, tokenizer=tokenizer)
+model.from_pretrained(model_path)
+model = model.eval()
+model = model.to(device)
+
+# load the image
+img_url = 'https://datasets-server.huggingface.co/assets/michelecafagna26/hl/--/default/train/0/image/image.jpg'
+raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
+
+
+# extract the prefix
+image = preprocess(raw_image).unsqueeze(0).to(device)
+with torch.no_grad():
+    prefix = clip_model.encode_image(image).to(
+        device, dtype=torch.float32
+    )
+    prefix_embed = model.clip_project(prefix).reshape(1, prefix_length, -1)
+
+# generate the caption   
+model.generate_beam(embed=prefix_embed)[0]
+
+
+# >> ""
+```
+
+## BibTex and citation info
+
+```BibTeX
+```

pytorch_model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:da58358afcd078516b09e046c18e3ebe925d9ef5b6bfd2c7272444739614d3d9
+size 636274141