README.md

---
tags:
- clip
- llm-jp-clip
- japanese-clip
library_name: open_clip
pipeline_tag: zero-shot-image-classification
license:
- apache-2.0
datasets:
- llm-jp/relaion2B-en-research-safe-japanese-translation
language:
- ja
---
# Model Card for llm-jp-clip-vit-base-patch16

# Model Details

Japanese CLIP model trained with [OpenCLIP](https://github.com/mlfoundations/open_clip) on [relaion2B-en-research-safe-japanese-translation](https://huggingface.co/datasets/llm-jp/relaion2B-en-research-safe-japanese-translation), a Japanese translation of the English subset of ReLAION-5B (https://huggingface.co/datasets/laion/relaion2B-en-research-safe), translated by [gemma-2-9b-it](https://huggingface.co/datasets/laion/relaion2B-en-research-safe).

The total number of parameters of this model is 248M.

# How to Use

## Installation

```bash
$ pip install open_clip_torch
```

## Zero-shot Image Classification
```python
import open_clip

model, preprocess = open_clip.create_model_from_pretrained('hf-hub:llm-jp/llm-jp-clip-vit-base-patch16')
tokenizer = open_clip.get_tokenizer('hf-hub:llm-jp/llm-jp-clip-vit-base-patch16')

import torch
from PIL import Image
import requests

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)
image = preprocess(image).unsqueeze(0)
text = tokenizer(["猫", "犬", "鳥"])

with torch.no_grad(), torch.cuda.amp.autocast():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)

    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", text_probs)
# Label probs: tensor([[9.9425e-01, 5.2273e-03, 5.2600e-04]])
```

Reference: 
- [Using OpenCLIP at Hugging Face](https://huggingface.co/docs/hub/en/open_clip), HuggingFace Docs
- OpenCLIP [repository](https://github.com/mlfoundations/open_clip)


# Training Details

## Model Architecture

- Text Encoder: RoBERTa base with llm-jp-tokenizer
- Image Encoder: ViT-B/16

## Training Data

This model is trained on [relaion2B-en-research-safe-japanese-translation](https://huggingface.co/datasets/llm-jp/relaion2B-en-research-safe-japanese-translation).
Due to a 70% success rate in image downloads, the dataset size was 1.45 billion samples, and we processed it over 9 epochs (13 billion samples in total).

# Evaluation

Evaluation Code: https://github.com/llm-jp/clip-eval

**Table:** Performance of each model in zero-shot image classification and image-text retrieval tasks. **Bold** indicates first place, and _underline_ indicates second place.


| Model                        | Params (M) | ImageNet | Recruit | CIFAR10 | CIFAR100 | Food101 | Caltech101 | XM3600 I → T | XM3600 T → I | Avg.  |
|-----------------------------|-------------|----------|---------|---------|----------|---------|------------|-------------|-------------|------|
| **Japanese CLIP**           |             |          |         |         |          |         |            |             |             |      |
| [Rinna ViT-B/16](https://huggingface.co/rinna/japanese-clip-vit-b-16)              | 196         | 50.6     | 39.9    | 90.7    | 64.0     | 53.2    | 84.6       | 53.8        | 54.0        | 61.4 |
| [Rinna ViT-B/16 cloob](https://huggingface.co/rinna/japanese-cloob-vit-b-16)        | 196         | 54.6     | 41.6    | 88.2    | 60.3     | 57.2    | 80.2       | 53.4        | 53.4        | 61.1 |
| [LY ViT-B/16](https://huggingface.co/line-corporation/clip-japanese-base)                 | 196         | 52.0     | **83.8** | 96.3    | 76.7     | 73.9    | **88.4**   | **76.9**    | **78.0**    | **78.3** |
| [**llm-jp-ViT-B/16**](https://huggingface.co/llm-jp/llm-jp-clip-vit-base-patch16)        | 248         | 54.2     | 59.4    | 91.8    | 69.2     | _82.2_   | 85.6       | 73.6        | 72.7        | 73.6 |
| [StabilityAI ViT-L/16](https://huggingface.co/stabilityai/japanese-stable-clip-vit-l-16)        | 414         | **62.4** | 70.5    | _97.6_   | **84.1** | 74.0    | 86.7       | 67.3        | 66.0        | 76.1 |
| [**llm-jp-ViT-L/14**](https://huggingface.co/llm-jp/llm-jp-clip-vit-large-patch14)        | 467         | _59.5_   | 62.9    | 96.4    | 77.0     | **88.2** | _87.8_      | 74.1        | _74.1_      | _77.5_ |
| **Multilingual CLIP**       |             |          |         |         |          |         |            |             |             |      |
| [SigLIP B/16-256 multi](https://huggingface.co/google/siglip-base-patch16-256-multilingual)       | 370         | 51.9     | 71.2    | 92.4    | 65.8     | 78.6    | 85.6       | 45.9        | 43.0        | 66.8 |
| [jina-clip-v2](https://huggingface.co/jinaai/jina-clip-v2)                | 865         | 35.8     | 48.1    | 95.1    | 58.3     | 52.0    | 69.4       | 67.3        | 66.4        | 61.6 |
| [LAION ViT-H/14 multi](https://huggingface.co/laion/CLIP-ViT-H-14-frozen-xlm-roberta-large-laion5B-s13B-b90k)        | 1193        | 53.0     | _74.5_   | **97.9** | _78.4_   | 74.3    | 85.1       | _75.0_      | 72.0        | 76.3 |


# LICENSE
[The Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0)


Please refer to the [Gemma Terms of Use](https://ai.google.dev/gemma/terms), as the training data was translated using gemma-2-9b-it. We utilizes Gemma solely for translation purposes. According to the definition of "Model Derivatives" in Section 1.1(e), our model does not fall under the category of a "model in order to cause that model to perform similarly to Gemma." Therefore, we have concluded that it is not necessary to inherit the Gemma license.

# Citation

Bibtex:
```
@inproceedings{sugiura2025clip,
author = {杉浦 一瑳 and 栗田 修平 and 小田 悠介 and 河原大輔 and 岡崎 直観},
month = mar,
series = {言語処理学会第31回年次大会 (NLP2025)},
title = {オープンLLMによる翻訳を活用した日本語 CLIP の開発},
year = {2025}
}

```