🦫 Beaver-Vision-11B

Beaver-Vision-11B is an Image-Text-to-Text chat assistant trained based on the LLaMA-3.2-11B-Vision (pretrained version) using the Align-Anything-Instruct dataset and Align-Anything framework.

Beaver-Vision-11B aims to enhance the instruction-following abilities of MLLMs (Multi-modal Large Language Models). Compared with LLaMA-3.2-11B-Vision-Instruct, Beaver-Vision-11B uses Align-Anything-Instruct dataset and post-training alignment method, achieving better performance. More importantly, Beaver-Vision-11B has open-sourced all of its training data, code, and evaluation scripts, providing greater convenience for the community and researchers.

• Developed by: the PKU-Alignment Team.
• Model Type: An auto-regressive multi-modal (Image-Text-to-Text) language model based on the transformer architecture.
• Fine-tuned from model: LLaMA-3.2-11B-Vision.

Model Sources

• Repository: https://github.com/PKU-Alignment/align-anything
• Dataset: https://huggingface.co/datasets/PKU-Alignment/Align-Anything

Evaluation Results

	Metrics	LLaMA-3.2-11B-Vision-Instruct	Beaver-Vision-11B
LLaVA-Bench-Coco	Rating	102.36	104.67
LLaVA-Bench-in-the-Wild	Rating	92.76	99.13
POPE	Accuracy	80.71	87.07
	Recall	70.95	83.02
	Precision	88.35	90.46
	F1 Score	78.65	86.55

Example

Comparison

Role	Content
User	What's the name of the restaurant serving these dishes?
LLaMA-3.2-11B-Vision-Instruct	The dish in the image is from a ramen restaurant called "Ippudo." Ippudo is known for its high-quality, authentic Japanese ramen. The bowl with the distinctive design and rich broth is characteristic of their signature Ramen.
GPT-4o	The restaurant serving these dishes is ICHIRAN.
Beaver-Vision-11B	The image shows that the restaurant is ICHIRAN. Because it appears to be a ramen dish served in a traditional Japanese-style bowl with various accompaniments such as green onions and possibly some kind of meat or broth flavoring. Additionally, the presence of the ICHIRAN logo on the dishware helps identify the restaurant.

In this example, LLaMA-3.2-11B-Vision-Instruct failed to recognize the logo text in the bowl and fell into hallucination; GPT-4o recognized accurately but did not provide more understandable information, while Beaver-Vision-11B was able to offer both accurate and detailed answers.

Further Examples

prompt What are the things I should be cautious about when I visit here?	prompt What is unusual about this image?	prompt Write an attractive product description for this.
response When visiting a place like this, which appears to be a serene lake with a dock and surrounded by forested mountains, you should be cautious of several factors: 1. Weather Conditions: Be aware of changing weather conditions such as sudden rainstorms or extreme temperatures, which can make activities dangerous. 2. Water Activities: If engaging in water activities like swimming or fishing, ensure you're safe around the water, especially if there are strong currents or deep areas. 3. Wildlife: The surrounding forest might have wildlife; stay informed about potential encounters and know how to react safely. 4. Injuries on the Dock: Walking on docks can be slippery, so watch your step to avoid falls or injuries. 5. Environmental Protection: Respect nature by not littering and staying on designated paths to protect local flora and fauna. 6. Physical Activity: Engage in physical activity responsibly, ensuring you’re prepared for outdoor exertion and have necessary supplies like food, water, and first aid. 7. Isolation: If venturing into remote areas, be mindful of isolation risks—let someone know your plans and expected return time. 8. Equipment Check: When using equipment such as kayaks or camping gear, ensure everything is in good condition before use. 9. Local Regulations: Follow any posted signs or regulations meant to preserve the environment and enhance safety for visitors. 10. Emergency Preparedness: Know basic emergency procedures and carry essential items like a map, compass, flashlight, and first-aid kit. By being cautious of these factors, you can enjoy your visit more safely.	response The unusual aspect of this image is that a person is ironing clothes on the back of a moving taxi. This is an unconventional and unsafe activity, as it involves using an electric iron in close proximity to other vehicles and potentially causing harm or damage. Additionally, the presence of the ironing board and iron on the side of a speeding vehicle defies common sense and safety protocols.	response Classic Mario Themed Mugs Step into the world of nostalgia with these vibrant Classic Mario themed mugs! Featuring the beloved character from the iconic Super Mario video game series, these mugs are a perfect blend of retro charm and modern appeal. - Vibrant Graphics: Each mug is adorned with high-quality prints of the classic Mario character, capturing his playful spirit and adventurous personality. - High-Quality Material: Made from durable ceramic, these mugs ensure longevity and resistance to wear and tear, making them ideal for both daily use and decorative purposes. - Perfect Gift: Whether you're a long-time fan of the Mario franchise or looking for a fun gift idea, these mugs are sure to delight gamers of all ages. - Versatile Use: Ideal for coffee, tea, hot chocolate, or even as a quirky storage container for pens and pencils, these mugs add a touch of gaming joy to any setting. Bring home a piece of gaming history with these Classic Mario themed mugs and let your morning routine be as exciting as rescuing a princess from a towering castle!

Usage

import requests
import torch
from PIL import Image
from transformers import MllamaForConditionalGeneration, AutoProcessor

model_id = "PKU-Alignment/Beaver-Vision-11B"

model = MllamaForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_id)

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
image = Image.open(requests.get(url, stream=True).raw)

messages = [
    {"role": "user", "content": [
        {"type": "image"},
        {"type": "text", "text": "If I had to write a haiku for this one, it would be: "}
    ]}
]
input_text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(
    image,
    input_text,
    add_special_tokens=False,
    return_tensors="pt"
).to(model.device)

output = model.generate(**inputs, max_new_tokens=30)
print(processor.decode(output[0], skip_special_tokens=True))

# The output:
# In the garden's embrace,
# Bunny in a blue coat,
# Spring's gentle whisper.

Citation

Please cite our work if you use the data or model in your paper.

@misc{align_anything,
  author = {PKU-Alignment Team},
  title = {Align Anything: training all modality models to follow instructions with unified language feedback},
  year = {2024},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/PKU-Alignment/align-anything}},
}

License

Beaver-Vision-11B is released under Apache License 2.0, and you also need to agree with LLAMA 3.2 COMMUNITY LICENSE.