README.md

---
inference: false
pipeline_tag: image-text-to-text
license: apache-2.0
datasets:
- VIMA/VIMA-Data
tags:
- llara
- llava
- robotics
- vlm
---
<br>
<be>

# LLaRA Model Card

This model is released with paper **[LLaRA: Supercharging Robot Learning Data for Vision-Language Policy](https://arxiv.org/abs/2406.20095)**

[Xiang Li](https://xxli.me)<sup>1</sup>, [Cristina Mata](https://openreview.net/profile?id=~Cristina_Mata1)<sup>1</sup>, [Jongwoo Park](https://github.com/jongwoopark7978)<sup>1</sup>, [Kumara Kahatapitiya](https://www3.cs.stonybrook.edu/~kkahatapitiy)<sup>1</sup>, [Yoo Sung Jang](https://yjang43.github.io/)<sup>1</sup>, [Jinghuan Shang](https://elicassion.github.io/)<sup>1</sup>, [Kanchana Ranasinghe](https://kahnchana.github.io/)<sup>1</sup>, [Ryan Burgert](https://ryanndagreat.github.io/)<sup>1</sup>, [Mu Cai](https://pages.cs.wisc.edu/~mucai/)<sup>2</sup>, [Yong Jae Lee](https://pages.cs.wisc.edu/~yongjaelee/)<sup>2</sup>, and [Michael S. Ryoo](http://michaelryoo.com/)<sup>1</sup>

<sup>1</sup>Stony Brook University  <sup>2</sup>University of Wisconsin-Madison 

## Model details

**Model type:**
D-RT2-Style is one of the baselines in our LLaRA paper, following the style of [RT2](https://robotics-transformer.github.io/).
This is an open-source visuomotor policy trained by fine-tuning [LLaVA-7b-v1.5](https://huggingface.co/liuhaotian/llava-v1.5-7b) on instruction-following data `D-RT2-Style`, converted from [VIMA-Data](https://huggingface.co/datasets/VIMA/VIMA-Data).
For the conversion code, please refer to [convert_vima.ipynb](https://github.com/LostXine/LLaRA/blob/main/datasets/convert_vima.ipynb)

**Model date:**
llava-1.5-7b-llara-D-RT2-Style-VIMA-80k was trained in June 2024.

**Paper or resources for more information:**
https://github.com/LostXine/LLaRA

**Where to send questions or comments about the model:**
https://github.com/LostXine/LLaRA/issues

## Intended use
**Primary intended uses:**
The primary use of LLaRA is research on large multimodal models for robotics.

**Primary intended users:**
The primary intended users of the model are researchers and hobbyists in robotics, computer vision, natural language processing, machine learning, and artificial intelligence.