README.md

---
language:
- en
pipeline_tag: text2text-generation
inference: false
license: mit
---
# ViPE-S-CTX7

<!-- Provide a quick summary of what the model is/does. -->

ViPE: Visualize Pretty-much Everything, is the first automated model for translating any arbitrary piece of text into a visualizable prompt.
It helps any text-to-image model in figurative or non-lexical language visualizations. It has been shown to be more robust than GPT3.5 Turbo (ChatGPT)
in generating depictable and semantically meaningful prompts.

### Model Description

<!-- Provide a longer summary of what this model is. -->

- **Developed by:** [Computer Graphics Group, University of Tuebingen](https://uni-tuebingen.de/fakultaeten/mathematisch-naturwissenschaftliche-fakultaet/fachbereiche/informatik/lehrstuehle/computergrafik/lehrstuhl/)
- **Model type:** Auto-Regressive
- **Language:** English
- **License:** MIT
- **Based on:** [GPT2-Small](https://huggingface.co/gpt2)
- **Versions:** [ViPE-M-CTX7](https://huggingface.co/fittar/ViPE-M-CTX7) (355M parameters) and [ViPE-S-CTX7](https://huggingface.co/fittar/ViPE-S-CTX7) (117M parameters)


### Model Sources

<!-- Provide the basic links for the model. -->

- **Repository:** [Github](https://github.com/Hazel1994/ViPE)
- **Paper:** [ViPE: Visualise Pretty-much Everything](https://aclanthology.org/2023.emnlp-main.333/) (**Outstanding Paper Award at EMNLP 2023**)

### Down Stream Applications
ViPE provides a robust backbone for many practical applications such as music video generation and creative writing.
- #### Music Video Genrations
  - **Repository:** [Github](https://github.com/Hazel1994/ViPE-Videos)
  - **Example Videos:** [ViPE Videos](https://www.youtube.com/playlist?list=PLvLHdI48ZdfaDMxPZIGHXrvsRkdADcMUh)
- #### Creative Writing
  - **Demo:** [Hugging Face Playground](https://huggingface.co/spaces/fittar/ViPE)
- #### Summagery: Document Summarization through Images
  - **Demo:** [Hugging Face Playground](https://huggingface.co/spaces/fittar/summagary)
  - **Repository:** [Github](https://github.com/Hazel1994/summagary)
  - **Examples:** [Summagery Videos](https://www.youtube.com/watch?v=mFMkE2waYGQ)
    
### Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

You can directly use the model to generate detailed prompts for any arbitrary text.


```python
from transformers import GPT2LMHeadModel, GPT2Tokenizer


def generate(text, model, tokenizer,device,do_sample,top_k=100, epsilon_cutoff=.00005, temperature=1):
    #mark the text with special tokens
    text=[tokenizer.eos_token +  i + tokenizer.eos_token for i in text]
    batch=tokenizer(text, padding=True, return_tensors="pt")

    input_ids = batch["input_ids"].to(device)
    attention_mask = batch["attention_mask"].to(device)

    #how many new tokens to generate at max
    max_prompt_length=50

    generated_ids = model.generate(input_ids=input_ids,attention_mask=attention_mask, max_new_tokens=max_prompt_length, do_sample=do_sample,top_k=top_k, epsilon_cutoff=epsilon_cutoff, temperature=temperature)
    #return only the generated prompts
    pred_caps = tokenizer.batch_decode(generated_ids[:, -(generated_ids.shape[1] - input_ids.shape[1]):], skip_special_tokens=True)

    return pred_caps

device='cpu'
model = GPT2LMHeadModel.from_pretrained('fittar/ViPE-S-CTX7')
model.to(device)

#ViPE-M's tokenizer is identical to that of GPT2-Small
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokenizer.pad_token = tokenizer.eos_token

# A list of abstract/figurative or any arbitrary combinations of keywords
texts=['lalala', 'I wanna start learning', 'free your mind; you will see the other side of life', 'brave; fantasy']

prompts=generate(texts,model,tokenizer,do_sample=True,device=device)
for t,p in zip(texts,prompts):
    print('{} --> {}'.format(t,p))

lalala -->  A group of dancers performing an extravagant traditional dance, lalala in Spanish
I wanna start learning -->  A student intently sitting at a desk, surrounded by books and notes
free your mind; you will see the other side of life -->  A view of the night sky, stars and planets shining bright, while a woman in a field of flowers looks up in awe
brave; fantasy -->  A knight in shining armor riding a gallant horse through a sunlit valley

```


### Recommendations

You can use either a comma or a semicolon to combine multiple keywords. for example ['dark, fantasy, brave'] or  ['This is gonna be the best day of my life; do you agree?'].
However, a semicolon draws a stronger boundary between the keywords and encourages the model to transfer the last keyword in a given context (previous keywords).


## Training Details

### Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
- [LyricCanvas dataset](https://huggingface.co/datasets/fittar/lyric_canvas): a synthetically generated dataset based on lyrics and synthetically generated prompts
  
### Training Procedure 

ViPE has been trained in the standard auto-regressive procedure: given a line (or lines) of lyrics as a prefix, the objective is to generate a plausible
prompt that is both despicable and semantically related to the given lyric(c). The loss function does not include the tokens corresponding to the lyrics. So ViPE
never generates any original lyrics and only learns to generate visually related prompts. 
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

## Evaluation
In all of the following evaluations, ViPE consistently demonstrates its robustness compared to ChatGPT and achieves performance that is competitive with that of human experts.

- ***Intrinsic evaluations***
  - General understanding of figurative language using [Fig-QA dataset](https://huggingface.co/datasets/nightingal3/fig-qa)
- ***Extrinsic evaluations***
  - Image-text Retrieval on the [HAIVMet dataset](https://aclanthology.org/2023.findings-acl.465.pdf)
  - Emotion visualizations: How well does ViPE transfer emotionally charged tweets into a depictable description of a scene in comparison with
    ChatGPT. The [Emotion dataset](https://huggingface.co/datasets/dair-ai/emotion) is utilized. 
- ***Human evaluations***
  - we conducted a user study involving 30 native English-speaking participants aged between 20 and 40. Participants were
  presented with 3 images and a metaphor from the HAIVMet dataset. They were asked to select the images that matches the metaphor the best.
  The images were generated using prompts from ViPE, ChatGPT, and human experts (HAIVMet).   

## Citation

If you find ViPE useful, please cite our paper.
```
@inproceedings{shahmohammadi-etal-2023-vipe,
    title = "{V}i{PE}: Visualise Pretty-much Everything",
    author = "Shahmohammadi, Hassan  and
      Ghosh, Adhiraj  and
      Lensch, Hendrik",
    editor = "Bouamor, Houda  and
      Pino, Juan  and
      Bali, Kalika",
    booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.emnlp-main.333",
    pages = "5477--5494"
}
```
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->


## Model Card Contact

[Hassan Shahmohammadi](https://fittar.me/)