--- language: - en pipeline_tag: text2text-generation inference: false license: mit --- # ViPE-S-CTX7 ViPE: Visualize Pretty-much Everything, is the first automated model for translating any arbitrary piece of text into a visualizable prompt. It helps any text-to-image model in figurative or non-lexical language visualizations. It has been shown to be more robust than GPT3.5 Turbo (ChatGPT) in generating depictable and semantically meaningful prompts. ### Model Description - **Developed by:** [Computer Graphics Group, University of Tuebingen](https://uni-tuebingen.de/fakultaeten/mathematisch-naturwissenschaftliche-fakultaet/fachbereiche/informatik/lehrstuehle/computergrafik/lehrstuhl/) - **Model type:** Auto-Regressive - **Language:** English - **License:** MIT - **Based on:** [GPT2-Small](https://huggingface.co/gpt2) - **Versions:** [ViPE-M-CTX7](https://huggingface.co/fittar/ViPE-M-CTX7) (355M parameters) and [ViPE-S-CTX7](https://huggingface.co/fittar/ViPE-S-CTX7) (117M parameters) ### Model Sources - **Repository:** [Github](https://github.com/Hazel1994/ViPE) - **Paper:** [ViPE: Visualise Pretty-much Everything](https://aclanthology.org/2023.emnlp-main.333/) (**Outstanding Paper Award at EMNLP 2023**) ### Down Stream Applications ViPE provides a robust backbone for many practical applications such as music video generation and creative writing. - #### Music Video Genrations - **Repository:** [Github](https://github.com/Hazel1994/ViPE-Videos) - **Example Videos:** [ViPE Videos](https://www.youtube.com/playlist?list=PLvLHdI48ZdfaDMxPZIGHXrvsRkdADcMUh) - #### Creative Writing - **Demo:** [Hugging Face Playground](https://huggingface.co/spaces/fittar/ViPE) - #### Summagery: Document Summarization through Images - **Demo:** [Hugging Face Playground](https://huggingface.co/spaces/fittar/summagary) - **Repository:** [Github](https://github.com/Hazel1994/summagary) - **Examples:** [Summagery Videos](https://www.youtube.com/watch?v=mFMkE2waYGQ) ### Direct Use You can directly use the model to generate detailed prompts for any arbitrary text. ```python from transformers import GPT2LMHeadModel, GPT2Tokenizer def generate(text, model, tokenizer,device,do_sample,top_k=100, epsilon_cutoff=.00005, temperature=1): #mark the text with special tokens text=[tokenizer.eos_token + i + tokenizer.eos_token for i in text] batch=tokenizer(text, padding=True, return_tensors="pt") input_ids = batch["input_ids"].to(device) attention_mask = batch["attention_mask"].to(device) #how many new tokens to generate at max max_prompt_length=50 generated_ids = model.generate(input_ids=input_ids,attention_mask=attention_mask, max_new_tokens=max_prompt_length, do_sample=do_sample,top_k=top_k, epsilon_cutoff=epsilon_cutoff, temperature=temperature) #return only the generated prompts pred_caps = tokenizer.batch_decode(generated_ids[:, -(generated_ids.shape[1] - input_ids.shape[1]):], skip_special_tokens=True) return pred_caps device='cpu' model = GPT2LMHeadModel.from_pretrained('fittar/ViPE-S-CTX7') model.to(device) #ViPE-M's tokenizer is identical to that of GPT2-Small tokenizer = GPT2Tokenizer.from_pretrained('gpt2') tokenizer.pad_token = tokenizer.eos_token # A list of abstract/figurative or any arbitrary combinations of keywords texts=['lalala', 'I wanna start learning', 'free your mind; you will see the other side of life', 'brave; fantasy'] prompts=generate(texts,model,tokenizer,do_sample=True,device=device) for t,p in zip(texts,prompts): print('{} --> {}'.format(t,p)) lalala --> A group of dancers performing an extravagant traditional dance, lalala in Spanish I wanna start learning --> A student intently sitting at a desk, surrounded by books and notes free your mind; you will see the other side of life --> A view of the night sky, stars and planets shining bright, while a woman in a field of flowers looks up in awe brave; fantasy --> A knight in shining armor riding a gallant horse through a sunlit valley ``` ### Recommendations You can use either a comma or a semicolon to combine multiple keywords. for example ['dark, fantasy, brave'] or ['This is gonna be the best day of my life; do you agree?']. However, a semicolon draws a stronger boundary between the keywords and encourages the model to transfer the last keyword in a given context (previous keywords). ## Training Details ### Training Data - [LyricCanvas dataset](https://huggingface.co/datasets/fittar/lyric_canvas): a synthetically generated dataset based on lyrics and synthetically generated prompts ### Training Procedure ViPE has been trained in the standard auto-regressive procedure: given a line (or lines) of lyrics as a prefix, the objective is to generate a plausible prompt that is both despicable and semantically related to the given lyric(c). The loss function does not include the tokens corresponding to the lyrics. So ViPE never generates any original lyrics and only learns to generate visually related prompts. ## Evaluation In all of the following evaluations, ViPE consistently demonstrates its robustness compared to ChatGPT and achieves performance that is competitive with that of human experts. - ***Intrinsic evaluations*** - General understanding of figurative language using [Fig-QA dataset](https://huggingface.co/datasets/nightingal3/fig-qa) - ***Extrinsic evaluations*** - Image-text Retrieval on the [HAIVMet dataset](https://aclanthology.org/2023.findings-acl.465.pdf) - Emotion visualizations: How well does ViPE transfer emotionally charged tweets into a depictable description of a scene in comparison with ChatGPT. The [Emotion dataset](https://huggingface.co/datasets/dair-ai/emotion) is utilized. - ***Human evaluations*** - we conducted a user study involving 30 native English-speaking participants aged between 20 and 40. Participants were presented with 3 images and a metaphor from the HAIVMet dataset. They were asked to select the images that matches the metaphor the best. The images were generated using prompts from ViPE, ChatGPT, and human experts (HAIVMet). ## Citation If you find ViPE useful, please cite our paper. ``` @inproceedings{shahmohammadi-etal-2023-vipe, title = "{V}i{PE}: Visualise Pretty-much Everything", author = "Shahmohammadi, Hassan and Ghosh, Adhiraj and Lensch, Hendrik", editor = "Bouamor, Houda and Pino, Juan and Bali, Kalika", booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing", month = dec, year = "2023", address = "Singapore", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2023.emnlp-main.333", pages = "5477--5494" } ``` ## Model Card Contact [Hassan Shahmohammadi](https://fittar.me/)