Hugging Face's logo

KBlueLeaf
/
TIPO-200M

Text Generation
Transformers
Safetensors
GGUF
English
llama
text-generation-inference
Inference Endpoints
Model card Files Community
TIPO-200M / README.md
KBlueLeaf's picture
KBlueLeaf
Create README.md
a8630de verified
preview code
|
raw
history blame
2.86 kB
metadata
license: other
datasets:
  - laion/conceptual-captions-12m-webdataset
  - CaptionEmporium/coyo-hd-11m-llavanext
  - KBlueLeaf/danbooru2023-metadata-database
  - graph-based-captions/GBC10M
language:
  - en
pipeline_tag: text-generation
library_name: transformers

TIPO: Text to Image with text presampling for Prompt Optimization

200M LLaMA arch model trained for TIPO.

Introduction

In this project, we introduce "TIPO" (Text to Image with text presampling for Prompt Optimization), an innovative framework designed to significantly enhance the quality and usability of Text-to-Image (T2I) generative models. TIPO utilizes the Large Language Models (LLMs) to perform "Text Presampling" within the inference pipeline of text-to-image generative modeling. By refining and extending user input prompts, TIPO enables generative models to produce superior results with minimal user effort, making T2I systems more accessible and effective for a wider range of users.

Usage

Use updated version of DTG extension (renamed to z-tipo-ext), current version of z-tipo-ext support stable-diffusion-webui, stable-diffusion-webui-forge and ComfyUI. SD-Next haven't been tested.

Metric

We have tested TIPO in several metric:

1. Aesthetic Score (Higher is Better)

We compute the Aesthetic Score using the Aesthetic Predictor V2.5. This metric is calculated on the short/truncated long test.

Aesthetic Score Distribution

Figure 1: Aesthetic Score distribution.

2. AI Corrupt Score (Higher is Better)

The AI Corrupt Score is obtained from the AICorruptMetrics in sdeval.

This metric is calculated on the short/truncated long test.

AI Corrupt Score Distribution

Figure 2: AI Corrupt Score distribution.

3. Frechet Dino Distance (FDD) on Scenery Tag Test

We use FDD on the Scenery Tag Test to demonstrate that when input prompts address a smaller distribution, the model struggles to generate images that reflect the true distribution. However, with TIPO, this issue is mitigated.

FDD Model <meta> scenery only <meta> scenery + TIPO
DinoV2 ViT-S 0.1917 0.1786
DinoV2 ViT-B 0.2002 0.1755
DinoV2 ViT-L 0.2017 0.1863
DinoV2 ViT-G 0.2359 0.2096

Table 1: Frechet Dino Distance (FDD) on Scenery Tag Test.

LICENSE

This model is released under Kohaku License 1.0
You can check the above provided URL or check the LICENSE file in this repo.