README.md

metadata

license: cc-by-nc-nd-4.0
datasets:
  - laion/conceptual-captions-12m-webdataset
  - CaptionEmporium/coyo-hd-11m-llavanext
  - KBlueLeaf/danbooru2023-metadata-database
  - graph-based-captions/GBC10M
language:
  - en
pipeline_tag: text-generation
library_name: transformers

[WIP] TIPO

What is this

TIPO is a tool to extend, generate, refine the input prompt for T2I models.
It can work on both Danbooru tags and Natural Language. Which means you can use it on almost all the existed T2I models.
You can take it as "pro max" version of DTG

Traning Detail

• Model Arch: LLaMA
• Size: 200M param
• Training Data:
  • Danbooru Metadata: 7.8M entries
  • CC12M/GBC10M: around 11M entries
  • Coyo11M: around 11M entries
• Training Procedure:
  • Danbooru + cc12m: 5epoch
  • Danbooru: 1epoch
  • Danbooru + cc12m + coyo11m: 3epoch (currently 2epoch, still training)
• Token Seen: Currently 35B token
• Cost Time: Around 2~3 week on 4x3090

How to use this model?

Although the official inference code with lot of formatting and automatic features is in private now.
You can still try to make your own inference interface based on format below:

quality: masterpiece
aspect ratio: 1.0
target: <|short|> <|tag_to_long|>
tag: 1girl, solo, dragon girl, dragon horns, dragon tail

Then you will get output like:

quality: masterpiece
aspect ratio: 1.0
target: <|short|> <|tag_to_long|>
tag: 1girl, solo, dragon girl, dragon horns, dragon tail, smile, ponytail, cleavage cutout, pointy ears, large breasts, black dress, white background, thighhighs, bare shoulders, tail, breasts, clothing cutout, simple background, blonde hair, long hair, blue eyes, looking at viewer, horns,
long: A young woman with blonde hair and cat ears on her head. she is wearing a black outfit with gold accents and has a sword in her right hand. the woman is sitting on top of a large orange snake that is coiled around her body. the snake appears to be attacking her, as if it is attacking her.

All the supported mode is:

None #Tags only, DTG mode
tag_to_long
long_to_tag
short_to_long
short_to_tag
tag_to_short_to_long
short_to_tag_to_long
short_to_long_to_tag

Brief Explaination of Possible "Weird" Output

The model is trained on "What we used for training T2I model", which is basically caption from VLM.
Since these VLM have lot of different hallucination, this project will also generate some content that "looks like have hallucination"
But since the T2I model we want to use also trained on these kind of things, it can still generate descent image, or even better.

For example:

• Lot of animal ears/horns feature will be captioned as "cat ears" in most of VLM, include GPT4o or Claude3.5 sonnet.

So if you met some weird output which looks like conflicting with tags, try to generate image from it first.
You should take the Natural Language part as "different English", since that's what we used for T2I currently...

Why inference code is private? When will it be open sourced?

1. This model/tool is still under development, currently is early Alpha version.
2. I'm doing some research and projects based on this.
3. The model is released under CC-BY-NC-ND License currently. If you have interest, you can implement inference by yourself.
4. Once the project/research are done, I will open source all these models/codes with Apache2 license.

Citation

@misc{TIPO2024,
  author       = {Shih-Ying Yeh},
  title        = {TIPO: Text to Image with text Presampling for Optimal prompting},
  howpublished = {\url{https://huggingface.co/KBlueLeaf/TIPO-200M-dev}},
  year         = {2024},
  note         = {Still under development},
}