File size: 2,736 Bytes
9659729 98364f5 9659729 89e39ca 81353b2 98364f5 29b56fd 98364f5 6ec0661 98364f5 c6c45f3 98364f5 c6c45f3 98364f5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 |
---
license: apache-2.0
datasets:
- yuvalkirstain/pickapic_v2
language:
- en
pipeline_tag: text-to-image
---
**Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation** (https://huggingface.co/papers/2402.10210)
# SPIN-Diffusion-iter3
This model is a self-play fine-tuned diffusion model at iteration 3 from [runwayml/stable-diffusion-v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5) using synthetic data based on the winner images of the [yuvalkirstain/pickapic_v2](https://huggingface.co/datasets/yuvalkirstain/pickapic_v2) dataset. We have also made a Gradio Demo at [UCLA-AGI/SPIN-Diffusion-demo-v1](https://huggingface.co/spaces/UCLA-AGI/SPIN-Diffusion-demo-v1).
## Model Details
### Model Description
- Model type: A diffusion model with unet fine-tuned, based on the structure of stable diffusion 1.5
- Language(s) (NLP): Primarily English
- License: Apache-2.0
- Finetuned from model: runwayml/stable-diffusion-v1-5
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2.0e-05
- train_batch_size: 8
- distributed_type: multi-GPU
- num_devices: 8
- train_gradient_accumulation_steps: 32
- total_train_batch_size: 2048
- optimizer: AdamW
- lr_scheduler: "linear"
- lr_warmup_steps: 200
- num_training_steps: 500
### Usage
To use the model, you must first load the SD1.5 base model and then substitute its unet with our fine-tuned version.
```python
from diffusers import StableDiffusionPipeline, UNet2DConditionModel
import torch
model_id = "runwayml/stable-diffusion-v1-5"
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
unet_id = "UCLA-AGI/SPIN-Diffusion-iter3"
unet = UNet2DConditionModel.from_pretrained(unet_id, subfolder="unet", torch_dtype=torch.float16)
pipe.unet = unet
###The rest of your generation code
```
### Evaluation Results on [Pick-a-pic test set](https://huggingface.co/datasets/yuvalkirstain/pickapic_v2/viewer/default/test_unique)
| Metric | Best of Five | Mean | Median
|-----------------------|---------------------------|--------|-------
| HPS | 0.28 | 0.27 | 0.27
| Aesthetic | 6.26 | 5.94 | 5.98
| Image Reward | 1.13 | 0.53 | 0.67
| Pickapic Score | 22.00 |21.36 | 21.46
## Citation
```
@misc{yuan2024self,
title={Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation},
author={Yuan, Huizhuo and Chen, Zixiang and Ji, Kaixuan and Gu, Quanquan},
year={2024},
eprint={2402.10210},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
```
|