File size: 3,980 Bytes
1bfa61c
 
 
 
 
f668844
1bfa61c
 
 
f668844
 
 
1bfa61c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
019ea83
1bfa61c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92

## Generate Example:
- Model: [andite/anything-v4.0](https://hf.co/andite/anything-v4.0)
- Prompt: `best quality, extremely detailed, cowboy shot`
- Negative_prompt: `cowboy, monochrome, lowres, bad anatomy, worst quality, low quality`
- Seed: 19 (cherry-picked)

|Control Image1|Control Image2|Generated|
|---|---|---|
|<img width="200" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/multi_controlnet/pac_pose_512x512.png">|<img width="200" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/multi_controlnet/pac_canny_512x512.png">|<img width="200" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/multi_controlnet/mc_pose_and_canny_result_19.png">|
|<img width="200" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/multi_controlnet/pac_pose_512x512.png">|(none)|<img width="200" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/multi_controlnet/mc_pose_only_result_19.png">|
|<img width="200" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/multi_controlnet/pac_canny_512x512.png">|(none)|<img width="200" src="https://huggingface.co/takuma104/controlnet_dev/resolve/main/multi_controlnet/mc_canny_only_result_19.png">|

- Pose & Canny Control Image Generated by [*Character bones that look like Openpose for blender _ Ver_4.7 Depth+Canny*](https://toyxyz.gumroad.com/l/ciojz) 


## Code
Using deveploment version [`StableDiffusionMultiControlNetPipeline`](https://github.com/takuma104/diffusers/tree/multi_controlnet)

```py
from stable_diffusion_multi_controlnet import StableDiffusionMultiControlNetPipeline
from stable_diffusion_multi_controlnet import ControlNetProcessor
from diffusers import ControlNetModel
import torch
from diffusers.utils import load_image

pipe = StableDiffusionMultiControlNetPipeline.from_pretrained(
    "andite/anything-v4.0", safety_checker=None, torch_dtype=torch.float16
).to("cuda")
pipe.scheduler = EulerDiscreteScheduler.from_config("andite/anything-v4.0", subfolder="scheduler")
pipe.enable_xformers_memory_efficient_attention()

controlnet_canny = ControlNetModel.from_pretrained(
    "lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16
).to("cuda")
controlnet_pose = ControlNetModel.from_pretrained(
    "lllyasviel/sd-controlnet-openpose", torch_dtype=torch.float16
).to("cuda")

canny_image = load_image('https://huggingface.co/takuma104/controlnet_dev/resolve/main/multi_controlnet/pac_canny_512x512.png').convert('RGB')
pose_image = load_image('https://huggingface.co/takuma104/controlnet_dev/resolve/main/multi_controlnet/pac_pose_512x512.png').convert('RGB')

prompt = "best quality, extremely detailed, cowboy shot"
negative_prompt = "cowboy, monochrome, lowres, bad anatomy, worst quality, low quality"
seed = 19

image = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    processors=[
        ControlNetProcessor(controlnet_pose, pose_image),
        # ControlNetProcessor(controlnet_canny, canny_image),
    ],
    generator=torch.Generator(device="cpu").manual_seed(seed),
    num_inference_steps=30,
    width=512,
    height=512,
).images[0]
image.save(f"./mc_pose_only_result_{seed}.png")

image = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    processors=[
        # ControlNetProcessor(controlnet_pose, pose_image),
        ControlNetProcessor(controlnet_canny, canny_image),
    ],
    generator=torch.Generator(device="cpu").manual_seed(seed),
    num_inference_steps=30,
    width=512,
    height=512,
).images[0]
image.save(f"./mc_canny_only_result_{seed}.png")

image = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    processors=[
        ControlNetProcessor(controlnet_pose, pose_image),
        ControlNetProcessor(controlnet_canny, canny_image),
    ],
    generator=torch.Generator(device="cpu").manual_seed(seed),
    num_inference_steps=30,
    width=512,
    height=512,
).images[0]
image.save(f"./mc_pose_and_canny_result_{seed}.png")


```