Nacholmo

/

controlnet-qr-pattern-sdxl

like 47

@Jagerius Are you trying to use a sdxl controlnet on a 1.5 model? What error outputs the terminal?

Try loading the 1.5 version instead

Nacholmo/controlnet-qr-pattern-v2

hi i have the same question, how would i load this controlnet model in python code? I'm using SDXL

@MaxTran96

hi i have the same question, how would i load this controlnet model in python code? I'm using SDXL

I'm not familiar with running models in that way, maybe I can suggest to use the automatic111 api.
Or use the diffusers library, here is an example from the docs:

from diffusers import StableDiffusionXLControlNetPipeline, ControlNetModel, UniPCMultistepScheduler
from diffusers.utils import load_image
import torch

base_model_path = "stabilityai/stable-diffusion-xl-base-1.0"
controlnet_path = "path to controlnet"

controlnet = ControlNetModel.from_pretrained(controlnet_path, torch_dtype=torch.float16)
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    base_model_path, controlnet=controlnet, torch_dtype=torch.float16
)

# speed up diffusion process with faster scheduler and memory optimization
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
# remove following line if xformers is not installed or when using Torch 2.0.
pipe.enable_xformers_memory_efficient_attention()
# memory optimization.
pipe.enable_model_cpu_offload()

control_image = load_image("./conditioning_image_1.png")
prompt = "pale golden rod circle with old lace background"

# generate image
generator = torch.manual_seed(0)
image = pipe(
    prompt, num_inference_steps=20, generator=generator, image=control_image
).images[0]
image.save("./output.png")

https://github.com/huggingface/diffusers/blob/main/examples/controlnet/README_sdxl.md

No response in a month, I'm closing this issue