Add support for Multiple ControlNetXSAdapters in SDXL pipeline

[naomili0924]

What does this PR do?

This PR is addressing the feature request from an open good-first issue: #8434

Fixes # (issue)

Before this change:

pipe = StableDiffusionXLControlNetXSPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", controlnet=depth_controlnet, torch_dtype=torch.float16
).to("cuda")

StableDiffusionXLControlNetXSPipeline loads unet base model and only support a single point injection from only one controlnet. With this new change, we allows the StableDiffusionXLControlNetXSPipeline to take a new UnetConditionModel named MultiControlUnetConditionModel and this new model is able to load weights from each controlnet model and handle bi-injection between unet and controlnet layer by layer.

Due to we didn't find any test repo to verify this change, we provide the following code and used it to verify our change (not included in this repo).

import torch
from diffusers.models.autoencoders import AutoencoderKL
from diffusers.models.controlnets import ControlNetXSAdapter
from diffusers.pipelines.controlnet_xs import StableDiffusionXLControlNetXSPipeline
from PIL import Image
from diffusers.utils import load_image
import cv2
import numpy as np

vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
canny_controlnet = ControlNetXSAdapter.from_pretrained(
    "UmerHA/Testing-ConrolNetXS-SDXL-canny", torch_dtype=torch.float16
).to("cuda")
depth_controlnet = ControlNetXSAdapter.from_pretrained(
    "UmerHA/Testing-ConrolNetXS-SDXL-depth", torch_dtype=torch.float16
).to("cuda")

pipe = StableDiffusionXLControlNetXSPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", controlnet=[depth_controlnet, canny_controlnet], torch_dtype=torch.float16
).to("cuda")

# Load your conditioning images
raw_image = load_image("/content/drive/MyDrive/diffusers/src/test.jpg").convert("RGB")

# Generate Canny edge
def get_canny(image):
    image_np = np.array(image.resize((512, 512)))
    image_gray = cv2.cvtColor(image_np, cv2.COLOR_RGB2GRAY)
    edges = cv2.Canny(image_gray, 100, 200)
    edges = np.stack([edges] * 3, axis=-1)  # Make it 3-channel
    return Image.fromarray(edges)

# Generate a fake depth map (in real cases use a depth estimator)
def get_fake_depth(image):
    image_np = np.array(image.resize((512, 512)))
    gray = cv2.cvtColor(image_np, cv2.COLOR_RGB2GRAY)
    depth = np.stack([gray] * 3, axis=-1)
    return Image.fromarray(depth)

canny_image = get_canny(raw_image)
depth_image = get_fake_depth(raw_image)

# Run inference
prompt = "A flower"
output = pipe(
    prompt=prompt,
    image=[canny_image, depth_image],  # order matches controlnet list
    controlnet_conditioning_scale = [0.2, 0.8],
    num_inference_steps=30,
    generator=torch.manual_seed(0),
)

output.images[0].save("output.png")

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[DN6]

Hi @naomili0924 thank you for putting the time into this, unfortunately we had to deprecate ControlnetXS due to low usage (You'll notice that the pipeline inherits from DeprecatedPipelineMixin.). We are not actively updating/adding features to it at this time.