用Stable Diffusion和ControlNet模型生成惊艳图片

原文地址：zaozaoliao.feishu.cn/wiki/L4Axwl…

「 效果展示 」

提示词	目标字样	原图	生成效果
This stunning image captures the beauty and innocence of youth, with vibrant colors and a serene seaside setting. High-definition photo of a breathtaking moment, perfect for any project that aims to evoke a sense of joy and tranquility.

「 预备工作 」

1. 开发环境 – 基于 Linux

• Ubuntu 20.04

• CUDA 11.4

• python 3.8

• torch 1.11.0+cu113

• Nvidia Tesla V100

2. 图片素材

准备一张自定义尺寸字体图片，工具任选，譬如美图秀秀、PS、Word、PPTX 均可，注意字体需要黑底白字，预先排好字体位置，字体最好免费可商用。

这里用的在线美图秀秀，字体 GreatVikes，分辨率 512*512。

3. 代码模型

	StableDiffusion	ControlNet
代码	AUTOMATIC1111/stable-diffusion-webui	Mikubill/sd-webui-controlnet
模型	beautifulRealistic_v60.safetensors	control_v1p_sd15_brightness.safetensors

SD 模型这里用的 Beautiful Realistic Asians，可根据个人需求选择，下载模型请移步大名鼎鼎的 C 站 civitai.com。

ControlNetcontrol_v1p_sd15_brightness，若更换模型，需下载对应代码的模型，否则加载模型会报错（参考本文最后的错误集锦）。

「 环境部署 」

1. StableDiffusion

代码： AUTOMATIC1111/stable-diffusion-webui

模型： beautifulRealistic_v60.safetensors

将下载好的 SD 模型放入 stable-diffusion-webui/models/Stable-diffusion 目录下即可

安装部署

# 代码下载
git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git

# 一键安装
./webui.sh

打开 WebUI

网页输入 127.0.0.1：xxx，即可打开 webUI 界面

2. ControlNet

代码： Mikubill/sd-webui-controlnet

模型： control_v1p_sd15_brightness.safetensors

将下载好的 ControlNet 模型放入/stable-diffusion-webui/models/ControlNet 即可。

安装部署

打开 WebUI 后，进入 Extensions -> Install from URL，输入 ControlNet 上面的代码链接，点击 Install 即可，安装好后重启 WebUI。

安装好后，即可看到 /stable-diffusion-webui/extensions 中多了 sd-webui-controlnet

「 图片生成 」

1. 示例 1

Prompt

Highest quality, ultra-high definition, masterpieces, 8k quality, 1girl, very delicate features, very detailed eyes and mouth, long hair, curly hair, delicate skin, big eyes, white sweater, necklace, street, Sunshine, standing on the beach with a backdrop of blue skies and ocean waves. This stunning image captures the beauty and innocence of youth, with vibrant colors and a serene seaside setting. High-definition photo of a breathtaking moment, perfect for any project that aims to evoke a sense of joy and tranquility.
Negative prompt: blurry, low quality, pixelated, (unattractive), (grimy), (old woman), (dark), (out of focus), (unclear), (dirty), ((extra arm)), ((extra leg)), ((extra nose)), ((extra mouth)), (out of frame), (bad composition), (too bright), (too dark), ((extra flowers)), ((extra hand)), (poor lighting), (bad color grading), (red-eyed), (morphed face), (unnatural posture), (awkward pose), (frozen animation), (poorly photoshopped), (low-res), (bad framing), (insipid)

Parameters

StableDiffusion

• checkpoint: beautifulRealistic_v60.safetensors

• Sampling method: DPM++ SDE Karras

• Sampling Steps: 30

ControlNet

• Enable

• Control Type: All

• Model: control_v1p_sd15_brightness.safetensors

• Control Weight: 0.4

• Ending Control Step: 0.6

2. 示例 2

Prompt

Highest quality, ultra-high definition, masterpieces, 8k quality, A man wearing a black jacket and jeans, very delicate features, very detailed eyes and mouth, with a black hat and sunglasses, holding a cigarette in his right hand and his left hand in his pocket, standing on the beach with blue skies and sea waves in the background. High-quality photo of a rugged man enjoying a smoke break by the sea, capturing the sense of freedom and relaxation. Shot in high definition, this image is perfect for any project that requires a cool and laid-back vibe.
Negative prompt: blurred, low quality, pixelated, (unattractive), (grimy), (old man), (dark), (out of focus), (unclear), (dirty), ((extra arm)), ((extra leg)), ((extra nose)), ((extra mouth)), (out of frame), (bad composition), (too bright), (too dark), ((extra cigarette)), ((extra hand)), (poor lighting), (bad color grading), (red-eyed), (morphed face), (unnatural posture), (awkward pose), (frozen animation), (poorly photoshopped), (low-res), (bad framing), (insipid)

Parameters

StableDiffusion

• checkpoint: beautifulRealistic_v60.safetensors

• Sampling method: DPM++ SDE Karras

• Sampling Steps: 30

ControlNet

• Enable

• Control Type: All

• Model: control_v1p_sd15_brightness.safetensors

• Control Weight: 0.4

• Ending Control Step: 0.6

3. 一些小 Tips

• 正反提示词：复制 prompt 到第一格空格后，直接点击右侧蓝色小箭头，即可自动拆分正反提示词；

• Control Weight：使用 ControlNet 权重，建议 0.4-0.7，参数越大，自定义的图片越明显；

Control Weight	0.4	0.6	0.8
Result

• Ending Control Step：ControlNet 终止介入的时机，建议 0.55-0.75。

Ending Control Step	0.4	0.6	0.9
Result

「 错误集锦 」

1. 执行 webui.sh 会安装虚拟环境，若报 No module named ‘importlib.metadata’ ，是因为系统不支持 python3.7，将 webui.sh 47 行改成 python3.8 或者 3.10 即可

# webui.sh 47行
# python3 executable
if [[ -z "${python_cmd}" ]]
then
    python_cmd="python3.8"
fi

2. 若前面修改为 python3.8，会导致 network.py 不支持 Network | None 用法，需载入 Union，且修改 103 行

# network.py的103行
from typing import Union
class ModuleType:
    def create_module(self, net: Network, weights: NetworkWeights) -> Union[Network, None]:#Network | None:
        return None

3. 虚拟环境默认安装 torch2.0，修改 launch_utils.py 的 273 行，以安装与 cuda 配套的指定版本，系统默认安装 torch 2.0.0，我本地为 cuda11.4，所以这里安装 torch==1.11.0+cu113

# launch_utils.py的273行
torch_command = os.environ.get('TORCH_COMMAND', f"pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html")

3. OSError: libtorch_cuda_cpp.so： cannot open shared object file: No such file or directory，注意 torch 版本

4. 若报下面这一坨问题，大概率是 ControlNet 模型不匹配问题

huggingface.co/ioclab/ioc-…

ERROR: ControlNet cannot find model config [/XXX/models/ControlNet/diffusion_pytorch_model.yaml] 
ERROR: ControlNet will use a WRONG config to load your model. 
ERROR: The WRONG config may not match your model. The generated results can be bad. 
ERROR: You are using a ControlNet model [diffusion_pytorch_model] without correct YAML config file. 
ERROR: The performance of this model may be worse than your expectation. 
ERROR: If this model cannot get good results, the reason is that you do not have a YAML file for the model. 

Hint: You can take a look at [/XXX/extensions/sd-webui-controlnet/models] to find many existing YAML files.
*** Error running process: /XXX/extensions/sd-webui-controlnet/scripts/controlnet.py
    Traceback (most recent call last):
        raise RuntimeError('Error(s) in loading state_dict for {}:nt{}'.format(
    RuntimeError: Error(s) in loading state_dict for ControlNet:
        Missing key(s) in state_dict: "time_embed.0.weight", "time_embed.0.bias", "time_embed.2.weight",