引言
Black Forest Labs 的新文本生成图像模型FLUX.1自推出以来就引起了开源社区很大的轰动,成为新一代的文生图模型的旗舰。虽然开源时间不久,但围绕Flux.1的社区开源生态发展迅猛,包括量化版本,GGUF版本,以及配套的Controlnet模型、LoRA模型均在各大开源平台上线,而配合各个模型的使用,对应的ComfyUI工作流也在蓬勃发展。
应近日来诸多社区开发者的学习诉求,特为大家推出一期 FLUX.1-LoRA/IP-adapter+ComfyUI +“顶流”IP「黑神话:悟空」的模型推理实战教程,玩转一把AI悟空生图 👀
PS:本教程只限社区学习使用,不可投入商业用途。
最佳实践
环境配置和安装:
- python 3.10及以上版本
- pytorch 1.12及以上版本,推荐2.0及以上版本
- 建议使用CUDA 11.4及以上
本文在魔搭社区免费提供的GPU免费算力上体验:
开发者也可以使用modelscope的官方镜像,在云端或自有的设备上体验。
GPU环境镜像(python3.10):
registry.cn-beijing.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.1.0-py310-torch2.1.2-tf2.14.0-1.13.1 registry.cn-hangzhou.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.1.0-py310-torch2.1.2-tf2.14.0-1.13.1 registry.us-west-1.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.1.0-py310-torch2.1.2-tf2.14.0-1.13.1
下载和部署ComfyUI
clone代码,并安装相关依赖,链接如下:
https://github.com/comfyanonymous/ComfyUI
https://github.com/ltdrdata/ComfyUI-Manager
https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes.git
# #@title Environment Setup from pathlib import Path OPTIONS = {} UPDATE_COMFY_UI = True #@param {type:"boolean"} INSTALL_COMFYUI_MANAGER = True #@param {type:"boolean"} INSTALL_CUSTOM_NODES_DEPENDENCIES = True #@param {type:"boolean"} INSTALL_ComfyUI_Comfyroll_CustomNodes = True #@param {type:"boolean"} INSTALL_x_flux_comfyui = True #@param {type:"boolean"} OPTIONS['UPDATE_COMFY_UI'] = UPDATE_COMFY_UI OPTIONS['INSTALL_COMFYUI_MANAGER'] = INSTALL_COMFYUI_MANAGER OPTIONS['INSTALL_CUSTOM_NODES_DEPENDENCIES'] = INSTALL_CUSTOM_NODES_DEPENDENCIES OPTIONS['INSTALL_ComfyUI_Comfyroll_CustomNodes'] = INSTALL_ComfyUI_Comfyroll_CustomNodes OPTIONS['INSTALL_x_flux_comfyui'] = INSTALL_x_flux_comfyui current_dir = !pwd WORKSPACE = f"{current_dir[0]}/ComfyUI" %cd /mnt/workspace/ ![ ! -d $WORKSPACE ] && echo -= Initial setup ComfyUI =- && git clone https://github.com/comfyanonymous/ComfyUI %cd $WORKSPACE if OPTIONS['UPDATE_COMFY_UI']: !echo "-= Updating ComfyUI =-" !git pull if OPTIONS['INSTALL_COMFYUI_MANAGER']: %cd custom_nodes ![ ! -d ComfyUI-Manager ] && echo -= Initial setup ComfyUI-Manager =- && git clone https://github.com/ltdrdata/ComfyUI-Manager %cd ComfyUI-Manager !git pull if OPTIONS['INSTALL_ComfyUI_Comfyroll_CustomNodes']: %cd .. !echo -= Initial setup ComfyUI_Comfyroll_CustomNodes =- && git clone https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes.git if OPTIONS['INSTALL_x_flux_comfyui']: !echo -= Initial setup x-flux-comfyui =- && git clone https://github.com/XLabs-AI/x-flux-comfyui.git if OPTIONS['INSTALL_CUSTOM_NODES_DEPENDENCIES']: !pwd !echo "-= Install custom nodes dependencies =-" ![ -f "custom_nodes/ComfyUI-Manager/scripts/colab-dependencies.py" ] && python "custom_nodes/ComfyUI-Manager/scripts/colab-dependencies.py" !pip install spandrel
下载一些模型(包含Flux.1基础模型,Lora,Controlnet,IP-adapter等),并存放到models目录的相关子目录下。小伙伴们可以选择自己希望使用的模型并下载。
感谢社区开发者们lora的贡献!
模型链接:
https://www.liblib.art/modelinfo/5e4a4cc0e3674818a9f8454a63cc0115
模型链接:
https://huggingface.co/wanghaofan/Black-Myth-Wukong-FLUX-LoRA
#@markdown ###Download standard resources %cd /mnt/workspace/ComfyUI ### FLUX1-DEV !modelscope download --model=AI-ModelScope/flux-fp8 --local_dir ./models/unet/ flux1-dev-fp8.safetensors ### clip !modelscope download --model=AI-ModelScope/flux_text_encoders --local_dir ./models/clip/ clip_l.safetensors !modelscope download --model=AI-ModelScope/flux_text_encoders --local_dir ./models/clip/ t5xxl_fp8_e4m3fn.safetensors ### vae !modelscope download --model=AI-ModelScope/FLUX.1-dev --local_dir ./models/vae/ ae.safetensors ### lora !modelscope download --model=FluxLora/flux-koda --local_dir ./models/loras/ araminta_k_flux_koda.safetensors !modelscope download --model=FluxLora/Black-Myth-Wukong-FLUX-LoRA --local_dir ./models/loras/ pytorch_lora_weights.safetensors !modelscope download --model=FluxLora/FLUX1_wukong_lora --local_dir ./models/loras/ FLUX1_wukong_lora.safetensors ### ip-adapter !modelscope download --model=FluxLora/flux-ip-adapter --local_dir ./models/xlabs/ipadapters/ flux-ip-adapter.safetensors !modelscope download --model=FluxLora/flux-ip-adapter --local_dir ./models/clip_vision/ clip_vision_l.safetensors
使用cloudflared运行ComfyUI
!wget "https://modelscope.oss-cn-beijing.aliyuncs.com/resource/cloudflared-linux-amd64.deb" !dpkg -i cloudflared-linux-amd64.deb %cd /mnt/workspace/ComfyUI import subprocess import threading import time import socket import urllib.request def iframe_thread(port): while True: time.sleep(0.5) sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) result = sock.connect_ex(('127.0.0.1', port)) if result == 0: break sock.close() print("\nComfyUI finished loading, trying to launch cloudflared (if it gets stuck here cloudflared is having issues)\n") p = subprocess.Popen(["cloudflared", "tunnel", "--url", "http://127.0.0.1:{}".format(port)], stdout=subprocess.PIPE, stderr=subprocess.PIPE) for line in p.stderr: l = line.decode() if "trycloudflare.com " in l: print("This is the URL to access ComfyUI:", l[l.find("http"):], end='') #print(l, end='') threading.Thread(target=iframe_thread, daemon=True, args=(8188,)).start() !python main.py --dont-print-server
load ComfyUI流程图,链接:
https://modelscope.oss-cn-beijing.aliyuncs.com/resource/workflow-flux-lora-simple.json
https://modelscope.oss-cn-beijing.aliyuncs.com/resource/workflow-flux-lora-wukong.json
https://modelscope.oss-cn-beijing.aliyuncs.com/resource/ip_adapter_workflow.json
lora流程图片如下:
ip-adapter流程图片如下:
显存占用:
注意,上传图片后,需要检查图片中的模型文件和下载存储的模型名称是否一致,可直接一键点击如ckpt_name关联到存储的模型文件名字。
作品分享
身披战衣,佛在心中
直面天命,浴火重生
带上眼镜,一起来玩魔搭!