01引 言
今天,Black Forest Labs官方发布FLUX.1 Tools,这是一套模型全家桶,旨在为FLUX.1基础文本转图像模型添加控制和可操纵性,从而实现对真实图像和生成图像的修改和重新创建。FLUX.1 Tools包含四个不同的功能:
FLUX.1 Fill:最先进的修复和修复模型,可以根据文本描述和二进制掩码编辑和扩展真实图像和生成的图像。
模型链接:
https://modelscope.cn/models/AI-ModelScope/FLUX.1-Fill-dev
FLUX.1 Depth:根据从输入图像和文本提示中提取的深度图训练模型,以实现结构引导。
模型链接:
https://modelscope.cn/models/AI-ModelScope/FLUX.1-Depth-dev-lora
FLUX.1 Canny:经过训练的模型,可根据从输入图像和文本提示中提取的 Canny 边缘实现结构引导。
模型链接:
https://modelscope.cn/models/AI-ModelScope/FLUX.1-Canny-dev-lora
FLUX.1 Redux:允许混合和重新创建输入图像和文本提示的适配器。
模型链接:
https://modelscope.cn/models/AI-ModelScope/FLUX.1-Redux-dev
02模型体验
FLUX.1 Fill
FLUX.1 Depth
FLUX.1 Canny:
FLUX.1 Redux:
最佳实践
环境安装:安装comfyui和对应的定制化节点
# #@title Environment Setup from pathlib import Path OPTIONS = {}UPDATE_COMFY_UI = True #@param {type:"boolean"}INSTALL_COMFYUI_MANAGER = True #@param {type:"boolean"}INSTALL_CUSTOM_NODES_DEPENDENCIES = True #@param {type:"boolean"}INSTALL_ComfyUI_CustomNodes = True #@param {type:"boolean"}INSTALL_x_flux_comfyui = True #@param {type:"boolean"}OPTIONS['UPDATE_COMFY_UI'] = UPDATE_COMFY_UIOPTIONS['INSTALL_COMFYUI_MANAGER'] = INSTALL_COMFYUI_MANAGEROPTIONS['INSTALL_CUSTOM_NODES_DEPENDENCIES'] = INSTALL_CUSTOM_NODES_DEPENDENCIESOPTIONS['INSTALL_ComfyUI_CustomNodes'] = INSTALL_ComfyUI_CustomNodesOPTIONS['INSTALL_x_flux_comfyui'] = INSTALL_x_flux_comfyui current_dir = !pwdWORKSPACE = f"{current_dir[0]}/ComfyUI" %cd /mnt/workspace/ ![ ! -d $WORKSPACE ] && echo -= Initial setup ComfyUI =- && git clone https://github.com/comfyanonymous/ComfyUI%cd $WORKSPACE if OPTIONS['UPDATE_COMFY_UI']: !echo "-= Updating ComfyUI =-" !git pull if OPTIONS['INSTALL_COMFYUI_MANAGER']: %cd custom_nodes ![ ! -d ComfyUI-Manager ] && echo -= Initial setup ComfyUI-Manager =- && git clone https://github.com/ltdrdata/ComfyUI-Manager %cd ComfyUI-Manager !git pull if OPTIONS['INSTALL_ComfyUI_CustomNodes']: %cd .. !echo -= Initial setup ComfyUI_Comfyroll_CustomNodes =- && git clone https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes.git !echo -= Initial setup ComfyUI_rgthree_comfy =- && git clone https://github.com/rgthree/rgthree-comfy.git !echo -= Initial setup ComfyUI_JPS =- && git clone https://github.com/JPS-GER/ComfyUI_JPS-Nodes.git !echo -= Initial setup ComfyUI_Custom_Scripts =- && git clone https://github.com/pythongosssss/ComfyUI-Custom-Scripts.git !echo -= Initial setup ComfyUI-KJNodes =- && git clone https://github.com/kijai/ComfyUI-KJNodes.git if OPTIONS['INSTALL_x_flux_comfyui']: !echo -= Initial setup x-flux-comfyui =- && git clone https://github.com/XLabs-AI/x-flux-comfyui.git if OPTIONS['INSTALL_CUSTOM_NODES_DEPENDENCIES']: !pwd !echo "-= Install custom nodes dependencies =-" ![ -f "custom_nodes/ComfyUI-Manager/scripts/colab-dependencies.py" ] && python "custom_nodes/ComfyUI-Manager/scripts/colab-dependencies.py" !pip install spandrel
模型下载:下载FLUX.1和tool全家桶
#@markdown ###Download standard resources%cd /mnt/workspace/ComfyUI ### FLUX1-DEV# !modelscope download --model=AI-ModelScope/FLUX.1-dev --local_dir ./models/unet/ flux1-dev.safetensors!modelscope download --model=AI-ModelScope/flux-fp8 --local_dir ./models/unet/ flux1-dev-fp8.safetensors ### clip!modelscope download --model=AI-ModelScope/flux_text_encoders --local_dir ./models/clip/ clip_l.safetensors!modelscope download --model=AI-ModelScope/flux_text_encoders --local_dir ./models/clip/ t5xxl_fp8_e4m3fn.safetensors ### vae!modelscope download --model=AI-ModelScope/FLUX.1-dev --local_dir ./models/vae/ ae.safetensors ### lora#!modelscope download --model=FluxLora/flux-koda --local_dir ./models/loras/ araminta_k_flux_koda.safetensors!modelscope download --model=AI-ModelScope/FLUX.1-Redux-dev --local_dir ./models/style_models flux1-redux-dev.safetensors!modelscope download --model=AI-ModelScope/FLUX.1-Fill-dev --local_dir ./models/diffusion_models/ flux1-fill-dev.safetensors!modelscope download --model=AI-ModelScope/FLUX.1-Canny-dev --local_dir ./models/diffusion_models/ flux1-canny-dev.safetensors!modelscope download --model=AI-ModelScope/FLUX.1-Depth-dev --local_dir ./models/diffusion_models/ flux1-depth-dev.safetensors!modelscope download --model=AI-ModelScope/FLUX.1-Canny-dev-lora --local_dir ./models/loras/ flux1-canny-dev-lora.safetensors!modelscope download --model=AI-ModelScope/FLUX.1-Depth-dev-lora --local_dir ./models/loras/ flux1-depth-dev-lora.safetensors!modelscope download --model=AI-ModelScope/sigclip_vision_384 --local_dir ./models/clip_vision/ sigclip_vision_patch14_384.safetensors
运行工作流,使用cloudflare运行工作流
#!wget "https://modelscope.oss-cn-beijing.aliyuncs.com/resource/cloudflared-linux-amd64.deb"!dpkg -i cloudflared-linux-amd64.deb %cd /mnt/workspace/ComfyUIimport subprocessimport threadingimport timeimport socketimport urllib.request def iframe_thread(port): while True: time.sleep(0.5) sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) result = sock.connect_ex(('127.0.0.1', port)) if result == 0: break sock.close() print("\nComfyUI finished loading, trying to launch cloudflared (if it gets stuck here cloudflared is having issues)\n") p = subprocess.Popen(["cloudflared", "tunnel", "--url", "http://127.0.0.1:{}".format(port)], stdout=subprocess.PIPE, stderr=subprocess.PIPE) for line in p.stderr: l = line.decode() if "trycloudflare.com " in l: print("This is the URL to access ComfyUI:", l[l.find("http"):], end='') #print(l, end='') threading.Thread(target=iframe_thread, daemon=True, args=(8188,)).start() !python main.py --dont-print-server
显存占用(fp8):
notebook分享地址:https://modelscope.cn/notebook/share/ipynb/39a1a470/comfyui-flux-tool.ipynb
工作流地址:
- https://comfyanonymous.github.io/ComfyUI_examples/flux/flux_fill_inpaint_example.png
- https://comfyanonymous.github.io/ComfyUI_examples/flux/flux_fill_outpaint_example.png
- https://comfyanonymous.github.io/ComfyUI_examples/flux/flux_redux_model_example.png
- https://comfyanonymous.github.io/ComfyUI_examples/flux/flux_canny_model_example.png
- https://comfyanonymous.github.io/ComfyUI_examples/flux/flux_depth_lora_example.png