How to Use ControlNet in ComfyUI: Step-by-Step Guide

ControlNet is one of the most transformative additions to the Stable Diffusion workflow. While a standard text-to-image generation gives you creative guidance through your prompt, ControlNet gives you structural control — the ability to specify the exact pose of a figure, the depth of a scene, or the edge structure of a composition. In ComfyUI, ControlNet is implemented through a small set of dedicated nodes that slot neatly into any existing workflow. This guide explains what ControlNet does, how to set it up, and how to build your first controlled generation.

What ControlNet Does

ControlNet works by feeding an additional conditioning image into the diffusion process. This conditioning image is not a photo reference in the usual sense — it is a preprocessed map extracted from an image that captures one specific structural property. The three most common types are:

• Pose (OpenPose): Detects the skeleton of a human figure as a stick diagram. ComfyUI will generate a person in the same pose as the original.
• Depth: Creates a greyscale depth map showing which parts of a scene are near or far. Useful for maintaining spatial relationships between objects.
• Edge (Canny): Detects hard edges in an image. Useful for preserving the overall shape and structure of an object or scene without locking in colour or texture.

The key insight is that ControlNet constrains the structure of the output while still allowing the model’s generative ability — guided by your text prompt — to fill in style, colour, lighting, and texture.

Downloading ControlNet Models

ControlNet requires two things: a ControlNet model file that matches your base model, and (usually) a preprocessor to extract the conditioning map from your input image.

ControlNet model files for SD 1.5 are available from the Lllyasviel repository on Hugging Face (lllyasviel/ControlNet-v1-1). For SDXL, look for the xinsir or diffusers repositories on Hugging Face. Files typically end in .safetensors and are named to reflect their type, such as control_v11p_sd15_openpose.safetensors.

Download the model files that match the types you want to use — OpenPose, Canny, depth, and so on. You do not need all of them; start with one type and expand from there.

Folder Structure

Place your downloaded ControlNet model files in:

ComfyUI/models/controlnet/

Create the folder if it does not already exist. Restart ComfyUI or use the refresh function in ComfyUI Manager to make newly added models appear in the node dropdowns.

The ControlNet Nodes in ComfyUI

A ControlNet workflow in ComfyUI typically uses two core nodes:

Load ControlNet Model

Found under Add Node > loaders > Load ControlNet Model. This node has a single dropdown where you select your ControlNet .safetensors file. Its output is a CONTROL_NET object.

Apply ControlNet

Found under Add Node > conditioning > Apply ControlNet. This node takes the following inputs:

• positive — your positive conditioning from the CLIP Text Encode node
• negative — your negative conditioning
• control_net — the output of the Load ControlNet Model node
• image — the preprocessed control image (the pose map, depth map, or edge map)

The Apply ControlNet node outputs new positive and negative conditioning values, which replace the originals feeding into your KSampler.

Preprocessors: Turning an Image Into a Control Map

If you already have a preprocessed control image (for example, you drew a stick figure pose yourself), you can load it directly. In most cases, however, you will want to extract the control map automatically from a reference photo. This requires a preprocessor node.

Preprocessors are not built into core ComfyUI — they come from the ComfyUI-ControlNet-Auxiliary-Preprocessors extension (install it via ComfyUI Manager). Once installed, you will find nodes such as:

• OpenposePreprocessor — detects human pose keypoints
• CannyEdgePreprocessor — applies Canny edge detection
• MiDaSDepthMapPreprocessor — generates a depth map using the MiDaS model

Add the appropriate preprocessor node, connect your input image (via a Load Image node), and connect its output image into the Apply ControlNet node’s image input.

Building a Basic Pose-Controlled Workflow

1. Load your base checkpoint with Load Checkpoint.
2. Add Load Image and select your reference photo.
3. Add OpenposePreprocessor, connect Load Image’s output to it.
4. Add Load ControlNet Model, select your OpenPose ControlNet file.
5. Add Apply ControlNet, connect your positive and negative text conditioning, the ControlNet model, and the preprocessor’s output image.
6. Connect Apply ControlNet’s positive and negative outputs to your KSampler.
7. Run as normal.

The generated image should adopt the pose from your reference photo while reflecting your text prompt for style and subject.

Strength and End Percent Settings

The Apply ControlNet node includes strength and end_percent parameters. Strength controls how strictly the output adheres to the control map — a value of 1.0 enforces the structure rigidly, while 0.5 allows more creative deviation. Values between 0.7 and 0.9 are a good starting point for most use cases.

End percent determines at what point in the diffusion process ControlNet stops influencing the output. Setting this to 0.8 means ControlNet is active for the first 80% of steps, allowing the final 20% to refine naturally. This often produces cleaner, more detailed results than leaving it at the full 1.0.

SDXL vs SD 1.5 ControlNet Models

Just as with LoRAs, ControlNet models are base-model specific. SD 1.5 ControlNet models will not work correctly with SDXL checkpoints. SDXL ControlNet models are available but are fewer in number and generally require higher VRAM. Always match the ControlNet model to your checkpoint’s base model.

Common Issues

• No visible effect: Check that the Apply ControlNet outputs are connected to the KSampler, not the original text conditioning nodes. It is easy to leave the original connections in place.
• Wrong ControlNet model type: Using a depth model with a pose map will produce incorrect results. Match the preprocessor type to the ControlNet model type.
• VRAM errors: ControlNet adds to your VRAM usage. If you are running close to your GPU’s limit, reduce image resolution or use --lowvram when launching ComfyUI.
• Preprocessor nodes missing: Install ComfyUI-ControlNet-Auxiliary-Preprocessors via ComfyUI Manager.

For a full index of every ComfyUI guide on Serverman, see the ComfyUI complete guide and hub.