Integrating ComfyUI & Local AI Inference into Nuke & Flow Production Tracking

Bridging Creative AI to Traditional VFX Pipelines

Generative AI and deep learning tools have exploded, but in most VFX facilities they remain siloed on an individual artist's gaming PC. Web interfaces like ComfyUI or WebUI are fantastic for experimentation, but they fail studio production requirements:

• They lack version control and audit trails.
• They do not adhere to studio OpenColorIO (ACEScg / linear) color spaces.
• They do not register passes or metadata into Autodesk Flow Production Tracking (ShotGrid).
• Models and prompts are not reproducible across a team.

Here is how we turn ComfyUI into an industrial, production-safe pipeline service.

Architecture: Headless ComfyUI Microservices

Instead of having artists manage Python virtual environments, CUDA drivers, and checkpoint checkpoints locally, we deploy Headless ComfyUI Microservice Workers on dedicated GPU nodes (either on-premise RTX 4090 / A6000 servers or AWS EC2 G5 instances).

[ Artist in Nuke ]
       │ Custom PySide Gizmo: Prompt, Mask & Parameters
       ▼
[ Pipeline Ingest API (FastAPI / Node.js) ]
       │ JSON Workflow Graph + OCIO Color Transform
       ▼
[ Headless ComfyUI Execution Daemon ]
       │ Dispatches to TensorRT / PyTorch Worker
       ▼
[ Output Processing & ShotGrid Register ]
       │ Generates EXR Multi-Channel + Shot Review Quicktime
       ▼
[ Result Loaded directly into Nuke Node Graph ]

Step 1: The Custom Nuke PySide Node

We built a custom Nuke Python Gizmo that mimics standard Nuke nodes. It provides:

• Input Pipe: Source Plate (RGB) and Roto/Mask (Alpha).
• Prompt & Denoising Controls: Clean UI sliders for CFG scale, seed, positive/negative prompts, and ControlNet strength (Depth, SoftEdge, Inpainting).
• Live Preview: Low-res preview in viewer within 1.5 seconds.
• Dispatch Button: Sends full 16-bit half-float frames to the farm worker.

# Example: Triggering Headless ComfyUI Node from Nuke Python API
import json
import urllib.request

def submit_nuke_comfy_job(node, prompt_text, denoise=0.45):
    plate_path = node.input(0).knob('file').value()
    frame = int(nuke.frame())
    
    payload = {
        "workflow": "vfx_plate_inpaint_v3",
        "parameters": {
            "source_image": f"{plate_path}.{frame:04d}.exr",
            "prompt": prompt_text,
            "denoise_strength": denoise,
            "output_format": "exr_acescg"
        }
    }
    
    req = urllib.request.Request(
        "http://ai-worker.internal.downzero.io:8188/api/submit",
        data=json.dumps(payload).encode('utf-8'),
        headers={'Content-Type': 'application/json'}
    )
    with urllib.request.urlopen(req) as resp:
        result = json.loads(resp.read().decode())
        return result['job_id']

Step 2: Automated Flow Production Tracking Registration

Every AI-assisted generation is logged as a distinct version in Autodesk Flow Production Tracking:

• The prompt, seed, model checkpoint hash, and LoRA weights are stored in custom fields.
• If a director asks for adjustments two weeks later, the exact state can be reproduced bit-for-bit.
• Client intellectual property is maintained safely within isolated VPC or local studio subnets.

Production Results

Artists no longer wait minutes or context-switch to separate web browsers. Matte painters and comp artists iterate on background extension variations, cleanup plates, and depth maps directly inside their primary toolset.