How Modly Integrates With ComfyUI for External Workflows: A Complete Guide

Modly integrates with ComfyUI through an experimental CLI layer that locates workflows, patches them with custom parameters, executes them on a live ComfyUI server, and routes the resulting images or 3-D assets into Modly's generation pipeline.

As the lightningpixel/modly repository evolves, developers increasingly need to bridge ComfyUI's powerful image generation graphs with Modly's 3-D mesh capabilities. This integration, though marked experimental, provides a stable mechanism for orchestrating complex pipelines without disrupting Modly's core API contract.

Loading and Locating ComfyUI Workflows

The integration begins with _load_comfy_workflow in tools/modly-cli/agent.py (lines 75-98). This function implements a tiered search strategy to find workflow JSON files:

• Default directories: User home, Documents, and Windows APPDATA
• Modly server: Remote JSON workflows fetched from the Modly backend

If multiple locations contain a workflow with the same name, Modly prioritizes local overrides, allowing rapid iteration without server round-trips.

Patching Workflows With Modly Parameters

Before execution, Modly modifies the raw ComfyUI graph through _patch_comfy_workflow (lines 101-135). This function performs targeted node manipulation:

• Text injection: Inserts prompt strings into ClipTextEncode nodes (or equivalent text input nodes)
• Seed rewriting: Updates random seed fields for reproducibility
• Preservation: Leaves all other graph topology intact

This surgical approach means existing ComfyUI workflows require zero modification to work with Modly—the CLI adapts them dynamically.

Executing Workflows on a Live ComfyUI Server

The _run_comfy_workflow function (lines 146-158) handles the runtime phase:

1. POST the patched JSON to http://127.0.0.1:8188/prompt
2. Extract the returned prompt_id
3. Poll /history/<prompt_id> until completion
4. Return the full history object containing outputs and metadata

The default polling behavior assumes a local ComfyUI instance, but the --comfy-url flag allows remote servers for distributed setups.

Extracting and Routing Outputs

Output handling diverges based on asset type in _download_comfy_image_output (lines 221-236) and related helpers:

Asset Type	Handler	Next Step
3-D file (.glb, .obj)	_download_comfy_ref	Report as final artifact
Image (.png, .jpg)	_download_comfy_image_output	Pass to _generate_one for image-to-3D conversion

This branching logic enables unified command interfaces regardless of whether the ComfyUI workflow terminates in raster images or mesh geometry.

CLI Commands for ComfyUI Integration

Two experimental sub-commands expose this functionality in tools/modly-cli/agent.py:

modly-cli experimental comfy-image

Retrieves the first image output from any ComfyUI workflow without triggering Modly's 3-D generation.

modly-cli experimental comfy-image \
  --workflow SimpleLandscape \
  --comfy-output /tmp/landscape.png \
  --comfy-url http://192.168.1.50:8188

Critical flags:

• --workflow <name|path>: Workflow identifier or direct JSON path
• --comfy-url: Override default 127.0.0.1:8188
• --comfy-output: Destination for downloaded image

modly-cli experimental generate-from-workflow

Full pipeline execution with automatic format detection.

# Workflow produces GLB directly

modly-cli experimental generate-from-workflow \
  --workflow Trellis2Workflow \
  --output result.glb \
  --prompt "A futuristic cityscape at sunset" \
  --seed 42

# Workflow produces image; Modly generates 3-D mesh

modly-cli experimental generate-from-workflow \
  --workflow SimplePortrait \
  --output result.glb \
  --timeout 300 \
  --poll 5

Additional flags:

• --timeout: Maximum wait seconds (default: 120)
• --poll: History polling interval in seconds
• --prompt / --seed: Override workflow defaults

Key Implementation Files

File	Purpose	Lines of Interest
tools/modly-cli/agent.py	Core integration logic	75-98 (loading), 101-135 (patching), 146-158 (execution), 221-236 (image extraction)
tools/modly-cli/SKILL.md	Experimental feature documentation	Entire file
README.md	High-level integration overview	Experimental ComfyUI Helpers section

Why This Integration Is Marked Experimental

According to the SKILL.md documentation in the lightningpixel/modly repository, the ComfyUI integration deliberately sits outside Modly's stability guarantees. This design choice achieves two objectives:

1. Core API protection: Breaking changes in ComfyUI's JSON format or REST API won't cascade into Modly's main contract
2. Rapid iteration: Power users can adopt advanced workflows immediately while the integration matures

The experimental status does not indicate instability in the current implementation—rather, it signals that command signatures and behavior may evolve based on community feedback.

Summary

• Workflow discovery spans local directories and Modly's server via _load_comfy_workflow
• Dynamic patching injects prompts and seeds without editing source JSON through _patch_comfy_workflow
• REST execution posts to ComfyUI's /prompt endpoint and polls /history via _run_comfy_workflow
• Smart routing sends 3-D assets directly to output or images through Modly's _generate_one path
• CLI exposure through comfy-image and generate-from-workflow sub-commands under the experimental namespace

Frequently Asked Questions

Does Modly require a local ComfyUI installation?

No. While the default --comfy-url points to 127.0.0.1:8188, any reachable ComfyUI server works. Remote GPU instances or containerized deployments are fully supported.

Can I use existing ComfyUI workflows without modification?

Yes. _patch_comfy_workflow performs runtime injection of prompts and seeds. The original JSON remains untouched, so workflows stay compatible with standalone ComfyUI usage.

What happens if a workflow produces multiple outputs?

The current implementation selects the first image asset for comfy-image and the first 3-D asset for generate-from-workflow. Subsequent outputs in the same history object are ignored.

Is the experimental status a stability concern?

The experimental designation primarily protects Modly's semantic versioning commitments. The agent.py implementation is production-tested for the supported feature set, but command names and flags may change in future releases before stabilization.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}