# How Modly Integrates With ComfyUI for External Workflows: A Complete Guide

> Learn how Modly integrates with ComfyUI for external workflows. Discover how Modly patches, executes, and routes ComfyUI workflows for seamless integration.

- Repository: [lightningpixel/modly](https://github.com/lightningpixel/modly)
- Tags: how-to-guide
- Published: 2026-08-20

---

**Modly integrates with ComfyUI through an experimental CLI layer that locates workflows, patches them with custom parameters, executes them on a live ComfyUI server, and routes the resulting images or 3-D assets into Modly's generation pipeline.**

As the `lightningpixel/modly` repository evolves, developers increasingly need to bridge ComfyUI's powerful image generation graphs with Modly's 3-D mesh capabilities. This integration, though marked **experimental**, provides a stable mechanism for orchestrating complex pipelines without disrupting Modly's core API contract.

## Loading and Locating ComfyUI Workflows

The integration begins with `_load_comfy_workflow` in [`tools/modly-cli/agent.py`](https://github.com/lightningpixel/modly/blob/main/tools/modly-cli/agent.py) (lines 75-98). This function implements a tiered search strategy to find workflow JSON files:

- **Default directories**: User home, `Documents`, and Windows `APPDATA`
- **Modly server**: Remote JSON workflows fetched from the Modly backend

If multiple locations contain a workflow with the same name, Modly prioritizes local overrides, allowing rapid iteration without server round-trips.

## Patching Workflows With Modly Parameters

Before execution, Modly modifies the raw ComfyUI graph through `_patch_comfy_workflow` (lines 101-135). This function performs targeted node manipulation:

- **Text injection**: Inserts prompt strings into `ClipTextEncode` nodes (or equivalent text input nodes)
- **Seed rewriting**: Updates random seed fields for reproducibility
- **Preservation**: Leaves all other graph topology intact

This surgical approach means existing ComfyUI workflows require zero modification to work with Modly—the CLI adapts them dynamically.

## Executing Workflows on a Live ComfyUI Server

The `_run_comfy_workflow` function (lines 146-158) handles the runtime phase:

1. POST the patched JSON to `http://127.0.0.1:8188/prompt`
2. Extract the returned `prompt_id`
3. Poll `/history/<prompt_id>` until completion
4. Return the full history object containing outputs and metadata

The default polling behavior assumes a local ComfyUI instance, but the `--comfy-url` flag allows remote servers for distributed setups.

## Extracting and Routing Outputs

Output handling diverges based on asset type in `_download_comfy_image_output` (lines 221-236) and related helpers:

| Asset Type | Handler | Next Step |
|------------|---------|-----------|
| 3-D file (`.glb`, `.obj`) | `_download_comfy_ref` | Report as final artifact |
| Image (`.png`, `.jpg`) | `_download_comfy_image_output` | Pass to `_generate_one` for image-to-3D conversion |

This branching logic enables unified command interfaces regardless of whether the ComfyUI workflow terminates in raster images or mesh geometry.

## CLI Commands for ComfyUI Integration

Two experimental sub-commands expose this functionality in [`tools/modly-cli/agent.py`](https://github.com/lightningpixel/modly/blob/main/tools/modly-cli/agent.py):

### `modly-cli experimental comfy-image`

Retrieves the first image output from any ComfyUI workflow without triggering Modly's 3-D generation.

```bash
modly-cli experimental comfy-image \
  --workflow SimpleLandscape \
  --comfy-output /tmp/landscape.png \
  --comfy-url http://192.168.1.50:8188

```

**Critical flags:**
- `--workflow <name|path>`: Workflow identifier or direct JSON path
- `--comfy-url`: Override default `127.0.0.1:8188`
- `--comfy-output`: Destination for downloaded image

### `modly-cli experimental generate-from-workflow`

Full pipeline execution with automatic format detection.

```bash

# Workflow produces GLB directly

modly-cli experimental generate-from-workflow \
  --workflow Trellis2Workflow \
  --output result.glb \
  --prompt "A futuristic cityscape at sunset" \
  --seed 42

# Workflow produces image; Modly generates 3-D mesh

modly-cli experimental generate-from-workflow \
  --workflow SimplePortrait \
  --output result.glb \
  --timeout 300 \
  --poll 5

```

**Additional flags:**
- `--timeout`: Maximum wait seconds (default: 120)
- `--poll`: History polling interval in seconds
- `--prompt` / `--seed`: Override workflow defaults

## Key Implementation Files

| File | Purpose | Lines of Interest |
|------|---------|-------------------|
| [`tools/modly-cli/agent.py`](https://github.com/lightningpixel/modly/blob/main/tools/modly-cli/agent.py) | Core integration logic | 75-98 (loading), 101-135 (patching), 146-158 (execution), 221-236 (image extraction) |
| [`tools/modly-cli/SKILL.md`](https://github.com/lightningpixel/modly/blob/main/tools/modly-cli/SKILL.md) | Experimental feature documentation | Entire file |
| [`README.md`](https://github.com/lightningpixel/modly/blob/main/README.md) | High-level integration overview | *Experimental ComfyUI Helpers* section |

## Why This Integration Is Marked Experimental

According to the [`SKILL.md`](https://github.com/lightningpixel/modly/blob/main/SKILL.md) documentation in the `lightningpixel/modly` repository, the ComfyUI integration deliberately sits outside Modly's stability guarantees. This design choice achieves two objectives:

1. **Core API protection**: Breaking changes in ComfyUI's JSON format or REST API won't cascade into Modly's main contract
2. **Rapid iteration**: Power users can adopt advanced workflows immediately while the integration matures

The experimental status does not indicate instability in the current implementation—rather, it signals that command signatures and behavior may evolve based on community feedback.

## Summary

- **Workflow discovery** spans local directories and Modly's server via `_load_comfy_workflow`
- **Dynamic patching** injects prompts and seeds without editing source JSON through `_patch_comfy_workflow`
- **REST execution** posts to ComfyUI's `/prompt` endpoint and polls `/history` via `_run_comfy_workflow`
- **Smart routing** sends 3-D assets directly to output or images through Modly's `_generate_one` path
- **CLI exposure** through `comfy-image` and `generate-from-workflow` sub-commands under the experimental namespace

## Frequently Asked Questions

### Does Modly require a local ComfyUI installation?

No. While the default `--comfy-url` points to `127.0.0.1:8188`, any reachable ComfyUI server works. Remote GPU instances or containerized deployments are fully supported.

### Can I use existing ComfyUI workflows without modification?

Yes. `_patch_comfy_workflow` performs runtime injection of prompts and seeds. The original JSON remains untouched, so workflows stay compatible with standalone ComfyUI usage.

### What happens if a workflow produces multiple outputs?

The current implementation selects the **first** image asset for `comfy-image` and the **first** 3-D asset for `generate-from-workflow`. Subsequent outputs in the same history object are ignored.

### Is the experimental status a stability concern?

The experimental designation primarily protects Modly's semantic versioning commitments. The [`agent.py`](https://github.com/lightningpixel/modly/blob/main/agent.py) implementation is production-tested for the supported feature set, but command names and flags may change in future releases before stabilization.