How the Modly Process Runner Manages Extension Subprocesses

Modly isolates each generator extension in its own Python virtual environment and communicates via a lightweight newline-delimited JSON protocol, orchestrated by the ExtensionProcess class in api/services/extension_process.py and the runner.py entry point.

The open-source Modly project (lightningpixel/modly) provides a robust sandbox for executing third-party generator extensions without compromising the stability of the main application. By delegating extension execution to subprocesses rather than loading code directly into the host process, the Modly process runner shields the UI and workspace from crashes, memory leaks, and dependency conflicts.

Architecture of the Subprocess Isolation

The subprocess system relies on two cooperating components. The parent process uses ExtensionProcess (defined in api/services/extension_process.py) to manage the lifecycle and communication channel, while the child process executes api/runner.py inside the extension's dedicated virtual environment.

ExtensionProcess implements the BaseGenerator interface from api/services/generators/base.py, allowing the GeneratorRegistry and the rest of the application to treat subprocess generators identically to in-process ones. This abstraction ensures that the UI and HTTP API remain agnostic to whether a generator runs in-process or in a sandboxed subprocess.

Subprocess Startup Flow

When instantiating a new extension, ExtensionProcess executes a rigorous five-stage initialization sequence to establish the isolated environment.

Environment Construction

The _build_env method clones the parent process's environment and injects extension-specific variables required by runner.py:

• EXTENSION_DIR: Root directory of the extension
• MODELS_DIR: Global storage path for shared models
• WORKSPACE_DIR: User workspace for generated outputs
• MODLY_API_DIR: Path to Modly's internal API modules
• MODEL_DIR: Optional extension-specific model directory

For embedded Python distributions lacking a CA bundle, the method also injects SSL_CERT_FILE to ensure HTTPS functionality within the subprocess.

Cross-Platform Python Resolution

The _venv_python property ensures the subprocess uses the correct interpreter across platforms. It returns venv/bin/python on Unix-like systems and venv/Scripts/python.exe on Windows, guaranteeing execution within the isolated virtual environment regardless of the host operating system.

Process Launch and Pipe Configuration

The runner invokes subprocess.Popen using the resolved Python executable with api/runner.py as the target script. The process configuration pipes stdin, stdout, and stderr to enable bidirectional JSON communication without network sockets or shared memory dependencies.

Asynchronous I/O Threads

Two daemon threads handle non-blocking communication:

• _read_loop: Continuously reads lines from the subprocess stdout, parses them as JSON, and pushes dictionaries into a thread-safe queue.Queue
• _stderr_loop: Forwards stderr streams to the parent process, handling both \n and \r characters to preserve interactive progress bars (such as tqdm output)

Ready Handshake Protocol

After launching, the parent blocks until receiving a message of type ready from the child. This JSON payload contains the generator's params_schema, which ExtensionProcess caches to serve subsequent schema queries without reloading the subprocess.

JSON Command Protocol

Once initialized, runner.py enters a command loop listening on stdin for structured JSON commands. Each command maps to specific methods on the generator class:

load: Invokes the generator's load() method and replies with {"type": "loaded"}.

generate: Decodes the base64-encoded input image, creates a per-request threading.Event for cancellation signaling, and calls generator.generate(). During execution, the runner streams progress updates as {"type": "progress", "pct": ..., "step": ...}. Upon completion, it sends either {"type": "done", "output_path": "..."} or {"type": "error", "message": "...", "traceback": "..."}.

cancel: Sets the cancellation event for the matching request ID. If the subprocess does not terminate within approximately three seconds, ExtensionProcess force-kills the process and raises GenerationCancelled to the caller.

unload: Calls unload() on the generator instance and acknowledges with {"type": "unloaded"}.

shutdown: Executes unload(), then breaks the main loop to exit the subprocess cleanly.

Resilience and Error Recovery

The Modly process runner implements sophisticated mechanisms to handle unreliable or malformed extension code.

Automatic Dependency Repair

When runner.py fails to start due to an ImportError, the _extract_missing_module function parses the traceback to identify the missing module (e.g., mapping PIL to Pillow). ExtensionProcess then installs the corresponding package via pip inside the extension's virtual environment and retries the launch automatically.

Graceful Cancellation with Timeout

Cancellation follows a two-phase approach. First, the parent sends a cancel command and waits up to three seconds for the generator to acknowledge and exit cleanly. If the subprocess remains unresponsive, ExtensionProcess issues a hard kill (SIGKILL on Unix, TerminateProcess on Windows), drains the internal message queue to prevent stale messages from contaminating future runs, and propagates a GenerationCancelled exception.

Integration with the Generator Registry

The GeneratorRegistry in api/services/generator_registry.py discovers available extensions and determines execution mode based on the presence of a virtual environment or build_vendor.py script. When subprocess mode is selected, the registry wraps the extension in an ExtensionProcess instance.

Because ExtensionProcess exposes the standard BaseGenerator methods—load, unload, generate, params_schema, and stop—the rest of Modly interacts with subprocess extensions using identical signatures to in-process generators, completely abstracting away the complexity of inter-process communication.

Working with Extension Subprocesses

The following examples demonstrate common interactions with the Modly process runner.

Launching an extension subprocess:

from services.extension_process import ExtensionProcess
from pathlib import Path
import json

ext_dir = Path("/path/to/extensions/my_extension")
manifest = json.loads((ext_dir / "manifest.json").read_text())
proc = ExtensionProcess(ext_dir, manifest)

proc.model_dir = Path("/home/user/.modly/models/my_extension")
proc.outputs_dir = Path("/home/user/.modly/workspace")
proc.load()  # Starts subprocess and loads the model

Generating with progress callbacks:

def progress(pct, step):
    print(f"{pct}% – {step}")

with open("input.png", "rb") as f:
    img_bytes = f.read()

params = {"prompt": "a futuristic car", "steps": 30}
output_path = proc.generate(img_bytes, params, progress_cb=progress)
print("Generated mesh saved to:", output_path)

Cancelling a long-running generation:

import threading

cancel_evt = threading.Event()
thread = threading.Thread(
    target=lambda: proc.generate(img_bytes, params, cancel_event=cancel_evt)
)
thread.start()

# ... after some time ...

cancel_evt.set()  # Requests cancellation; hard-kill follows if needed

thread.join()

Summary

• Modly isolates extensions using Python virtual environments and subprocesses managed by ExtensionProcess in api/services/extension_process.py.
• Communication occurs via newline-delimited JSON over stdin/stdout, with api/runner.py serving as the subprocess entry point.
• The startup sequence includes environment variable injection, cross-platform Python resolution via _venv_python, and a ready handshake that caches the generator schema.
• Commands include load, generate, cancel, unload, and shutdown, with cooperative cancellation and automatic hard-kill timeouts after approximately three seconds.
• Missing dependencies are auto-detected by _extract_missing_module and installed via pip before retrying failed launches.
• The GeneratorRegistry treats subprocess instances as standard BaseGenerator implementations, providing seamless integration with the Modly UI and HTTP API.

Frequently Asked Questions

How does Modly handle missing Python packages in extension subprocesses?

When runner.py crashes on startup with an ImportError, the _extract_missing_module function parses the traceback to map the missing import to a known pip package name. ExtensionProcess automatically installs the dependency inside the extension's virtual environment using pip, then retries the subprocess launch without requiring manual intervention.

What protocol does Modly use to communicate with extension subprocesses?

Modly uses a lightweight, newline-delimited JSON protocol transmitted over standard input and output streams. The parent process sends command objects (such as {"cmd": "generate", ...}) to the child's stdin, and the child responds with structured JSON messages to stdout, including types like ready, progress, done, error, and loaded.

How does Modly ensure cross-platform compatibility for the Python interpreter path?

The _venv_python property in ExtensionProcess dynamically resolves the correct interpreter path based on the operating system. It returns venv/bin/python for Unix-like systems and venv/Scripts/python.exe for Windows, ensuring the subprocess always executes within the correct virtual environment regardless of platform.

What happens if an extension subprocess hangs during generation?

If a generation task does not respond to a cancellation request within approximately three seconds, ExtensionProcess terminates the subprocess forcefully. It then drains the internal message queue to remove stale messages, clears the internal state, and raises a GenerationCancelled exception to notify the caller that the operation was aborted.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}