# How the Modly Process Runner Manages Extension Subprocesses

> Learn how Modly manages extension subprocesses by isolating them in Python virtual environments and communicating via JSON. Discover the ExtensionProcess class and runner.py in this technical breakdown.

- Repository: [lightningpixel/modly](https://github.com/lightningpixel/modly)
- Tags: internals
- Published: 2026-08-15

---

**Modly isolates each generator extension in its own Python virtual environment and communicates via a lightweight newline-delimited JSON protocol, orchestrated by the `ExtensionProcess` class in [`api/services/extension_process.py`](https://github.com/lightningpixel/modly/blob/main/api/services/extension_process.py) and the [`runner.py`](https://github.com/lightningpixel/modly/blob/main/runner.py) entry point.**

The open-source Modly project (lightningpixel/modly) provides a robust sandbox for executing third-party generator extensions without compromising the stability of the main application. By delegating extension execution to subprocesses rather than loading code directly into the host process, the **Modly process runner** shields the UI and workspace from crashes, memory leaks, and dependency conflicts.

## Architecture of the Subprocess Isolation

The subprocess system relies on two cooperating components. The parent process uses `ExtensionProcess` (defined in [`api/services/extension_process.py`](https://github.com/lightningpixel/modly/blob/main/api/services/extension_process.py)) to manage the lifecycle and communication channel, while the child process executes [`api/runner.py`](https://github.com/lightningpixel/modly/blob/main/api/runner.py) inside the extension's dedicated virtual environment.

`ExtensionProcess` implements the `BaseGenerator` interface from [`api/services/generators/base.py`](https://github.com/lightningpixel/modly/blob/main/api/services/generators/base.py), allowing the `GeneratorRegistry` and the rest of the application to treat subprocess generators identically to in-process ones. This abstraction ensures that the UI and HTTP API remain agnostic to whether a generator runs in-process or in a sandboxed subprocess.

## Subprocess Startup Flow

When instantiating a new extension, `ExtensionProcess` executes a rigorous five-stage initialization sequence to establish the isolated environment.

### Environment Construction

The `_build_env` method clones the parent process's environment and injects extension-specific variables required by [`runner.py`](https://github.com/lightningpixel/modly/blob/main/runner.py):

- `EXTENSION_DIR`: Root directory of the extension
- `MODELS_DIR`: Global storage path for shared models
- `WORKSPACE_DIR`: User workspace for generated outputs
- `MODLY_API_DIR`: Path to Modly's internal API modules
- `MODEL_DIR`: Optional extension-specific model directory

For embedded Python distributions lacking a CA bundle, the method also injects `SSL_CERT_FILE` to ensure HTTPS functionality within the subprocess.

### Cross-Platform Python Resolution

The `_venv_python` property ensures the subprocess uses the correct interpreter across platforms. It returns `venv/bin/python` on Unix-like systems and `venv/Scripts/python.exe` on Windows, guaranteeing execution within the isolated virtual environment regardless of the host operating system.

### Process Launch and Pipe Configuration

The runner invokes `subprocess.Popen` using the resolved Python executable with [`api/runner.py`](https://github.com/lightningpixel/modly/blob/main/api/runner.py) as the target script. The process configuration pipes `stdin`, `stdout`, and `stderr` to enable bidirectional JSON communication without network sockets or shared memory dependencies.

### Asynchronous I/O Threads

Two daemon threads handle non-blocking communication:

- `_read_loop`: Continuously reads lines from the subprocess stdout, parses them as JSON, and pushes dictionaries into a thread-safe `queue.Queue`
- `_stderr_loop`: Forwards stderr streams to the parent process, handling both `\n` and `\r` characters to preserve interactive progress bars (such as `tqdm` output)

### Ready Handshake Protocol

After launching, the parent blocks until receiving a message of type `ready` from the child. This JSON payload contains the generator's `params_schema`, which `ExtensionProcess` caches to serve subsequent schema queries without reloading the subprocess.

## JSON Command Protocol

Once initialized, [`runner.py`](https://github.com/lightningpixel/modly/blob/main/runner.py) enters a command loop listening on stdin for structured JSON commands. Each command maps to specific methods on the generator class:

**load**: Invokes the generator's `load()` method and replies with `{"type": "loaded"}`.

**generate**: Decodes the base64-encoded input image, creates a per-request `threading.Event` for cancellation signaling, and calls `generator.generate()`. During execution, the runner streams progress updates as `{"type": "progress", "pct": ..., "step": ...}`. Upon completion, it sends either `{"type": "done", "output_path": "..."}` or `{"type": "error", "message": "...", "traceback": "..."}`.

**cancel**: Sets the cancellation event for the matching request ID. If the subprocess does not terminate within approximately three seconds, `ExtensionProcess` force-kills the process and raises `GenerationCancelled` to the caller.

**unload**: Calls `unload()` on the generator instance and acknowledges with `{"type": "unloaded"}`.

**shutdown**: Executes `unload()`, then breaks the main loop to exit the subprocess cleanly.

## Resilience and Error Recovery

The Modly process runner implements sophisticated mechanisms to handle unreliable or malformed extension code.

### Automatic Dependency Repair

When [`runner.py`](https://github.com/lightningpixel/modly/blob/main/runner.py) fails to start due to an `ImportError`, the `_extract_missing_module` function parses the traceback to identify the missing module (e.g., mapping `PIL` to `Pillow`). `ExtensionProcess` then installs the corresponding package via pip inside the extension's virtual environment and retries the launch automatically.

### Graceful Cancellation with Timeout

Cancellation follows a two-phase approach. First, the parent sends a cancel command and waits up to three seconds for the generator to acknowledge and exit cleanly. If the subprocess remains unresponsive, `ExtensionProcess` issues a hard kill (`SIGKILL` on Unix, `TerminateProcess` on Windows), drains the internal message queue to prevent stale messages from contaminating future runs, and propagates a `GenerationCancelled` exception.

## Integration with the Generator Registry

The `GeneratorRegistry` in [`api/services/generator_registry.py`](https://github.com/lightningpixel/modly/blob/main/api/services/generator_registry.py) discovers available extensions and determines execution mode based on the presence of a virtual environment or [`build_vendor.py`](https://github.com/lightningpixel/modly/blob/main/build_vendor.py) script. When subprocess mode is selected, the registry wraps the extension in an `ExtensionProcess` instance.

Because `ExtensionProcess` exposes the standard `BaseGenerator` methods—`load`, `unload`, `generate`, `params_schema`, and `stop`—the rest of Modly interacts with subprocess extensions using identical signatures to in-process generators, completely abstracting away the complexity of inter-process communication.

## Working with Extension Subprocesses

The following examples demonstrate common interactions with the Modly process runner.

Launching an extension subprocess:

```python
from services.extension_process import ExtensionProcess
from pathlib import Path
import json

ext_dir = Path("/path/to/extensions/my_extension")
manifest = json.loads((ext_dir / "manifest.json").read_text())
proc = ExtensionProcess(ext_dir, manifest)

proc.model_dir = Path("/home/user/.modly/models/my_extension")
proc.outputs_dir = Path("/home/user/.modly/workspace")
proc.load()  # Starts subprocess and loads the model

```

Generating with progress callbacks:

```python
def progress(pct, step):
    print(f"{pct}% – {step}")

with open("input.png", "rb") as f:
    img_bytes = f.read()

params = {"prompt": "a futuristic car", "steps": 30}
output_path = proc.generate(img_bytes, params, progress_cb=progress)
print("Generated mesh saved to:", output_path)

```

Cancelling a long-running generation:

```python
import threading

cancel_evt = threading.Event()
thread = threading.Thread(
    target=lambda: proc.generate(img_bytes, params, cancel_event=cancel_evt)
)
thread.start()

# ... after some time ...

cancel_evt.set()  # Requests cancellation; hard-kill follows if needed

thread.join()

```

## Summary

- Modly isolates extensions using Python virtual environments and subprocesses managed by `ExtensionProcess` in [`api/services/extension_process.py`](https://github.com/lightningpixel/modly/blob/main/api/services/extension_process.py).
- Communication occurs via newline-delimited JSON over stdin/stdout, with [`api/runner.py`](https://github.com/lightningpixel/modly/blob/main/api/runner.py) serving as the subprocess entry point.
- The startup sequence includes environment variable injection, cross-platform Python resolution via `_venv_python`, and a ready handshake that caches the generator schema.
- Commands include `load`, `generate`, `cancel`, `unload`, and `shutdown`, with cooperative cancellation and automatic hard-kill timeouts after approximately three seconds.
- Missing dependencies are auto-detected by `_extract_missing_module` and installed via pip before retrying failed launches.
- The `GeneratorRegistry` treats subprocess instances as standard `BaseGenerator` implementations, providing seamless integration with the Modly UI and HTTP API.

## Frequently Asked Questions

### How does Modly handle missing Python packages in extension subprocesses?

When [`runner.py`](https://github.com/lightningpixel/modly/blob/main/runner.py) crashes on startup with an `ImportError`, the `_extract_missing_module` function parses the traceback to map the missing import to a known pip package name. `ExtensionProcess` automatically installs the dependency inside the extension's virtual environment using pip, then retries the subprocess launch without requiring manual intervention.

### What protocol does Modly use to communicate with extension subprocesses?

Modly uses a lightweight, newline-delimited JSON protocol transmitted over standard input and output streams. The parent process sends command objects (such as `{"cmd": "generate", ...}`) to the child's stdin, and the child responds with structured JSON messages to stdout, including types like `ready`, `progress`, `done`, `error`, and `loaded`.

### How does Modly ensure cross-platform compatibility for the Python interpreter path?

The `_venv_python` property in `ExtensionProcess` dynamically resolves the correct interpreter path based on the operating system. It returns `venv/bin/python` for Unix-like systems and `venv/Scripts/python.exe` for Windows, ensuring the subprocess always executes within the correct virtual environment regardless of platform.

### What happens if an extension subprocess hangs during generation?

If a generation task does not respond to a cancellation request within approximately three seconds, `ExtensionProcess` terminates the subprocess forcefully. It then drains the internal message queue to remove stale messages, clears the internal state, and raises a `GenerationCancelled` exception to notify the caller that the operation was aborted.