# How Modly’s Python Subprocess Management Handles Startup, Health Checks, and Termination

> Discover how Modly's Python subprocess management ensures reliable startup, health checks, and graceful termination with its ExtensionProcess class for robust application development.

- Repository: [lightningpixel/modly](https://github.com/lightningpixel/modly)
- Tags: internals
- Published: 2026-08-15

---

**Modly uses a dedicated `ExtensionProcess` class that launches isolated Python subprocesses, validates startup health via a JSON "ready" handshake, and enforces clean termination through graceful unload requests, hard SIGKILL stops, and timeout-protected cancellation logic.**

Modly's architecture runs every extension (custom models or processing scripts) inside an isolated virtual-environment subprocess. This design prevents dependency conflicts and provides fine-grained lifecycle control. The `ExtensionProcess` class in [`api/services/extension_process.py`](https://github.com/lightningpixel/modly/blob/main/api/services/extension_process.py) orchestrates the complete subprocess management pipeline—from environment setup to guaranteed cleanup.

## ExtensionProcess Architecture

The `ExtensionProcess` class serves as the primary interface for Python subprocess management. Each instance encapsulates a running extension, providing methods for startup, communication, health monitoring, and termination.

Key responsibilities include:

- Building isolated virtual environments per extension
- Spawning subprocesses with `subprocess.Popen`
- Parsing JSON messages from the child's stdout via a background reader thread
- Implementing auto-repair for missing dependencies
- Providing multiple termination strategies with graceful degradation

## Subprocess Startup and Health Checks

Modly's startup protocol ensures an extension is fully loaded and functional before marking it as ready for use.

### Environment Construction and Launch

The `_start()` method (lines 94–135) constructs the execution context:

```python

# Simplified excerpt showing core launch logic

env = os.environ.copy()
env["PYTHONPATH"] = self._build_python_path(
    ext_dir=self.ext_dir,
    model_dir=self.model_dir,
    workspace=self.workspace
)

self._proc = subprocess.Popen(
    [sys.executable, "-m", "api.runner"],
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,
    env=env,
    cwd=str(self.ext_dir)
)

```

The runner module at [`api/runner.py`](https://github.com/lightningpixel/modly/blob/main/api/runner.py) executes inside this subprocess and becomes responsible for loading the extension's generator and emitting status messages.

### Asynchronous Message Reading

Immediately after launch, `_start()` spawns a **reader thread** (lines 84–95) that continuously reads from the child's stdout:

```python
def _reader_thread():
    for line in iter(self._proc.stdout.readline, b""):
        try:
            msg = json.loads(line.decode("utf-8"))
            self._queue.put(msg)
        except json.JSONDecodeError:
            # Log malformed lines but continue reading

            self._logger.warning(f"Malformed JSON from runner: {line!r}")

```

This thread pushes parsed JSON objects onto a `queue.Queue`, decoupling I/O from the main control flow and preventing pipe buffer deadlocks.

### The Ready Handshake

The startup health check blocks on receiving a `"type": "ready"` message (lines 123–133):

```python
def _start(self):
    # ... launch subprocess and reader thread ...

    
    ready_msg = self._recv(timeout=30.0)  # blocks here

    if ready_msg.get("type") != "ready":
        raise ExtensionStartupError(f"Unexpected first message: {ready_msg}")
    
    self._params_schema = ready_msg["params_schema"]
    self._loaded = True
    self._logger.info(f"Extension ready (PID {self._proc.pid})")

```

The `params_schema` contained in this message describes the generator's configurable parameters, enabling the parent to validate subsequent generation requests.

### Auto-Repair for Missing Dependencies

If startup fails due to an `ImportError`, Modly attempts **automatic recovery** (lines 135–141):

```python
if "No module named" in str(e):
    missing_module = self._extract_missing_module(e)
    package = self._AUTO_REPAIR_PACKAGE_MAP.get(missing_module)
    
    if package and self._repair_attempts < 3:
        self._install_package(package)  # pip install in extension venv

        self._repair_attempts += 1
        return self._start()  # retry

```

The `_AUTO_REPAIR_PACKAGE_MAP` maps common missing modules (e.g., `numpy`, `torch`, `trimesh`) to their PyPI package names. This self-healing mechanism reduces manual intervention for extensions with undeclared dependencies.

## Subprocess Termination Strategies

Modly implements **three distinct termination pathways** to handle cooperative shutdowns, forced cleanup, and stuck operations.

### Graceful Unload

The `unload()` method (lines 66–75) attempts cooperative termination:

```python
def unload(self) -> None:
    if not self._loaded or self._proc is None:
        return  # idempotent no-op

    
    self._send({"action": "unload"})
    self._recv(timeout=5.0)  # wait for acknowledgment

    self._loaded = False
    self._logger.info("Extension unloaded gracefully")

```

This sends an `"unload"` JSON command to the child, expecting a response before the subprocess exits voluntarily. The method is idempotent—calling it on an already-dead process has no effect.

### Hard Stop

The `stop()` method (lines 66–83) provides **immediate termination**:

```python
def stop(self) -> None:
    if self._proc is None:
        return
    
    self._proc.kill()  # SIGKILL on Unix, TerminateProcess on Windows

    self._proc.wait(timeout=5.0)
    
    self._drain_queue()  # prevent stale messages

    self._proc = None
    self._loaded = False
    self._logger.info("Extension process killed")

```

Hard stops are used when the UI needs to reclaim memory or when a cooperative shutdown is impossible. The 5-second wait timeout prevents indefinite blocking, and `_drain_queue()` clears any pending messages to avoid contamination of future extension instances.

### Generation Cancellation with Grace Period

The most complex termination scenario occurs during **active generation**. The `generate()` method implements a two-phase cancellation (lines 94–124 and 154–165):

```python
def generate(self, params: dict) -> Iterator[bytes]:
    self._ensure_started()
    self._send({"action": "generate", "params": params})
    
    try:
        for msg in self._message_stream():
            if msg["type"] == "chunk":
                yield base64.b64decode(msg["data"])
            elif msg["type"] == "done":
                return
    except GenerationCancelled:
        # Cancellation requested from outside

        self._send({"action": "cancel"})
        
        # Wait for acknowledgment with timeout

        try:
            self._recv(timeout=3.0)  # grace period

        except TimeoutError:
            self._logger.warning("Cancel grace period expired, forcing kill")
            self.stop()  # hard kill to break stuck native calls

        
        raise

```

The **3-second grace period** protects against hung native operations—common in geometric processing like Marching Cubes implementations that don't check Python signals. If the child fails to acknowledge cancellation within this window, `stop()` terminates it unconditionally.

## CLI Health Check Pattern

The Modly CLI ([`tools/modly-cli/agent.py`](https://github.com/lightningpixel/modly/blob/main/tools/modly-cli/agent.py)) extends this subprocess management pattern to the backend server itself:

```python

# Simplified from agent.py

backend_proc = subprocess.Popen(
    [sys.executable, "-m", "api.main"],
    env=build_backend_env()
)

# Poll HTTP health endpoint instead of JSON stdout

for attempt in range(max_retries):
    try:
        resp = requests.get("http://localhost:8000/status/health", timeout=2)
        if resp.status_code == 200:
            break  # backend healthy

    except requests.ConnectionError:
        time.sleep(poll_interval)
else:
    backend_proc.kill()
    raise BackendStartupError("Health check failed after maximum retries")

```

This HTTP-based health check mirrors the extension ready-message mechanism, demonstrating architectural consistency across Modly's subprocess boundaries.

## Key Implementation Files

| Path | Purpose |
|------|---------|
| [`api/services/extension_process.py`](https://github.com/lightningpixel/modly/blob/main/api/services/extension_process.py) | Core `ExtensionProcess` class with startup, health checks, and all termination pathways |
| [`api/runner.py`](https://github.com/lightningpixel/modly/blob/main/api/runner.py) | Subprocess entry point; loads generators and emits the `"ready"` message |
| [`tools/modly-cli/agent.py`](https://github.com/lightningpixel/modly/blob/main/tools/modly-cli/agent.py) | CLI subprocess management and HTTP health polling for backend server |
| [`api/routers/status.py`](https://github.com/lightningpixel/modly/blob/main/api/routers/status.py) | FastAPI health check endpoint (`/status/health`) used by CLI |

## Summary

- **Startup**: `ExtensionProcess._start()` launches subprocesses with environment isolation, validates health through a blocking JSON "ready" handshake, and auto-repairs missing dependencies up to three attempts
- **Health Monitoring**: A background reader thread consumes stdout as a JSON stream, using `queue.Queue` for thread-safe message passing
- **Graceful Termination**: `unload()` sends cooperative `"unload"` commands with acknowledgment timeout
- **Forced Termination**: `stop()` issues SIGKILL, waits 5 seconds, and drains message queue to prevent state contamination
- **Protected Cancellation**: Active generations can be cancelled with a 3-second grace period before automatic hard-kill, preventing indefinite hangs in native code

## Frequently Asked Questions

### How does Modly prevent extension dependency conflicts?

Each extension runs in its own Python virtual environment with an isolated `PYTHONPATH`. The `ExtensionProcess` constructs environment variables in `_start()` that prioritize extension-local packages over system packages, ensuring version-specific requirements don't collide.

### What happens if an extension hangs during generation?

The cancellation protocol in `generate()` first sends a `"cancel"` command and waits 3 seconds. If no acknowledgment arrives—indicating the child is stuck in uninterruptible native code—the parent automatically executes `stop()` to SIGKILL the process and raise `GenerationCancelled` to the caller.

### Can the auto-repair mechanism handle any import error?

No. Auto-repair only triggers for `ImportError` messages matching the pattern `No module named {module}` where `{module}` exists in the internal `_AUTO_REPAIR_PACKAGE_MAP`. Unknown modules or non-import startup failures propagate as `ExtensionStartupError` without automatic retry.

### Why does Modly use stdout for health checks instead of stderr or a socket?

Stdout provides a simple, portable communication channel that works without additional dependencies or firewall considerations. The reader thread approach with JSON line protocol ensures structured messaging while remaining compatible with standard Python logging to stderr—keeping diagnostic output separate from control messages.