# How Modly Workflow Runs Track Status and Handle Cancellation

> Learn how Modly workflow runs track progress and handle cancellation using its in-memory job dictionary, global cancellation set, and threading Events for robust control. Abort generation with the cancel endpoint.

- Repository: [lightningpixel/modly](https://github.com/lightningpixel/modly)
- Tags: internals
- Published: 2026-08-15

---

**Modly uses an in-memory job dictionary (`_jobs`), a global cancellation set (`_cancelled`), and threading Events to track workflow run progress and allow clients to abort generation through the `/workflow-runs/{run_id}/cancel` endpoint.**

The `lightningpixel/modly` repository implements a lightweight, in-process job manager for 3D generation workflows. When clients initiate workflow runs via the API, the system tracks status through shared state containers defined in the generation router and exposes endpoints to query progress or cancel execution.

## Understanding Modly's Job Tracking Architecture

### The JobStatus Model and In-Memory Storage

Modly stores all active workflow runs in a global dictionary named `_jobs` defined in [`api/routers/generation.py`](https://github.com/lightningpixel/modly/blob/main/api/routers/generation.py) (lines 17‑20). Each entry maps a UUID to a **JobStatus** object that tracks `status`, `progress`, `step`, `output_url`, and `error` fields. This model, defined in [`schemas/generation.py`](https://github.com/lightningpixel/modly/blob/main/schemas/generation.py), serves as the single source of truth for run state across the application.

The system also maintains two parallel structures for lifecycle management: `_cancelled` (a set of run IDs marked for cancellation) and `_cancel_events` (a dictionary mapping run IDs to threading Events).

### UUID-Based Run Identification

Every workflow run receives a unique identifier generated upon creation. In [`api/routers/workflow_runs.py`](https://github.com/lightningpixel/modly/blob/main/api/routers/workflow_runs.py) (lines 54‑58), the `POST /workflow-runs/from-image` endpoint generates this UUID, instantiates a `JobStatus` object with initial status, and stores it in `_jobs` before returning the `run_id` to the client.

## How Modly Tracks Workflow Run Status

### Creating a Workflow Run

When a client submits an image to start generation, the router immediately returns a run ID while the actual processing occurs asynchronously. The background coroutine `_run_generation` in [`api/routers/generation.py`](https://github.com/lightningpixel/modly/blob/main/api/routers/generation.py) receives this ID and updates the shared job object via the `progress_cb` callback.

```python
import httpx, uuid, json

async def create_run():
    async with httpx.AsyncClient(base_url="http://localhost:8000") as client:
        with open("my_image.png", "rb") as f:
            files = {"image": ("my_image.png", f, "image/png")}
            data = {"model_id": "sf3d", "params": json.dumps({"remesh": "quad"})}
            resp = await client.post("/workflow-runs/from-image", files=files, data=data)
        return resp.json()["run_id"]

```

### Polling Status via GET /workflow-runs/{run_id}

The `get_run` function in [`api/routers/workflow_runs.py`](https://github.com/lightningpixel/modly/blob/main/api/routers/workflow_runs.py) (lines 65‑84) handles status retrieval. It queries the `_jobs` dictionary for the provided UUID, builds a `WorkflowRunStatus` model from the stored `JobStatus`, and returns the current state. If the job completed successfully, the response includes a `scene_candidate` field pointing to the generated workspace file.

```python
async def poll_status(client, run_id):
    while True:
        status = await client.get(f"/workflow-runs/{run_id}")
        info = status.json()
        print(f"Status: {info['status']}  Progress: {info['progress']}%")
        if info["status"] in ("done", "error", "cancelled"):
            break
        await asyncio.sleep(1)
    return info

```

### Real-Time Progress Updates

The `_run_generation` coroutine continuously updates the job object through the `progress_cb` callback defined in [`api/routers/generation.py`](https://github.com/lightningpixel/modly/blob/main/api/routers/generation.py) (lines 29‑34). This callback modifies `job.progress`, `job.step`, and `job.status` as the generation pipeline advances through stages such as image preprocessing, model inference, and mesh post-processing.

## Workflow Run Cancellation Mechanism

### The Three-Layer Cancellation Signal

The `cancel_run` function in [`api/routers/workflow_runs.py`](https://github.com/lightningpixel/modly/blob/main/api/routers/workflow_runs.py) (lines 86‑107) implements a robust cancellation strategy that operates on three levels:

1. **Set Membership** – Adds the `run_id` to the global `_cancelled` set for quick lookup
2. **Event Signaling** – Sets the threading Event stored in `_cancel_events` for the specific run ID
3. **Process Termination** – Kills the underlying generator subprocess via `gen._proc.kill()` and clears generator state

### Background Task Polling

The `_run_generation` function checks for cancellation signals at critical points in the execution flow (lines 60‑62 and 78‑92 in [`api/routers/generation.py`](https://github.com/lightningpixel/modly/blob/main/api/routers/generation.py)). Before and after the expensive model inference step, it verifies whether the run ID exists in `_cancelled` or whether the associated Event has been set. If detected, the function returns early and updates the job status to **cancelled**.

### Process Termination

For workflows utilizing external generators (such as Stable Fast 3D), the cancellation endpoint accesses the active generator instance through [`services/generator_registry.py`](https://github.com/lightningpixel/modly/blob/main/services/generator_registry.py). If `gen._proc` exists and is running, the system sends a kill signal to immediately terminate the subprocess, preventing resource waste on abandoned jobs.

```python
await client.post(f"/workflow-runs/{run_id}/cancel")

# Subsequent polls return: {"status": "cancelled"}

```

## Summary

- Modly stores workflow run state in an in-memory `_jobs` dictionary using the `JobStatus` model defined in [`schemas/generation.py`](https://github.com/lightningpixel/modly/blob/main/schemas/generation.py)
- The `GET /workflow-runs/{run_id}` endpoint queries this store via the `get_run` function in [`api/routers/workflow_runs.py`](https://github.com/lightningpixel/modly/blob/main/api/routers/workflow_runs.py)
- Cancellation works through a combination of set membership checks (`_cancelled`), threading Events (`_cancel_events`), and subprocess termination
- Background generation occurs in `_run_generation` within [`api/routers/generation.py`](https://github.com/lightningpixel/modly/blob/main/api/routers/generation.py), which polls cancellation signals before and after model inference
- Completed jobs are eventually purged by `_purge_old_jobs` to prevent memory leaks

## Frequently Asked Questions

### How does Modly store workflow run status internally?

Modly uses a global dictionary named `_jobs` defined in [`api/routers/generation.py`](https://github.com/lightningpixel/modly/blob/main/api/routers/generation.py) that maps UUIDs to `JobStatus` objects. These objects track `status`, `progress`, `step`, `output_url`, and error information in-memory. This approach provides low-latency status updates without requiring an external database.

### What happens to the subprocess when I cancel a workflow run?

The `POST /workflow-runs/{run_id}/cancel` endpoint kills the generator subprocess via `gen._proc.kill()` if it exists, sets a threading Event for the specific run ID, and adds the ID to a global cancellation set. This three-layer approach ensures that even if the subprocess termination fails, the background task will detect the cancellation signal and terminate gracefully.

### Can I poll workflow status from multiple clients simultaneously?

Yes, because Modly stores job status in shared memory (`_jobs`), any client can query `GET /workflow-runs/{run_id}` to receive the current state. The endpoint simply reads from the shared dictionary and builds a `WorkflowRunStatus` model, making the system inherently safe for concurrent status polling across multiple HTTP connections.

### What status values can a workflow run have?

According to the source code in [`schemas/generation.py`](https://github.com/lightningpixel/modly/blob/main/schemas/generation.py) and [`api/routers/generation.py`](https://github.com/lightningpixel/modly/blob/main/api/routers/generation.py), jobs typically progress through states including `pending`, `running`, `done`, `error`, and `cancelled`. The `_run_generation` coroutine updates these values via the `progress_cb` callback as the workflow advances through its pipeline stages.