How Modly Workflow Runs Track Status and Handle Cancellation

Modly uses an in-memory job dictionary (_jobs), a global cancellation set (_cancelled), and threading Events to track workflow run progress and allow clients to abort generation through the /workflow-runs/{run_id}/cancel endpoint.

The lightningpixel/modly repository implements a lightweight, in-process job manager for 3D generation workflows. When clients initiate workflow runs via the API, the system tracks status through shared state containers defined in the generation router and exposes endpoints to query progress or cancel execution.

Understanding Modly's Job Tracking Architecture

The JobStatus Model and In-Memory Storage

Modly stores all active workflow runs in a global dictionary named _jobs defined in api/routers/generation.py (lines 17‑20). Each entry maps a UUID to a JobStatus object that tracks status, progress, step, output_url, and error fields. This model, defined in schemas/generation.py, serves as the single source of truth for run state across the application.

The system also maintains two parallel structures for lifecycle management: _cancelled (a set of run IDs marked for cancellation) and _cancel_events (a dictionary mapping run IDs to threading Events).

UUID-Based Run Identification

Every workflow run receives a unique identifier generated upon creation. In api/routers/workflow_runs.py (lines 54‑58), the POST /workflow-runs/from-image endpoint generates this UUID, instantiates a JobStatus object with initial status, and stores it in _jobs before returning the run_id to the client.

How Modly Tracks Workflow Run Status

Creating a Workflow Run

When a client submits an image to start generation, the router immediately returns a run ID while the actual processing occurs asynchronously. The background coroutine _run_generation in api/routers/generation.py receives this ID and updates the shared job object via the progress_cb callback.

import httpx, uuid, json

async def create_run():
    async with httpx.AsyncClient(base_url="http://localhost:8000") as client:
        with open("my_image.png", "rb") as f:
            files = {"image": ("my_image.png", f, "image/png")}
            data = {"model_id": "sf3d", "params": json.dumps({"remesh": "quad"})}
            resp = await client.post("/workflow-runs/from-image", files=files, data=data)
        return resp.json()["run_id"]

Polling Status via GET /workflow-runs/{run_id}

The get_run function in api/routers/workflow_runs.py (lines 65‑84) handles status retrieval. It queries the _jobs dictionary for the provided UUID, builds a WorkflowRunStatus model from the stored JobStatus, and returns the current state. If the job completed successfully, the response includes a scene_candidate field pointing to the generated workspace file.

async def poll_status(client, run_id):
    while True:
        status = await client.get(f"/workflow-runs/{run_id}")
        info = status.json()
        print(f"Status: {info['status']}  Progress: {info['progress']}%")
        if info["status"] in ("done", "error", "cancelled"):
            break
        await asyncio.sleep(1)
    return info

Real-Time Progress Updates

The _run_generation coroutine continuously updates the job object through the progress_cb callback defined in api/routers/generation.py (lines 29‑34). This callback modifies job.progress, job.step, and job.status as the generation pipeline advances through stages such as image preprocessing, model inference, and mesh post-processing.

Workflow Run Cancellation Mechanism

The Three-Layer Cancellation Signal

The cancel_run function in api/routers/workflow_runs.py (lines 86‑107) implements a robust cancellation strategy that operates on three levels:

1. Set Membership – Adds the run_id to the global _cancelled set for quick lookup
2. Event Signaling – Sets the threading Event stored in _cancel_events for the specific run ID
3. Process Termination – Kills the underlying generator subprocess via gen._proc.kill() and clears generator state

Background Task Polling

The _run_generation function checks for cancellation signals at critical points in the execution flow (lines 60‑62 and 78‑92 in api/routers/generation.py). Before and after the expensive model inference step, it verifies whether the run ID exists in _cancelled or whether the associated Event has been set. If detected, the function returns early and updates the job status to cancelled.

Process Termination

For workflows utilizing external generators (such as Stable Fast 3D), the cancellation endpoint accesses the active generator instance through services/generator_registry.py. If gen._proc exists and is running, the system sends a kill signal to immediately terminate the subprocess, preventing resource waste on abandoned jobs.

await client.post(f"/workflow-runs/{run_id}/cancel")

# Subsequent polls return: {"status": "cancelled"}

Summary

• Modly stores workflow run state in an in-memory _jobs dictionary using the JobStatus model defined in schemas/generation.py
• The GET /workflow-runs/{run_id} endpoint queries this store via the get_run function in api/routers/workflow_runs.py
• Cancellation works through a combination of set membership checks (_cancelled), threading Events (_cancel_events), and subprocess termination
• Background generation occurs in _run_generation within api/routers/generation.py, which polls cancellation signals before and after model inference
• Completed jobs are eventually purged by _purge_old_jobs to prevent memory leaks

Frequently Asked Questions

How does Modly store workflow run status internally?

Modly uses a global dictionary named _jobs defined in api/routers/generation.py that maps UUIDs to JobStatus objects. These objects track status, progress, step, output_url, and error information in-memory. This approach provides low-latency status updates without requiring an external database.

What happens to the subprocess when I cancel a workflow run?

The POST /workflow-runs/{run_id}/cancel endpoint kills the generator subprocess via gen._proc.kill() if it exists, sets a threading Event for the specific run ID, and adds the ID to a global cancellation set. This three-layer approach ensures that even if the subprocess termination fails, the background task will detect the cancellation signal and terminate gracefully.

Can I poll workflow status from multiple clients simultaneously?

Yes, because Modly stores job status in shared memory (_jobs), any client can query GET /workflow-runs/{run_id} to receive the current state. The endpoint simply reads from the shared dictionary and builds a WorkflowRunStatus model, making the system inherently safe for concurrent status polling across multiple HTTP connections.

What status values can a workflow run have?

According to the source code in schemas/generation.py and api/routers/generation.py, jobs typically progress through states including pending, running, done, error, and cancelled. The _run_generation coroutine updates these values via the progress_cb callback as the workflow advances through its pipeline stages.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}