How to Debug CAD Generation Repair Loop Failures in Text-to-CAD

To debug CAD generation repair loop failures in the earthtojake/text-to-cad repository, inspect the .generation.lock.json heartbeat lock files to identify stale locks from crashed processes, verify the _heartbeat thread is updating timestamps every second, and manually remove orphaned locks when the _finish() cleanup fails.

The text-to-cad pipeline relies on a generation-status lock mechanism to prevent concurrent runs and track real-time progress. When repair loops stall or fail repeatedly, the issue often stems from lock files in generation_status.py that persist after process crashes or forced terminations. Understanding how to inspect these locks and verify the heartbeat thread activity is critical for resolving generation bottlenecks.

Understanding the Generation Status Lock Mechanism

The CAD generation pipeline coordinates concurrent runs using a JSON-based lock file created alongside each output. When a generation job starts, the track_generation_run context manager in generation_status.py creates a lock file named <output>.generation.lock.json (e.g., model.glb.generation.lock.json).

The lock contains a JSON payload with the run ID (combining PID and UUID from line 61), process ID (PID), status (running or finished), and timestamps. A dedicated heartbeat thread (_heartbeat, lines 110-112) refreshes the updatedAt field every _HEARTBEAT_INTERVAL_SEC (typically one second) while the job remains active. When generation completes, the _finish() method (lines 98-108) writes a final "finished" status and removes the lock file.

Locating and Inspecting Lock Files

Find Lock Files Alongside Outputs

The lock path is constructed by generation_lock_path() (lines 27-30) and resides in the same directory as the generated output:

lock_path = generation_lock_path(output_path, run_id)

For a model output at models/box.glb, expect a lock at models/box.glb.generation.lock.json.

Decode the JSON Payload

Open the lock file to verify the internal state:

import json, pathlib
lock = pathlib.Path("models/box.generation.lock.json")
print(json.dumps(json.loads(lock.read_text()), indent=2))

Verify three critical fields:

• status – Should read "running" during active generation or "finished" during cleanup.
• updatedAt – ISO timestamp refreshed every second by the heartbeat.
• pid – The process ID that created the lock.

If updatedAt exceeds 30 seconds, the heartbeat thread has likely stopped, indicating a hung or crashed process.

Diagnosing Heartbeat and Cleanup Failures

Identify Stale Locks by Timestamp

Stale locks occur when the updatedAt timestamp is significantly older than _HEARTBEAT_INTERVAL_SEC. Compare the timestamp against current system time; a gap exceeding 30 seconds suggests the heartbeat thread exited silently or the process terminated without cleanup.

Verify the Heartbeat Thread

The heartbeat thread runs while self._stop remains unset (lines 110-112). To diagnose silent thread exits:

1. Add temporary debug prints inside _heartbeat to trace execution.
2. Run generation with PYTHONFAULTHANDLER=1 to capture tracebacks from thread crashes.
3. Check that the thread is not blocked by I/O operations on the lock file.

Check for Orphaned Locks

Orphaned locks persist when _finish() (lines 98-108) cannot delete the file due to permission errors or SIGKILL forced termination. The fallback logic writes a final "finished" status before removal, but if the file remains, manual intervention is required.

Advanced Debugging Techniques

Monitor Filesystem Events in Real Time

Verify the heartbeat is actually writing to disk using filesystem watchers:

• Linux: inotifywait -m models/box.glb.generation.lock.json
• macOS: fswatch -o models/box.glb.generation.lock.json

These tools confirm whether the heartbeat thread updates the lock file every second as expected.

Interactive Debugger Session

Wrap the track_generation_run context manager in pdb to step through the lifecycle:

from pathlib import Path
from cadpy.generation_status import GenerationOutput, track_generation_run
import pdb

output = GenerationOutput(Path("models/box.glb"), kind="glb")
pdb.set_trace()
with track_generation_run(
    source_path=Path("models/box.step"),
    generator="my_cad_generator",
    outputs=[output],
):
    # CAD generation code here

    pass

Step through _start(), _heartbeat, and _finish() to observe lock creation and deletion in real time.

Validate Repository Symlink Configuration

Ensure the repository's symlink layout documented in AGENTS.md points to the source packages/cadpy implementation rather than a stale bundled copy. Editing the wrong target without re-bundling can leave outdated lock-handling code in the runtime, causing the repair loop to use obsolete logic from skills/dxf/scripts/packages/cadpy/src/cadpy/generation_status.py instead of the canonical packages/cadpy/src/cadpy/generation_status.py.

Automated Cleanup of Stale Locks

For recurring stale lock issues, implement a cleanup utility that removes locks older than a configurable threshold:

import pathlib, json, time

def purge_stale_locks(root: pathlib.Path, max_age_sec: int = 60):
    for lock in root.rglob("*.generation.lock.json"):
        try:
            payload = json.loads(lock.read_text())
            updated = time.strptime(payload["updatedAt"], "%Y-%m-%dT%H:%M:%SZ")
            if time.time() - time.mktime(updated) > max_age_sec:
                lock.unlink()
                print(f"Removed stale lock: {lock}")
        except Exception as exc:
            print(f"Failed to process {lock}: {exc}")

purge_stale_locks(pathlib.Path("models"))

Run this script before starting new generation batches to clear debris from previous crashes.

Summary

• Lock files are JSON files named <output>.generation.lock.json created by generation_lock_path() in generation_status.py.
• Heartbeat verification requires checking that updatedAt timestamps refresh every second; stale timestamps indicate thread failure.
• Orphaned locks result from _finish() cleanup failures or forced terminations and require manual deletion.
• Run-ID uniqueness combines PID and UUID (line 61), but rapid reruns in the same process may cause false positives.
• Repository integrity depends on correct symlinks to packages/cadpy as documented in AGENTS.md.

Frequently Asked Questions

How do I manually remove a stale generation lock file?

Locate the .generation.lock.json file alongside your output (e.g., model.glb.generation.lock.json) and delete it directly. The lock is safe to remove if the updatedAt timestamp is older than 30 seconds and no corresponding process with the listed PID is running. Use the purge_stale_locks() script to automate this across directories.

Why does the heartbeat thread stop updating the lock file?

The heartbeat thread (lines 110-112 in generation_status.py) stops when self._stop is set or when an unhandled exception occurs. Silent exits often result from thread-killing signals or blocking I/O operations. Run your generation with PYTHONFAULTHANDLER=1 to capture stack traces that reveal the exact failure point.

What causes run-ID collisions in the lock mechanism?

The run identifier combines the process PID and a UUID (line 61), making collisions extremely unlikely. However, if you manually rerun a generation within the same process using identical parameters before the previous cleanup completes, the system may mistake the old lock for an active one. Always launch generations in fresh processes to ensure unique run IDs.

How do repository symlinks affect lock file behavior?

The text-to-cad repository uses symlinks to reference packages/cadpy from multiple skill directories. If you edit skills/dxf/scripts/packages/cadpy/src/cadpy/generation_status.py instead of the source packages/cadpy/src/cadpy/generation_status.py, the runtime may execute stale code from a bundled copy. Verify your changes are applied to the canonical source path and re-bundle according to AGENTS.md guidelines.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}