# How to Debug CAD Generation Repair Loop Failures in Text-to-CAD

> Debug CAD generation repair loop failures in text-to-CAD by inspecting heartbeat lock files and removing orphaned locks to ensure smooth generation.

- Repository: [earthtojake/text-to-cad](https://github.com/earthtojake/text-to-cad)
- Tags: how-to-guide
- Published: 2026-07-23

---

**To debug CAD generation repair loop failures in the earthtojake/text-to-cad repository, inspect the [`.generation.lock.json`](https://github.com/earthtojake/text-to-cad/blob/main/.generation.lock.json) heartbeat lock files to identify stale locks from crashed processes, verify the `_heartbeat` thread is updating timestamps every second, and manually remove orphaned locks when the `_finish()` cleanup fails.**

The text-to-cad pipeline relies on a generation-status lock mechanism to prevent concurrent runs and track real-time progress. When repair loops stall or fail repeatedly, the issue often stems from lock files in [`generation_status.py`](https://github.com/earthtojake/text-to-cad/blob/main/generation_status.py) that persist after process crashes or forced terminations. Understanding how to inspect these locks and verify the heartbeat thread activity is critical for resolving generation bottlenecks.

## Understanding the Generation Status Lock Mechanism

The CAD generation pipeline coordinates concurrent runs using a JSON-based lock file created alongside each output. When a generation job starts, the `track_generation_run` context manager in [`generation_status.py`](https://github.com/earthtojake/text-to-cad/blob/main/generation_status.py) creates a lock file named `<output>.generation.lock.json` (e.g., [`model.glb.generation.lock.json`](https://github.com/earthtojake/text-to-cad/blob/main/model.glb.generation.lock.json)).

The lock contains a JSON payload with the **run ID** (combining PID and UUID from line 61), **process ID (PID)**, **status** (`running` or `finished`), and **timestamps**. A dedicated heartbeat thread (`_heartbeat`, lines 110-112) refreshes the `updatedAt` field every `_HEARTBEAT_INTERVAL_SEC` (typically one second) while the job remains active. When generation completes, the `_finish()` method (lines 98-108) writes a final `"finished"` status and removes the lock file.

## Locating and Inspecting Lock Files

### Find Lock Files Alongside Outputs

The lock path is constructed by `generation_lock_path()` (lines 27-30) and resides in the same directory as the generated output:

```python
lock_path = generation_lock_path(output_path, run_id)

```

For a model output at `models/box.glb`, expect a lock at [`models/box.glb.generation.lock.json`](https://github.com/earthtojake/text-to-cad/blob/main/models/box.glb.generation.lock.json).

### Decode the JSON Payload

Open the lock file to verify the internal state:

```python
import json, pathlib
lock = pathlib.Path("models/box.generation.lock.json")
print(json.dumps(json.loads(lock.read_text()), indent=2))

```

Verify three critical fields:

- **`status`** – Should read `"running"` during active generation or `"finished"` during cleanup.
- **`updatedAt`** – ISO timestamp refreshed every second by the heartbeat.
- **`pid`** – The process ID that created the lock.

If `updatedAt` exceeds 30 seconds, the heartbeat thread has likely stopped, indicating a hung or crashed process.

## Diagnosing Heartbeat and Cleanup Failures

### Identify Stale Locks by Timestamp

Stale locks occur when the `updatedAt` timestamp is significantly older than `_HEARTBEAT_INTERVAL_SEC`. Compare the timestamp against current system time; a gap exceeding 30 seconds suggests the heartbeat thread exited silently or the process terminated without cleanup.

### Verify the Heartbeat Thread

The heartbeat thread runs while `self._stop` remains unset (lines 110-112). To diagnose silent thread exits:

1. Add temporary debug prints inside `_heartbeat` to trace execution.
2. Run generation with `PYTHONFAULTHANDLER=1` to capture tracebacks from thread crashes.
3. Check that the thread is not blocked by I/O operations on the lock file.

### Check for Orphaned Locks

Orphaned locks persist when `_finish()` (lines 98-108) cannot delete the file due to permission errors or SIGKILL forced termination. The fallback logic writes a final `"finished"` status before removal, but if the file remains, manual intervention is required.

## Advanced Debugging Techniques

### Monitor Filesystem Events in Real Time

Verify the heartbeat is actually writing to disk using filesystem watchers:

- **Linux**: `inotifywait -m models/box.glb.generation.lock.json`
- **macOS**: `fswatch -o models/box.glb.generation.lock.json`

These tools confirm whether the heartbeat thread updates the lock file every second as expected.

### Interactive Debugger Session

Wrap the `track_generation_run` context manager in `pdb` to step through the lifecycle:

```python
from pathlib import Path
from cadpy.generation_status import GenerationOutput, track_generation_run
import pdb

output = GenerationOutput(Path("models/box.glb"), kind="glb")
pdb.set_trace()
with track_generation_run(
    source_path=Path("models/box.step"),
    generator="my_cad_generator",
    outputs=[output],
):
    # CAD generation code here

    pass

```

Step through `_start()`, `_heartbeat`, and `_finish()` to observe lock creation and deletion in real time.

### Validate Repository Symlink Configuration

Ensure the repository's symlink layout documented in [`AGENTS.md`](https://github.com/earthtojake/text-to-cad/blob/main/AGENTS.md) points to the source `packages/cadpy` implementation rather than a stale bundled copy. Editing the wrong target without re-bundling can leave outdated lock-handling code in the runtime, causing the repair loop to use obsolete logic from [`skills/dxf/scripts/packages/cadpy/src/cadpy/generation_status.py`](https://github.com/earthtojake/text-to-cad/blob/main/skills/dxf/scripts/packages/cadpy/src/cadpy/generation_status.py) instead of the canonical [`packages/cadpy/src/cadpy/generation_status.py`](https://github.com/earthtojake/text-to-cad/blob/main/packages/cadpy/src/cadpy/generation_status.py).

## Automated Cleanup of Stale Locks

For recurring stale lock issues, implement a cleanup utility that removes locks older than a configurable threshold:

```python
import pathlib, json, time

def purge_stale_locks(root: pathlib.Path, max_age_sec: int = 60):
    for lock in root.rglob("*.generation.lock.json"):
        try:
            payload = json.loads(lock.read_text())
            updated = time.strptime(payload["updatedAt"], "%Y-%m-%dT%H:%M:%SZ")
            if time.time() - time.mktime(updated) > max_age_sec:
                lock.unlink()
                print(f"Removed stale lock: {lock}")
        except Exception as exc:
            print(f"Failed to process {lock}: {exc}")

purge_stale_locks(pathlib.Path("models"))

```

Run this script before starting new generation batches to clear debris from previous crashes.

## Summary

- **Lock files** are JSON files named `<output>.generation.lock.json` created by `generation_lock_path()` in [`generation_status.py`](https://github.com/earthtojake/text-to-cad/blob/main/generation_status.py).
- **Heartbeat verification** requires checking that `updatedAt` timestamps refresh every second; stale timestamps indicate thread failure.
- **Orphaned locks** result from `_finish()` cleanup failures or forced terminations and require manual deletion.
- **Run-ID uniqueness** combines PID and UUID (line 61), but rapid reruns in the same process may cause false positives.
- **Repository integrity** depends on correct symlinks to `packages/cadpy` as documented in [`AGENTS.md`](https://github.com/earthtojake/text-to-cad/blob/main/AGENTS.md).

## Frequently Asked Questions

### How do I manually remove a stale generation lock file?

Locate the [`.generation.lock.json`](https://github.com/earthtojake/text-to-cad/blob/main/.generation.lock.json) file alongside your output (e.g., [`model.glb.generation.lock.json`](https://github.com/earthtojake/text-to-cad/blob/main/model.glb.generation.lock.json)) and delete it directly. The lock is safe to remove if the `updatedAt` timestamp is older than 30 seconds and no corresponding process with the listed PID is running. Use the `purge_stale_locks()` script to automate this across directories.

### Why does the heartbeat thread stop updating the lock file?

The heartbeat thread (lines 110-112 in [`generation_status.py`](https://github.com/earthtojake/text-to-cad/blob/main/generation_status.py)) stops when `self._stop` is set or when an unhandled exception occurs. Silent exits often result from thread-killing signals or blocking I/O operations. Run your generation with `PYTHONFAULTHANDLER=1` to capture stack traces that reveal the exact failure point.

### What causes run-ID collisions in the lock mechanism?

The run identifier combines the process PID and a UUID (line 61), making collisions extremely unlikely. However, if you manually rerun a generation within the same process using identical parameters before the previous cleanup completes, the system may mistake the old lock for an active one. Always launch generations in fresh processes to ensure unique run IDs.

### How do repository symlinks affect lock file behavior?

The `text-to-cad` repository uses symlinks to reference `packages/cadpy` from multiple skill directories. If you edit [`skills/dxf/scripts/packages/cadpy/src/cadpy/generation_status.py`](https://github.com/earthtojake/text-to-cad/blob/main/skills/dxf/scripts/packages/cadpy/src/cadpy/generation_status.py) instead of the source [`packages/cadpy/src/cadpy/generation_status.py`](https://github.com/earthtojake/text-to-cad/blob/main/packages/cadpy/src/cadpy/generation_status.py), the runtime may execute stale code from a bundled copy. Verify your changes are applied to the canonical source path and re-bundle according to [`AGENTS.md`](https://github.com/earthtojake/text-to-cad/blob/main/AGENTS.md) guidelines.