# How Cua's Trajectory Recording and Replay System Works for Agent Debugging

> Debug agents effectively with Cua's trajectory recording and replay system. See how Cua captures and replays agent actions for clear visualization.

- Repository: [Cua/cua](https://github.com/trycua/cua)
- Tags: internals
- Published: 2026-04-27

---

**Cua captures every agent action as structured JSON and PNG files in timestamped session directories, then replays them through a temporary local HTTP server for visualization in a web-based trajectory viewer.**

The **trycua/cua** repository implements a file-based trajectory recording and replay system that enables comprehensive debugging of language-model-driven agents without requiring persistent network services. By persisting every `cua do` command to disk in `~/.cua/trajectories/`, the system creates a complete audit trail that developers can inspect locally or visualize through an integrated browser interface.

## Session Management and State Tracking

The trajectory system maintains persistent state across commands using a hidden JSON file located at `~/.cua/do_target.json`.

### Loading and Initializing Sessions

When any `cua do` command executes, the system invokes `_load_state()` from [`libs/python/cua-cli/cua_cli/utils/trajectory_recorder.py`](https://github.com/trycua/cua/blob/main/libs/python/cua-cli/cua_cli/utils/trajectory_recorder.py) (lines 40-46) to read the state file. If the file is missing or malformed, the function returns an empty dictionary to start fresh.

The `ensure_session(state)` function (lines 57-77) then checks for an existing `trajectory_session` key in the state dictionary. If the referenced directory exists, it is reused; otherwise, the function creates a new timestamped folder at `~/.cua/trajectories/<machine>/<YYYYMMDD-HHMMSS>/` and writes the path back to [`do_target.json`](https://github.com/trycua/cua/blob/main/do_target.json).

### Resetting Sessions on VM Switch

When users execute `cua do switch`, the system invokes `reset_session(state)` defined in [`libs/python/cua-cli/cua_cli/commands/do.py`](https://github.com/trycua/cua/blob/main/libs/python/cua-cli/cua_cli/commands/do.py) (lines 74-80). This removes the `trajectory_session` key from the state file, ensuring the next `cua do` command initializes a fresh trajectory directory for the new machine context.

## Turn-Level Recording Implementation

Every agent action triggers the `_maybe_record_turn()` helper, which persists the interaction to disk without blocking the main command execution.

### The Recording Wrapper

After each sub-command completes, the system calls `_maybe_record_turn()` with the action type, parameters, and optional screenshot bytes:

```python
def _maybe_record_turn(args, state, action_type, action_params, screenshot_bytes=None):
    if getattr(args, "no_record", False):
        return
    try:
        from cua_cli.utils.trajectory_recorder import ensure_session, record_turn
        session_dir = ensure_session(state)
        record_turn(session_dir, action_type, action_params, screenshot_bytes)
    except Exception:
        pass  # Recording failures never break the main command

```

The function silently catches all exceptions to ensure recording failures never interrupt the primary agent workflow.

### File Structure and JSON Payload

The `record_turn()` function creates a numbered subdirectory `turn_###` (e.g., `turn_001`, `turn_002`) containing two files:

1. **screenshot.png**: The visual state captured at action time
2. **turn_###_agent_response.json**: A structured document containing the model identifier, timestamps, and the complete action dictionary

```python
def record_turn(session_dir, action_type, action_params, screenshot_bytes=None):
    turn_num = get_next_turn_number(session_dir)
    turn_dir = session_dir / f"turn_{turn_num:03d}"
    turn_dir.mkdir(parents=True, exist_ok=True)

    if screenshot_bytes:
        (turn_dir / "screenshot.png").write_bytes(screenshot_bytes)

    response_json = {
        "model": model_id,
        "timestamp": timestamp,
        "action": _build_action_dict(action_type, action_params)
    }
    (turn_dir / f"{turn_dir.name}_agent_response.json").write_text(
        json.dumps(response_json, indent=2)
    )

```

To skip recording for sensitive operations, append the `--no-record` flag:

```bash
cua do type --text "confidential data" --no-record

```

## Replaying and Viewing Trajectories

The `cua trajectory` command suite, implemented in [`libs/python/cua-cli/cua_cli/commands/trajectory.py`](https://github.com/trycua/cua/blob/main/libs/python/cua-cli/cua_cli/commands/trajectory.py), provides CLI tools for inspection and visualization.

### Listing Recorded Sessions

The `cua trajectory ls` command calls `list_trajectories(machine)`, which walks the `~/.cua/trajectories/` directory, counts `turn_` subdirectories, and parses timestamps from directory names. Use the `--json` flag for machine-readable output:

```bash
cua trajectory ls --json

```

### Launching the Trajectory Viewer

The `cua trajectory view` command transforms local files into a browser-based interface through four distinct steps:

1. **Session Resolution**: `_resolve_session()` identifies the target (latest, specific machine, or explicit path)
2. **Compression**: `zip_trajectory(session_dir)` creates a ZIP archive containing the full session tree
3. **Server Startup**: A Python `http.server` subprocess with CORS headers serves the ZIP file; the PID is stored in `~/.cua/trajectory_server.pid`
4. **Browser Launch**: The CLI constructs a viewer URL and opens it via `webbrowser.open()`:

```python
zip_path = zip_trajectory(session_dir)
viewer_url = f"https://cua.ai/trajectory-viewer?zip={quote(zip_url, safe='')}"
webbrowser.open(viewer_url)

```

The viewer at `cua.ai/trajectory-viewer` unpacks the ZIP client-side and renders each turn with its screenshot and agent reasoning, enabling step-by-step debugging of the model's decision path.

### Cleanup and Server Management

Manage local resources with these commands:

```bash

# Delete sessions older than 30 days with confirmation

cua trajectory clean --older-than 30

# Force delete all sessions for a specific machine

cua trajectory clean --machine my-vm-01 -y

# Stop the background HTTP server

cua trajectory stop

```

## Summary

- **File-based architecture**: All trajectory data persists as local JSON and PNG files in `~/.cua/trajectories/<machine>/<timestamp>/`, requiring no external database or network daemon.
- **Automatic session management**: The system creates timestamped directories per machine and maintains state via `~/.cua/do_target.json`, resetting only when switching VMs via `cua do switch`.
- **Non-blocking recording**: The `_maybe_record_turn()` wrapper captures screenshots and structured action data silently, with a `--no-record` flag available for privacy-sensitive operations.
- **Zero-upload replay**: The `cua trajectory view` command zips sessions, starts a temporary CORS-enabled HTTP server on localhost, and launches the web viewer, keeping all data local to your machine.

## Frequently Asked Questions

### Where does Cua store trajectory recordings?

Cua stores all trajectory data in the `~/.cua/trajectories/<machine>/<timestamp>/` directory hierarchy according to the trycua/cua source code. Each turn saves as a numbered subdirectory containing `screenshot.png` and `turn_###_agent_response.json` files. Session state is tracked in `~/.cua/do_target.json`.

### Can I view trajectories without uploading data to external servers?

Yes. The `cua trajectory view` command starts a local HTTP server on your machine and generates a URL pointing to `127.0.0.1`. The web viewer at `cua.ai/trajectory-viewer` downloads and unpacks the ZIP file client-side in your browser, ensuring no trajectory data leaves your local network.

### How do I disable recording for specific commands?

Append the `--no-record` flag to any `cua do` command to skip trajectory persistence for that specific action. For example: `cua do type --text "password" --no-record`. The `_maybe_record_turn()` function checks this flag before invoking the recording logic.

### What happens to trajectory sessions when switching VMs?

When you run `cua do switch`, the system calls `reset_session()` from [`do.py`](https://github.com/trycua/cua/blob/main/do.py) (lines 74-80) to clear the current trajectory session from the state file. The next `cua do` command automatically creates a new timestamped directory for the new machine context, preventing trajectories from mixing across different VMs.