How Cua's Trajectory Recording and Replay System Works for Agent Debugging

Cua captures every agent action as structured JSON and PNG files in timestamped session directories, then replays them through a temporary local HTTP server for visualization in a web-based trajectory viewer.

The trycua/cua repository implements a file-based trajectory recording and replay system that enables comprehensive debugging of language-model-driven agents without requiring persistent network services. By persisting every cua do command to disk in ~/.cua/trajectories/, the system creates a complete audit trail that developers can inspect locally or visualize through an integrated browser interface.

Session Management and State Tracking

The trajectory system maintains persistent state across commands using a hidden JSON file located at ~/.cua/do_target.json.

Loading and Initializing Sessions

When any cua do command executes, the system invokes _load_state() from libs/python/cua-cli/cua_cli/utils/trajectory_recorder.py (lines 40-46) to read the state file. If the file is missing or malformed, the function returns an empty dictionary to start fresh.

The ensure_session(state) function (lines 57-77) then checks for an existing trajectory_session key in the state dictionary. If the referenced directory exists, it is reused; otherwise, the function creates a new timestamped folder at ~/.cua/trajectories/<machine>/<YYYYMMDD-HHMMSS>/ and writes the path back to do_target.json.

Resetting Sessions on VM Switch

When users execute cua do switch, the system invokes reset_session(state) defined in libs/python/cua-cli/cua_cli/commands/do.py (lines 74-80). This removes the trajectory_session key from the state file, ensuring the next cua do command initializes a fresh trajectory directory for the new machine context.

Turn-Level Recording Implementation

Every agent action triggers the _maybe_record_turn() helper, which persists the interaction to disk without blocking the main command execution.

The Recording Wrapper

After each sub-command completes, the system calls _maybe_record_turn() with the action type, parameters, and optional screenshot bytes:

def _maybe_record_turn(args, state, action_type, action_params, screenshot_bytes=None):
    if getattr(args, "no_record", False):
        return
    try:
        from cua_cli.utils.trajectory_recorder import ensure_session, record_turn
        session_dir = ensure_session(state)
        record_turn(session_dir, action_type, action_params, screenshot_bytes)
    except Exception:
        pass  # Recording failures never break the main command

The function silently catches all exceptions to ensure recording failures never interrupt the primary agent workflow.

File Structure and JSON Payload

The record_turn() function creates a numbered subdirectory turn_### (e.g., turn_001, turn_002) containing two files:

1. screenshot.png: The visual state captured at action time
2. turn_###_agent_response.json: A structured document containing the model identifier, timestamps, and the complete action dictionary

def record_turn(session_dir, action_type, action_params, screenshot_bytes=None):
    turn_num = get_next_turn_number(session_dir)
    turn_dir = session_dir / f"turn_{turn_num:03d}"
    turn_dir.mkdir(parents=True, exist_ok=True)

    if screenshot_bytes:
        (turn_dir / "screenshot.png").write_bytes(screenshot_bytes)

    response_json = {
        "model": model_id,
        "timestamp": timestamp,
        "action": _build_action_dict(action_type, action_params)
    }
    (turn_dir / f"{turn_dir.name}_agent_response.json").write_text(
        json.dumps(response_json, indent=2)
    )

To skip recording for sensitive operations, append the --no-record flag:

cua do type --text "confidential data" --no-record

Replaying and Viewing Trajectories

The cua trajectory command suite, implemented in libs/python/cua-cli/cua_cli/commands/trajectory.py, provides CLI tools for inspection and visualization.

Listing Recorded Sessions

The cua trajectory ls command calls list_trajectories(machine), which walks the ~/.cua/trajectories/ directory, counts turn_ subdirectories, and parses timestamps from directory names. Use the --json flag for machine-readable output:

cua trajectory ls --json

Launching the Trajectory Viewer

The cua trajectory view command transforms local files into a browser-based interface through four distinct steps:

1. Session Resolution: _resolve_session() identifies the target (latest, specific machine, or explicit path)
2. Compression: zip_trajectory(session_dir) creates a ZIP archive containing the full session tree
3. Server Startup: A Python http.server subprocess with CORS headers serves the ZIP file; the PID is stored in ~/.cua/trajectory_server.pid
4. Browser Launch: The CLI constructs a viewer URL and opens it via webbrowser.open():

zip_path = zip_trajectory(session_dir)
viewer_url = f"https://cua.ai/trajectory-viewer?zip={quote(zip_url, safe='')}"
webbrowser.open(viewer_url)

The viewer at cua.ai/trajectory-viewer unpacks the ZIP client-side and renders each turn with its screenshot and agent reasoning, enabling step-by-step debugging of the model's decision path.

Cleanup and Server Management

Manage local resources with these commands:

# Delete sessions older than 30 days with confirmation

cua trajectory clean --older-than 30

# Force delete all sessions for a specific machine

cua trajectory clean --machine my-vm-01 -y

# Stop the background HTTP server

cua trajectory stop

Summary

• File-based architecture: All trajectory data persists as local JSON and PNG files in ~/.cua/trajectories/<machine>/<timestamp>/, requiring no external database or network daemon.
• Automatic session management: The system creates timestamped directories per machine and maintains state via ~/.cua/do_target.json, resetting only when switching VMs via cua do switch.
• Non-blocking recording: The _maybe_record_turn() wrapper captures screenshots and structured action data silently, with a --no-record flag available for privacy-sensitive operations.
• Zero-upload replay: The cua trajectory view command zips sessions, starts a temporary CORS-enabled HTTP server on localhost, and launches the web viewer, keeping all data local to your machine.

Frequently Asked Questions

Where does Cua store trajectory recordings?

Cua stores all trajectory data in the ~/.cua/trajectories/<machine>/<timestamp>/ directory hierarchy according to the trycua/cua source code. Each turn saves as a numbered subdirectory containing screenshot.png and turn_###_agent_response.json files. Session state is tracked in ~/.cua/do_target.json.

Can I view trajectories without uploading data to external servers?

Yes. The cua trajectory view command starts a local HTTP server on your machine and generates a URL pointing to 127.0.0.1. The web viewer at cua.ai/trajectory-viewer downloads and unpacks the ZIP file client-side in your browser, ensuring no trajectory data leaves your local network.

How do I disable recording for specific commands?

Append the --no-record flag to any cua do command to skip trajectory persistence for that specific action. For example: cua do type --text "password" --no-record. The _maybe_record_turn() function checks this flag before invoking the recording logic.

What happens to trajectory sessions when switching VMs?

When you run cua do switch, the system calls reset_session() from do.py (lines 74-80) to clear the current trajectory session from the state file. The next cua do command automatically creates a new timestamped directory for the new machine context, preventing trajectories from mixing across different VMs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}