Debugging Daemonized Blueprints by Inspecting Logs in DimOS

Use the dimos log CLI commands to tail structured JSON logs from background processes, leveraging per-run directories under ~/.local/state/dimos/logs/ to isolate and debug individual blueprint executions without terminal attachment.

DimOS (the Dimensional Agentic Operating System) executes robot blueprints as daemonized processes that completely detach from the terminal, making structured log inspection the primary method for debugging runtime failures. When a blueprint runs with the --daemon flag, all stdout and stderr are redirected to /dev/null while the system writes detailed JSON events to rotating log files. Understanding the logging architecture in the dimensionalOS/dimos repository enables you to trace module initialization, capture stack traces, and correlate process lifecycles using the built-in CLI utilities.

DimOS Logging Architecture

Structured JSON Log Format

DimOS configures structlog to emit machine-readable JSON lines through a rotating file handler. The central configuration resides in dimos/utils/logging_config.py, where the set_run_log_dir() function (lines 44-57) dynamically rewires file handlers to a per-run directory before the blueprint initializes. The helper _get_log_file_path() (lines 101-114) determines the final destination, defaulting to $XDG_STATE_HOME/dimos/logs/ or ~/.local/state/dimos/logs/ when the environment variable is unset.

The _configure_structlog() function (lines 16-24 and 40-46) builds a processor chain that includes a JSON renderer, ensuring every log event contains structured fields like timestamp, level, logger, and event. This results in a rotating JSON log named main.jsonl that captures output from the core system, individual modules, and LLM agents.

Per-Run Directory Isolation

Each daemonized execution receives a unique identifier generated by generate_run_id() in dimos/core/run_registry.py (lines 77-82), combining a timestamp with the blueprint name. The CLI creates an isolated log directory under ~/.local/state/dimos/logs/<run-id>/ (lines 39-41 in run_registry.py) and exports DIMOS_RUN_LOG_DIR so all worker processes inherit the path (lines 56-58 and 80-85 in dimos/robot/cli/dimos.py). This isolation prevents log collisions and simplifies correlation between a specific process instance and its output history.

Daemonization Flow and Output Redirection

The Double-Fork Mechanism

The actual backgrounding logic lives in dimos/core/daemon.py within the daemonize() function (lines 52-83). This implementation uses a standard Unix double-fork technique:

def daemonize(log_dir: Path) -> None:
    """Double-fork daemonize the current process."""
    log_dir.mkdir(parents=True, exist_ok=True)

    # First fork - detach from terminal

    pid = os.fork()
    if pid > 0:
        os._exit(0)

    os.setsid()

    # Second fork - prevent terminal reacquisition

    pid = os.fork()
    if pid > 0:
        os._exit(0)

    # Redirect stdio to /dev/null

    sys.stdout.flush()
    sys.stderr.flush()
    devnull = open(os.devnull)
    os.dup2(devnull.fileno(), sys.stdin.fileno())
    os.dup2(devnull.fileno(), sys.stdout.fileno())
    os.dup2(devnull.fileno(), sys.stderr.fileno())
    devnull.close()

The first fork creates a child that detaches from the controlling terminal, while the second fork ensures the daemon cannot reacquire a TTY (lines 52-74). Immediately after forking, the function redirects stdin, stdout, and stderr to /dev/null (lines 75-83), meaning the structured log file becomes the only observable output stream.

Pre-Daemon Health Checks

Before invoking daemonize(), the CLI executes a health check on the ModuleCoordinator via coordinator.health_check() (lines 74-78 in dimos/robot/cli/dimos.py). If any worker fails to initialize, the daemon is never created and the CLI returns an immediate error message. This guard prevents "ghost" processes and ensures startup failures surface in the foreground with actionable context.

Inspecting Logs via CLI

Locating Active Log Streams

After launching a daemonized blueprint, the CLI prints the run ID and log directory path. You can query this metadata at any time using:

dimos status           # Display PID, run ID, and health status

dimos log              # Print the last 50 lines in human-readable format

dimos log -f           # Tail the log in real-time (follow mode)

These commands read the DIMOS_RUN_LOG_DIR environment variable and parse the JSON lines using the compact console processor defined in logging_config.py (lines 71-100), rendering timestamps, severity levels, and key-value pairs.

Filtering and Searching JSON Events

Since logs are structured JSON, you can export raw events with --json and pipe them to filtering tools:

# Show only error-level events

dimos log --json | jq 'select(.level=="error")'

# Find messages from the navigation module

dimos log --json | jq 'select(.logger | contains("navigation"))'

# Extract stack traces

dimos log --json | jq 'select(.level=="error") | .exception'

The exception field is populated by structlog.processors.format_exc_info (line 37 of logging_config.py), preserving full Python tracebacks for post-mortem analysis.

Log Rotation Behavior

The rotating file handler in logging_config.py (lines 66-70) caps each file at 10 MiB while retaining 20 backups (main.jsonl.1, main.jsonl.2, etc.). If debugging requires historical data beyond the current file, check for rotated archives in the same run directory.

Practical Debugging Workflow

1. Launch with Daemon Flag and Verify

Start your blueprint in the background and capture the run ID:

dimos run unitree-go2-agentic --daemon

Immediately verify process health:

dimos status               # Confirm PID and "healthy" status

dimos log -f               # Watch for "All modules started" confirmation

If the health check fails during launch, the CLI prints the error immediately rather than daemonizing, and the log shows which module failed registration.

2. Examine Startup Sequences

Filter for initialization events to verify module loading order:

dimos log --json | jq 'select(.event | test("Starting"))'

Missing entries from specific modules (e.g., navigation, perception) indicate worker processes crashed before completing registration with the ModuleCoordinator.

3. Diagnose Runtime Crashes

When a module dies unexpectedly, locate the stack trace:

dimos log --json | jq 'select(.level=="error") | {time: .timestamp, msg: .event, exc: .exception}'

Cross-reference the pid field in log entries with dimos status to determine if the main process or a specific worker thread failed.

4. Increase Verbosity for Deep Debugging

Enable debug-level logging for specific runs by setting the environment variable before launch:

DIMOS_LOG_LEVEL=debug dimos run unitree-go2-agentic --daemon

The setup_logger() function propagates this level to all structlog processors, writing additional debug entries to main.jsonl that reveal internal state transitions and LLM prompt/response cycles.

5. Clean Up Stale Entries

List all daemonized runs and remove corrupted entries:

dimos list                # Show active runs only

dimos list --all          # Include dead entries

dimos stop <run-id>       # Terminate and unregister a specific run

The registry automatically prunes stale entries via cleanup_stale() in run_registry.py (lines 16-30), but manual cleanup via dimos stop resolves issues with orphaned log directories.

Summary

• Isolated logging: Each daemonized blueprint writes to a unique directory under ~/.local/state/dimos/logs/<run-id>/, configured by set_run_log_dir() in logging_config.py.
• Daemon transparency: The double-fork in daemonize() (lines 52-83) redirects all stdio to /dev/null, making the structured JSON log at main.jsonl the sole debugging interface.
• Health validation: Pre-daemon health checks in the CLI prevent silent startup failures by validating the ModuleCoordinator before backgrounding.
• CLI toolkit: Use dimos log for human-readable output, dimos log --json for structured queries, and dimos status to correlate PIDs with log directories.
• Rotation aware: Logs rotate at 10 MiB with 20 backups; use jq to filter historical data across rotated files.

Frequently Asked Questions

Where are DimOS daemon logs stored?

DimOS stores daemon logs in per-run directories under ~/.local/state/dimos/logs/<run-id>/ (or $XDG_STATE_HOME/dimos/logs/ if set). Each directory contains a main.jsonl file with structured JSON events. The specific path is printed when you launch with --daemon and can be retrieved via dimos status.

How do I view logs from a daemonized blueprint in real-time?

Use the dimos log -f command to tail the log file continuously. This monitors the main.jsonl file in the current run's directory and outputs new entries as they are written by the background process, formatted for human readability with timestamps and severity levels.

What happens if my blueprint crashes immediately after daemonization?

The CLI runs a health check on the ModuleCoordinator before calling daemonize(). If initialization fails, the error prints to your terminal and the process does not background. If the daemon starts but crashes later, the stack trace appears in the JSON log's exception field, accessible via dimos log --json | jq 'select(.exception)'.

How do I filter logs to find errors from a specific module?

Since DimOS uses JSON structured logging, pipe the output to jq to filter by logger name or severity. For example, run dimos log --json | jq 'select(.logger | contains("navigation"))' to isolate messages from the navigation module, or add and .level=="error" to find only its error events.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}