# Debugging Daemonized Blueprints by Inspecting Logs in DimOS

> Master debugging daemonized blueprints in DimOS. Inspect structured JSON logs with dimos log CLI and isolate runs in per-run directories for efficient troubleshooting.

- Repository: [Dimensional/dimos](https://github.com/dimensionalOS/dimos)
- Tags: best-practices
- Published: 2026-03-15

---

**Use the `dimos log` CLI commands to tail structured JSON logs from background processes, leveraging per-run directories under `~/.local/state/dimos/logs/` to isolate and debug individual blueprint executions without terminal attachment.**

DimOS (the Dimensional Agentic Operating System) executes robot blueprints as **daemonized processes** that completely detach from the terminal, making structured log inspection the primary method for debugging runtime failures. When a blueprint runs with the `--daemon` flag, all stdout and stderr are redirected to `/dev/null` while the system writes detailed JSON events to rotating log files. Understanding the logging architecture in the `dimensionalOS/dimos` repository enables you to trace module initialization, capture stack traces, and correlate process lifecycles using the built-in CLI utilities.

## DimOS Logging Architecture

### Structured JSON Log Format

DimOS configures **structlog** to emit machine-readable JSON lines through a rotating file handler. The central configuration resides in [`dimos/utils/logging_config.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/utils/logging_config.py), where the `set_run_log_dir()` function (lines 44-57) dynamically rewires file handlers to a per-run directory before the blueprint initializes. The helper `_get_log_file_path()` (lines 101-114) determines the final destination, defaulting to `$XDG_STATE_HOME/dimos/logs/` or `~/.local/state/dimos/logs/` when the environment variable is unset.

The `_configure_structlog()` function (lines 16-24 and 40-46) builds a processor chain that includes a JSON renderer, ensuring every log event contains structured fields like `timestamp`, `level`, `logger`, and `event`. This results in a **rotating JSON log** named `main.jsonl` that captures output from the core system, individual modules, and LLM agents.

### Per-Run Directory Isolation

Each daemonized execution receives a unique identifier generated by `generate_run_id()` in [`dimos/core/run_registry.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/core/run_registry.py) (lines 77-82), combining a timestamp with the blueprint name. The CLI creates an isolated log directory under `~/.local/state/dimos/logs/<run-id>/` (lines 39-41 in [`run_registry.py`](https://github.com/dimensionalOS/dimos/blob/main/run_registry.py)) and exports `DIMOS_RUN_LOG_DIR` so all worker processes inherit the path (lines 56-58 and 80-85 in [`dimos/robot/cli/dimos.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/robot/cli/dimos.py)). This isolation prevents log collisions and simplifies correlation between a specific process instance and its output history.

## Daemonization Flow and Output Redirection

### The Double-Fork Mechanism

The actual backgrounding logic lives in [`dimos/core/daemon.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/core/daemon.py) within the `daemonize()` function (lines 52-83). This implementation uses a standard Unix double-fork technique:

```python
def daemonize(log_dir: Path) -> None:
    """Double-fork daemonize the current process."""
    log_dir.mkdir(parents=True, exist_ok=True)

    # First fork - detach from terminal

    pid = os.fork()
    if pid > 0:
        os._exit(0)

    os.setsid()

    # Second fork - prevent terminal reacquisition

    pid = os.fork()
    if pid > 0:
        os._exit(0)

    # Redirect stdio to /dev/null

    sys.stdout.flush()
    sys.stderr.flush()
    devnull = open(os.devnull)
    os.dup2(devnull.fileno(), sys.stdin.fileno())
    os.dup2(devnull.fileno(), sys.stdout.fileno())
    os.dup2(devnull.fileno(), sys.stderr.fileno())
    devnull.close()

```

The first fork creates a child that detaches from the controlling terminal, while the second fork ensures the daemon cannot reacquire a TTY (lines 52-74). Immediately after forking, the function redirects `stdin`, `stdout`, and `stderr` to `/dev/null` (lines 75-83), meaning **the structured log file becomes the only observable output stream**.

### Pre-Daemon Health Checks

Before invoking `daemonize()`, the CLI executes a health check on the `ModuleCoordinator` via `coordinator.health_check()` (lines 74-78 in [`dimos/robot/cli/dimos.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/robot/cli/dimos.py)). If any worker fails to initialize, the daemon is never created and the CLI returns an immediate error message. This guard prevents "ghost" processes and ensures startup failures surface in the foreground with actionable context.

## Inspecting Logs via CLI

### Locating Active Log Streams

After launching a daemonized blueprint, the CLI prints the run ID and log directory path. You can query this metadata at any time using:

```bash
dimos status           # Display PID, run ID, and health status

dimos log              # Print the last 50 lines in human-readable format

dimos log -f           # Tail the log in real-time (follow mode)

```

These commands read the `DIMOS_RUN_LOG_DIR` environment variable and parse the JSON lines using the compact console processor defined in [`logging_config.py`](https://github.com/dimensionalOS/dimos/blob/main/logging_config.py) (lines 71-100), rendering timestamps, severity levels, and key-value pairs.

### Filtering and Searching JSON Events

Since logs are structured JSON, you can export raw events with `--json` and pipe them to filtering tools:

```bash

# Show only error-level events

dimos log --json | jq 'select(.level=="error")'

# Find messages from the navigation module

dimos log --json | jq 'select(.logger | contains("navigation"))'

# Extract stack traces

dimos log --json | jq 'select(.level=="error") | .exception'

```

The `exception` field is populated by `structlog.processors.format_exc_info` (line 37 of [`logging_config.py`](https://github.com/dimensionalOS/dimos/blob/main/logging_config.py)), preserving full Python tracebacks for post-mortem analysis.

### Log Rotation Behavior

The rotating file handler in [`logging_config.py`](https://github.com/dimensionalOS/dimos/blob/main/logging_config.py) (lines 66-70) caps each file at **10 MiB** while retaining **20 backups** (`main.jsonl.1`, `main.jsonl.2`, etc.). If debugging requires historical data beyond the current file, check for rotated archives in the same run directory.

## Practical Debugging Workflow

### 1. Launch with Daemon Flag and Verify

Start your blueprint in the background and capture the run ID:

```bash
dimos run unitree-go2-agentic --daemon

```

Immediately verify process health:

```bash
dimos status               # Confirm PID and "healthy" status

dimos log -f               # Watch for "All modules started" confirmation

```

If the health check fails during launch, the CLI prints the error immediately rather than daemonizing, and the log shows which module failed registration.

### 2. Examine Startup Sequences

Filter for initialization events to verify module loading order:

```bash
dimos log --json | jq 'select(.event | test("Starting"))'

```

Missing entries from specific modules (e.g., `navigation`, `perception`) indicate worker processes crashed before completing registration with the `ModuleCoordinator`.

### 3. Diagnose Runtime Crashes

When a module dies unexpectedly, locate the stack trace:

```bash
dimos log --json | jq 'select(.level=="error") | {time: .timestamp, msg: .event, exc: .exception}'

```

Cross-reference the `pid` field in log entries with `dimos status` to determine if the main process or a specific worker thread failed.

### 4. Increase Verbosity for Deep Debugging

Enable debug-level logging for specific runs by setting the environment variable before launch:

```bash
DIMOS_LOG_LEVEL=debug dimos run unitree-go2-agentic --daemon

```

The `setup_logger()` function propagates this level to all structlog processors, writing additional `debug` entries to `main.jsonl` that reveal internal state transitions and LLM prompt/response cycles.

### 5. Clean Up Stale Entries

List all daemonized runs and remove corrupted entries:

```bash
dimos list                # Show active runs only

dimos list --all          # Include dead entries

dimos stop <run-id>       # Terminate and unregister a specific run

```

The registry automatically prunes stale entries via `cleanup_stale()` in [`run_registry.py`](https://github.com/dimensionalOS/dimos/blob/main/run_registry.py) (lines 16-30), but manual cleanup via `dimos stop` resolves issues with orphaned log directories.

## Summary

- **Isolated logging**: Each daemonized blueprint writes to a unique directory under `~/.local/state/dimos/logs/<run-id>/`, configured by `set_run_log_dir()` in [`logging_config.py`](https://github.com/dimensionalOS/dimos/blob/main/logging_config.py).
- **Daemon transparency**: The double-fork in `daemonize()` (lines 52-83) redirects all stdio to `/dev/null`, making the structured JSON log at `main.jsonl` the sole debugging interface.
- **Health validation**: Pre-daemon health checks in the CLI prevent silent startup failures by validating the `ModuleCoordinator` before backgrounding.
- **CLI toolkit**: Use `dimos log` for human-readable output, `dimos log --json` for structured queries, and `dimos status` to correlate PIDs with log directories.
- **Rotation aware**: Logs rotate at 10 MiB with 20 backups; use `jq` to filter historical data across rotated files.

## Frequently Asked Questions

### Where are DimOS daemon logs stored?

DimOS stores daemon logs in per-run directories under `~/.local/state/dimos/logs/<run-id>/` (or `$XDG_STATE_HOME/dimos/logs/` if set). Each directory contains a `main.jsonl` file with structured JSON events. The specific path is printed when you launch with `--daemon` and can be retrieved via `dimos status`.

### How do I view logs from a daemonized blueprint in real-time?

Use the `dimos log -f` command to tail the log file continuously. This monitors the `main.jsonl` file in the current run's directory and outputs new entries as they are written by the background process, formatted for human readability with timestamps and severity levels.

### What happens if my blueprint crashes immediately after daemonization?

The CLI runs a health check on the `ModuleCoordinator` before calling `daemonize()`. If initialization fails, the error prints to your terminal and the process does not background. If the daemon starts but crashes later, the stack trace appears in the JSON log's `exception` field, accessible via `dimos log --json | jq 'select(.exception)'`.

### How do I filter logs to find errors from a specific module?

Since DimOS uses JSON structured logging, pipe the output to `jq` to filter by logger name or severity. For example, run `dimos log --json | jq 'select(.logger | contains("navigation"))'` to isolate messages from the navigation module, or add `and .level=="error"` to find only its error events.