# CBM_DIAGNOSTICS: What Information Is Captured and How It Is Tracked in Codebase Memory MCP

> Explore CBM_DIAGNOSTICS to understand what health metrics and query stats are captured and tracked in Codebase Memory MCP. Learn how runtime data is collected and stored.

- Repository: [Martin Vogel/codebase-memory-mcp](https://github.com/DeusData/codebase-memory-mcp)
- Tags: deep-dive
- Published: 2026-07-07

---

**The CBM_DIAGNOSTICS subsystem captures runtime health metrics and query-level statistics every 5 seconds, writing JSON snapshots and NDJSON trajectory logs to `/tmp` while tracking data via atomic counters and background threads.**

The **CBM_DIAGNOSTICS** feature in the [DeusData/codebase-memory-mcp](https://github.com/DeusData/codebase-memory-mcp) repository provides continuous visibility into server health and query performance. When enabled via the `CBM_DIAGNOSTICS` environment variable, the system periodically records memory usage, file descriptor counts, and query execution statistics to temporary files for both live monitoring and post-mortem analysis.

## Data Captured in Diagnostic Snapshots

Each diagnostic cycle writes a comprehensive JSON snapshot containing process-level metrics and cumulative query statistics.

### System and Memory Metrics

The diagnostics harvest platform-specific data through **mimalloc** and filesystem traversal:

- **`rss_bytes`** – Current resident set size of the process, obtained via `mi_process_info()` with a fallback to `cbm_mem_rss()` when mimalloc reports zero (lines 28-44 of [`src/foundation/diagnostics.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/foundation/diagnostics.c))
- **`peak_rss_bytes`** – Peak RSS observed during the process lifetime
- **`heap_committed_bytes`** – Current heap-committed size reported by mimalloc (`current_commit`)
- **`peak_committed_bytes`** – Peak heap-committed size (`peak_commit`)
- **`page_faults`** – Total page-fault count for the process
- **`fd_count`** – Number of open file descriptors, determined by scanning `/proc/self/fd` on Linux or `/dev/fd` on macOS in `count_open_fds()` (lines 64-88 of [`src/foundation/diagnostics.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/foundation/diagnostics.c))

### Query Execution Statistics

Query tracking relies on atomic counters updated after each tool invocation:

- **`query_count`** – Total tool-call queries processed since startup (`g_query_stats.count`)
- **`query_errors`** – Number of queries returning an error status (`g_query_stats.errors`)
- **`query_total_us`** – Cumulative query execution time in microseconds (`g_query_stats.time_us`)
- **`query_avg_us`** – Computed average query time (`qtime / qcount`)
- **`query_max_us`** – Maximum single-query duration observed (`g_query_stats.max_us`)

Additional fields include **`uptime_s`** (seconds since server start) and **`pid`** (process ID).

### Output File Locations

The subsystem maintains two distinct outputs:

1. **JSON snapshot** – `/tmp/cbm-diagnostics-<pid>.json` (atomic write via rename from `.tmp`)
2. **NDJSON trajectory** – `/tmp/cbm-diagnostics-<pid>.ndjson` (append-only stream rotated at **8 MiB** via `DIAG_NDJSON_CAP_BYTES`)

The trajectory file preserves one compact line per 5-second interval (using the field name `queries` instead of `query_count`) and survives process termination for shipping to support teams.

## How Diagnostics Are Tracked

The implementation relies on thread-safe atomic operations and a dedicated background writer thread.

### Atomic Query Statistics

A global structure `cbm_query_stats_t g_query_stats` (declared in [`src/foundation/diagnostics.h`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/foundation/diagnostics.h)) maintains atomic counters for query telemetry. Each tool execution triggers `cbm_diag_record_query()` from [`src/mcp/mcp.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/mcp/mcp.c) (line 6104), which atomically updates:

- `count` (total queries)
- `errors` (failure count)
- `time_us` (cumulative duration)
- `max_us` (peak duration)

This lock-free approach ensures minimal overhead on the request path.

### Background Writer Thread

The **`cbm_diag_start()`** function spawns `g_diag_thread` during server initialization ([`src/main.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/main.c)). The thread executes `diag_thread_fn`, which calls `write_diagnostics()` every **`DIAG_INTERVAL_S`** (5 seconds) until `cbm_diag_stop()` sets the atomic flag `g_diag_stop`.

The thread lifecycle is protected by `g_diag_started` to prevent double-initialization, and the thread is joined cleanly during shutdown.

### Platform-Specific Metric Collection

Memory metrics prioritize **mimalloc** via `mi_process_info()` but implement a fallback chain:

- **Primary**: `mi_process_info()` for RSS, committed heap, and page faults
- **Fallback**: `cbm_mem_rss()` when mimalloc returns zero values
- **File descriptors**: `count_open_fds()` performs directory enumeration on `/proc/self/fd` (Linux) or `/dev/fd` (macOS)

All writes use atomic rename operations (`*.tmp` to final path) to prevent readers from observing partial JSON.

## Enabling CBM_DIAGNOSTICS

Activation requires only an environment variable before server launch:

```bash
export CBM_DIAGNOSTICS=1   # or "true"

./cbm-server

```

If the variable is absent or set to any other value, `cbm_diag_start()` returns `false` and the background thread is never created, leaving zero runtime overhead.

## Reading Diagnostic Output Programmatically

### Parse JSON Snapshots in Python

Monitor live RSS and query rates by reading the latest snapshot:

```python
import json
import pathlib
import time

def load_latest_diagnostics():
    """Load the most recent CBM diagnostics snapshot."""
    pid = pathlib.Path("/tmp").glob("cbm-diagnostics-*.json")
    path = max(pid, key=lambda p: p.stat().st_mtime)
    with open(path) as f:
        return json.load(f)

while True:
    data = load_latest_diagnostics()
    print(f"Uptime: {data['uptime_s']}s, "
          f"RSS: {data['rss_bytes']/1e6:.1f} MiB, "
          f"Queries: {data['query_count']}")
    time.sleep(5)

```

### Analyze NDJSON Trajectories in Node.js

Perform post-mortem leak analysis on the trajectory file:

```javascript
const fs = require('fs');
const path = '/tmp/cbm-diagnostics-12345.ndjson'; // Replace with actual PID

const lines = fs.readFileSync(path, 'utf8').trim().split('\n');
for (const line of lines) {
  const rec = JSON.parse(line);
  console.log(`t=${rec.uptime_s}s rss=${rec.rss_bytes}B queries=${rec.queries}`);
}

```

### Shell Script Activation

Embed diagnostics enablement in startup scripts:

```bash
#!/usr/bin/env bash
export CBM_DIAGNOSTICS=1
exec ./cbm-server

```

## Summary

- **CBM_DIAGNOSTICS** captures 12 distinct metrics including RSS, heap commitment, file descriptor counts, and query latency statistics every 5 seconds.
- Data is tracked via atomic counters in `g_query_stats` and platform-specific APIs (`mi_process_info()`, `count_open_fds()`), then persisted to `/tmp/cbm-diagnostics-<pid>.json` and `.ndjson`.
- The system uses a dedicated background thread (`g_diag_thread`) spawned by `cbm_diag_start()` with lock-free updates from `cbm_diag_record_query()` in [`src/mcp/mcp.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/mcp/mcp.c).
- NDJSON files rotate at 8 MiB to prevent unbounded growth while preserving historical trajectory data for analysis.
- Activation requires only `export CBM_DIAGNOSTICS=1` before launching the server.

## Frequently Asked Questions

### What is the performance impact of enabling CBM_DIAGNOSTICS?

The overhead is minimal. Query statistics use atomic operations (lock-free) updated via `cbm_diag_record_query()`, and the background thread sleeps for 5 seconds between cycles. File writes are atomic renames (for JSON) or append operations (for NDJSON), causing no blocking on the main request path.

### How long are diagnostic files retained?

The JSON snapshot is deleted when the process exits, but the NDJSON trajectory file persists in `/tmp` until manually removed or rotated. The trajectory rotates internally when exceeding 8 MiB (`DIAG_NDJSON_CAP_BYTES`), but older rotations are not automatically cleaned up by the current implementation.

### Can I change the 5-second sampling interval?

The interval is hardcoded as `DIAG_INTERVAL_S` (5 seconds) in [`src/foundation/diagnostics.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/foundation/diagnostics.c). To modify this, you must recompile the server after changing the constant definition, as there is no runtime configuration option for the sampling frequency.

### Why does the NDJSON file use different field names than the JSON snapshot?

The trajectory format uses `queries` (instead of `query_count`) and omits several fields to minimize line length for high-frequency logging. Both files contain the core metrics (`rss_bytes`, `uptime_s`, `pid`), but the NDJSON prioritizes compactness for long-running historical analysis while the JSON snapshot provides the complete current state.