CBM_DIAGNOSTICS: What Information Is Captured and How It Is Tracked in Codebase Memory MCP

The CBM_DIAGNOSTICS subsystem captures runtime health metrics and query-level statistics every 5 seconds, writing JSON snapshots and NDJSON trajectory logs to /tmp while tracking data via atomic counters and background threads.

The CBM_DIAGNOSTICS feature in the DeusData/codebase-memory-mcp repository provides continuous visibility into server health and query performance. When enabled via the CBM_DIAGNOSTICS environment variable, the system periodically records memory usage, file descriptor counts, and query execution statistics to temporary files for both live monitoring and post-mortem analysis.

Data Captured in Diagnostic Snapshots

Each diagnostic cycle writes a comprehensive JSON snapshot containing process-level metrics and cumulative query statistics.

System and Memory Metrics

The diagnostics harvest platform-specific data through mimalloc and filesystem traversal:

• rss_bytes – Current resident set size of the process, obtained via mi_process_info() with a fallback to cbm_mem_rss() when mimalloc reports zero (lines 28-44 of src/foundation/diagnostics.c)
• peak_rss_bytes – Peak RSS observed during the process lifetime
• heap_committed_bytes – Current heap-committed size reported by mimalloc (current_commit)
• peak_committed_bytes – Peak heap-committed size (peak_commit)
• page_faults – Total page-fault count for the process
• fd_count – Number of open file descriptors, determined by scanning /proc/self/fd on Linux or /dev/fd on macOS in count_open_fds() (lines 64-88 of src/foundation/diagnostics.c)

Query Execution Statistics

Query tracking relies on atomic counters updated after each tool invocation:

• query_count – Total tool-call queries processed since startup (g_query_stats.count)
• query_errors – Number of queries returning an error status (g_query_stats.errors)
• query_total_us – Cumulative query execution time in microseconds (g_query_stats.time_us)
• query_avg_us – Computed average query time (qtime / qcount)
• query_max_us – Maximum single-query duration observed (g_query_stats.max_us)

Additional fields include uptime_s (seconds since server start) and pid (process ID).

Output File Locations

The subsystem maintains two distinct outputs:

1. JSON snapshot – /tmp/cbm-diagnostics-<pid>.json (atomic write via rename from .tmp)
2. NDJSON trajectory – /tmp/cbm-diagnostics-<pid>.ndjson (append-only stream rotated at 8 MiB via DIAG_NDJSON_CAP_BYTES)

The trajectory file preserves one compact line per 5-second interval (using the field name queries instead of query_count) and survives process termination for shipping to support teams.

How Diagnostics Are Tracked

The implementation relies on thread-safe atomic operations and a dedicated background writer thread.

Atomic Query Statistics

A global structure cbm_query_stats_t g_query_stats (declared in src/foundation/diagnostics.h) maintains atomic counters for query telemetry. Each tool execution triggers cbm_diag_record_query() from src/mcp/mcp.c (line 6104), which atomically updates:

• count (total queries)
• errors (failure count)
• time_us (cumulative duration)
• max_us (peak duration)

This lock-free approach ensures minimal overhead on the request path.

Background Writer Thread

The cbm_diag_start() function spawns g_diag_thread during server initialization (src/main.c). The thread executes diag_thread_fn, which calls write_diagnostics() every DIAG_INTERVAL_S (5 seconds) until cbm_diag_stop() sets the atomic flag g_diag_stop.

The thread lifecycle is protected by g_diag_started to prevent double-initialization, and the thread is joined cleanly during shutdown.

Platform-Specific Metric Collection

Memory metrics prioritize mimalloc via mi_process_info() but implement a fallback chain:

• Primary: mi_process_info() for RSS, committed heap, and page faults
• Fallback: cbm_mem_rss() when mimalloc returns zero values
• File descriptors: count_open_fds() performs directory enumeration on /proc/self/fd (Linux) or /dev/fd (macOS)

All writes use atomic rename operations (*.tmp to final path) to prevent readers from observing partial JSON.

Enabling CBM_DIAGNOSTICS

Activation requires only an environment variable before server launch:

export CBM_DIAGNOSTICS=1   # or "true"

./cbm-server

If the variable is absent or set to any other value, cbm_diag_start() returns false and the background thread is never created, leaving zero runtime overhead.

Reading Diagnostic Output Programmatically

Parse JSON Snapshots in Python

Monitor live RSS and query rates by reading the latest snapshot:

import json
import pathlib
import time

def load_latest_diagnostics():
    """Load the most recent CBM diagnostics snapshot."""
    pid = pathlib.Path("/tmp").glob("cbm-diagnostics-*.json")
    path = max(pid, key=lambda p: p.stat().st_mtime)
    with open(path) as f:
        return json.load(f)

while True:
    data = load_latest_diagnostics()
    print(f"Uptime: {data['uptime_s']}s, "
          f"RSS: {data['rss_bytes']/1e6:.1f} MiB, "
          f"Queries: {data['query_count']}")
    time.sleep(5)

Analyze NDJSON Trajectories in Node.js

Perform post-mortem leak analysis on the trajectory file:

const fs = require('fs');
const path = '/tmp/cbm-diagnostics-12345.ndjson'; // Replace with actual PID

const lines = fs.readFileSync(path, 'utf8').trim().split('\n');
for (const line of lines) {
  const rec = JSON.parse(line);
  console.log(`t=${rec.uptime_s}s rss=${rec.rss_bytes}B queries=${rec.queries}`);
}

Shell Script Activation

Embed diagnostics enablement in startup scripts:

#!/usr/bin/env bash
export CBM_DIAGNOSTICS=1
exec ./cbm-server

Summary

• CBM_DIAGNOSTICS captures 12 distinct metrics including RSS, heap commitment, file descriptor counts, and query latency statistics every 5 seconds.
• Data is tracked via atomic counters in g_query_stats and platform-specific APIs (mi_process_info(), count_open_fds()), then persisted to /tmp/cbm-diagnostics-<pid>.json and .ndjson.
• The system uses a dedicated background thread (g_diag_thread) spawned by cbm_diag_start() with lock-free updates from cbm_diag_record_query() in src/mcp/mcp.c.
• NDJSON files rotate at 8 MiB to prevent unbounded growth while preserving historical trajectory data for analysis.
• Activation requires only export CBM_DIAGNOSTICS=1 before launching the server.

Frequently Asked Questions

What is the performance impact of enabling CBM_DIAGNOSTICS?

The overhead is minimal. Query statistics use atomic operations (lock-free) updated via cbm_diag_record_query(), and the background thread sleeps for 5 seconds between cycles. File writes are atomic renames (for JSON) or append operations (for NDJSON), causing no blocking on the main request path.

How long are diagnostic files retained?

The JSON snapshot is deleted when the process exits, but the NDJSON trajectory file persists in /tmp until manually removed or rotated. The trajectory rotates internally when exceeding 8 MiB (DIAG_NDJSON_CAP_BYTES), but older rotations are not automatically cleaned up by the current implementation.

Can I change the 5-second sampling interval?

The interval is hardcoded as DIAG_INTERVAL_S (5 seconds) in src/foundation/diagnostics.c. To modify this, you must recompile the server after changing the constant definition, as there is no runtime configuration option for the sampling frequency.

Why does the NDJSON file use different field names than the JSON snapshot?

The trajectory format uses queries (instead of query_count) and omits several fields to minimize line length for high-frequency logging. Both files contain the core metrics (rss_bytes, uptime_s, pid), but the NDJSON prioritizes compactness for long-running historical analysis while the JSON snapshot provides the complete current state.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}