How to Debug Indexing Issues and Capture Performance Diagnostics in Codebase-Memory-MCP

Enable MCP_LOG=debug and MCP_INDEXER_LOG=trace to capture verbose output, set MCP_METRICS=1 to write performance data to CSV, and query the HTTP endpoints /status and /coverage for real-time diagnostics.

The Codebase-Memory-MCP system builds a cross-language codebase graph by orchestrating a native indexer subprocess and persisting results in SQLite. When you need to debug indexing issues or capture diagnostics for performance problems, the project provides built-in instrumentation via environment variables, CLI flags, and a live HTTP diagnostics interface.

Diagnostic Architecture Overview

The system collects diagnostics through several coordinated components:

MCP CLI Entry Point (src/mcp/mcp.c): The main driver parses arguments, spawns the native indexer, and orchestrates the pipeline. It handles --log-level flags and writes the metrics.csv and coverage reports.

Index Supervisor (src/mcp/index_supervisor.c): This component supervises the indexer subprocess, restarting it on crash and forwarding the child’s stdout/stderr to the parent process.

HTTP Diagnostics Server (src/ui/http_server.c): Implements REST endpoints including /status, /metrics, and /coverage that expose live indexing state and historical data.

Enabling Verbose Logging

CLI and Supervisor Logs

Set the MCP_LOG environment variable to debug or pass --log-level=debug to see the full command line passed to the indexer and high-level progress messages. In src/mcp/mcp.c, the argument parser routes these flags to the logging subsystem.

MCP_LOG=debug ./mcp index /path/to/repo

Indexer Process Tracing

To capture the native indexer’s internal operations—including every file it opens, language detection events, and parse errors—set MCP_INDEXER_LOG=trace. The supervisor in src/mcp/index_supervisor.c forwards the child’s output to the parent’s stdout.

MCP_INDEXER_LOG=trace ./mcp index /path/to/repo

Capturing Performance Metrics

Enable MCP_METRICS=1 to flush timing and memory statistics to a CSV file after each run. You can also use --metrics-file=<path> to specify a custom location. The metrics struct in src/mcp/mcp.c accumulates wall-clock time, CPU usage, and bytes read across phases.

MCP_METRICS=1 ./mcp index /path/to/repo --metrics-file=./run-metrics.csv

The resulting CSV includes columns such as discover_ms, parse_ms, db_write_ms, and memory_used, matching the format found in scripts/soak-ql-mac-fixed/metrics.csv. Analyze this file to identify bottlenecks in the discovery, parsing, or database write phases.

Analyzing Coverage Reports and Missed Files

After indexing, MCP generates a coverage CSV listing files the indexer could not fully process, such as binary files, parse errors, or symlink loops. This file is written to the coverage/ directory by default. In src/mcp/mcp.c, the code path that adds a "missed-graph skeleton" populates these entries with reasons for partial coverage.

When a file cannot be fully indexed, the system creates a missed-graph node recorded in the coverage data. The React UI component in graph-ui/src/components/GraphTab.tsx visualizes these as a "Missed skeleton" cluster, helping you spot patterns like large generated headers or unsupported language constructs.

Query the coverage data via the HTTP UI:

curl http://localhost:8080/coverage | head

Live HTTP Diagnostics

Start the HTTP server with --http-port=<port> to expose real-time endpoints:

./mcp serve --http-port=8080

Available endpoints:

• /status: Returns JSON with indexedProjects, indexingInProgress, and indexingFailed fields, corresponding to the UI strings defined in graph-ui/src/lib/i18n.ts.
• /metrics: Exposes current performance counters.
• /coverage: Returns the coverage CSV data.

Query the status endpoint for a quick health check:

curl http://localhost:8080/status | jq .

Step-by-Step Debugging Workflow

Follow this sequence to diagnose indexing failures or performance degradation:

1. Enable verbose logging: Run with MCP_LOG=debug MCP_INDEXER_LOG=trace to see the exact commands and file operations.
2. Capture metrics: Set MCP_METRICS=1 and specify a metrics file to record per-phase timings.
3. Inspect coverage: Check the generated CSV under coverage/ or via /coverage to identify parse_partial entries.
4. Query HTTP status: Start the server with --http-port and monitor /status for real-time state.
5. Visualize missed files: Open the React UI and examine the "Missed skeleton" panel rendered by GraphTab.tsx to identify file patterns causing issues.

Common Pitfalls and Solutions

Indexer exits with code 137: This indicates an out-of-memory (OOM) error while parsing a huge file. Check metrics.csv for a spike in memory_used and consider adjusting the MCP_MAX_FILE_SIZE limit defined in src/foundation/limits.h.

Many parse_partial entries in coverage: Files contain syntax errors or unsupported language constructs. Review the specific paths in the coverage CSV and verify if the language front-end is registered in the CBM grammar table under internal/cbm/vendored/grammars/.

UI shows "Indexing failed" without details: The indexer process terminated unexpectedly. Enable MCP_INDEXER_LOG=trace to see the stderr output, and check the supervisor’s restart logic in src/mcp/index_supervisor.c.

Unexpectedly slow performance: Excessive symlink traversal or redundant re-parsing of unchanged files. Verify that the incremental pipeline in src/pipeline/pipeline_incremental.c is skipping unchanged files correctly, and ensure the timestamp cache is not being cleared on each run.

Summary

• Enable MCP_LOG=debug for CLI verbosity and MCP_INDEXER_LOG=trace for indexer subprocess details.
• Set MCP_METRICS=1 to generate performance CSVs with per-phase timing and memory usage from src/mcp/mcp.c.
• Inspect coverage CSVs to find partially indexed files and visualize them in the UI via the missed-graph skeleton.
• Use --http-port to expose /status, /metrics, and /coverage endpoints for real-time monitoring.
• Check exit code 137 in metrics data to diagnose OOM errors, and verify incremental indexing logic in src/pipeline/pipeline_incremental.c for performance issues.

Frequently Asked Questions

How do I enable debug logging for the MCP indexer?

Set the environment variable MCP_LOG=debug or pass the --log-level=debug flag when running the index command. This outputs high-level progress messages and the exact command line passed to the indexer subprocess. For deeper tracing of the native indexer itself, add MCP_INDEXER_LOG=trace, which the supervisor in src/mcp/index_supervisor.c forwards from the child process.

What does exit code 137 indicate when running the indexer?

Exit code 137 indicates the process was killed by the system, typically due to an out-of-memory (OOM) error while parsing a large file. Examine the memory_used column in your metrics CSV to confirm a spike, and consider reducing the MCP_MAX_FILE_SIZE value in src/foundation/limits.h or increasing available system memory.

Where can I find the performance metrics after an indexing run?

When you set MCP_METRICS=1, the system writes a CSV file to the path specified by --metrics-file (or a default location). This file, structured like scripts/soak-ql-mac-fixed/metrics.csv, contains columns for discover_ms, parse_ms, db_write_ms, and memory_used, allowing you to pinpoint which phase of indexing consumes the most resources.

How do I view which files were only partially indexed?

Partially indexed files appear in the coverage CSV generated under the coverage/ directory. These entries are created in src/mcp/mcp.c when the indexer cannot fully process a file. You can view this data via the /coverage HTTP endpoint or visualize it in the React UI under the "Missed skeleton" cluster handled by graph-ui/src/components/GraphTab.tsx.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}