How to Ingest Runtime Traces in Codebase-Memory-MCP: A Complete Guide

Codebase-Memory-MCP ingests runtime traces by parsing line-oriented JSON through a dedicated C library that validates events and persists them to an on-disk SQLite store, accessible via both CLI and Python API.

Codebase-Memory-MCP (CBM) is an open-source tool maintained in the DeusData/codebase-memory-mcp repository that captures and replays program execution behavior through structured runtime traces. The system persists dynamic execution data using a specialized ingestion pipeline built around a small C library. Understanding how to ingest runtime traces in Codebase-Memory-MCP enables retrospective analysis of application behavior without re-executing the original binary.

The Four-Step Trace Ingestion Pipeline

The trace ingestion workflow follows a linear pipeline from instrumentation to persistent storage. Each stage utilizes specific components from the CBM core libraries.

Step 1: Capture Execution Events with Instrumentation

The process begins by instrumenting your target binary to emit structured events. When built with CBM support, setting the environment variable CBM_TRACE=1 causes the binary to output JSON objects to STDOUT. The schema for these events is defined in internal/cbm/trace.h, which specifies required fields including event, timestamp, pid, tid, location, and optional payload data.

For convenience, the repository provides scripts/trace-collector.py, a Python wrapper that handles instrumentation and collection automatically:

python3 scripts/trace-collector.py \
    --binary ./my-app \
    --output my-app.trace \
    --json-events exec,load,exit

This wrapper prepends monotonic timestamps to each line to ensure proper event ordering during ingestion.

Step 2: Collect Line-Oriented JSON Output

Each trace event represents a single JSON object terminated by a newline. A typical entry follows this structure:

{
  "event": "function_entry",
  "timestamp": 1720203542.123,
  "pid": 1234,
  "tid": 7,
  "location": "src/foo.c:42",
  "payload": { "func": "do_work", "args": [1, 2] }
}

The collector streams these objects into a .trace file while the target executes, creating a line-oriented JSON log ready for parsing.

Step 3: Execute the Ingestion Command

With the trace file prepared, invoke the ingest-trace sub-command registered in src/main.c. This command calls trace_ingest_file() implemented in src/cbm/trace_ingest.c:

codebase-memory-mcp ingest-trace --store /path/to/store my-app.trace

Internally, the ingestion engine uses the jansson library to parse each line. The parser in trace_parse.c validates required fields against the schema in internal/cbm/trace.h, and inserts valid records into the SQLite-based store via store_insert_trace(). Malformed lines trigger warnings to STDERR but do not halt the import process.

Step 4: Verify the Import

Confirm successful ingestion using the store-ls utility or direct SQL queries. The store listing tool is implemented in src/cbm/store.c:

codebase-memory-mcp store-ls --store /path/to/store --table traces

Alternatively, verify record counts with standard SQL:

codebase-memory-mcp store-ls --store /path/to/store --sql "SELECT COUNT(*) FROM traces"

Programmatic Trace Analysis with Python

Beyond the CLI, CBM exposes a Python API through pkg/pypi/src/codebase_memory_mcp/__init__.py. After ingestion, query your traces programmatically:

from codebase_memory_mcp import Store

store = Store(path="/path/to/store")
cursor = store.conn.execute(
    "SELECT event, location FROM traces WHERE event='function_entry'"
)

for event, location in cursor.fetchall():
    print(f"{event} @ {location}")

The Store class provides direct access to the underlying SQLite connection, enabling complex analytical queries against your runtime data.

Key Source Files and Architecture

Understanding the ingestion pipeline requires familiarity with these critical components:

• internal/cbm/trace.h: Defines the JSON schema and C structs for trace events.
• src/cbm/trace_ingest.c: Contains trace_ingest_file(), the core ingestion routine.
• trace_parse.c: Validates incoming JSON lines against the schema before storage.
• src/cbm/store.c: Implements persistent storage operations including store_insert_trace().
• scripts/trace-collector.py: Python wrapper for capturing binary output.
• src/main.c: CLI entry point that registers the ingest-trace sub-command.

Summary

• Instrumentation: Set CBM_TRACE=1 or use scripts/trace-collector.py to capture JSON events from instrumented binaries.
• Format: Each line must contain valid JSON with event, timestamp, pid, tid, and location fields as defined in internal/cbm/trace.h.
• Ingestion: Run codebase-memory-mcp ingest-trace to parse and store records via src/cbm/trace_ingest.c.
• Storage: Data persists to an SQLite store, typically located at $XDG_CACHE_HOME/codebase-memory-mcp/<version>/store.
• Verification: Use store-ls or the Python Store class to validate and query imported traces.

Frequently Asked Questions

What JSON schema does Codebase-Memory-MCP require for trace ingestion?

The schema requires a line-oriented JSON format where each object contains the fields event, timestamp, pid, tid, and location, with an optional payload field for additional metadata. This schema is defined in internal/cbm/trace.h and enforced by the parser logic in trace_parse.c.

Can I ingest traces from applications not built with CBM instrumentation?

Yes. While native CBM instrumentation (triggered by CBM_TRACE=1) automatically emits the correct JSON format, any application that produces line-oriented JSON matching the schema can be ingested. Simply write the output to a file and run the ingest-trace command against it.

How does the ingestion handle malformed JSON lines?

The trace_ingest_file() function in src/cbm/trace_ingest.c validates each line individually using the jansson library and the validator in trace_parse.c. Malformed entries trigger a warning to STDERR but are skipped, allowing the ingestion to continue processing valid records without aborting the entire batch.

Where does Codebase-Memory-MCP store ingested trace data?

By default, traces persist to an SQLite database located at $XDG_CACHE_HOME/codebase-memory-mcp/<version>/store, though you can specify a custom path using the --store flag during ingestion. The Python API in pkg/pypi/src/codebase_memory_mcp/__init__.py provides direct access to this storage layer for post-ingest analysis.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}