# How to Ingest Runtime Traces in Codebase-Memory-MCP: A Complete Guide

> Learn to ingest runtime traces in Codebase-Memory-MCP using the C library and Python API for event validation and SQLite persistence. A complete guide for developers.

- Repository: [Martin Vogel/codebase-memory-mcp](https://github.com/DeusData/codebase-memory-mcp)
- Tags: how-to-guide
- Published: 2026-07-04

---

**Codebase-Memory-MCP ingests runtime traces by parsing line-oriented JSON through a dedicated C library that validates events and persists them to an on-disk SQLite store, accessible via both CLI and Python API.**

Codebase-Memory-MCP (CBM) is an open-source tool maintained in the DeusData/codebase-memory-mcp repository that captures and replays program execution behavior through structured runtime traces. The system persists dynamic execution data using a specialized ingestion pipeline built around a small C library. Understanding how to ingest runtime traces in Codebase-Memory-MCP enables retrospective analysis of application behavior without re-executing the original binary.

## The Four-Step Trace Ingestion Pipeline

The trace ingestion workflow follows a linear pipeline from instrumentation to persistent storage. Each stage utilizes specific components from the CBM core libraries.

### Step 1: Capture Execution Events with Instrumentation

The process begins by instrumenting your target binary to emit structured events. When built with CBM support, setting the environment variable `CBM_TRACE=1` causes the binary to output JSON objects to STDOUT. The schema for these events is defined in [`internal/cbm/trace.h`](https://github.com/DeusData/codebase-memory-mcp/blob/main/internal/cbm/trace.h), which specifies required fields including `event`, `timestamp`, `pid`, `tid`, `location`, and optional `payload` data.

For convenience, the repository provides [`scripts/trace-collector.py`](https://github.com/DeusData/codebase-memory-mcp/blob/main/scripts/trace-collector.py), a Python wrapper that handles instrumentation and collection automatically:

```bash
python3 scripts/trace-collector.py \
    --binary ./my-app \
    --output my-app.trace \
    --json-events exec,load,exit

```

This wrapper prepends monotonic timestamps to each line to ensure proper event ordering during ingestion.

### Step 2: Collect Line-Oriented JSON Output

Each trace event represents a single JSON object terminated by a newline. A typical entry follows this structure:

```json
{
  "event": "function_entry",
  "timestamp": 1720203542.123,
  "pid": 1234,
  "tid": 7,
  "location": "src/foo.c:42",
  "payload": { "func": "do_work", "args": [1, 2] }
}

```

The collector streams these objects into a `.trace` file while the target executes, creating a line-oriented JSON log ready for parsing.

### Step 3: Execute the Ingestion Command

With the trace file prepared, invoke the `ingest-trace` sub-command registered in [`src/main.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/main.c). This command calls `trace_ingest_file()` implemented in [`src/cbm/trace_ingest.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/cbm/trace_ingest.c):

```bash
codebase-memory-mcp ingest-trace --store /path/to/store my-app.trace

```

Internally, the ingestion engine uses the `jansson` library to parse each line. The parser in [`trace_parse.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/trace_parse.c) validates required fields against the schema in [`internal/cbm/trace.h`](https://github.com/DeusData/codebase-memory-mcp/blob/main/internal/cbm/trace.h), and inserts valid records into the SQLite-based store via `store_insert_trace()`. Malformed lines trigger warnings to STDERR but do not halt the import process.

### Step 4: Verify the Import

Confirm successful ingestion using the `store-ls` utility or direct SQL queries. The store listing tool is implemented in [`src/cbm/store.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/cbm/store.c):

```bash
codebase-memory-mcp store-ls --store /path/to/store --table traces

```

Alternatively, verify record counts with standard SQL:

```bash
codebase-memory-mcp store-ls --store /path/to/store --sql "SELECT COUNT(*) FROM traces"

```

## Programmatic Trace Analysis with Python

Beyond the CLI, CBM exposes a Python API through [`pkg/pypi/src/codebase_memory_mcp/__init__.py`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pkg/pypi/src/codebase_memory_mcp/__init__.py). After ingestion, query your traces programmatically:

```python
from codebase_memory_mcp import Store

store = Store(path="/path/to/store")
cursor = store.conn.execute(
    "SELECT event, location FROM traces WHERE event='function_entry'"
)

for event, location in cursor.fetchall():
    print(f"{event} @ {location}")

```

The `Store` class provides direct access to the underlying SQLite connection, enabling complex analytical queries against your runtime data.

## Key Source Files and Architecture

Understanding the ingestion pipeline requires familiarity with these critical components:

- **[`internal/cbm/trace.h`](https://github.com/DeusData/codebase-memory-mcp/blob/main/internal/cbm/trace.h)**: Defines the JSON schema and C structs for trace events.
- **[`src/cbm/trace_ingest.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/cbm/trace_ingest.c)**: Contains `trace_ingest_file()`, the core ingestion routine.
- **[`trace_parse.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/trace_parse.c)**: Validates incoming JSON lines against the schema before storage.
- **[`src/cbm/store.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/cbm/store.c)**: Implements persistent storage operations including `store_insert_trace()`.
- **[`scripts/trace-collector.py`](https://github.com/DeusData/codebase-memory-mcp/blob/main/scripts/trace-collector.py)**: Python wrapper for capturing binary output.
- **[`src/main.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/main.c)**: CLI entry point that registers the `ingest-trace` sub-command.

## Summary

- **Instrumentation**: Set `CBM_TRACE=1` or use [`scripts/trace-collector.py`](https://github.com/DeusData/codebase-memory-mcp/blob/main/scripts/trace-collector.py) to capture JSON events from instrumented binaries.
- **Format**: Each line must contain valid JSON with `event`, `timestamp`, `pid`, `tid`, and `location` fields as defined in [`internal/cbm/trace.h`](https://github.com/DeusData/codebase-memory-mcp/blob/main/internal/cbm/trace.h).
- **Ingestion**: Run `codebase-memory-mcp ingest-trace` to parse and store records via [`src/cbm/trace_ingest.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/cbm/trace_ingest.c).
- **Storage**: Data persists to an SQLite store, typically located at `$XDG_CACHE_HOME/codebase-memory-mcp/<version>/store`.
- **Verification**: Use `store-ls` or the Python `Store` class to validate and query imported traces.

## Frequently Asked Questions

### What JSON schema does Codebase-Memory-MCP require for trace ingestion?

The schema requires a line-oriented JSON format where each object contains the fields `event`, `timestamp`, `pid`, `tid`, and `location`, with an optional `payload` field for additional metadata. This schema is defined in [`internal/cbm/trace.h`](https://github.com/DeusData/codebase-memory-mcp/blob/main/internal/cbm/trace.h) and enforced by the parser logic in [`trace_parse.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/trace_parse.c).

### Can I ingest traces from applications not built with CBM instrumentation?

Yes. While native CBM instrumentation (triggered by `CBM_TRACE=1`) automatically emits the correct JSON format, any application that produces line-oriented JSON matching the schema can be ingested. Simply write the output to a file and run the `ingest-trace` command against it.

### How does the ingestion handle malformed JSON lines?

The `trace_ingest_file()` function in [`src/cbm/trace_ingest.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/cbm/trace_ingest.c) validates each line individually using the `jansson` library and the validator in [`trace_parse.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/trace_parse.c). Malformed entries trigger a warning to STDERR but are skipped, allowing the ingestion to continue processing valid records without aborting the entire batch.

### Where does Codebase-Memory-MCP store ingested trace data?

By default, traces persist to an SQLite database located at `$XDG_CACHE_HOME/codebase-memory-mcp/<version>/store`, though you can specify a custom path using the `--store` flag during ingestion. The Python API in [`pkg/pypi/src/codebase_memory_mcp/__init__.py`](https://github.com/DeusData/codebase-memory-mcp/blob/main/pkg/pypi/src/codebase_memory_mcp/__init__.py) provides direct access to this storage layer for post-ingest analysis.