How Dump Verification Ensures Data Integrity After Indexing in codebase-memory-mcp

The codebase-memory-mcp repository implements a post-dump plausibility gate that compares committed in-memory node counts against actual persisted SQLite rows, flagging data loss when the ratio falls below a configurable threshold.

After index_repository completes, the Memory-Code-Profiler (MCP) must confirm that the SQLite database accurately reflects the indexed data. According to the DeusData/codebase-memory-mcp source code, the system performs a lightweight sanity-check via src/foundation/dump_verify.c that acts as a final gate before returning success to the client.

The Post-Dump Plausibility Gate

The verification process follows a pipeline to detect substantial data loss before the JSON response is returned.

Gathering Expected Counts

When indexing finishes, the pipeline reports the number of nodes and edges it committed in memory via cbm_pipeline_get_committed_counts. This establishes the baseline expectation for what should exist in the database.

Reading the Persisted Store

The MCP opens the per-project SQLite store and counts actual rows using cbm_store_count_nodes and cbm_store_count_edges. These functions query the persisted state in src/mcp/mcp.c to determine what actually survived the write operation.

Applying the Ratio Gate

The core verification logic resides in cbm_dump_verify_is_degraded in src/foundation/dump_verify.c. This function compares persisted counts against committed counts using a configurable ratio threshold. The default minimum ratio is 0.5 (50%), read from the environment variable CBM_DUMP_VERIFY_MIN_RATIO. The gate automatically disables itself for very small repositories (≤ 50 nodes) or when the ratio is set to ≤ 0.

Recovery and Re-verification

If the initial check flags degradation, the system attempts self-healing. The store is checkpointed via cbm_store_checkpoint, counts are re-read, and the gate is re-evaluated. This handles transient corruption or stale reads that might occur during the initial verification.

Reporting the Outcome

When degradation is confirmed, the JSON response includes a hint field explaining the likely cause (such as a hard-killed sibling process), and a warning is logged via cbm_log_warn. If the store is missing entirely, a different hint is provided instructing the caller to re-run index_repository.

Key Implementation Details

The verification system spans multiple files with specific responsibilities.

The Gate Implementation

In src/foundation/dump_verify.c, the cbm_dump_verify_is_degraded function implements the plausibility logic:

bool cbm_dump_verify_is_degraded(int committed_nodes, int persisted_nodes,
                                 double ratio, int min_floor) {
    if (ratio <= 0.0)                return false;   // gate disabled
    if (committed_nodes < 0)        return false;   // no dump performed
    if (committed_nodes <= min_floor) return false; // tiny repo – skip
    if (persisted_nodes < 0)        return true;    // error reading count
    return (double)persisted_nodes < (double)committed_nodes * ratio;
}

Configuring the Verification Threshold

The ratio threshold is configurable via environment variables, parsed in cbm_dump_verify_min_ratio:

double cbm_dump_verify_min_ratio(void) {
    char buf[CBM_SZ_32];
    if (cbm_safe_getenv("CBM_DUMP_VERIFY_MIN_RATIO", buf, sizeof(buf), NULL) != NULL) {
        char *end = NULL;
        double r = strtod(buf, &end);
        if (end != buf && r >= 0.0 && r <= 1.0) return r;
        cbm_log_warn("dump_verify.env.invalid", "value", buf, "fallback", "0.5");
    }
    return CBM_DUMP_VERIFY_DEFAULT_RATIO; // 0.5
}

Integration in the Indexing Pipeline

The verification gate is invoked in src/mcp/mcp.c after the indexing pipeline completes:

int exp_nodes = -1, exp_edges = -1;
cbm_pipeline_get_committed_counts(p, &exp_nodes, &exp_edges);

const double ratio = cbm_dump_verify_min_ratio();
const int min_floor = CBM_DUMP_VERIFY_MIN_FLOOR;

cbm_store_t *store = resolve_store(srv, project_name);
int nodes = 0, edges = 0;
bool degraded = false;

if (store) {
    nodes = cbm_store_count_nodes(store, project_name);
    edges = cbm_store_count_edges(store, project_name);
    if (cbm_dump_verify_is_degraded(exp_nodes, nodes, ratio, min_floor)) {
        (void)cbm_store_checkpoint(store);          // flush & reopen
        nodes = cbm_store_count_nodes(store, project_name);
        edges = cbm_store_count_edges(store, project_name);
        degraded = cbm_dump_verify_is_degraded(exp_nodes, nodes, ratio, min_floor);
    }
}

User-Facing Error Reporting

When verification fails, the system adds actionable hints to the JSON response:

if (degraded) {
    yyjson_mut_obj_add_str(doc, root, "hint",
        "Persisted far fewer nodes than indexed — likely durability loss from a "
        "hard‑killed sibling process. Re‑run index_repository(...) to rebuild.");
    cbm_log_warn("dump.verify", "expected_nodes", exp_buf,
                 "persisted_nodes", got_buf);
}

Summary

• Ratio-based checking: The system compares committed node counts against persisted rows using a configurable threshold (default 0.5) to detect substantial data loss.
• Automatic bypass for small repos: Repositories with ≤ 50 nodes skip verification to prevent false positives on tiny codebases.
• Self-healing attempt: If initial verification fails, the system checkpoints the store and re-verifies before marking the dump as degraded.
• Clear failure signaling: Degraded states trigger JSON hints and log warnings, instructing users to re-run index_repository when data integrity is compromised.

Frequently Asked Questions

What is the default minimum ratio for dump verification?

The default minimum ratio is 0.5 (50%), meaning the system tolerates up to 50% data loss before flagging degradation. This value is defined as CBM_DUMP_VERIFY_DEFAULT_RATIO in src/foundation/dump_verify.c and can be overridden via the CBM_DUMP_VERIFY_MIN_RATIO environment variable.

Why does verification skip small repositories?

Repositories with 50 or fewer nodes bypass the ratio gate to prevent false positives. Tiny codebases can exhibit statistical anomalies during persistence that don't indicate actual corruption, so the min_floor parameter (set to 50) disables the check for these cases.

How does the system handle transient corruption during verification?

If the initial check detects degradation, the MCP performs a checkpoint and re-verification cycle. It calls cbm_store_checkpoint to flush and reopen the database, then re-counts the nodes and edges. This handles transient states or stale reads that might occur immediately after the initial write.

What happens when dump verification fails?

When verification fails, the JSON response includes a hint field explaining that "persisted far fewer nodes than indexed" were stored, typically indicating a hard-killed sibling process. A warning is logged via cbm_log_warn, and the caller is instructed to re-run index_repository to rebuild the database with correct data.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}