Team-Shared Graph Artifact Format and Incremental Indexing in codebase-memory-mcp

The team-shared graph artifact format uses a JSON manifest and binary payload stored in .codebase-memory to enable O(Δ) incremental indexing, allowing teams to merge only changed files into existing code graphs rather than rebuilding from scratch.

The codebase-memory-mcp repository provides a high-performance indexing system designed for large-scale collaborative development. Its team-shared graph artifact format creates a portable, compact representation of repository code graphs that teams can exchange and re-import. By leveraging incremental indexing through binary graph merging, the system reduces computational overhead from O(N) full re-parses to O(Δ) delta updates, where Δ represents only the modified files.

Understanding the Team-Shared Graph Artifact Structure

The artifact persists in a hidden .codebase-memory directory at the repository root, implementing a three-file architecture that balances human-readable metadata with efficient binary storage.

The .codebase-memory Directory Layout

Each artifact deployment generates the following structure:

• artifact.json: A lightweight manifest tracking schema version and integrity checksums
• graph.bin: Binary serialization containing the full code graph (nodes, edges, symbols)
• metadata.json: Optional human-readable annotations including generation date and tool version

The artifact.json Manifest

This JSON file serves as the authoritative source for version compatibility and data integrity. Located at .codebase-memory/artifact.json, it contains four critical fields that the import routine validates:

• schema_version: Guarantees forward compatibility; cbm_artifact_import() aborts on mismatch
• commit: The Git hash representing the artifact's generation state
• original_size: Uncompressed graph size used for sanity checks during import
• checksum: SHA-256 hash validating the integrity of graph.bin

A minimal manifest appears as follows:

{
  "schema_version": 2,
  "commit": "a1b2c3d4e5f6…",
  "original_size": 24576,
  "checksum": "9f86d081884c7d659a2feaa0c55ad015"
}

Binary Graph Storage in graph.bin

The graph.bin file contains a space-efficient, versioned binary serialization designed for memory-mapped random access. According to the implementation in graph/graph_io.c, this format supports fast in-place updates during incremental merge operations, allowing the system to modify existing graphs without full deserialization.

How Incremental Indexing Works in codebase-memory-mcp

The incremental indexing workflow transforms expensive full-repository scans into targeted delta updates. This six-stage process leverages specific functions across the codebase to achieve O(Δ) complexity.

Step 1: Change Detection with cbm_indexer_scan

The workflow initiates in src/indexer.c where cbm_indexer_scan() identifies modified files using Git diffs or file-system watchers. This function filters the repository to isolate only the files requiring re-indexing, establishing the delta set for subsequent processing.

Step 2: Loading Existing Artifacts via cbm_artifact_import

In pipeline/artifact.c, the cbm_artifact_import() function parses artifact.json and validates the stored commit hash against the current HEAD. If hashes match, the existing graph loads via memory-mapping, avoiding redundant parsing and enabling the incremental pathway.

Step 3: Computing Delta Graphs with cbm_graph_merge_incremental

The cbm_graph_merge_incremental() function in graph/graph_merge.c merges partial graph fragments from newly modified files into the existing binary payload. This in-place merge operation preserves the existing graph structure while incorporating delta changes, operating directly on the memory-mapped binary.

Step 4: Atomic Serialization via cbm_graph_serialize

Following the merge, cbm_graph_serialize() in graph/graph_io.c writes the updated graph to a temporary file before atomically renaming it to graph.bin. This write-temp-then-rename pattern prevents corruption during write operations by ensuring that incomplete writes never overwrite valid data.

Step 5: Manifest Updates with cbm_artifact_export

The cbm_artifact_export() function in pipeline/artifact.c regenerates artifact.json with the new commit hash, updated size metrics, and a freshly computed SHA-256 checksum. This ensures the manifest reflects the current graph state and maintains integrity for subsequent team shares.

Step 6: Team Propagation via cbm_artifact_push

Finally, cbm_artifact_push() in pipeline/artifact_sync.c handles optional remote synchronization, committing the updated artifact to the repository or uploading to shared storage buckets for team-wide access.

Failure Handling and Data Integrity

The system implements multiple safeguards to prevent silent corruption and data loss. If cbm_artifact_import() detects a schema version mismatch in artifact.json, it aborts immediately with the artifact_schema_version_mismatch error code. Checksum failures trigger automatic fallback to full re-indexing, ensuring corrupted artifacts never propagate. All write operations across the codebase use atomic rename patterns, guaranteeing that failed writes leave the previous artifact intact.

Working with the C API

Export a fast artifact containing only delta changes using the flag CBM_ARTIFACT_FAST:

int rc = cbm_artifact_export(db, repo_path, "my-project", CBM_ARTIFACT_FAST);
if (rc != 0) {
    fprintf(stderr, "Export failed: %s\n", cbm_artifact_export_last_error());
}

Import a shared artifact and perform incremental indexing:

sqlite3 *import_db;
int rc = cbm_artifact_import(repo_path, &import_db);
if (rc == 0) {
    cbm_indexer_scan_and_merge(import_db, repo_path);
}

Verify artifact availability before operations:

if (cbm_artifact_exists(repo_path)) {
    printf("Artifact available for incremental indexing.\n");
}

Summary

• The team-shared graph artifact format stores repository code graphs in .codebase-memory using JSON manifests and binary payloads that teams can exchange and re-import
• Incremental indexing reduces complexity from O(N) to O(Δ) by merging only changed files via cbm_graph_merge_incremental() in graph/graph_merge.c
• Data integrity is enforced through SHA-256 checksums, schema version validation in cbm_artifact_import(), and atomic write operations in cbm_graph_serialize()
• The C API provides functions like cbm_artifact_export() and cbm_artifact_push() for managing artifact lifecycle and team synchronization

Frequently Asked Questions

What is the team-shared graph artifact format?

The team-shared graph artifact format is a JSON-based storage system used by codebase-memory-mcp to persist code-graph representations. It consists of an artifact.json manifest and a graph.bin binary payload stored in the .codebase-memory directory, enabling teams to share pre-computed graph data without rebuilding from source on every workstation.

How does incremental indexing reduce indexing time?

Incremental indexing reduces time complexity from O(N) to O(Δ) by only processing modified files detected by cbm_indexer_scan(). Instead of re-parsing the entire repository, the system loads the existing binary graph via cbm_artifact_import() and merges new fragments using cbm_graph_merge_incremental(), dramatically speeding up operations in large monorepos.

What happens if the artifact schema version doesn't match?

When cbm_artifact_import() in pipeline/artifact.c detects a schema version mismatch between the runtime and the artifact.json manifest, it aborts immediately with an artifact_schema_version_mismatch error. This prevents incompatible binary formats from causing undefined behavior or data corruption during the import process.

How does the system prevent artifact corruption during writes?

All write operations use a write-temp-then-rename pattern implemented in cbm_graph_serialize() within graph/graph_io.c. The system writes to a temporary file first, then performs an atomic rename to graph.bin. If the rename fails, the previous artifact remains untouched, ensuring data consistency even during system crashes or disk full errors.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}