How to Contribute to the Development of the Graph Storage Implementation in Codebase-Memory-MCP

You can contribute to the graph storage implementation by extending secondary indexes in src/graph_buffer/graph_buffer.c, optimizing edge deduplication via make_edge_key, or enhancing the merge logic that combines parallel worker buffers.

The graph storage layer in DeusData/codebase-memory-mcp is encapsulated in the graph-buffer module, which maintains an in-memory directed property graph that the indexing pipeline flushes to SQLite. To contribute effectively to the development of the graph storage implementation, you must understand the cbm_gbuf_t architecture, its hash-table based indexing system, and the merge semantics that enable parallel processing.

Understanding the Graph-Buffer Architecture

Core Data Structures

The central structure cbm_gbuf_t (defined in src/graph_buffer/graph_buffer.h) manages all graph entities. It is allocated via cbm_gbuf_new or cbm_gbuf_new_shared_ids and contains:

• Node storage: Heap-allocated cbm_gbuf_node_t objects stored in gb->nodes (a CBMDynArray), indexed by qualified name (node_by_qn) and numeric ID (node_by_id).
• Edge storage: Heap-allocated cbm_gbuf_edge_t objects tracking source/target IDs, types, and JSON properties, stored in gb->edges and deduplicated via edge_by_key.

Primary and Secondary Indexes

The implementation uses the generic CBMHashTable API (located in internal/cbm/hash_table.c) for O(1) lookups:

• Primary indexes: node_by_qn and node_by_id provide direct node access.
• Secondary indexes: Enable complex queries through nodes_by_label, nodes_by_name, edges_by_source_type, edges_by_target_type, and edges_by_type.

Memory Optimization Features

The graph buffer employs string interning via intern_pool (accessed through gb_intern) to reduce memory fragmentation for repetitive strings like labels and file paths. It also stores semantic embeddings through cbm_gbuf_store_vector and cbm_gbuf_store_token_vector before flushing to SQLite.

Key Contribution Areas

Adding Secondary Indexes

To expose new query capabilities (e.g., "edges by property value"), add a hash table (e.g., edges_by_property) and update register_edge_in_indexes and make_edge_key in src/graph_buffer/graph_buffer.c to populate the index during edge insertion.

Optimizing Edge Deduplication

Refine the make_edge_key function to handle additional edge-type-specific fields, or make the deduplication hash table configurable. This affects how cbm_gbuf_insert_edge prevents duplicate edges in edge_by_key.

Performance Tuning

Replace the generic CBMHashTable with cache-friendly alternatives, or optimize the CBMDynArray growth strategy in foundation/dyn_array.h. These changes impact allocation patterns in cbm_gbuf_new and node/edge insertion performance.

Testing and Serialization

Add unit tests for edge insertion, deletion, and merge scenarios in tests/test_graph_buffer.c. For serialization enhancements, modify the "Dump / Flush" section functions (build_dump_nodes, build_dump_edges) to support new SQLite schema columns.

Implementation Guide

Basic API Usage

Here's a minimal example demonstrating the public API declared in src/graph_buffer/graph_buffer.h:

/* Create a new buffer for a project named "demo" */
cbm_gbuf_t *gb = cbm_gbuf_new("demo", "/path/to/project");

/* Insert a node – a Go package named "utils" */
int64_t node_id = cbm_gbuf_upsert_node(
    gb,
    "Package",                     /* label */
    "utils",                       /* name */
    "demo.utils",                  /* qualified name */
    "utils.go",                    /* file_path */
    1, 42,                         /* start/end line */
    "{\"doc\":\"utility package\"}" /* optional JSON properties */
);

/* Insert an edge recording a function call */
cbm_gbuf_insert_edge(
    gb,
    node_id,                       /* source */
    node_id,                       /* target – self-call for demo */
    "CALLS",                       /* edge type */
    "{\"caller\":\"main\",\"callee\":\"helper\"}"
);

/* Merge a second buffer from a parallel worker */
cbm_gbuf_t *worker_gb = /* ... */;
cbm_gbuf_merge(gb, worker_gb);

/* Persist to SQLite */
cbm_gbuf_dump_to_sqlite(gb, "/tmp/demo.db");

/* Cleanup */
cbm_gbuf_free(gb);
cbm_gbuf_free(worker_gb);

Merge and Cascade-Delete Logic

The cbm_gbuf_merge function allows parallel workers to safely combine independent buffers. For incremental re-indexing, the cascade-delete logic (implemented in cascade_delete_edges and *_delete_by_* helpers) removes nodes and edges belonging to specific files.

Building and Testing Your Changes

To compile and test your modifications:

# Build the core library

make -f Makefile.cbm

# Run the graph-buffer unit tests

./tests/test_graph_buffer

The test suite validates node insertion, edge deduplication, cascade deletes, and merge algorithms. When adding features, extend tests/test_graph_buffer.c following the existing patterns.

Critical Source Files

File	Purpose
src/graph_buffer/graph_buffer.h	Public API definitions and data structures
src/graph_buffer/graph_buffer.c	Implementation of node/edge management, indexes, and dump logic
internal/cbm/hash_table.h / hash_table.c	Generic hash table for all indexes
foundation/dyn_array.h	Macro-based dynamic array for node/edge storage
tests/test_graph_buffer.c	Unit tests for graph buffer functionality
src/pipeline/*	Passes like pass_calls.c that populate the buffer
src/mcp/*	Orchestration layers managing buffer merging

Summary

• The graph storage implementation centers on cbm_gbuf_t in src/graph_buffer/graph_buffer.c, using CBMHashTable for indexing and CBMDynArray for storage.
• Key contribution points include adding secondary indexes via register_edge_in_indexes, optimizing make_edge_key for deduplication, and improving the cbm_gbuf_merge logic.
• Testing occurs in tests/test_graph_buffer.c, while serialization logic resides in the dump functions at the end of graph_buffer.c.
• The graph-buffer module serves as the in-memory representation that feeds the SQLite persistence layer, making it critical to the indexing pipeline.

Frequently Asked Questions

What role does the graph-buffer module play in the indexing pipeline?

The graph-buffer module provides an in-memory directed property graph representation that parallel pipeline passes populate using cbm_gbuf_upsert_node and cbm_gbuf_insert_edge. Once processing completes, cbm_gbuf_dump_to_sqlite serializes this graph into the final SQLite database tables (nodes, edges, vectors, and token_vectors).

How do I add a new secondary index to support custom queries?

Define a new hash table field in cbm_gbuf_t, initialize it in cbm_gbuf_new, and update register_edge_in_indexes (or the node equivalent) to populate it during insertion. You must also update deletion logic to remove entries from your new index when nodes or edges are deleted via the cascade-delete helpers.

What testing infrastructure should I use for graph-buffer changes?

The repository includes a dedicated test suite in tests/test_graph_buffer.c that validates node insertion, edge deduplication, merge operations, and cascade deletes. Run tests with ./tests/test_graph_buffer after building with make -f Makefile.cbm. Add new test cases following the existing C unit testing patterns.

How does the merge logic handle conflicts between parallel workers?

The cbm_gbuf_merge function combines buffers from parallel workers by inserting all nodes and edges from the source buffer into the destination. Edge deduplication relies on the edge_by_key hash table, which uses make_edge_key to generate unique keys. If identical edges exist in both buffers, the hash table prevents duplicates, ensuring idempotent merges.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}