# How the Graph Buffer Manages In-Memory Nodes Before SQLite Persistence

> Discover how the graph buffer manages in-memory nodes using heap allocation, hash tables, and string interning before flushing to SQLite. Understand the RAM-to-disk process.

- Repository: [Martin Vogel/codebase-memory-mcp](https://github.com/DeusData/codebase-memory-mcp)
- Tags: internals
- Published: 2026-07-07

---

**The graph buffer maintains nodes in RAM using heap-allocated structures, hash-table indexing, and string interning until `cbm_write_db` flushes the entire structure to SQLite.**

The **Codebase Memory MCP** project uses an in-memory graph buffer to stage semantic code entities before persisting them to disk. Located in [`src/graph_buffer/graph_buffer.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/graph_buffer/graph_buffer.c), this temporary storage layer handles node allocation, deduplication, and relationship tracking while extraction pipelines analyze source code. Understanding this architecture reveals how the system balances memory efficiency with fast incremental updates.

## Core Architecture of the Graph Buffer

The graph buffer serves as the transient authority for all graph operations during a pipeline run. It employs several specialized data structures to ensure O(1) lookups and deterministic behavior.

### Heap-Allocated Node Storage

Each extracted entity becomes a `cbm_gbuf_node_t` allocated individually on the heap. Unlike arena allocators, individual `calloc` calls ensure pointer stability even when the dynamic array holding node pointers grows or reallocates.

```c
cbm_gbuf_node_t *node = calloc(CBM_ALLOC_ONE, sizeof(cbm_gbuf_node_t));

```

This stability is critical because hash tables store direct pointers to nodes; any memory movement would invalidate the indexes.

### Hash Table Indexing Strategy

The buffer maintains multiple hash tables for rapid lookups without scanning the entire node array:

- **Primary QN index** (`node_by_qn`): Maps fully-qualified names (e.g., `my_pkg.my_func`) directly to node pointers.
- **ID index** (`by_id`): Maps sequential integer IDs to nodes for edge resolution.
- **Secondary indexes** (`nodes_by_label`, `nodes_by_name`): Group nodes by type or simple name for filtering operations.

```c
gb->node_by_qn = cbm_ht_create(CBM_SZ_256);
gb->nodes_by_label = cbm_ht_create(CBM_SZ_32);

```

These indexes enable the extraction passes to check for existing entities in constant time before creating duplicates.

### String Interning Pool

To minimize memory overhead, repetitive strings (labels, file paths, edge types) are stored once in an **intern pool** (`intern_pool`). All nodes reference the same immutable pooled copy via the `gb_intern` function.

```c
const char *gb_intern(cbm_gbuf_t *gb, const char *s) { … }
gb->intern_pool = cbm_ht_create(CBM_SZ_1K);

```

This guarantees that pointer comparisons can substitute for string equality checks during deduplication, and it prevents redundant allocations of common strings like `"Function"` or `"CALLS"`.

## Node Lifecycle and Upsert Logic

The buffer uses a deterministic **upsert** pattern to handle incremental updates from multiple extraction passes without creating redundant entities.

### Deterministic Node Upserts via cbm_gbuf_upsert_node

When extraction passes discover entities, they call `cbm_gbuf_upsert_node` rather than simple insertion. This function first queries the `node_by_qn` hash table to detect duplicates.

If a node with the same qualified name exists, the buffer applies **canonical ordering** rules to decide whether to keep the existing node or replace it:

1. **Smallest file path** (lexicographically) wins
2. If tied, **highest start line** wins
3. If still tied, **lexical name/label** breaks the tie

This eliminates nondeterministic "last-writer-wins" behavior that would make graph generation flaky across different runs.

```c
cbm_gbuf_node_t *existing = cbm_ht_get(gb->node_by_qn, qualified_name);
/* canonical ordering code selects winner */

```

### Edge Management and Deduplication

Relationships are inserted via `cbm_gbuf_insert_edge`, which similarly allocates heap memory and indexes edges by a composite key combining source ID, target ID, type, and optional local name.

```c
cbm_gbuf_edge_t *edge = calloc(CBM_ALLOC_ONE, sizeof(cbm_gbuf_edge_t));
cbm_ht_set(gb->edge_by_key, strdup(key), edge);

```

The composite key guarantees idempotency; attempting to insert the same relationship twice silently deduplicates to the first instance.

## Persistence Workflow

When the extraction pipeline completes, the buffer transitions from active memory store to serialization source.

### Flushing to SQLite

The `cbm_write_db` function in [`src/store/sqlite_writer.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/store/sqlite_writer.c) orchestrates persistence. It iterates over the buffer using `cbm_gbuf_foreach_node` and `cbm_gbuf_foreach_edge`, inserting records into the SQLite `nodes` and `edges` tables respectively.

```c
void cbm_write_db(cbm_gbuf_t *gb, …) {
    cbm_gbuf_foreach_node(gb, write_node_callback, db);
    cbm_gbuf_foreach_edge(gb, write_edge_callback, db);
    release_gbuf_indexes(gb);
}

```

This traversal preserves the graph structure while translating in-memory pointers to database foreign keys.

### Index Cleanup After Persistence

Following a successful write, `release_gbuf_indexes` destroys the primary hash tables (`node_by_qn`, `edge_by_key`, etc.) because the SQLite file becomes the authoritative store. The raw node and edge arrays remain allocated until `cbm_gbuf_free` releases all heap memory, allowing for post-write analysis or reporting if needed.

## Practical Implementation Examples

Initializing a graph buffer for a new project:

```c
cbm_gbuf_t *gb = cbm_gbuf_new("my-project", "/path/to/repo");

```

Upserting a function node with metadata:

```c
int64_t fn_id = cbm_gbuf_upsert_node(
    gb,
    "Function",
    "my_func",
    "my_pkg.my_func",
    "src/my_pkg/file.c",
    42,
    48,
    "{\"return_type\":\"int\"}"
);

```

Creating a relationship between entities:

```c
cbm_gbuf_insert_edge(
    gb,
    fn_id,
    other_fn_id,
    "CALLS",
    "{\"confidence\":1.0}"
);

```

Persisting the entire graph:

```c
int rc = cbm_write_db(gb, "output.db", "my-project");
if (rc == 0) {
    cbm_gbuf_free(gb);
}

```

## Summary

- **Individual heap allocation** via `calloc` ensures pointer stability for hash table indexes.
- **Multiple hash tables** (`node_by_qn`, `by_id`, `edge_by_key`) provide O(1) lookups and deduplication.
- **String interning** reduces memory footprint and enables pointer-based comparisons.
- **Canonical ordering rules** in `cbm_gbuf_upsert_node` guarantee deterministic graph generation.
- **Separation of concerns** leaves persistence logic strictly in [`src/store/sqlite_writer.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/store/sqlite_writer.c), invoked via `cbm_write_db`.

## Frequently Asked Questions

### What happens when two extraction passes try to create the same node?

The `cbm_gbuf_upsert_node` function detects existing entries via the `node_by_qn` hash table. It applies canonical ordering—smallest file path, then highest start line, then lexical name—to deterministically select which version persists, preventing duplicate nodes.

### Why does the buffer use individual calloc calls instead of arena allocation?

Individual `calloc` calls guarantee pointer stability. Since hash tables store direct pointers to `cbm_gbuf_node_t` structures, any reallocation of the underlying array (as happens with dynamic arrays) would invalidate those pointers. Heap-allocated nodes remain at fixed addresses regardless of array growth.

### How does string interning reduce memory usage?

The `gb_intern` function stores each unique string exactly once in `intern_pool`. All nodes referencing that string share the same pointer, eliminating redundant allocations for common values like `"Function"`, `"Class"`, or repeated file paths. This also allows fast pointer equality checks instead of string comparisons.

### When are the hash table indexes released?

After `cbm_write_db` successfully writes all nodes and edges to SQLite, it calls `release_gbuf_indexes` to free the primary hash tables. At this point, the SQLite database becomes the authoritative store, and the buffer retains only the raw node/edge arrays for final cleanup.