Why the Index Supervisor Releases Memory Back to the OS

The index supervisor releases memory back to the OS by spawning indexing work in a child process that terminates after completion, allowing the operating system to automatically reclaim all allocated memory while the parent process cleans up only temporary files and response buffers.

The codebase-memory-mcp repository by DeusData implements a robust memory management strategy in its indexing pipeline. The index supervisor handles resource-intensive repository scanning by spawning isolated worker processes. This architecture ensures that the long-running MCP server or CLI parent process remains lightweight by delegating memory-heavy operations to ephemeral child processes.

Child Process Isolation

The index supervisor delegates all actual indexing work to a separate child process rather than executing it within the main MCP server thread. When the supervisor needs to index a repository, it re-invokes the same binary with the --index-worker flag using the cbm_subprocess_run() function.

This approach provides critical isolation benefits:

• Crash containment – Segmentation faults, aborts, or infinite loops in the indexer remain confined to the child process
• Hang protection – The supervisor monitors the child via a quiet-timeout mechanism and can terminate stuck workers
• Resource boundaries – Memory exhaustion in the worker cannot destabilize the parent MCP server

Automatic OS-Level Memory Reclamation

When the child process exits—whether it completes cleanly with CBM_PROC_CLEAN or is killed by the supervisor due to timeout—the operating system automatically reclaims all memory allocated by that process. This happens without any explicit free operations from the parent code.

This process-exit reclamation is significantly more reliable than manual memory management for complex indexing operations involving:

• Large file buffers
• Tree-sitter AST allocations
• Symbol table hash maps
• String pools for code snippets

By terminating the child, the supervisor ensures complete memory hygiene for the bulk of the work, preventing fragmentation and leaks in the long-running parent process.

Explicit Parent Process Cleanup

While the OS handles the child's memory, the supervisor must explicitly clean up resources created in the parent process. In src/mcp/index_supervisor.c, the parent performs targeted cleanup after reaping the child.

Temporary File Removal

The supervisor generates temporary files for inter-process communication that must be deleted after the worker finishes:

/* Spawn the worker and wait for it to finish */
int rc = cbm_subprocess_run(&opts, &r);   // Child process created here

/* After child exits, remove temporary communication files */
(void)remove(resp_path);

if (r.outcome == CBM_PROC_CLEAN && !cbm_profile_active) {
    (void)remove(log_path);   // Clean run → delete log file
}

Response Buffer Management

The parent allocates memory only for the final response string, which it reads from the temporary file:

if (r.outcome == CBM_PROC_CLEAN) {
    result->response = slurp_file(resp_path);   // Allocate response string
}

This buffer is later freed through cbm_index_worker_result_free():

free(result->response);   // Explicit free in cbm_index_worker_result_free()

Implementation Details in codebase-memory-mcp

The memory release strategy is implemented across three critical files:

• src/mcp/index_supervisor.h – Defines the cbm_subprocess_run() API and the cbm_index_worker_result structure that tracks outcomes and response data
• src/mcp/index_supervisor.c – Contains the fork-exec logic, quiet-timeout handling, and the cleanup sequence for temporary files and response buffers
• src/mcp/mcp.c – Orchestrates the supervision by calling the index supervisor when handling index_repository requests from the MCP protocol

Summary

• The index supervisor isolates memory-intensive indexing work in a child process to protect the parent MCP server
• Process termination triggers automatic memory reclamation by the operating system, which is more thorough than manual cleanup
• Explicit cleanup in the parent removes only temporary files (resp_path, log_path) and the response string buffer
• This architecture allows the MCP server to handle thousands of indexing operations without accumulating memory overhead

Frequently Asked Questions

How does the index supervisor prevent memory leaks?

The supervisor prevents memory leaks by delegating all indexing work to a child process. When the child terminates, the operating system automatically reclaims every byte of memory allocated by that process, regardless of whether the code properly called free(). This eliminates the possibility of leak accumulation in the long-running parent process.

What files does the supervisor clean up after indexing?

According to the source code in src/mcp/index_supervisor.c, the supervisor removes the response path (resp_path) unconditionally after reading the result, and removes the log path (log_path) only when the outcome is CBM_PROC_CLEAN and profiling is not active. These temporary files facilitate communication between the parent and child processes.

Why use a child process instead of threads for indexing?

The codebase uses a child process instead of threads to achieve complete isolation. A crashing or memory-leaking worker cannot corrupt the parent process state. Additionally, the supervisor can set timeouts and forcibly terminate hung workers—capabilities that are difficult to implement safely with threads in C. The process boundary also provides the automatic memory release mechanism described above.

What happens if the index worker exceeds the quiet-timeout?

If the index worker exceeds the configured quiet-timeout, the supervisor forcibly terminates the child process. Upon termination, the operating system immediately reclaims all memory allocated by the worker. The supervisor then reports the timeout outcome and cleans up the temporary files, returning the MCP server to a clean state without memory pollution from the aborted operation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}