What Is the Index Supervisor in Codebase-Memory-MCP and When Does It Spawn Child Processes?

The index supervisor is a crash-isolation layer that spawns supervised worker subprocesses to handle repository indexing only when running in the host MCP process, preventing individual file failures from crashing the main server.

The Codebase-Memory-MCP server from DeusData uses the index supervisor to protect the long-running MCP process from crashes or hangs caused by problematic files during indexing. This lightweight orchestration layer ensures that the actual index_repository operation runs inside a separate binary process, isolating catastrophic failures from the main application.

Architecture of the Index Supervisor

Purpose and Design Goals

The index supervisor serves as a defensive barrier between the MCP server and the native indexing engine. Its primary responsibility is to execute the heavy indexing work inside a supervised worker subprocess—the same binary re-invoked with cli --index-worker … arguments—so that segmentation faults or infinite loops in the indexer never terminate the parent process.

This design prioritizes resilience over performance: while spawning a subprocess adds overhead, it guarantees that a single malformed file cannot bring down the entire MCP service.

Host Marking and Safety Guards

To prevent recursive spawning and ensure only the real MCP binary can delegate work, the supervisor implements a host-marking protocol. In src/main.c at approximately line 182, the real MCP binary calls cbm_index_supervisor_mark_host() at the start of main():

/* Mark this process as the host - only hosts may spawn workers */
cbm_index_supervisor_mark_host();   // src/main.c: ≈182

Embedded callers or library consumers never invoke this function, which prevents them from accidentally spawning workers. The supervisor tracks this state through internal globals (g_host_marked and g_worker_active) that distinguish between the host process and worker children.

When Does the Index Supervisor Spawn Child Processes?

Decision Logic in cbm_index_supervisor_should_wrap

The function cbm_index_supervisor_should_wrap() in src/mcp/index_supervisor.c (lines 73-86) determines whether to wrap the indexing call in a subprocess. It returns true only when all three conditions are met:

1. The process is the marked host (g_host_marked is set)
2. The process is not already an active worker (g_worker_active is false)
3. The kill-switch environment variable CBM_INDEX_SUPERVISOR is not set to 0

This triple-check ensures that workers cannot spawn additional workers, and users can disable supervision entirely by setting CBM_INDEX_SUPERVISOR=0.

Trigger Points in the Codebase

The supervisor is consulted at two critical points in the indexing path:

• File system watcher: In src/watcher/watcher.c, the watcher_index_fn callback triggers supervised indexing when repository changes are detected
• MCP request handler: In src/mcp/mcp.c at approximately line 4855, the function cbm_mcp_index_run_supervised_path checks the supervisor before executing index commands

When cbm_index_supervisor_should_wrap() returns true, these paths invoke cbm_index_spawn_worker() to create the isolated process.

How the Spawn Mechanics Work

Building the Worker Arguments

The cbm_index_spawn_worker() function in src/mcp/index_supervisor.c (lines 58-88 and 147-188) constructs a specific argument array to re-execute the current binary as a worker:

int cbm_index_spawn_worker(const char *args_json, bool single_thread,
                           const char *marker_file, const char *quarantine_file,
                           cbm_index_worker_result_t *result) {
    /* Build argv for the child process */
    const char *argv[8];
    int n = 0;
    argv[n++] = self;               // resolved binary path
    argv[n++] = "cli";
    argv[n++] = "--index-worker";
    argv[n++] = "index_repository";
    argv[n++] = args_json;
    argv[n++] = "--response-out";
    argv[n++] = resp_path;
    argv[n]   = NULL;
    
    /* fork + exec performed by cbm_proc_spawn ... */
}

The function resolves the binary path, creates temporary files for the worker's response and logs, sets optional probe environment variables, and then calls fork followed by exec.

Process Creation and Fallback Behavior

If the spawn succeeds, the parent process waits to reap the child, reads the response from the temporary file if the exit was clean, and continues operation. However, if spawning fails—for example, if the binary path cannot be resolved—the supervisor degrades gracefully to an in-process indexing run (see src/main.c lines 92-94). This fallback ensures that indexing remains functional even when process supervision is unavailable due to system constraints.

Summary

• The index supervisor isolates indexing crashes by running index_repository in a separate subprocess
• It only activates in the host MCP process (marked via cbm_index_supervisor_mark_host()) and never inside existing workers
• Spawning occurs when repository changes are detected by the watcher or during MCP request handling, provided the CBM_INDEX_SUPERVISOR environment variable is not disabled
• The worker process receives arguments via cli --index-worker index_repository with JSON configuration and writes responses to temporary files
• Failed spawns automatically fall back to in-process execution to maintain availability

Frequently Asked Questions

What happens if the index supervisor cannot spawn a worker process?

If cbm_index_spawn_worker() fails to resolve the binary path or execute the fork operation, the supervisor degrades to in-process execution. As implemented in src/main.c lines 92-94, the code detects the spawn failure and proceeds to run the indexing operation directly within the parent process, ensuring that repository indexing continues uninterrupted even when subprocess isolation is unavailable.

How does the index supervisor prevent recursive spawning?

The supervisor uses a state flag g_worker_active that is set when the current process is operating as a worker. The function cbm_index_supervisor_should_wrap() explicitly checks this flag and returns false if g_worker_active is true, preventing workers from spawning additional workers. Additionally, only processes that have called cbm_index_supervisor_mark_host()—typically done only in the real MCP binary's main() function—are permitted to spawn, while embedded library callers never set this mark.

Can I disable the index supervisor behavior?

Yes, you can disable the supervisor by setting the environment variable CBM_INDEX_SUPERVISOR to 0. When this kill-switch is detected, cbm_index_supervisor_should_wrap() returns false regardless of other state conditions, forcing all indexing operations to run in-process without subprocess isolation.

How can I monitor how many times the supervisor has spawned workers?

The supervisor exposes test hooks via cbm_index_supervisor_spawn_count() and cbm_index_supervisor_spawn_st_count() (defined in src/mcp/index_supervisor.c lines 50-64). These functions return the number of spawn attempts made, allowing the test suite to verify that embedding callers never trigger spawns and helping operators monitor indexing activity in production environments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}