What Is the Index Supervisor Subprocess in codebase-memory-mcp?

The index supervisor subprocess is a crash-isolation mechanism that runs the indexing pipeline in a separate worker process to protect the main MCP server from hangs or crashes caused by malformed source files.

The index supervisor subprocess sits between the MCP server and the actual indexing logic. According to the codebase-memory-mcp source code, it wraps the index_repository operation in a child process that can be killed if it hangs, ensuring the parent process remains stable even when processing pathological codebases.

How the Index Supervisor Subprocess Works

The supervisor operates through a six-stage pipeline that determines when to spawn a worker and how to manage its lifecycle.

Marking the Host Binary

The system first distinguishes the real binary from embedders or test binaries. In src/main.c, the supervisor calls cbm_index_supervisor_mark_host() early in the startup sequence to flag the genuine codebase-memory-mcp binary as the host.

/* At the very start of main() in src/main.c (lines 674-680) */
cbm_index_supervisor_mark_host();

This mark enables the supervisor gate only for the production binary, preventing accidental worker spawning in testing contexts.

Deciding When to Wrap

Before handling an index request, the code checks cbm_index_supervisor_should_wrap() in src/mcp/mcp.c (line 4794). The function returns true only when three conditions are met:

1. The host has been marked via cbm_index_supervisor_mark_host()
2. The current process is not already a worker
3. The environment variable CBM_INDEX_SUPERVISOR is not set to 0

This logic prevents recursive worker spawning and provides a kill-switch for debugging.

Spawning the Worker

When wrapping is required, cbm_index_spawn_worker() constructs a command line that re-invokes the same binary with the cli --index-worker flag:

<self> cli --index-worker index_repository <args_json> --response-out <tmp_file>

The function launches this via cbm_subprocess_run and passes the JSON arguments containing repository paths and configuration options.

Monitoring and Timeout Handling

The supervisor monitors the worker using a quiet-timeout mechanism. If the worker produces no new log lines within the configured window (default 15 minutes, customizable via CBM_INDEX_WORKER_TIMEOUT_S), the supervisor treats this as a hang and terminates the child process.

Collecting Results

On clean exit, the worker writes its JSON response to a temporary file specified by --response-out. The supervisor reads this file using slurp_file() into result->response. It also logs the child’s exit status and retains worker logs for post-mortem analysis unless the run was clean and profiling is disabled.

Implementation Details

The public API in src/mcp/index_supervisor.h provides the hooks needed to implement this pattern:

/* 1. Mark the real binary as host */
cbm_index_supervisor_mark_host();

/* 2. Check if we need to wrap this request */
if (cbm_index_supervisor_should_wrap()) {
    const char *args_json = "{\"repo_path\":\"/path/to/repo\"}";
    cbm_index_worker_result_t result = {0};

    /* 3. Spawn the supervised worker */
    if (cbm_index_spawn_worker(args_json,
                               /*single_thread=*/false,
                               /*marker_file=*/NULL,
                               /*quarantine_file=*/NULL,
                               &result) == 0) {
        printf("Worker succeeded: %s\n", result.response);
        cbm_index_worker_result_free(&result);
    } else {
        /* Fallback: run indexing in-process if spawning failed */
        run_index_in_process(args_json);
    }
}

In the worker process (the same binary re-invoked with --index-worker), the code identifies its role and sets the output path:

int main(int argc, char **argv) {
    /* Tell the supervisor we are a worker and where to write the response */
    cbm_index_set_worker_role(true, "/tmp/worker.response");
    
    /* Normal CLI handling continues */
    return cbm_cli_main(argc, argv);
}

Key Source Files

The implementation spans these critical files:

• src/mcp/index_supervisor.h — Public API declarations for cbm_index_supervisor_mark_host(), cbm_index_supervisor_should_wrap(), and cbm_index_spawn_worker()
• src/mcp/index_supervisor.c — Core implementation of the supervisor logic and subprocess management
• src/main.c — Marks the host early in program startup (lines 674-680)
• src/mcp/mcp.c — Calls the wrap check before indexing (line 4794)
• tests/test_mcp.c — Unit tests exercising supervisor counters g_spawn_count and g_spawn_st_count

Summary

• The index supervisor subprocess isolates the index_repository operation to prevent crashes from propagating to the main MCP server.
• It uses cbm_index_supervisor_should_wrap() to determine when to spawn a worker, checking host marks and environment kill-switches.
• Workers are launched via cbm_index_spawn_worker() with a 15-minute quiet timeout (configurable via CBM_INDEX_WORKER_TIMEOUT_S).
• Results are passed back through temporary JSON files, with logs preserved for debugging failures.
• Test hooks g_spawn_count and g_spawn_st_count ensure only parallel spawns occur in production.

Frequently Asked Questions

What triggers the index supervisor subprocess to spawn a worker?

The supervisor spawns a worker when cbm_index_supervisor_should_wrap() returns true. This requires that the host binary has been marked via cbm_index_supervisor_mark_host(), the current process is not already a worker, and the environment variable CBM_INDEX_SUPERVISOR is not set to 0.

How does the supervisor detect hangs in the indexing worker?

The supervisor monitors the worker for a quiet-timeout of 15 minutes by default. If the worker produces no new log lines within this window, the supervisor assumes the process is hung and terminates it. You can adjust this threshold by setting the CBM_INDEX_WORKER_TIMEOUT_S environment variable.

Can the index supervisor be disabled?

Yes. Setting the environment variable CBM_INDEX_SUPERVISOR=0 disables the wrapping mechanism, causing cbm_index_supervisor_should_wrap() to return false. This forces the indexing logic to run in-process, which is useful for debugging but removes the crash protection.

What happens if the worker subprocess fails to spawn?

If cbm_index_spawn_worker() returns a non-zero exit code, the code can fall back to running the indexing logic in the current process via run_index_in_process(). This ensures that indexing can still proceed even when subprocess creation fails, though without the isolation guarantees.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}