How Codebase-Memory-MCP Handles Background Auto-Sync and Git Polling

Codebase-Memory-MCP maintains an up-to-date code index by running a continuous background watcher service that polls Git repositories at adaptive intervals, detects HEAD changes and dirty working trees, and triggers re-indexing automatically.

Codebase-Memory-MCP is an open-source Model Context Protocol (MCP) server that keeps AI coding assistants synchronized with your codebase. According to the DeusData/codebase-memory-mcp source code, the system implements a robust background auto-sync mechanism called the watcher that continuously monitors project directories for Git changes without blocking the main application thread.

Core Architecture and Project State

The watcher implementation resides in src/watcher/ and maintains a dedicated state structure for every monitored project. As defined in watcher.c lines 38‑48, the project_state_t struct tracks:

• last_head – The last known Git HEAD hash to detect branch switches or new commits
• is_git – A boolean flag indicating whether the directory is a valid Git repository
• next_poll_ns – The timestamp for the next scheduled poll
• interval_ms – The adaptive poll interval that scales with project size

This per-project state allows the watcher to treat each directory independently, applying different polling frequencies based on the repository's file count and current Git status.

Adaptive Poll Interval Strategy

To balance responsiveness against system load, Codebase-Memory-MCP implements an adaptive polling algorithm that increases the interval as project size grows. The constants defined in watcher.c establish the baseline behavior:

#define POLL_BASE_MS   5000            // 5 s base
#define POLL_FILE_STEP 500             // +1 s per 500 files
#define POLL_MAX_MS   60000            // cap at 60 s

The function cbm_watcher_poll_interval_ms() at lines 98‑104 computes the interval using the formula:

ms = POLL_BASE_MS + (file_count / POLL_FILE_STEP) * 1000
ms = min(ms, POLL_MAX_MS)

A small project with fewer than 500 files polls every 5 seconds, while a monorepo with thousands of files may poll only once per minute. This approach prevents excessive CPU and disk usage on large codebases while ensuring rapid detection of changes in smaller repositories.

Git Change Detection Pipeline

The watcher detects three categories of changes to determine if re-indexing is required. Each check uses raw Git command execution rather than library bindings to minimize dependencies.

Repository Validation

Before polling, is_git_repo() at watcher.c lines 18‑31 executes git rev-parse --git-dir. If the command exits with a non-zero status, the project is marked as non-Git and the watcher skips further Git checks for that cycle.

HEAD Comparison

To detect new commits or branch switches, git_head() at lines 33‑50 runs git rev-parse HEAD and compares the output string against the stored last_head value in the project state. A mismatch indicates the repository has moved forward and requires re-indexing.

Working Tree Status

Even when HEAD remains constant, uncommitted changes must be captured. The function git_is_dirty() at lines 53‑84 executes git status --porcelain --untracked-files=normal to check for modified, staged, or untracked files. On POSIX platforms, it additionally runs git submodule foreach to detect changes within submodules that git status alone would miss.

If either the HEAD hash changed or the working tree is dirty, the watcher invokes the user-provided cbm_index_fn callback to trigger the re-indexing pipeline.

The Polling Loop and Lifecycle

The watcher exposes two primary modes of operation through the public API in watcher.h.

Single Poll Execution

cbm_watcher_poll_once() (declared at lines 54‑57) iterates through the internal hash table of watched projects, executes the Git detection pipeline for each, and invokes the index callback for any projects requiring updates. It returns an integer count of how many projects were re-indexed during that cycle.

Continuous Background Loop

For autonomous operation, cbm_watcher_run() at lines 58‑63 enters an indefinite loop that:

1. Calculates sleep duration based on each project's next_poll_ns timestamp
2. Sleeps for the shortest pending interval
3. Updates next_poll_ns based on the adaptive interval calculation
4. Checks for a thread-safe stop signal set by cbm_watcher_stop()

This design allows the watcher to run on a dedicated thread while remaining responsive to shutdown requests.

Stale Root Pruning

When project directories are deleted or moved, the watcher must clean up cached metadata to prevent database bloat. As implemented in watcher.c lines 77‑84, the system tracks consecutive "missing-root" poll failures. After MISSING_ROOT_DELETE_AFTER (3) consecutive failures and a grace period of PRUNE_GRACE_DEFAULT_S (600 seconds), the watcher deletes the project's cached database and removes the watch entry. This protects against accidental data loss during temporary filesystem unmounts or network disconnects.

Implementation Example

The following pattern demonstrates creating a watcher, registering a callback, and running the background loop:

static int index_project(const char *name, const char *root, void *ud) {
    // Trigger MCP’s indexing pipeline for this project.
    return cbm_mcp_index(root);
}

int main(void) {
    cbm_store_t *store = cbm_store_open("cbm.db");
    cbm_watcher_t *watcher = cbm_watcher_new(store, index_project, NULL);

    cbm_watcher_watch(watcher, "my-app", "/home/user/my-app");
    cbm_watcher_run(watcher, 5000);   // Run until cbm_watcher_stop() is called
    cbm_watcher_free(watcher);
    cbm_store_close(store);
}

For testing or manual control, you can invoke a single poll cycle without blocking:

int changed = cbm_watcher_poll_once(watcher);
printf("Re‑indexed %d project(s) this cycle\n", changed);

Summary

• Adaptive intervals scale from 5 seconds to 60 seconds based on file count, calculated by cbm_watcher_poll_interval_ms() in watcher.c lines 98‑104.
• Git detection relies on git rev-parse for HEAD validation and git status --porcelain for dirty checking, with submodule awareness on POSIX systems.
• State management uses project_state_t to track per-project HEAD hashes, Git status, and scheduling metadata.
• Automatic cleanup removes stale entries after three consecutive missing-root polls and a 10-minute grace period to prevent orphaned database entries.
• Thread-safe lifecycle allows graceful shutdown via cbm_watcher_stop() while the background loop runs indefinitely through cbm_watcher_run().

Frequently Asked Questions

How does the adaptive polling interval work in Codebase-Memory-MCP?

The watcher calculates the poll interval using the formula POLL_BASE_MS + (file_count / 500) * 1000, capping the result at POLL_MAX_MS (60 seconds). This means a repository with 2,500 files polls every 10 seconds, while a 50,000-file monorepo polls every 60 seconds. This logic resides in watcher.c lines 98‑104.

What Git commands does the watcher use to detect changes?

The system executes three distinct Git commands: git rev-parse --git-dir to validate the repository, git rev-parse HEAD to read the current HEAD hash, and git status --porcelain --untracked-files=normal to detect working tree modifications. On POSIX systems, it also runs git submodule foreach to catch changes in submodules.

Can I manually trigger a poll instead of running the continuous loop?

Yes. While cbm_watcher_run() provides the continuous background loop, you can call cbm_watcher_poll_once() to execute a single synchronization cycle across all watched projects. This function returns the count of projects that required re-indexing and is useful for testing or cron-based invocation.

How does the system handle deleted or moved project directories?

The watcher tracks consecutive poll failures where the root directory is missing. After MISSING_ROOT_DELETE_AFTER (3) consecutive failures and a grace period of PRUNE_GRACE_DEFAULT_S (600 seconds), it automatically deletes the cached database for that project and removes the watch entry, as implemented in watcher.c lines 77‑84.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}