How the Background Watcher Monitors and Responds to Git Changes in codebase-memory-mcp

The background watcher polls Git repositories using an adaptive interval, detects changes by comparing HEAD hashes and checking dirty status via git status --porcelain, and triggers a re-index callback whenever the working tree or commit history changes.

The DeusData/codebase-memory-mcp project implements a robust background watcher that keeps an in-memory index synchronized with Git repositories. This component runs as a background thread to continuously monitor registered projects for any modifications in their commit history or working tree. Understanding how the background watcher monitors Git changes is essential for developing tools that require real-time codebase awareness.

Per-Project State Management

When a project is registered via cbm_watcher_watch() in src/watcher/watcher.c (lines 24–33), the watcher allocates a project_state_t structure to track its metadata. This structure stores the project name, root path, the last known HEAD hash, a flag indicating whether the path is a valid Git repository, an adaptive poll interval, and the next scheduled poll deadline (next_poll_ns).

This stateful approach allows the watcher to maintain a baseline for each repository, enabling efficient incremental checks rather than full rescans. The per-project state persists in memory for the lifetime of the watcher, ensuring that historical context is available for comparison during each poll cycle.

Polling Loop and Git Detection

The core polling logic resides in cbm_watcher_poll_once(), which iterates over all registered projects. The function respects each project's adaptive interval, calculated via cbm_watcher_poll_interval_ms() (lines 82–88 in src/watcher/watcher.c), to balance responsiveness with system resource usage.

For each project, the polling loop executes two distinct phases: baseline initialization for new entries, and change detection for existing ones.

Baseline Initialization

When a project is first encountered, the watcher calls init_baseline() to establish the reference state. This process involves three key Git operations:

1. is_git_repo() (lines 102–115): Validates whether the path contains a .git directory or file.
2. git_head() (lines 92–135): Retrieves the current commit hash via git rev-parse HEAD and stores it as the baseline.
3. git_file_count() (lines 98–120): Counts tracked files to inform the adaptive polling interval calculation.

Once the baseline is established, the watcher transitions to active monitoring.

Change Detection Logic

On subsequent polls, check_changes() (lines 86–102) executes two verification steps:

• HEAD Comparison: The function re-invokes git_head() to retrieve the current commit hash. If this value differs from the stored HEAD hash, the watcher detects a commit, checkout, or pull operation.
• Working Tree Inspection: The function executes git status --porcelain via git_is_dirty() to identify uncommitted modifications, untracked files, or changes within submodules.

If either the HEAD hash has changed or the repository is dirty, the watcher flags the project as requiring re-indexing.

Triggering the Index Callback

When changes are detected, the watcher invokes the user-provided index callback (index_fn) at the call site located in src/watcher/watcher.c (lines 45–58). This callback is responsible for synchronizing the in-memory representation with the current repository state.

After a successful re-index, the watcher updates the stored HEAD hash and file count, then recalculates the adaptive polling interval based on the current repository size.

Adaptive Intervals and Run Loop

The watcher implements adaptive polling to optimize resource consumption. Larger repositories with more tracked files receive longer intervals between checks, while smaller projects are polled more frequently. The default base interval is 5 seconds, though this scales dynamically based on the file count reported by git_file_count().

The cbm_watcher_run() function drives the periodic polling loop. It checks an atomic stopped flag to determine when to terminate, sleeping in short chunks defined by SLEEP_CHUNK_MS to ensure responsive shutdown when cbm_watcher_stop() is called (see lines 21–30 and 31–46 in src/watcher/watcher.c). This architecture prevents the watcher from blocking indefinitely during sleep, allowing immediate termination upon request.

Implementation Example

The following example demonstrates how to initialize the watcher, register a callback, and manage the lifecycle:

/* 1. Define the index callback that will be run on a change */
static int reindex_project(const char *project_name,
                           const char *root_path,
                           void *user_data)
{
    /* user_data could hold a reference to the store or any context */
    printf("Re‑indexing %s (%s)\n", project_name, root_path);
    /* …perform the expensive indexing work here… */
    return 0;   // success
}

/* 2. Create the watcher and start watching a repository */
cbm_store_t *store = cbm_store_new();                // (store creation omitted)
cbm_watcher_t *watcher = cbm_watcher_new(store,
                                         reindex_project,
                                         NULL);

/* Add a project to watch – the path must be a valid Git repo */
cbm_watcher_watch(watcher, "my‑project", "/home/user/my-project");

/* 3. Run the watcher in a background thread (or call directly) */
int rc = cbm_watcher_run(watcher, 0);   // 0 → use default base interval
if (rc != 0) {
    fprintf(stderr, "Watcher failed to start\n");
}

/* 4. When the application is shutting down */
cbm_watcher_stop(watcher);   // request a clean shutdown
cbm_watcher_free(watcher);

Summary

• The background watcher maintains per-project state in project_state_t structures to track Git metadata and polling schedules.
• Change detection relies on comparing stored HEAD hashes and executing git status --porcelain via git_is_dirty() to identify dirty working trees.
• The adaptive polling interval scales with repository size, calculated by cbm_watcher_poll_interval_ms() using tracked file counts.
• Upon detecting changes, the watcher triggers the registered index_fn callback to re-index the project, then refreshes the baseline state.
• The run loop in cbm_watcher_run() uses chunked sleeping and an atomic stopped flag to enable immediate shutdown via cbm_watcher_stop().

Frequently Asked Questions

How does the watcher distinguish between new commits and uncommitted changes?

The watcher uses two separate mechanisms: it compares the current HEAD hash against the stored hash to detect commits, checkouts, or pulls, while simultaneously executing git status --porcelain via git_is_dirty() to detect uncommitted modifications in the working tree. Either condition triggers the re-index callback.

What is the default polling frequency, and can it be customized?

The default base interval is 5 seconds, passed to cbm_watcher_run(). However, the actual interval per project is adaptive, calculated by cbm_watcher_poll_interval_ms() based on the number of tracked files. You can specify a custom base interval by passing a non-zero value to cbm_watcher_run() instead of 0.

Which source files define the watcher API and implementation?

The public API is declared in src/watcher/watcher.h, which defines the watcher handle, lifecycle functions, and polling helpers. The full implementation, including project_state_t management, Git detection helpers, and the run-loop logic, resides in src/watcher/watcher.c.

How does the watcher ensure clean shutdown without blocking?

The implementation uses an atomic stopped flag checked by cbm_watcher_run() combined with SLEEP_CHUNK_MS micro-sleeps. This allows cbm_watcher_stop() to request termination, which the loop detects quickly rather than waiting for a full polling interval to complete.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}