How the Background File Watcher in codebase-memory-mcp Detects and Handles Git Changes

The background file watcher polls repositories in a dedicated thread, detecting changes through HEAD movement and dirty-state hashing, then triggers re-indexing while adapting its polling interval to repository size.

The codebase-memory-mcp repository implements a robust background file watcher that continuously monitors Git repositories for changes. This component runs in a dedicated thread and periodically checks registered projects to determine when the codebase has evolved. Understanding its detection mechanisms reveals how the system maintains an up-to-date memory index without excessive CPU or disk usage.

The Dual-Signal Detection Strategy

The watcher employs two distinct Git-based signals to detect repository modifications. This dual approach ensures that both committed changes and working tree modifications trigger appropriate re-indexing.

HEAD Movement Detection

The first signal monitors the HEAD reference to detect new commits, branch checkouts, or pulls. In src/watcher/watcher.c, the function git_head() (lines 47-65) executes git rev-parse HEAD to retrieve the current commit hash.

Within check_changes() (lines 22-33), the watcher compares the freshly retrieved hash against the stored last_head value:

if (git_head(state->root_path, head, sizeof(head)) == 0) {
    if (strcmp(head, state->last_head) != 0) {
        state->changed = true;
        state->pending_head = head;
    }
}

Any mismatch indicates repository movement and sets the changed flag.

Dirty-State Signature Hashing

The second signal captures working tree modifications through a content hash. The git_dirty_signature() function (lines 17-34) generates a 64-bit FNV-1a hash by running git status --porcelain -uall -z and folding each entry with the file's size and mtime using sig_fold_path_stat() and sig_fold().

In check_changes() (lines 38-44), the watcher validates the current signature:

uint64_t sig = git_dirty_signature(state->root_path);
if (sig != state->last_dirty_sig) {
    state->changed = true;
    state->pending_dirty_sig = sig;
}

This approach efficiently detects uncommitted changes without scanning the entire file tree.

Adaptive Polling and Performance Optimization

The watcher dynamically adjusts its polling interval based on repository size to balance responsiveness against resource consumption. The function cbm_watcher_poll_interval_ms() (lines 12-17) calculates the interval using the formula:

Interval = 5 seconds + (1 second per 500 files)

The implementation uses git_file_count() (lines 7-29) to determine the tracked file count, capping the maximum interval at 60 seconds. This ensures that large repositories with thousands of files do not overwhelm the system with excessive polling while smaller projects remain responsive.

Change Handling and Baseline Commitment

When check_changes() detects modification, the watcher invokes the user-supplied indexing callback (index_fn). After successful re-indexing returns rc == 0, the system commits the observed baselines to prevent duplicate triggers:

• last_head updates to pending_head (lines 59-61)
• last_dirty_sig updates to pending_dirty_sig (line 62)

This atomic commitment ensures that only net-new changes trigger subsequent re-indexing operations. The watcher also handles edge cases such as missing root directories and graceful shutdowns, skipping change checks entirely for non-Git projects (line 32).

Implementation Files and API

The detection logic resides in the following source files:

• src/watcher/watcher.c – Core implementation including git_head(), git_dirty_signature(), and the polling loop
• src/watcher/watcher.h – Public interface for cbm_watcher_t creation and configuration
• tests/test_watcher.c – Unit tests validating change detection and baseline behavior

To register a project for monitoring:

cbm_watcher_t *watcher = cbm_watcher_new(store, my_index_fn, NULL);
cbm_watcher_watch(watcher, "my-project", "/path/to/repo");

The watcher then manages all subsequent detection automatically according to the adaptive polling schedule.

Summary

• The watcher runs in a dedicated thread and polls each registered project periodically
• Two signals detect changes: HEAD movement via git rev-parse HEAD and dirty-state via FNV-1a hashing of git status output
• check_changes() compares current states against stored baselines (last_head and last_dirty_sig)
• Successful re-indexing updates the baselines to prevent duplicate processing
• Polling intervals adapt dynamically based on file count (5s base + 1s per 500 files, max 60s)
• Non-Git projects are gracefully skipped during the poll cycle

Frequently Asked Questions

How does the watcher distinguish between committed changes and uncommitted working tree modifications?

The watcher uses separate detection mechanisms for each case. HEAD movement detection via git_head() identifies new commits, checkouts, and pulls by comparing commit hashes. Dirty-state signature detection via git_dirty_signature() captures working tree changes by hashing the output of git status --porcelain combined with file metadata. Both signals operate independently, ensuring that either committed or uncommitted changes trigger re-indexing.

What happens to the stored baselines after a successful re-index?

After the indexing callback returns successfully (rc == 0), the watcher commits the pending baselines observed during detection. Specifically, last_head updates to pending_head and last_dirty_sig updates to pending_dirty_sig as implemented in src/watcher/watcher.c lines 59-62. This prevents the same changes from triggering repeated indexing operations on subsequent poll cycles.

How does the polling interval scale with repository size?

The system implements adaptive polling through cbm_watcher_poll_interval_ms() in src/watcher/watcher.c. The interval calculates as 5 seconds plus 1 second for every 500 tracked files, as determined by git_file_count(), with a hard cap at 60 seconds. This algorithm ensures small repositories receive frequent checks while massive codebases avoid excessive polling overhead.

Does the background file watcher support non-Git projects?

Yes, the watcher gracefully handles non-Git directories by simply skipping the change detection logic. According to the implementation in src/watcher/watcher.c (line 32), if a project root lacks a Git repository, the watcher bypasses the check_changes() routine for that specific path, allowing the system to monitor mixed environments without errors.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}