How the File Watcher and Auto-Index Feature Works in Codebase-Memory MCP

The file watcher monitors Git repositories through adaptive polling and FNV-1a dirty-state signatures, automatically triggering auto-index callbacks only after confirming successful baseline commits.

The codebase-memory-mcp repository implements a high-performance incremental indexing pipeline written in C. Its file watcher and auto-index mechanism, located in src/watcher/, provides continuous change detection while minimizing CPU usage through snapshot-based concurrency and adaptive polling intervals.

Core Architecture and Registration

The watcher centers on the cbm_watcher_t opaque struct defined in src/watcher/watcher.h (lines 45-48). You initialize a watcher using cbm_watcher_new, which accepts a storage backend and an indexing callback:

cbm_watcher_t *watcher = cbm_watcher_new(store, my_index_callback, NULL);

The callback must conform to the cbm_index_fn signature: int (*cbm_index_fn)(const char *project, const char *root, void *userdata). It receives the project name, repository root path, and an opaque user data pointer.

Register projects for monitoring with cbm_watcher_watch:

cbm_watcher_watch(watcher, "my-project", "/path/to/repo");

This allocates a project_state_t entry (see state_new in watcher.c lines 70-75) and copies the root path into the watcher's internal hash table.

The Polling Loop and Snapshotting

The background execution loop runs via cbm_watcher_run(watcher, 0), where the second argument specifies a base interval (zero selects the 5-second default). This function repeatedly invokes cbm_watcher_poll_once until cbm_watcher_stop sets the atomic stopped flag.

To prevent lock contention, cbm_watcher_poll_once (lines 145-185 in watcher.c) operates on a snapshot of the current project list. It copies all active project_state_t* pointers under a brief global lock, then processes each entry without holding that lock. This allows the main thread to add or remove watches concurrently without blocking the poll cycle.

Change Detection and Baseline Management

Each project maintains a baseline representing the last known good state. During the first poll, init_baseline executes git rev-parse --git-dir to confirm the directory is a Git repository, then caches the current HEAD hash via git_head (lines 560-585) and counts tracked files.

Subsequent polls detect changes through two parallel checks in poll_project (lines 120-130):

1. HEAD comparison: Calls git_head again; a differing SHA indicates a commit, checkout, or pull operation.
2. Dirty-state signature: Invokes git_dirty_signature (lines 850-900), which runs git status --porcelain -uall -z and computes an FNV-1a hash over each entry combined with file size and mtime via sig_fold_path_stat.

If either the HEAD hash or the 64-bit dirty signature differs from the committed baseline, the watcher invokes the index_fn callback. A return value of zero signals success, prompting the watcher to commit the new baseline (last_head, last_dirty_sig). Non-zero return values trigger retry logic without updating the baseline, ensuring no changes are lost during indexing failures.

Adaptive Polling Strategy

CPU load scales with repository size through the cbm_watcher_poll_interval_ms function (lines 42-48). The interval calculates as:

base = 5000ms + (file_count / 500) * 1000ms
capped at 60000ms

This adapts the poll frequency based on tracked file count, mirroring the Go reference implementation. Large repositories poll less frequently to reduce system overhead, while small projects maintain responsiveness with 5-second intervals.

Stale-Root Pruning and Safety Guards

The watcher protects against data loss through stale-root pruning. In poll_project, the root_status check validates directory existence. If ENOENT or ENOTDIR persists for MISSING_ROOT_DELETE_AFTER (default 3) consecutive polls and the grace period (PRUNE_GRACE_DEFAULT_S, 10 minutes) expires, prune_missing_project (lines 67-80, 110-125) removes the cached database entry and the watch registration.

Advanced deployments can register a mutation guard via cbm_watcher_set_project_mutation_guard:

cbm_watcher_set_project_mutation_guard(watcher, prune_begin, prune_end, prune_notified, ctx);

This allows custom locking or logging before the watcher deletes project data, as demonstrated in tests/test_watcher.c around line 425.

Implementation Example

Embed the watcher in a daemon with custom indexing logic:

#include <watcher/watcher.h>
#include <store/store.h>

static int my_index(const char *project, const char *root, void *ud) {
    printf("Re-indexing %s at %s\n", project, root);
    /* Return 0 for success, >0 to retry, <0 for fatal error */
    return 0;
}

int main(void) {
    cbm_store_t *store = cbm_store_open_memory();
    cbm_watcher_t *watcher = cbm_watcher_new(store, my_index, NULL);
    
    cbm_watcher_watch(watcher, "backend", "/repos/backend");
    cbm_watcher_watch(watcher, "frontend", "/repos/frontend");
    
    /* Blocks until cbm_watcher_stop is called */
    cbm_watcher_run(watcher, 0);
    
    cbm_watcher_free(watcher);
    cbm_store_close(store);
    return 0;
}

To shut down from a signal handler or management thread:

void request_shutdown(cbm_watcher_t *watcher) {
    cbm_watcher_stop(watcher); /* Cancels ongoing git subprocesses */
}

Summary

• Registration: Use cbm_watcher_new and cbm_watcher_watch to bind indexing callbacks to repository paths.
• Concurrency: Snapshot-based polling in cbm_watcher_poll_once prevents lock contention during project enumeration.
• Detection: Changes are identified via HEAD hash comparison and FNV-1a dirty-state signatures computed from git status output and file metadata.
• Reliability: Baselines update only after successful callback completion (return 0), with automatic retry for transient failures.
• Efficiency: Adaptive intervals scale from 5 seconds to 60 seconds based on file count, reducing overhead for large codebases.
• Safety: Stale-root pruning removes entries only after sustained absence and configurable grace periods, preventing accidental data deletion.

Frequently Asked Questions

How does the watcher detect changes without missing rapid edits?

The dirty-state signature combines Git's porcelain status with file size and mtime into an FNV-1a hash. This captures both committed changes (via HEAD hash) and working directory modifications. Because the watcher only commits a new baseline after your callback returns success (0), rapid successive edits collapse into a single re-index operation rather than triggering redundant updates.

What happens if the indexing callback fails or crashes?

If the cbm_index_fn callback returns a positive value, the watcher treats this as a transient failure and retries the same project on the next poll cycle without committing the new baseline. A negative return value aborts that specific poll iteration but retains the watch. The previous baseline remains valid, ensuring no changes are marked as processed until the indexer confirms success.

Why does the poll interval increase with repository size?

The cbm_watcher_poll_interval_ms calculation adds one second per 500 tracked files (capped at 60 seconds) to minimize CPU and I/O pressure on large monorepos. According to the implementation in src/watcher/watcher.c, this mirrors the project's Go watcher reference and prevents the system from saturating disk resources scanning thousands of files at high frequency.

Can the watcher handle non-Git directories or disappearing projects?

The watcher strictly requires Git repositories and validates this via git rev-parse during baseline initialization. For disappearing roots, the root_status and prune_missing_project logic removes a project only after MISSING_ROOT_DELETE_AFTER (3) consecutive failed polls and a 10-minute grace period. This prevents accidental deletion of cached data during temporary filesystem unmounts or network outages.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}