# How the File Watcher and Auto-Index Feature Works in Codebase-Memory MCP

> Learn how the file watcher and auto-index feature in Codebase-Memory MCP works. Discover adaptive polling and dirty-state signatures for efficient Git repository monitoring and automatic indexing.

- Repository: [Martin Vogel/codebase-memory-mcp](https://github.com/DeusData/codebase-memory-mcp)
- Tags: internals
- Published: 2026-07-30

---

**The file watcher monitors Git repositories through adaptive polling and FNV-1a dirty-state signatures, automatically triggering auto-index callbacks only after confirming successful baseline commits.**

The `codebase-memory-mcp` repository implements a high-performance incremental indexing pipeline written in C. Its **file watcher** and **auto-index** mechanism, located in `src/watcher/`, provides continuous change detection while minimizing CPU usage through snapshot-based concurrency and adaptive polling intervals.

## Core Architecture and Registration

The watcher centers on the `cbm_watcher_t` opaque struct defined in [`src/watcher/watcher.h`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/watcher/watcher.h) (lines 45-48). You initialize a watcher using `cbm_watcher_new`, which accepts a storage backend and an indexing callback:

```c
cbm_watcher_t *watcher = cbm_watcher_new(store, my_index_callback, NULL);

```

The callback must conform to the `cbm_index_fn` signature: `int (*cbm_index_fn)(const char *project, const char *root, void *userdata)`. It receives the project name, repository root path, and an opaque user data pointer.

Register projects for monitoring with `cbm_watcher_watch`:

```c
cbm_watcher_watch(watcher, "my-project", "/path/to/repo");

```

This allocates a `project_state_t` entry (see `state_new` in [`watcher.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/watcher.c) lines 70-75) and copies the root path into the watcher's internal hash table.

## The Polling Loop and Snapshotting

The background execution loop runs via `cbm_watcher_run(watcher, 0)`, where the second argument specifies a base interval (zero selects the 5-second default). This function repeatedly invokes `cbm_watcher_poll_once` until `cbm_watcher_stop` sets the atomic `stopped` flag.

To prevent lock contention, `cbm_watcher_poll_once` (lines 145-185 in [`watcher.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/watcher.c)) operates on a **snapshot** of the current project list. It copies all active `project_state_t*` pointers under a brief global lock, then processes each entry without holding that lock. This allows the main thread to add or remove watches concurrently without blocking the poll cycle.

## Change Detection and Baseline Management

Each project maintains a **baseline** representing the last known good state. During the first poll, `init_baseline` executes `git rev-parse --git-dir` to confirm the directory is a Git repository, then caches the current HEAD hash via `git_head` (lines 560-585) and counts tracked files.

Subsequent polls detect changes through two parallel checks in `poll_project` (lines 120-130):

1. **HEAD comparison**: Calls `git_head` again; a differing SHA indicates a commit, checkout, or pull operation.
2. **Dirty-state signature**: Invokes `git_dirty_signature` (lines 850-900), which runs `git status --porcelain -uall -z` and computes an FNV-1a hash over each entry combined with file size and mtime via `sig_fold_path_stat`.

If either the HEAD hash or the 64-bit dirty signature differs from the committed baseline, the watcher invokes the `index_fn` callback. A return value of zero signals success, prompting the watcher to **commit** the new baseline (`last_head`, `last_dirty_sig`). Non-zero return values trigger retry logic without updating the baseline, ensuring no changes are lost during indexing failures.

## Adaptive Polling Strategy

CPU load scales with repository size through the `cbm_watcher_poll_interval_ms` function (lines 42-48). The interval calculates as:

```

base = 5000ms + (file_count / 500) * 1000ms
capped at 60000ms

```

This adapts the poll frequency based on tracked file count, mirroring the Go reference implementation. Large repositories poll less frequently to reduce system overhead, while small projects maintain responsiveness with 5-second intervals.

## Stale-Root Pruning and Safety Guards

The watcher protects against data loss through **stale-root pruning**. In `poll_project`, the `root_status` check validates directory existence. If `ENOENT` or `ENOTDIR` persists for `MISSING_ROOT_DELETE_AFTER` (default 3) consecutive polls and the grace period (`PRUNE_GRACE_DEFAULT_S`, 10 minutes) expires, `prune_missing_project` (lines 67-80, 110-125) removes the cached database entry and the watch registration.

Advanced deployments can register a mutation guard via `cbm_watcher_set_project_mutation_guard`:

```c
cbm_watcher_set_project_mutation_guard(watcher, prune_begin, prune_end, prune_notified, ctx);

```

This allows custom locking or logging before the watcher deletes project data, as demonstrated in [`tests/test_watcher.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/tests/test_watcher.c) around line 425.

## Implementation Example

Embed the watcher in a daemon with custom indexing logic:

```c
#include <watcher/watcher.h>
#include <store/store.h>

static int my_index(const char *project, const char *root, void *ud) {
    printf("Re-indexing %s at %s\n", project, root);
    /* Return 0 for success, >0 to retry, <0 for fatal error */
    return 0;
}

int main(void) {
    cbm_store_t *store = cbm_store_open_memory();
    cbm_watcher_t *watcher = cbm_watcher_new(store, my_index, NULL);
    
    cbm_watcher_watch(watcher, "backend", "/repos/backend");
    cbm_watcher_watch(watcher, "frontend", "/repos/frontend");
    
    /* Blocks until cbm_watcher_stop is called */
    cbm_watcher_run(watcher, 0);
    
    cbm_watcher_free(watcher);
    cbm_store_close(store);
    return 0;
}

```

To shut down from a signal handler or management thread:

```c
void request_shutdown(cbm_watcher_t *watcher) {
    cbm_watcher_stop(watcher); /* Cancels ongoing git subprocesses */
}

```

## Summary

- **Registration**: Use `cbm_watcher_new` and `cbm_watcher_watch` to bind indexing callbacks to repository paths.
- **Concurrency**: Snapshot-based polling in `cbm_watcher_poll_once` prevents lock contention during project enumeration.
- **Detection**: Changes are identified via HEAD hash comparison and FNV-1a dirty-state signatures computed from `git status` output and file metadata.
- **Reliability**: Baselines update only after successful callback completion (return 0), with automatic retry for transient failures.
- **Efficiency**: Adaptive intervals scale from 5 seconds to 60 seconds based on file count, reducing overhead for large codebases.
- **Safety**: Stale-root pruning removes entries only after sustained absence and configurable grace periods, preventing accidental data deletion.

## Frequently Asked Questions

### How does the watcher detect changes without missing rapid edits?

The **dirty-state signature** combines Git's porcelain status with file size and mtime into an FNV-1a hash. This captures both committed changes (via HEAD hash) and working directory modifications. Because the watcher only commits a new baseline after your callback returns success (0), rapid successive edits collapse into a single re-index operation rather than triggering redundant updates.

### What happens if the indexing callback fails or crashes?

If the `cbm_index_fn` callback returns a positive value, the watcher treats this as a transient failure and retries the same project on the next poll cycle without committing the new baseline. A negative return value aborts that specific poll iteration but retains the watch. The previous baseline remains valid, ensuring no changes are marked as processed until the indexer confirms success.

### Why does the poll interval increase with repository size?

The `cbm_watcher_poll_interval_ms` calculation adds one second per 500 tracked files (capped at 60 seconds) to minimize CPU and I/O pressure on large monorepos. According to the implementation in [`src/watcher/watcher.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/watcher/watcher.c), this mirrors the project's Go watcher reference and prevents the system from saturating disk resources scanning thousands of files at high frequency.

### Can the watcher handle non-Git directories or disappearing projects?

The watcher strictly requires Git repositories and validates this via `git rev-parse` during baseline initialization. For disappearing roots, the `root_status` and `prune_missing_project` logic removes a project only after `MISSING_ROOT_DELETE_AFTER` (3) consecutive failed polls and a 10-minute grace period. This prevents accidental deletion of cached data during temporary filesystem unmounts or network outages.