How Auto-Index and Git-Watcher Detect File Changes in Codebase-Memory-MCP

The DeusData/codebase-memory-mcp repository uses a dual-stage detection system where auto-index performs a one-time background scan at session startup, while the Git-watcher continuously polls via git rev-parse HEAD and git status --porcelain to detect commit changes and dirty worktree states, triggering re-indexing whenever the FNV-1a signature of file metadata changes.

Codebase-Memory-MCP implements an efficient file change detection architecture that avoids costly filesystem scans by leveraging Git plumbing commands. Both the auto-index and git-watcher features rely on Git-native tooling—specifically git rev-parse, git status, and git ls-files—to monitor repository state with minimal overhead. The system divides responsibilities between a one-shot initialization thread and a long-running polling daemon, coordinating through shared pipeline functions in src/mcp/mcp.c and src/watcher/watcher.c.

Auto-Index: One-Shot Detection at Session Startup

The auto-index feature triggers a background indexing run when a new session starts, but only if no existing database is detected. This mechanism executes in a separate thread and serves as the initial bootstrap before continuous monitoring begins.

Session Initialization and Database Check

When the server creates a session in src/mcp/mcp.c, the maybe_auto_index() function checks for an existing .db file using cbm_file_size(db_check). If a database already exists, the system skips indexing and immediately registers the project with the Git-watcher via register_watcher_if_enabled().

Configuration and File Count Guards

Before launching the indexing thread, the system validates two configuration parameters via cbm_config_get_bool() and cbm_config_get_int():

• auto_index: Boolean flag defaulting to off
• auto_index_limit: Integer defaulting to 50000 files

The function cbm_mcp_auto_index_within_file_limit() executes git ls-files -z with a 5-second timeout to count tracked files. Projects exceeding the limit abort automatically to prevent resource exhaustion.

Background Thread Execution

If all guards pass, cbm_thread_create() launches autoindex_thread, which instantiates a full-mode pipeline via cbm_pipeline_new() and executes cbm_pipeline_run(). Upon successful completion, the thread calls register_watcher_if_enabled() to transition the project to continuous monitoring mode.

Git-Watcher: Continuous Change Detection

While auto-index handles initial ingestion, the Git-watcher provides ongoing surveillance through an adaptive polling mechanism defined in src/watcher/watcher.c.

Project Registration and Baseline Establishment

The cbm_watcher_watch() function stores a project_state_t entry for each monitored project. During the first poll cycle, init_baseline() executes three Git commands to establish the reference state:

1. git rev-parse --git-dir to verify repository validity
2. git rev-parse HEAD to capture the current commit hash
3. git ls-files -z to enumerate tracked files

Adaptive Polling Intervals

Rather than using a fixed polling rate, the watcher calculates dynamic intervals via cbm_watcher_poll_interval_ms(). The formula adds 1 second per 500 tracked files to a base of 5 seconds, capping at 60 seconds. This prevents excessive CPU usage in large monorepos while maintaining responsiveness in smaller projects.

Dual-Strategy Change Detection

On each poll, check_changes() evaluates two independent signals:

HEAD Movement Detection: The watcher compares the output of git rev-parse HEAD against the stored last_head value. Any divergence—caused by new commits, checkouts, or pulls—triggers a re-index.

Dirty-State Signature Calculation: For uncommitted changes, the system executes git status --porcelain -uall -z and computes an FNV-1a hash using sig_fold_path_stat macros that incorporate file paths, sizes, and modification times. If the resulting signature differs from last_dirty_sig, the worktree is considered modified.

Re-Index Triggering and State Commitment

When either detection method identifies changes, poll_project() invokes the server-provided index_fn callback—the same function used by auto-index. Only after successful completion (rc == 0) does the watcher commit the new HEAD hash and dirty signature to the baseline, ensuring atomic state transitions.

Stale Project Pruning

The watcher includes garbage collection via prune_missing_project(). If a project's root directory disappears for a configurable number of polls (MISSING_ROOT_DELETE_AFTER) and exceeds the grace period (PRUNE_GRACE_DEFAULT_S), the system calls delete_cached_project_db() to remove the cached database and deletes the entry from the watch list.

Configuration and Integration Examples

Enabling Auto-Index via CLI

Configure the system to automatically index projects on session startup:

# Enable auto-index globally (disabled by default)

codebase-memory-mcp config set auto_index true

# Set maximum file count threshold (default: 50000)

codebase-memory-mcp config set auto_index_limit 20000

# Start session to trigger background indexing

codebase-memory-mcp session start /path/to/project

When the session initializes, maybe_auto_index() in src/mcp/mcp.c launches the background thread. Upon completion, the project automatically registers with the Git-watcher for continuous monitoring.

Programmatic Watcher Registration

Integrate the watcher directly using the C API:

// Register a project for continuous monitoring
bool success = cbm_watcher_watch(
    srv->watcher,
    "project-name",           // Unique project identifier
    "/absolute/path/to/repo"  // Repository root
);

if (!success) {
    fprintf(stderr, "Watcher registration failed\n");
}

This call creates a project_state_t entry in the watcher's hash table, beginning the adaptive polling cycle.

Custom Index Callback Implementation

Define a custom indexing function to handle detected changes:

int custom_indexer(const char *project, const char *root, void *userdata)
{
    // Execute full indexing pipeline
    cbm_pipeline_t *pipeline = cbm_pipeline_new(root, NULL, CBM_MODE_FULL);
    int result = cbm_pipeline_run(pipeline);
    return result; // Return 0 on success
}

// Initialize watcher with custom callback
cbm_watcher_t *watcher = cbm_watcher_new(store, custom_indexer, NULL);

The watcher invokes this callback whenever check_changes() detects HEAD movement or dirty-state modifications.

Summary

• Auto-index performs a one-time background scan at session startup using git ls-files to count and index tracked files, running in a separate thread spawned by cbm_thread_create() in src/mcp/mcp.c.
• Git-watcher continuously monitors repositories through adaptive polling in src/watcher/watcher.c, calculating intervals based on repository size (5s base + 1s per 500 files, capped at 60s).
• Change detection relies on two Git-native signals: HEAD hash comparison via git rev-parse HEAD and dirty-state signatures computed via git status --porcelain with FNV-1a hashing.
• Re-indexing occurs only when the watcher detects state changes, invoking the same pipeline used by auto-index through the configurable index_fn callback.
• Resource protection includes file-count limits, configurable timeouts, and automatic pruning of missing projects via prune_missing_project() after grace periods.

Frequently Asked Questions

How does the Git-watcher detect uncommitted changes?

The Git-watcher detects uncommitted changes by executing git status --porcelain -uall -z and computing an FNV-1a hash of the output using sig_fold_path_stat. This hash incorporates file paths, sizes, and modification times into a last_dirty_sig value. When the computed signature differs from the stored baseline, the system recognizes the worktree as dirty and triggers re-indexing through the index_fn callback.

What happens if a project has too many files for auto-indexing?

If cbm_mcp_auto_index_within_file_limit() determines the file count exceeds the auto_index_limit configuration (defaulting to 50000 files) via git ls-files -z, the auto-index aborts before launching the background thread. The system still registers the project with the Git-watcher if enabled, allowing manual or on-demand indexing while preventing resource exhaustion during automatic scans.

Can the polling interval be customized?

The polling interval adapts automatically based on repository size through cbm_watcher_poll_interval_ms() in src/watcher/watcher.c. While not directly configurable via user settings, the algorithm uses a base of 5 seconds plus 1 second per 500 tracked files, capped at 60 seconds. This ensures large monorepos don't overwhelm system resources while smaller projects receive frequent checks.

How does the system handle deleted or moved project directories?

The Git-watcher implements stale-root pruning via prune_missing_project() in src/watcher/watcher.c. If git rev-parse --git-dir fails or the directory disappears for more than MISSING_ROOT_DELETE_AFTER polls (with a grace period of PRUNE_GRACE_DEFAULT_S), the watcher automatically calls delete_cached_project_db() to remove the cached database and deletes the project entry from the watch list.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}