How the Auto-Sync Background Watcher Optimizes Performance in DeusData/codebase-memory-mcp

The auto-sync background watcher optimizes performance through adaptive polling intervals, FNV-1a dirty-state hashing, lock-free snapshotting, and deferred memory management, enabling continuous monitoring of multiple repositories with minimal CPU and I/O overhead.

The auto-sync background watcher is the core mechanism in DeusData/codebase-memory-mcp (Instagit) that continuously monitors watched projects for changes. Built in src/watcher/watcher.c, it balances responsiveness with resource efficiency using seven distinct optimization strategies that scale from small repositories to massive codebases containing thousands of files.

Adaptive Poll Intervals Based on Repository Size

The watcher implements dynamic interval scaling to prevent excessive CPU usage on large repositories. According to the source in src/watcher/watcher.c (lines 12‑19, 86‑89), the algorithm begins with a 5-second base interval and adds 1 second for every 500 tracked files, capping at 60 seconds maximum.

This cbm_watcher_poll_interval_ms calculation ensures small repositories remain responsive while preventing the background thread from saturating CPU cores when monitoring directories with tens of thousands of files. When initializing the watcher, passing 0 to cbm_watcher_run() activates this adaptive behavior:

cbm_watcher_t *w = cbm_watcher_new(store, my_index_fn, NULL);
cbm_watcher_watch(w, "my-project", "/path/to/repo");
cbm_watcher_run(w, 0);   // 0 → use adaptive base interval

Dirty-State Signature Detection

Rather than re-indexing on every poll cycle, the watcher computes a compact FNV-1a hash of the current repository state to detect actual changes. As implemented in lines 67‑77 and 108‑122, the system runs git status --porcelain -uall -z and combines the output with each file’s size and mtime to generate a git_dirty_signature.

The watcher compares this against a cached last_dirty_sig (lines 138‑140). Only when signatures differ does the system trigger a full re-index, eliminating redundant work on repositories that remain dirty between poll cycles.

Baseline Caching for At-Least-Once Delivery

The baseline caching mechanism records the current HEAD hash and dirty-state signature immediately following the first successful poll (lines 78‑85). Subsequent polls compare against these baselines, and the system updates them only after a successful re-index completes (lines 142‑158, 236‑252).

This ensures the watcher maintains at-least-once delivery semantics without losing intermediate changes that occur during lengthy indexing operations.

Stale-Root Pruning with Configurable Grace Windows

Projects whose root directories disappear are not immediately purged. Instead, the watcher tracks missing_root_count across consecutive polls (lines 91‑99, 107‑113). After three consecutive "root missing" detections and a configurable grace window (default 10 minutes or CBM_WATCHER_PRUNE_GRACE_S), the system prunes the project and deletes its cached database (lines 155‑171, 190‑210).

You can adjust this grace period at runtime:

/* Set a 5-minute grace period instead of default 600 seconds */
setenv("CBM_WATCHER_PRUNE_GRACE_S", "300", 1);

This prevents endless polling of deleted repositories while safeguarding against accidental data loss from transient filesystem issues.

Lock-Free Snapshotting for Concurrent Access

To minimize contention, the watcher implements lock-free snapshotting for its project list. Each poll begins by briefly acquiring a lock, copying all project pointers into a temporary array, and immediately releasing the lock (lines 76‑82, 98‑104, 124‑130).

All subsequent Git commands and indexing operations execute outside the critical section, allowing the main thread to add or remove watched projects without blocking the background worker. This architecture keeps latency low even when monitoring dozens of repositories simultaneously.

Deferred Free List for Safe Memory Management

When a project is unwatched or pruned, the watcher places its state structure on a deferred free list rather than immediately deallocating memory (lines 74‑78, 60‑73, 92‑100). The actual freeing occurs only after the current poll snapshot finishes processing.

This eliminates race conditions between the removal thread and the background polling loop without requiring long-duration lock holds that would degrade performance.

Responsive Shutdown with Chunked Sleeping

The run loop sleeps in SLEEP_CHUNK_MS increments of 500 milliseconds (lines 99‑101, 532‑564) rather than sleeping for the full poll duration. This chunked approach allows the watcher to detect shutdown requests quickly, ensuring the background thread terminates within half a second of receiving the stop signal rather than waiting for the entire adaptive interval to elapse.

Triggering Immediate Manual Syncs

For scenarios requiring instant synchronization, the API provides cbm_watcher_touch() to bypass the interval calculation:

/* Manually trigger an immediate poll for a specific project */
cbm_watcher_touch(w, "my-project");   // resets back-off, next poll runs now

This resets the internal timer and forces the next poll cycle to execute immediately, useful for CI/CD integrations or manual refresh operations.

Summary

• Adaptive intervals scale from 5 seconds to 60 seconds based on file count (500 files = +1s), preventing CPU saturation on large repositories.
• FNV-1a dirty-state signatures cached in last_dirty_sig eliminate redundant re-indexing of unchanged dirty trees.
• Baseline caching stores HEAD hashes and signatures post-index, ensuring no changes are lost during concurrent modifications.
• Stale-root pruning requires three consecutive misses plus a 10-minute grace period (CBM_WATCHER_PRUNE_GRACE_S) before deleting project data.
• Lock-free snapshotting copies project pointers under brief locks, executing Git operations outside critical sections to maintain concurrency.
• Deferred free lists postpone deallocation until after poll completion, eliminating race conditions without extended locking.
• 500ms sleep chunks enable sub-second shutdown responsiveness while maintaining the adaptive polling schedule.

Frequently Asked Questions

How does the auto-sync background watcher avoid re-indexing unchanged files?

The watcher computes an FNV-1a hash of the git status --porcelain -uall -z output combined with file sizes and mtimes to generate a compact git_dirty_signature. It compares this against the cached last_dirty_sig (as seen in src/watcher/watcher.c, lines 108‑122). Only when these signatures differ does the system trigger a re-index, eliminating unnecessary work on repositories that remain dirty between poll cycles.

What happens when a watched project's root directory is deleted?

The watcher increments a missing_root_count for each poll where the root is inaccessible. After three consecutive misses and a grace period (default 600 seconds configured via CBM_WATCHER_PRUNE_GRACE_S), the system prunes the project and deletes its cached database (lines 190‑210). This prevents indefinite polling of dead projects while protecting against accidental deletion due to transient network or filesystem issues.

How can I adjust the polling behavior for very large repositories?

The polling interval automatically scales by adding 1 second for every 500 tracked files up to a 60-second maximum. For manual control, you can trigger immediate polls using cbm_watcher_touch(), or modify the base interval logic by adjusting the adaptive scaling parameters in watcher.c (lines 86‑89). The architecture intentionally caps intervals to ensure changes are detected within one minute even on massive repositories.

Is the background watcher thread-safe during concurrent modifications?

Yes. The implementation uses lock-free snapshotting where the poll loop briefly locks only to copy project pointers into a temporary array (lines 76‑82), then releases the lock before executing Git commands. Combined with the deferred free list mechanism (lines 60‑73) that queues deletions until after the current poll completes, the system maintains thread safety without blocking the main thread during long-running index operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}