How AgentsView Detects and Syncs New Agent Sessions: The File Watcher Mechanism Explained

AgentsView uses a debounced file watcher built on Go's fsnotify library to monitor session directories, automatically detect new agent sessions, and trigger database synchronization through a callback-driven pipeline.

The kenn-io/agentsview repository maintains a real-time SQLite store by watching directories that contain agent session files. The file watcher mechanism lives in internal/sync/watcher.go and wraps the underlying fsnotify package with recursive directory monitoring, resource exhaustion handling, and intelligent debouncing to batch rapid filesystem changes.

Core Architecture in internal/sync/watcher.go

The watcher implementation centers on a struct that manages an fsnotify.Watcher instance alongside configuration for debounce intervals and exclusion patterns. When initialized via NewWatcher (lines 47-69), the system stores a change callback function (onChange) that bridges filesystem events to the sync engine. This design keeps the watcher agnostic of agent-specific logic while providing the hooks necessary to trigger database updates.

The architecture supports both deep recursive watching and shallow monitoring. For massive directory trees, WatchRecursiveBudgeted (lines 80-124) walks the filesystem and adds each sub-directory to the underlying watch list, tracking root paths for exclusion checks. Conversely, WatchShallow (lines 30-38) registers only the top-level directory, relying on the auto-watch feature to catch dynamically created subdirectories.

Step-by-Step Detection Flow

Initializing the Watcher with NewWatcher

The NewWatcher function accepts three parameters: a debounce duration, an onChange callback, and a slice of exclude patterns. It constructs the underlying fsnotify.Watcher, normalizes exclusion globs (stripping leading "./" and deduplicating entries), and returns a configured watcher ready for directory registration.

// Create a watcher with 200ms debounce, ignoring version control and temp folders
watcher, err := sync.NewWatcher(
    200*time.Millisecond,
    func(paths []string) {
        // This callback receives batched changed paths
        engine.SyncPaths(paths)
    },
    []string{".git", "node_modules", "**/tmp/**"},
)

Registering Directories: Recursive and Shallow Modes

The watcher provides two registration strategies. WatchRecursive traverses the entire directory tree, while WatchRecursiveBudgeted stops when the OS limit for file descriptors is reached, setting a ResourceExhausted flag for the sync engine to handle via fallback polling. For large but shallow trees, WatchShallow monitors only the specified root, though the watcher will still auto-add subdirectories that belong to recursive roots to avoid shadowing.

The Event Loop and handleEvent

After calling Start(), a dedicated goroutine executes the loop function (lines 55-80). This loop selects across three channels: filesystem events, errors, and stop signals. When fsnotify reports a write, create, remove, or rename operation, handleEvent (lines 83-99) validates the event type and records the affected path in a thread-safe pending map with a timestamp.

Auto-Watching New Subdirectories

When a Create event targets a directory, the watchIfDir helper (lines 105-119) automatically adds it to the monitoring list. This mechanism catches dynamically created session-date folders such as sessions/2024/09/15/ without requiring manual re-registration. The function consults shouldExclude to skip paths matching user-defined glob patterns and avoids adding directories under shallow-only roots.

Debouncing and Batch Flushing

To prevent sync storms during rapid file modifications, the watcher implements a debounce mechanism. A ticker fires at the configured interval, triggering flush (lines 130-165). This function scans the pending map, selects paths whose timestamps exceed the debounce duration, removes them from the map, and invokes the onChange callback with the complete batch of changed paths. This coalesces multiple rapid events into a single database synchronization operation.

Triggering the Sync via Callback

The onChange callback closure is registered by the sync engine in cmd/agentsview/main.go. When invoked, it receives a slice of absolute paths that changed during the debounce window. The engine parses these session files, updates SQLite tables, and optionally pushes changes to PostgreSQL. The watcher itself remains unaware of agent semantics; it purely reports filesystem changes.

Key Design Features

Resource-Aware Recursion: The WatchRecursiveBudgeted method tracks OS file descriptor limits. When exhaustion occurs, it marks the error condition and allows the sync engine to fall back to periodic polling for the remaining subtree.

Exclusion Pattern Matching: Users specify glob-style patterns (e.g., **/tmp/**) that the watcher normalizes and validates via shouldExclude before adding directories to the monitor list.

Thread-Safe Bookkeeping: Two mutexes protect critical state: rootsMu guards the list of watched roots, while mu protects the pending change map. This ensures safe concurrent access between the fsnotify goroutine and the debounce ticker.

Shallow Root Handling: The watcher distinguishes between recursive and shallow roots. When auto-watching new directories, it checks against the shallow root list to prevent over-monitoring massive subtrees that the user explicitly marked as shallow.

Integration Example

The following example demonstrates setting up the watcher to monitor agent session directories and trigger synchronization:

package main

import (
    "log"
    "time"
    
    "github.com/kenn-io/agentsview/internal/sync"
)

func main() {
    // Initialize with 200ms debounce and exclusions
    watcher, err := sync.NewWatcher(
        200*time.Millisecond,
        func(paths []string) {
            log.Printf("Syncing %d changed paths", len(paths))
            // Engine processes session files and updates SQLite
            syncEngine.SyncPaths(paths)
        },
        []string{".git", "node_modules", "*.tmp"},
    )
    if err != nil {
        log.Fatal(err)
    }
    
    // Watch session directory recursively
    err = watcher.WatchRecursive("/home/user/.agentsview/sessions")
    if err != nil {
        log.Fatal(err)
    }
    
    watcher.Start()
    defer watcher.Stop()
    
    // Block or run other services...
    select {}
}

Summary

• The file watcher mechanism resides in internal/sync/watcher.go and wraps Go's fsnotify package.
• NewWatcher initializes the system with debounce configuration and exclusion patterns.
• WatchRecursive and WatchShallow provide flexible directory monitoring strategies.
• The event loop processes filesystem events and populates a pending map with timestamps.
• watchIfDir auto-registers newly created subdirectories, enabling dynamic session folder detection.
• The flush method debounces rapid changes, batching them into single sync operations.
• An onChange callback bridges filesystem events to the sync engine in cmd/agentsview/main.go.
• Thread-safe mutexes and resource exhaustion handling ensure robust production operation.

Frequently Asked Questions

How does the watcher handle newly created session directories?

When the filesystem reports a Create event for a directory, the watchIfDir function (lines 105-119 in internal/sync/watcher.go) automatically adds that directory to the fsnotify watch list. This enables the system to detect agent sessions written to date-organized folders like sessions/2024/09/15/ without requiring manual restarts or re-registration of watch paths.

What happens when the OS file descriptor limit is reached?

The WatchRecursiveBudgeted method tracks the number of directories being watched and stops recursion when the OS limit approaches. It sets a ResourceExhausted flag on the watcher, which the sync engine checks in cmd/agentsview/main.go. The engine then falls back to periodic polling for the unwatched portion of the directory tree, ensuring no sessions are missed while respecting system limits.

How does the debounce mechanism improve performance?

The watcher accumulates filesystem events in a pending map and only invokes the sync callback after the configured debounce interval (typically 200ms) has elapsed without new changes. Implemented in the flush method (lines 130-165), this coalesces rapid bursts of file operations—such as atomic rewrites or batch session imports—into a single database transaction, reducing SQLite lock contention and PostgreSQL sync overhead.

Can the watcher exclude specific directories from monitoring?

Yes. The NewWatcher constructor accepts a slice of glob patterns (e.g., ".git", "node_modules", "**/tmp/**"). These patterns are normalized and stored in the watcher struct. Before adding any directory to the fsnotify watch list, the shouldExclude method checks against these patterns, allowing users to skip version control folders, temporary directories, or large asset caches that would otherwise consume file descriptors and trigger unnecessary sync operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}