How Hister's File Watcher Uses fsnotify to Monitor Filesystem Changes

Hister wraps the fsnotify library in files/files.go to recursively watch configured directories, debounce rapid write events (200 ms), and trigger indexing callbacks while supporting dynamic directory additions and graceful shutdown via context cancellation.

Hister, the open-source indexing tool maintained at asciimoo/hister, relies on a custom file-watching subsystem to detect filesystem changes in real time. At the core of this implementation is the popular fsnotify library, wrapped in a dedicated module that translates low-level OS events into high-level indexing operations. The WatchDirectories function in files/files.go orchestrates the entire process, from recursive directory registration to event debouncing and cleanup.

Initializing the fsnotify Watcher

The entry point for Hister’s file watcher is the WatchDirectories function, which accepts a context.Context, a slice of directory configurations, and callback functions for file changes and removals.

According to the source code in files/files.go (lines 57‑60), the initialization follows this sequence:

1. Create the underlying watcher using fsnotify.NewWatcher().
2. Register all configured directories and their subdirectories via walkAndWatch.
3. Start the event loop that monitors watcher.Events and watcher.Errors.

The watcher instance persists for the lifetime of the context, ensuring that all goroutines and OS resources are tied to the parent cancellation signal.

Recursive Directory Registration with walkAndWatch

Before the event loop begins, Hister must register every directory that should be monitored. The walkAndWatch function (lines 71‑73) performs a depth-first traversal using filepath.WalkDir starting from each root directory defined in the configuration.

During traversal, the watcher applies three layers of filtering:

• Hidden directories (those starting with a dot) are skipped.
• Known cache and dependency folders defined in the skipDirs variable—such as node_modules, .git, and vendor—are ignored.
• User-defined exclude patterns are evaluated by shouldSkipDir to respect custom ignore rules.

Only directories passing all filters are added to the fsnotify.Watcher instance, preventing OS watch limit exhaustion on large codebases.

The Event Loop Architecture

Once directories are registered, WatchDirectories enters a blocking select loop (lines 74‑96) that consumes two channels: watcher.Events for filesystem activity and watcher.Errors for operational failures.

The loop runs until the supplied context is cancelled, at which point it exits cleanly and closes the watcher (lines 76‑78). Inside the loop, events are dispatched to specialized handlers based on their operation type.

Handling Write Events with Debouncing

To avoid spamming the indexer when an editor performs multiple rapid writes during a single save operation, Hister implements a 200 ms debounce timer for write events.

The handleWrite function (lines 94‑114) maintains a map of *time.Timer instances keyed by file path. When a fsnotify.Write event arrives:

• If a timer exists for that path, it is reset.
• If no timer exists, a new one is created.
• When the timer expires, the user-provided callback function is invoked with the absolute file path.

This ensures that only the final state of a file is indexed, not intermediate writes.

Dynamic Directory Watching on Create Events

Hister supports on-the-fly directory registration when new folders are created inside already-watched paths. The handleCreate function (lines 38‑48) inspects every fsnotify.Create event:

• If the created entry is a directory, it is immediately added to the watcher via watcher.Add().
• If the created entry is a file that matches the directory’s include filters, the callback is triggered directly (lines 50‑54).

This mechanism allows Hister to index new project directories without requiring a process restart.

Processing Remove and Rename Events

When files or directories are deleted or moved, Hister receives fsnotify.Remove or fsnotify.Rename events. The handleRemove function (lines 16‑28) processes these events by checking the parent directory’s configuration:

• If the directory is configured with delete_on_remove: true, the onRemove callback is executed to purge the path from the index.
• The watcher automatically cleans up internal state for removed paths, though explicit unwatching is handled defensively.

Configuration-Driven Filtering and Path Validation

Hister exposes two utility functions to ensure events belong to valid watch targets:

• HasPathPrefix validates that an event’s path originates from a configured root directory.
• DirectoryMatchesPath checks whether a file matches the include/exclude patterns defined for its parent directory.

These helpers, combined with shouldSkipDir, ensure that build artifacts, hidden files, and dependency caches never trigger indexing operations, keeping the system efficient even on large monorepos.

Graceful Shutdown Implementation

The watcher respects Go’s idiomatic context pattern for lifecycle management. When the parent context is cancelled—either through OS signals or application shutdown—the event loop detects ctx.Done() and initiates cleanup:

ctx, cancel := context.WithCancel(context.Background())
defer cancel()

err := files.WatchDirectories(
    ctx,
    config.Load().Directories,                    // []*config.Directory
    func(path string) { indexer.Index(path) },    // index new/changed files
    func(path string) { indexer.Remove(path) },   // delete removed files
)
if err != nil {
    log.Fatal().Err(err).Msg("file watcher failed")
}

During shutdown (lines 76‑78), the loop exits and watcher.Close() is called, releasing all OS file descriptors and terminating background goroutines safely.

Summary

• Hister’s file watcher is implemented in files/files.go as a thin wrapper around fsnotify.NewWatcher().
• The WatchDirectories function recursively registers directories using walkAndWatch, filtering hidden paths and cache folders via shouldSkipDir and skipDirs.
• Write events are debounced for 200 ms in handleWrite to prevent duplicate indexing during rapid saves.
• Create events trigger handleCreate, which dynamically adds new subdirectories to the watcher without restarting the process.
• Remove and rename events invoke handleRemove, which respects the delete_on_remove configuration flag.
• The entire subsystem shuts down gracefully when the parent context.Context is cancelled.

Frequently Asked Questions

How does Hister prevent duplicate indexing when files are saved rapidly?

Hister implements a debounce mechanism in the handleWrite function using a 200 ms time.Timer. Each write event resets the timer for that specific file path, and the indexing callback only fires after the debounce interval expires without new writes. This consolidates multiple rapid fsnotify.Write events—common during IDE auto-saves—into a single indexing operation.

Can Hister detect and watch directories created after the process starts?

Yes. The handleCreate function monitors fsnotify.Create events and checks whether the new entry is a directory. If so, it immediately calls watcher.Add() to register the path, enabling Hister to index content in newly created folders without requiring a process restart or manual rescans.

What happens when a watched directory is deleted while Hister is running?

When a directory is removed, the operating system sends a fsnotify.Remove or fsnotify.Rename event. Hister’s handleRemove function checks the parent directory’s delete_on_remove configuration; if enabled, it invokes the onRemove callback to purge the directory’s contents from the search index. The fsnotify library automatically cleans up internal watches for deleted paths.

How does Hister exclude dependency folders like node_modules from monitoring?

During the initial walkAndWatch traversal and when handling create events, Hister checks paths against a built-in list of cache directories (skipDirs) and user-defined exclude patterns via shouldSkipDir. Directories matching these rules—such as node_modules, .git, or vendor—are never passed to watcher.Add(), conserving OS watch descriptors and preventing unnecessary indexing of third-party code.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}