# How Hister's File Watcher Uses fsnotify to Monitor Filesystem Changes

> Discover how Hister's file watcher leverages fsnotify to recursively monitor filesystem changes, debounce events, and trigger callbacks for efficient indexing. Learn about its dynamic directory support and graceful shutdown.

- Repository: [Adam Tauber/hister](https://github.com/asciimoo/hister)
- Tags: internals
- Published: 2026-09-01

---

**Hister wraps the fsnotify library in [`files/files.go`](https://github.com/asciimoo/hister/blob/main/files/files.go) to recursively watch configured directories, debounce rapid write events (200 ms), and trigger indexing callbacks while supporting dynamic directory additions and graceful shutdown via context cancellation.**

Hister, the open-source indexing tool maintained at `asciimoo/hister`, relies on a custom file-watching subsystem to detect filesystem changes in real time. At the core of this implementation is the popular **fsnotify** library, wrapped in a dedicated module that translates low-level OS events into high-level indexing operations. The `WatchDirectories` function in [`files/files.go`](https://github.com/asciimoo/hister/blob/main/files/files.go) orchestrates the entire process, from recursive directory registration to event debouncing and cleanup.

## Initializing the fsnotify Watcher

The entry point for Hister’s file watcher is the `WatchDirectories` function, which accepts a `context.Context`, a slice of directory configurations, and callback functions for file changes and removals.

According to the source code in [`files/files.go`](https://github.com/asciimoo/hister/blob/main/files/files.go) (lines 57‑60), the initialization follows this sequence:

1. **Create the underlying watcher** using `fsnotify.NewWatcher()`.
2. **Register all configured directories** and their subdirectories via `walkAndWatch`.
3. **Start the event loop** that monitors `watcher.Events` and `watcher.Errors`.

The watcher instance persists for the lifetime of the context, ensuring that all goroutines and OS resources are tied to the parent cancellation signal.

## Recursive Directory Registration with walkAndWatch

Before the event loop begins, Hister must register every directory that should be monitored. The `walkAndWatch` function (lines 71‑73) performs a depth-first traversal using `filepath.WalkDir` starting from each root directory defined in the configuration.

During traversal, the watcher applies three layers of filtering:

- **Hidden directories** (those starting with a dot) are skipped.
- **Known cache and dependency folders** defined in the `skipDirs` variable—such as `node_modules`, `.git`, and `vendor`—are ignored.
- **User-defined exclude patterns** are evaluated by `shouldSkipDir` to respect custom ignore rules.

Only directories passing all filters are added to the `fsnotify.Watcher` instance, preventing OS watch limit exhaustion on large codebases.

## The Event Loop Architecture

Once directories are registered, `WatchDirectories` enters a blocking select loop (lines 74‑96) that consumes two channels: `watcher.Events` for filesystem activity and `watcher.Errors` for operational failures.

The loop runs until the supplied context is cancelled, at which point it exits cleanly and closes the watcher (lines 76‑78). Inside the loop, events are dispatched to specialized handlers based on their operation type.

### Handling Write Events with Debouncing

To avoid spamming the indexer when an editor performs multiple rapid writes during a single save operation, Hister implements a **200 ms debounce timer** for write events.

The `handleWrite` function (lines 94‑114) maintains a map of `*time.Timer` instances keyed by file path. When a `fsnotify.Write` event arrives:

- If a timer exists for that path, it is reset.
- If no timer exists, a new one is created.
- When the timer expires, the user-provided `callback` function is invoked with the absolute file path.

This ensures that only the final state of a file is indexed, not intermediate writes.

### Dynamic Directory Watching on Create Events

Hister supports **on-the-fly directory registration** when new folders are created inside already-watched paths. The `handleCreate` function (lines 38‑48) inspects every `fsnotify.Create` event:

- If the created entry is a directory, it is immediately added to the watcher via `watcher.Add()`.
- If the created entry is a file that matches the directory’s include filters, the callback is triggered directly (lines 50‑54).

This mechanism allows Hister to index new project directories without requiring a process restart.

### Processing Remove and Rename Events

When files or directories are deleted or moved, Hister receives `fsnotify.Remove` or `fsnotify.Rename` events. The `handleRemove` function (lines 16‑28) processes these events by checking the parent directory’s configuration:

- If the directory is configured with `delete_on_remove: true`, the `onRemove` callback is executed to purge the path from the index.
- The watcher automatically cleans up internal state for removed paths, though explicit unwatching is handled defensively.

## Configuration-Driven Filtering and Path Validation

Hister exposes two utility functions to ensure events belong to valid watch targets:

- **`HasPathPrefix`** validates that an event’s path originates from a configured root directory.
- **`DirectoryMatchesPath`** checks whether a file matches the include/exclude patterns defined for its parent directory.

These helpers, combined with `shouldSkipDir`, ensure that build artifacts, hidden files, and dependency caches never trigger indexing operations, keeping the system efficient even on large monorepos.

## Graceful Shutdown Implementation

The watcher respects Go’s idiomatic context pattern for lifecycle management. When the parent context is cancelled—either through OS signals or application shutdown—the event loop detects `ctx.Done()` and initiates cleanup:

```go
ctx, cancel := context.WithCancel(context.Background())
defer cancel()

err := files.WatchDirectories(
    ctx,
    config.Load().Directories,                    // []*config.Directory
    func(path string) { indexer.Index(path) },    // index new/changed files
    func(path string) { indexer.Remove(path) },   // delete removed files
)
if err != nil {
    log.Fatal().Err(err).Msg("file watcher failed")
}

```

During shutdown (lines 76‑78), the loop exits and `watcher.Close()` is called, releasing all OS file descriptors and terminating background goroutines safely.

## Summary

- Hister’s file watcher is implemented in [`files/files.go`](https://github.com/asciimoo/hister/blob/main/files/files.go) as a thin wrapper around `fsnotify.NewWatcher()`.
- The `WatchDirectories` function recursively registers directories using `walkAndWatch`, filtering hidden paths and cache folders via `shouldSkipDir` and `skipDirs`.
- Write events are debounced for **200 ms** in `handleWrite` to prevent duplicate indexing during rapid saves.
- Create events trigger `handleCreate`, which dynamically adds new subdirectories to the watcher without restarting the process.
- Remove and rename events invoke `handleRemove`, which respects the `delete_on_remove` configuration flag.
- The entire subsystem shuts down gracefully when the parent `context.Context` is cancelled.

## Frequently Asked Questions

### How does Hister prevent duplicate indexing when files are saved rapidly?

Hister implements a **debounce mechanism** in the `handleWrite` function using a 200 ms `time.Timer`. Each write event resets the timer for that specific file path, and the indexing callback only fires after the debounce interval expires without new writes. This consolidates multiple rapid `fsnotify.Write` events—common during IDE auto-saves—into a single indexing operation.

### Can Hister detect and watch directories created after the process starts?

Yes. The `handleCreate` function monitors `fsnotify.Create` events and checks whether the new entry is a directory. If so, it immediately calls `watcher.Add()` to register the path, enabling Hister to index content in newly created folders without requiring a process restart or manual rescans.

### What happens when a watched directory is deleted while Hister is running?

When a directory is removed, the operating system sends a `fsnotify.Remove` or `fsnotify.Rename` event. Hister’s `handleRemove` function checks the parent directory’s `delete_on_remove` configuration; if enabled, it invokes the `onRemove` callback to purge the directory’s contents from the search index. The fsnotify library automatically cleans up internal watches for deleted paths.

### How does Hister exclude dependency folders like node_modules from monitoring?

During the initial `walkAndWatch` traversal and when handling create events, Hister checks paths against a built-in list of cache directories (`skipDirs`) and user-defined exclude patterns via `shouldSkipDir`. Directories matching these rules—such as `node_modules`, `.git`, or `vendor`—are never passed to `watcher.Add()`, conserving OS watch descriptors and preventing unnecessary indexing of third-party code.