# How the LLM-Wiki Source Folder Auto-Watch Feature Works: Implementation and Configuration

> Discover how LLM-Wiki's source folder auto-watch uses Rust file watching and TypeScript rules to efficiently queue files for knowledge graph ingestion. Learn implementation and configuration.

- Repository: [nash_su/llm_wiki](https://github.com/nashsu/llm_wiki)
- Tags: internals
- Published: 2026-09-13

---

**The source folder auto-watch feature uses a Rust-native file system watcher to monitor directory changes, filters events through configurable TypeScript rules, and automatically queues valid files for parsing and ingestion into the knowledge graph.**

The auto-watch capability in `nashsu/llm_wiki` creates a real-time bridge between your local source directories and the application's ingestion pipeline. Implemented across the TypeScript frontend and Rust backend, the system watches for file system events, applies strict filtering logic based on user-defined configuration, and triggers automated processing workflows. This architecture ensures that documentation updates and code changes are indexed immediately without manual intervention.

## Configuration Layer: SourceWatchConfig Schema and Defaults

The feature centers around the `SourceWatchConfig` interface, which defines rules for what the watcher should monitor and ignore.

### Default Configuration Storage

Out-of-the-box settings are stored in [`src/lib/source-watch-defaults.json`](https://github.com/nashsu/llm_wiki/blob/main/src/lib/source-watch-defaults.json) and exposed through [`src/lib/source-watch-config.ts`](https://github.com/nashsu/llm_wiki/blob/main/src/lib/source-watch-config.ts) as the `DEFAULT_SOURCE_WATCH_CONFIG` constant. This configuration object specifies:

- `includeExtensions`: Array of file extensions to explicitly watch
- `excludeExtensions`: Array of extensions to ignore
- `excludeDirs`: Directory names to recursively skip (e.g., `node_modules`, `.git`)
- `excludeGlobs`: Glob patterns for path-based exclusion
- `maxFileSizeMb`: Maximum file size threshold (capped at 4096 MB)
- `parsingConcurrency`: Parallel parsing limit (capped at 8 workers)
- `ingestConcurrency`: Parallel ingestion limit
- `persistExtractedMarkdown`: Boolean flag to save extracted content

### Configuration Normalization and Safety Limits

When users provide partial configurations, the `normalizeSourceWatchConfig()` function in [`src/lib/source-watch-config.ts`](https://github.com/nashsu/llm_wiki/blob/main/src/lib/source-watch-config.ts) merges inputs with defaults while enforcing safety boundaries. The function automatically caps `parsingConcurrency` at 8 to prevent resource exhaustion and limits `maxFileSizeMb` to 4096 to avoid memory issues with the Rust file system watcher.

## Filtering Logic: The isPathAllowedBySourceWatch Decision Tree

The core filtering mechanism resides in `isPathAllowedBySourceWatch(path, config)`, which executes a five-stage validation pipeline to determine if a file event warrants processing:

1. **Exclude Directory Check**: Validates against `excludeDirs` using `pathMatchesExcludedDir()`
2. **Glob Pattern Matching**: Tests against `excludeGlobs` using `matchesGlob()`
3. **Hidden File Detection**: Automatically rejects files starting with `.` (dotfiles)
4. **Extension Exclusion**: Removes files matching `excludeExtensions` via `getSourceWatchExtension()`
5. **Extension Inclusion**: If `includeExtensions` is defined, only allows matches; otherwise permits all remaining files

### Helper Utilities for Path Validation

The filtering system relies on three specialized utilities in [`src/lib/source-watch-config.ts`](https://github.com/nashsu/llm_wiki/blob/main/src/lib/source-watch-config.ts):

- **`getSourceWatchExtension(path)`**: Extracts and normalizes file extensions for comparison
- **`matchesGlob(path, pattern)`**: Implements glob pattern matching against file paths
- **`pathMatchesExcludedDir(path, excludeDirs)`**: Checks if any path segment matches excluded directory names

## Runtime Architecture: From Rust File System Events to TypeScript Ingestion

The auto-watch system spans the Tauri frontend-backend boundary, utilizing the `notify` crate for efficient native file system monitoring.

### Starting the Native Watcher via Tauri Commands

Watcher lifecycle management occurs in [`src/commands/file-sync.ts`](https://github.com/nashsu/llm_wiki/blob/main/src/commands/file-sync.ts), which exposes two primary functions:

- **`start_project_file_watcher`**: Invokes the Tauri backend command to instantiate a recursive file system watcher on the project's source directory
- **`stop_project_file_watcher`**: Terminates the active watcher and cleans up resources

On the Rust side in [`src-tauri/src/lib.rs`](https://github.com/nashsu/llm_wiki/blob/main/src-tauri/src/lib.rs), these commands initialize the `notify` crate's recursive watcher, which monitors the configured directory tree at the OS level.

### Event Flow Through the Project-File-Sync Store

When the Rust backend detects a file change, it emits events to the TypeScript frontend via Tauri's invoke system. The [`src/lib/project-file-sync.ts`](https://github.com/nashsu/llm_wiki/blob/main/src/lib/project-file-sync.ts) module receives these events and:

1. Parses the file path from the event payload
2. Calls `isPathAllowedBySourceWatch()` to apply configuration filters
3. Enqueues allowed files into the processing queue while respecting concurrency limits

### The Source-Lifecycle Ingestion Pipeline

Queued files enter [`src/lib/source-lifecycle.ts`](https://github.com/nashsu/llm_wiki/blob/main/src/lib/source-lifecycle.ts), which orchestrates the complete ingestion workflow:

- **Preprocessing**: Hands files to [`src/lib/source-preprocess.ts`](https://github.com/nashsu/llm_wiki/blob/main/src/lib/source-preprocess.ts) for content extraction
- **Concurrency Control**: Respects `parsingConcurrency` and `ingestConcurrency` limits from the normalized config to prevent system overload
- **Persistence**: Conditionally saves extracted Markdown when `persistExtractedMarkdown` is enabled
- **Knowledge Graph Updates**: Integrates processed content into the wiki's searchable index

## Implementation Example: Configuring and Activating the Watcher

The following example demonstrates loading configuration, validating a file path, and initiating the watch process:

```typescript
import { invoke } from '@tauri-apps/api/tauri';
import { 
  normalizeSourceWatchConfig, 
  isPathAllowedBySourceWatch,
  DEFAULT_SOURCE_WATCH_CONFIG 
} from '@/lib/source-watch-config';

// 1. Initialize configuration with user overrides
const userConfig = {
  includeExtensions: ['md', 'txt', 'rs'],
  excludeDirs: ['target', 'node_modules'],
  maxFileSizeMb: 50,
  parsingConcurrency: 4
};
const watchConfig = normalizeSourceWatchConfig(userConfig);

// 2. Validate a file path before processing
const changedFile = '/home/user/project/src/main.rs';
if (isPathAllowedBySourceWatch(changedFile, watchConfig)) {
  console.log('File allowed for ingestion:', changedFile);
}

// 3. Start the native file system watcher
await invoke('start_project_file_watcher', { projectId: 'my-wiki-project' });

```

## Summary

- The **source folder auto-watch feature** combines Rust's `notify` crate with TypeScript configuration management to monitor directories in real-time.
- **File filtering** occurs through `isPathAllowedBySourceWatch()` in [`src/lib/source-watch-config.ts`](https://github.com/nashsu/llm_wiki/blob/main/src/lib/source-watch-config.ts), which applies a five-stage validation pipeline including extension checks and glob matching.
- **Configuration normalization** enforces safety limits on concurrency (max 8 parsing workers) and file sizes (max 4096 MB) to prevent resource exhaustion.
- **Event flow** moves from Rust backend watchers through Tauri invokes to [`src/lib/project-file-sync.ts`](https://github.com/nashsu/llm_wiki/blob/main/src/lib/project-file-sync.ts), which queues valid files for the [`src/lib/source-lifecycle.ts`](https://github.com/nashsu/llm_wiki/blob/main/src/lib/source-lifecycle.ts) ingestion pipeline.
- **Lifecycle management** commands `start_project_file_watcher` and `stop_project_file_watcher` in [`src/commands/file-sync.ts`](https://github.com/nashsu/llm_wiki/blob/main/src/commands/file-sync.ts) control the native watcher state.

## Frequently Asked Questions

### What file types does the source folder auto-watch feature support?

The watcher supports any file type defined in the `includeExtensions` array of the `SourceWatchConfig`. By default, the system can monitor all files unless specifically excluded, though in practice, LLM-Wiki configures specific extensions like `.md`, `.txt`, and source code files based on the project's ingestion requirements.

### How does the watcher prevent system overload from large directories or files?

The system implements multiple safeguards: `maxFileSizeMb` prevents processing files exceeding the configured limit (capped at 4096 MB), `excludeDirs` automatically skips directories like `node_modules` and `.git`, and `parsingConcurrency` limits parallel processing to a maximum of 8 concurrent workers to manage CPU and memory usage.

### Can I watch multiple source folders simultaneously across different projects?

Yes, each project can maintain its own `SourceWatchConfig` and watcher instance. The `start_project_file_watcher` and `stop_project_file_watcher` commands accept a `projectId` parameter, allowing the Rust backend to manage separate watcher instances for different directories concurrently.

### Where is the actual file system watching logic implemented in the codebase?

The native file system watching logic resides in the Rust backend at [`src-tauri/src/lib.rs`](https://github.com/nashsu/llm_wiki/blob/main/src-tauri/src/lib.rs) using the `notify` crate, while the TypeScript frontend controls the watcher via [`src/commands/file-sync.ts`](https://github.com/nashsu/llm_wiki/blob/main/src/commands/file-sync.ts). Event filtering and processing logic is implemented in [`src/lib/source-watch-config.ts`](https://github.com/nashsu/llm_wiki/blob/main/src/lib/source-watch-config.ts) and [`src/lib/source-lifecycle.ts`](https://github.com/nashsu/llm_wiki/blob/main/src/lib/source-lifecycle.ts) respectively.