How the LLM-Wiki Source Folder Auto-Watch Feature Works: Implementation and Configuration

The source folder auto-watch feature uses a Rust-native file system watcher to monitor directory changes, filters events through configurable TypeScript rules, and automatically queues valid files for parsing and ingestion into the knowledge graph.

The auto-watch capability in nashsu/llm_wiki creates a real-time bridge between your local source directories and the application's ingestion pipeline. Implemented across the TypeScript frontend and Rust backend, the system watches for file system events, applies strict filtering logic based on user-defined configuration, and triggers automated processing workflows. This architecture ensures that documentation updates and code changes are indexed immediately without manual intervention.

Configuration Layer: SourceWatchConfig Schema and Defaults

The feature centers around the SourceWatchConfig interface, which defines rules for what the watcher should monitor and ignore.

Default Configuration Storage

Out-of-the-box settings are stored in src/lib/source-watch-defaults.json and exposed through src/lib/source-watch-config.ts as the DEFAULT_SOURCE_WATCH_CONFIG constant. This configuration object specifies:

• includeExtensions: Array of file extensions to explicitly watch
• excludeExtensions: Array of extensions to ignore
• excludeDirs: Directory names to recursively skip (e.g., node_modules, .git)
• excludeGlobs: Glob patterns for path-based exclusion
• maxFileSizeMb: Maximum file size threshold (capped at 4096 MB)
• parsingConcurrency: Parallel parsing limit (capped at 8 workers)
• ingestConcurrency: Parallel ingestion limit
• persistExtractedMarkdown: Boolean flag to save extracted content

Configuration Normalization and Safety Limits

When users provide partial configurations, the normalizeSourceWatchConfig() function in src/lib/source-watch-config.ts merges inputs with defaults while enforcing safety boundaries. The function automatically caps parsingConcurrency at 8 to prevent resource exhaustion and limits maxFileSizeMb to 4096 to avoid memory issues with the Rust file system watcher.

Filtering Logic: The isPathAllowedBySourceWatch Decision Tree

The core filtering mechanism resides in isPathAllowedBySourceWatch(path, config), which executes a five-stage validation pipeline to determine if a file event warrants processing:

1. Exclude Directory Check: Validates against excludeDirs using pathMatchesExcludedDir()
2. Glob Pattern Matching: Tests against excludeGlobs using matchesGlob()
3. Hidden File Detection: Automatically rejects files starting with . (dotfiles)
4. Extension Exclusion: Removes files matching excludeExtensions via getSourceWatchExtension()
5. Extension Inclusion: If includeExtensions is defined, only allows matches; otherwise permits all remaining files

Helper Utilities for Path Validation

The filtering system relies on three specialized utilities in src/lib/source-watch-config.ts:

• getSourceWatchExtension(path): Extracts and normalizes file extensions for comparison
• matchesGlob(path, pattern): Implements glob pattern matching against file paths
• pathMatchesExcludedDir(path, excludeDirs): Checks if any path segment matches excluded directory names

Runtime Architecture: From Rust File System Events to TypeScript Ingestion

The auto-watch system spans the Tauri frontend-backend boundary, utilizing the notify crate for efficient native file system monitoring.

Starting the Native Watcher via Tauri Commands

Watcher lifecycle management occurs in src/commands/file-sync.ts, which exposes two primary functions:

• start_project_file_watcher: Invokes the Tauri backend command to instantiate a recursive file system watcher on the project's source directory
• stop_project_file_watcher: Terminates the active watcher and cleans up resources

On the Rust side in src-tauri/src/lib.rs, these commands initialize the notify crate's recursive watcher, which monitors the configured directory tree at the OS level.

Event Flow Through the Project-File-Sync Store

When the Rust backend detects a file change, it emits events to the TypeScript frontend via Tauri's invoke system. The src/lib/project-file-sync.ts module receives these events and:

1. Parses the file path from the event payload
2. Calls isPathAllowedBySourceWatch() to apply configuration filters
3. Enqueues allowed files into the processing queue while respecting concurrency limits

The Source-Lifecycle Ingestion Pipeline

Queued files enter src/lib/source-lifecycle.ts, which orchestrates the complete ingestion workflow:

• Preprocessing: Hands files to src/lib/source-preprocess.ts for content extraction
• Concurrency Control: Respects parsingConcurrency and ingestConcurrency limits from the normalized config to prevent system overload
• Persistence: Conditionally saves extracted Markdown when persistExtractedMarkdown is enabled
• Knowledge Graph Updates: Integrates processed content into the wiki's searchable index

Implementation Example: Configuring and Activating the Watcher

The following example demonstrates loading configuration, validating a file path, and initiating the watch process:

import { invoke } from '@tauri-apps/api/tauri';
import { 
  normalizeSourceWatchConfig, 
  isPathAllowedBySourceWatch,
  DEFAULT_SOURCE_WATCH_CONFIG 
} from '@/lib/source-watch-config';

// 1. Initialize configuration with user overrides
const userConfig = {
  includeExtensions: ['md', 'txt', 'rs'],
  excludeDirs: ['target', 'node_modules'],
  maxFileSizeMb: 50,
  parsingConcurrency: 4
};
const watchConfig = normalizeSourceWatchConfig(userConfig);

// 2. Validate a file path before processing
const changedFile = '/home/user/project/src/main.rs';
if (isPathAllowedBySourceWatch(changedFile, watchConfig)) {
  console.log('File allowed for ingestion:', changedFile);
}

// 3. Start the native file system watcher
await invoke('start_project_file_watcher', { projectId: 'my-wiki-project' });

Summary

• The source folder auto-watch feature combines Rust's notify crate with TypeScript configuration management to monitor directories in real-time.
• File filtering occurs through isPathAllowedBySourceWatch() in src/lib/source-watch-config.ts, which applies a five-stage validation pipeline including extension checks and glob matching.
• Configuration normalization enforces safety limits on concurrency (max 8 parsing workers) and file sizes (max 4096 MB) to prevent resource exhaustion.
• Event flow moves from Rust backend watchers through Tauri invokes to src/lib/project-file-sync.ts, which queues valid files for the src/lib/source-lifecycle.ts ingestion pipeline.
• Lifecycle management commands start_project_file_watcher and stop_project_file_watcher in src/commands/file-sync.ts control the native watcher state.

Frequently Asked Questions

What file types does the source folder auto-watch feature support?

The watcher supports any file type defined in the includeExtensions array of the SourceWatchConfig. By default, the system can monitor all files unless specifically excluded, though in practice, LLM-Wiki configures specific extensions like .md, .txt, and source code files based on the project's ingestion requirements.

How does the watcher prevent system overload from large directories or files?

The system implements multiple safeguards: maxFileSizeMb prevents processing files exceeding the configured limit (capped at 4096 MB), excludeDirs automatically skips directories like node_modules and .git, and parsingConcurrency limits parallel processing to a maximum of 8 concurrent workers to manage CPU and memory usage.

Can I watch multiple source folders simultaneously across different projects?

Yes, each project can maintain its own SourceWatchConfig and watcher instance. The start_project_file_watcher and stop_project_file_watcher commands accept a projectId parameter, allowing the Rust backend to manage separate watcher instances for different directories concurrently.

Where is the actual file system watching logic implemented in the codebase?

The native file system watching logic resides in the Rust backend at src-tauri/src/lib.rs using the notify crate, while the TypeScript frontend controls the watcher via src/commands/file-sync.ts. Event filtering and processing logic is implemented in src/lib/source-watch-config.ts and src/lib/source-lifecycle.ts respectively.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}