How to Configure Hister to Index Local Files from Specific Directories

Configure local file indexing in Hister by defining one or more Directory entries under the indexer.directories section of your ~/.config/hister/config.yml file, specifying paths, filetypes, and filtering patterns.

Hister is an open-source document search engine that crawls and indexes local files for fast full-text search. To make Hister index files from specific directories on your system, you must define directory configurations in the YAML configuration file, which the indexer reads at startup to determine which files to process. This guide explains the configuration structure and indexing process based on the actual Hister source code.

Understanding the Directory Configuration Structure

Hister discovers files to index through the Directory struct defined in config/config.go (lines 107‑115). Each directory entry in your configuration maps to this struct, allowing granular control over what gets indexed and how.

The Directory struct supports the following fields:

• Path – The absolute path to scan, supporting ~/ expansion for home directories.
• Filetypes – Optional list of extensions (e.g., pdf, txt, md) that must match the file’s extension (case‑insensitive).
• Patterns – Optional glob patterns (e.g., *.journal) that act as an allow‑list when present.
• Excludes – Glob patterns that exclude matching files from indexing.
• IncludeHidden – Boolean flag; when false (default), files starting with . are ignored.
• DeleteOnRemove – When true, files removed from the source directory are also removed from the index.
• User – The Hister user ID that owns the indexed documents.

During the indexing process, the function walkDirectoryFiles in server/indexer/files.go (lines 90‑106) reads these fields to filter files before calling Indexer.IndexFile for each accepted path.

Configuring Indexer Directories in config.yml

To index custom local folders, add one or more directory entries under indexer.directories in your configuration file (default location ~/.config/hister/config.yml). Hister automatically expands ~/ to the user's home directory using the ExpandHome function implemented in files/files.go.

Here is a sample configuration that indexes two distinct folders with different rules:

indexer:
  detect_languages: true
  keep_stopwords: false
  max_file_size_mb: 5      # reject files larger than 5 MiB

  directories:
    - path: ~/Documents/Research   # expands `~` automatically

      label: research
      filetypes: [pdf, docx, md]   # only these extensions are indexed

      patterns: ["*2023*"]        # only files containing "2023" in the name

      excludes: ["secret_*.pdf"]  # ignore files that match this glob

      include_hidden: false
      delete_on_remove: true
      user: alice                  # documents will belong to user "alice"

    - path: /var/www/html
      label: website
      filetypes: [html, css, js]
      include_hidden: true
      delete_on_remove: false
      user: bob

Each directory entry operates independently, allowing you to apply different filtering rules, ownership, and cleanup behaviors to separate file collections.

How Hister Indexes Files from Configured Directories

When you run the indexer, Hister walks every directory listed in config.Indexer.Directories sequentially. The walkDirectoryFiles function applies the filtering logic in the following order:

1. Checks if the file matches the Filetypes list (if specified).
2. Validates against Patterns glob filters (if specified).
3. Excludes files matching any Excludes patterns.
4. Skips hidden files unless IncludeHidden is true.
5. Verifies file size against max_file_size_mb.

Files passing all filters are queued for indexing via Indexer.IndexFile. If DeleteOnRemove is enabled, the indexer also monitors for file deletions to maintain index consistency.

Running the Indexer

After editing the configuration file, trigger the indexing process using the CLI commands defined in cmd/index.go.

Index all configured directories for the current user:

hister index --global

Index only for a specific user (useful when multiple users share a configuration):

hister index --user alice

Programmatic Configuration

You can also construct directory configurations programmatically in Go when integrating Hister into custom applications:

dir := &config.Directory{
    Path:           "~/Projects/Notes",
    Label:          "notes",
    Filetypes:      []string{"md", "txt"},
    Patterns:       []string{"*2024*"},
    Excludes:       []string{"draft_*"},
    IncludeHidden:  false,
    DeleteOnRemove: true,
    User:           "bob",
}
cfg.Indexer.Directories = append(cfg.Indexer.Directories, dir)

Summary

• Directory configuration in Hister uses the Directory struct in config/config.go to define scan paths and filtering rules.
• Path expansion supports ~/ notation, resolved by the ExpandHome function in files/files.go.
• File filtering combines Filetypes, Patterns, Excludes, and IncludeHidden settings applied during the directory walk in server/indexer/files.go.
• Ownership and cleanup are controlled per directory via the User and DeleteOnRemove fields.
• Trigger indexing manually using hister index --global or hister index --user <name> after updating ~/.config/hister/config.yml.

Frequently Asked Questions

How do I add multiple directories to Hister?

Add multiple entries to the indexer.directories list in your config.yml. Each entry is an independent configuration with its own path, filters, and user assignment. Hister processes all configured directories in sequence during the indexing run.

What file types does Hister support for indexing?

Hister supports any file type, but the Filetypes field restricts indexing to specific extensions you define (e.g., pdf, txt, md, html). The comparison is case‑insensitive, and if you omit the Filetypes list, all file types are considered for indexing (subject to other filters).

How does Hister handle hidden files?

By default, Hister ignores hidden files (those starting with .). Set include_hidden: true in your directory configuration to index hidden files. This setting is useful for documentation stored in .github directories or dotfiles you wish to search.

Can I exclude specific files or patterns from indexing?

Yes, use the Excludes field with glob patterns such as temp_* or *.bak to skip matching files. You can also combine Patterns (allow‑list) and Excludes (deny‑list) for precise control over which files enter the search index.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}