# How to Configure Hister to Index Local Files from Specific Directories

> Learn how to configure Hister to index local files from specific directories. Define paths, filetypes, and patterns in your config yml for efficient indexing.

- Repository: [Adam Tauber/hister](https://github.com/asciimoo/hister)
- Tags: how-to-guide
- Published: 2026-08-27

---

**Configure local file indexing in Hister by defining one or more `Directory` entries under the `indexer.directories` section of your `~/.config/hister/config.yml` file, specifying paths, filetypes, and filtering patterns.**

Hister is an open-source document search engine that crawls and indexes local files for fast full-text search. To make Hister index files from specific directories on your system, you must define **directory configurations** in the YAML configuration file, which the indexer reads at startup to determine which files to process. This guide explains the configuration structure and indexing process based on the actual Hister source code.

## Understanding the Directory Configuration Structure

Hister discovers files to index through the `Directory` struct defined in [`config/config.go`](https://github.com/asciimoo/hister/blob/main/config/config.go) ([lines 107‑115](https://github.com/asciimoo/hister/blob/master/config/config.go#L107-L115)). Each directory entry in your configuration maps to this struct, allowing granular control over what gets indexed and how.

The `Directory` struct supports the following fields:

- **Path** – The absolute path to scan, supporting `~/` expansion for home directories.
- **Filetypes** – Optional list of extensions (e.g., `pdf`, `txt`, `md`) that must match the file’s extension (case‑insensitive).
- **Patterns** – Optional glob patterns (e.g., `*.journal`) that act as an allow‑list when present.
- **Excludes** – Glob patterns that exclude matching files from indexing.
- **IncludeHidden** – Boolean flag; when `false` (default), files starting with `.` are ignored.
- **DeleteOnRemove** – When `true`, files removed from the source directory are also removed from the index.
- **User** – The Hister user ID that owns the indexed documents.

During the indexing process, the function `walkDirectoryFiles` in [`server/indexer/files.go`](https://github.com/asciimoo/hister/blob/main/server/indexer/files.go) ([lines 90‑106](https://github.com/asciimoo/hister/blob/master/server/indexer/files.go#L90-L106)) reads these fields to filter files before calling `Indexer.IndexFile` for each accepted path.

## Configuring Indexer Directories in config.yml

To index custom local folders, add one or more directory entries under `indexer.directories` in your configuration file (default location `~/.config/hister/config.yml`). Hister automatically expands `~/` to the user's home directory using the `ExpandHome` function implemented in [`files/files.go`](https://github.com/asciimoo/hister/blob/main/files/files.go).

Here is a sample configuration that indexes two distinct folders with different rules:

```yaml
indexer:
  detect_languages: true
  keep_stopwords: false
  max_file_size_mb: 5      # reject files larger than 5 MiB

  directories:
    - path: ~/Documents/Research   # expands `~` automatically

      label: research
      filetypes: [pdf, docx, md]   # only these extensions are indexed

      patterns: ["*2023*"]        # only files containing "2023" in the name

      excludes: ["secret_*.pdf"]  # ignore files that match this glob

      include_hidden: false
      delete_on_remove: true
      user: alice                  # documents will belong to user "alice"

    - path: /var/www/html
      label: website
      filetypes: [html, css, js]
      include_hidden: true
      delete_on_remove: false
      user: bob

```

Each directory entry operates independently, allowing you to apply different filtering rules, ownership, and cleanup behaviors to separate file collections.

## How Hister Indexes Files from Configured Directories

When you run the indexer, Hister walks every directory listed in `config.Indexer.Directories` sequentially. The `walkDirectoryFiles` function applies the filtering logic in the following order:

1. Checks if the file matches the `Filetypes` list (if specified).
2. Validates against `Patterns` glob filters (if specified).
3. Excludes files matching any `Excludes` patterns.
4. Skips hidden files unless `IncludeHidden` is `true`.
5. Verifies file size against `max_file_size_mb`.

Files passing all filters are queued for indexing via `Indexer.IndexFile`. If `DeleteOnRemove` is enabled, the indexer also monitors for file deletions to maintain index consistency.

## Running the Indexer

After editing the configuration file, trigger the indexing process using the CLI commands defined in [`cmd/index.go`](https://github.com/asciimoo/hister/blob/main/cmd/index.go).

Index all configured directories for the current user:

```bash
hister index --global

```

Index only for a specific user (useful when multiple users share a configuration):

```bash
hister index --user alice

```

### Programmatic Configuration

You can also construct directory configurations programmatically in Go when integrating Hister into custom applications:

```go
dir := &config.Directory{
    Path:           "~/Projects/Notes",
    Label:          "notes",
    Filetypes:      []string{"md", "txt"},
    Patterns:       []string{"*2024*"},
    Excludes:       []string{"draft_*"},
    IncludeHidden:  false,
    DeleteOnRemove: true,
    User:           "bob",
}
cfg.Indexer.Directories = append(cfg.Indexer.Directories, dir)

```

## Summary

- **Directory configuration** in Hister uses the `Directory` struct in [`config/config.go`](https://github.com/asciimoo/hister/blob/main/config/config.go) to define scan paths and filtering rules.
- **Path expansion** supports `~/` notation, resolved by the `ExpandHome` function in [`files/files.go`](https://github.com/asciimoo/hister/blob/main/files/files.go).
- **File filtering** combines `Filetypes`, `Patterns`, `Excludes`, and `IncludeHidden` settings applied during the directory walk in [`server/indexer/files.go`](https://github.com/asciimoo/hister/blob/main/server/indexer/files.go).
- **Ownership and cleanup** are controlled per directory via the `User` and `DeleteOnRemove` fields.
- **Trigger indexing** manually using `hister index --global` or `hister index --user <name>` after updating `~/.config/hister/config.yml`.

## Frequently Asked Questions

### How do I add multiple directories to Hister?

Add multiple entries to the `indexer.directories` list in your [`config.yml`](https://github.com/asciimoo/hister/blob/main/config.yml). Each entry is an independent configuration with its own path, filters, and user assignment. Hister processes all configured directories in sequence during the indexing run.

### What file types does Hister support for indexing?

Hister supports any file type, but the `Filetypes` field restricts indexing to specific extensions you define (e.g., `pdf`, `txt`, `md`, `html`). The comparison is case‑insensitive, and if you omit the `Filetypes` list, all file types are considered for indexing (subject to other filters).

### How does Hister handle hidden files?

By default, Hister ignores hidden files (those starting with `.`). Set `include_hidden: true` in your directory configuration to index hidden files. This setting is useful for documentation stored in `.github` directories or dotfiles you wish to search.

### Can I exclude specific files or patterns from indexing?

Yes, use the `Excludes` field with glob patterns such as `temp_*` or `*.bak` to skip matching files. You can also combine `Patterns` (allow‑list) and `Excludes` (deny‑list) for precise control over which files enter the search index.