# How to Configure Hister for Sensitive Content Detection and Blocking

> Learn to configure Hister for sensitive content detection and blocking using regular expressions in config.yaml. Protect your data effectively.

- Repository: [Adam Tauber/hister](https://github.com/asciimoo/hister)
- Tags: how-to-guide
- Published: 2026-08-27

---

**Hister blocks documents that match user‑defined regular expressions via the `sensitive_content_patterns` map in [`config.yaml`](https://github.com/asciimoo/hister/blob/main/config.yaml), returning `ErrSensitiveContent` unless you override the check with the `--allow‑sensitive` flag.**

Hister is a Go‑based indexing engine that can automatically detect and exclude sensitive content from your search index. By configuring regular expression patterns in the configuration file, you prevent documents containing credit card numbers, API keys, or other PII from being indexed. This guide explains how to set up sensitive content detection using the actual source implementation in `asciimoo/hister`.

## Define Sensitive Content Patterns in Configuration

The configuration structure in [`config/config.go`](https://github.com/asciimoo/hister/blob/main/config/config.go) declares the `SensitiveContentPatterns` field as a map of string keys to regular expression values.

```go
SensitiveContentPatterns map[string]string `yaml:"sensitive_content_patterns"`

```

Map keys serve as human‑readable identifiers (e.g., `credit_card`, `ssn`), while values contain the raw regex strings. During application startup, the indexer reads this map and compiles the expressions into a single optimized pattern matcher.

## Pattern Compilation and Loading

The indexer handles pattern compilation in [`server/indexer/indexer.go`](https://github.com/asciimoo/hister/blob/main/server/indexer/indexer.go) (lines 331‑339). It aggregates all user‑defined patterns into a single alternation group to improve matching performance.

```go
sp := make([]string, 0, len(cfg.SensitiveContentPatterns))
for _, p := range cfg.SensitiveContentPatterns {
    sp = append(sp, p)
}
sensitivePattern := regexp.MustCompile(fmt.Sprintf("(%s)", strings.Join(sp, "|")))
idx.sensitivePattern = sensitivePattern

```

This approach creates one `*regexp.Regexp` instance that matches any of the provided patterns, reducing memory overhead and CPU usage during document processing.

## Document Processing and Detection Logic

When Hister processes a document, the `ProcessWithSensitivePatternContext` function in [`server/document/document.go`](https://github.com/asciimoo/hister/blob/main/server/document/document.go) evaluates the compiled regex against the document’s HTML content (or extracted text for remote files).

```go
if !d.SkipSensitiveCheck && sensitivePattern != nil && sensitivePattern.MatchString(d.HTML) {
    log.Debug().Msg("Sensitive content detected")
    return ErrSensitiveContent
}

```

Documents return the error `ErrSensitiveContent` immediately upon matching, preventing them from entering the index. The check respects a per‑document `skip_sensitive_check` flag that allows specific documents to bypass filtering when necessary.

## CLI Overrides and API Handling

By default, the HTTP handlers in [`server/endpoints.go`](https://github.com/asciimoo/hister/blob/main/server/endpoints.go) propagate rejection errors to clients and log the event:

```go
log.Warn().Str("URL", d.URL).Msg("rejected document: sensitive content")

```

Hister provides two command‑line flags to override detection behavior:

- **`--allow-sensitive`** – Defined in [`cmd/index.go`](https://github.com/asciimoo/hister/blob/main/cmd/index.go) (line 422), this flag forces indexing even when content matches sensitive patterns.
- **`--exclude-sensitive`** – Defined in [`cmd/root.go`](https://github.com/asciimoo/hister/blob/main/cmd/root.go) (line 361), this flag skips checks during re‑indexing operations.

Use these flags when you need to temporarily index documents that would otherwise be blocked by your security policies.

## Practical Configuration Examples

Create a [`config.yaml`](https://github.com/asciimoo/hister/blob/main/config.yaml) that targets common sensitive data formats:

```yaml
sensitive_content_patterns:
  credit_card: '\b(?:\d[ -]*?){13,16}\b'
  ssn: '\b\d{3}-\d{2}-\d{4}\b'
  api_key: '(?i)apikey\s*[:=]\s*[A-Za-z0-9]{32}'

```

Run the indexer with detection enabled (default behavior):

```bash
hister index

```

Force indexing of documents that match sensitive patterns:

```bash
hister index --allow-sensitive

```

Skip sensitive checks while re‑indexing existing data:

```bash
hister reindex --exclude-sensitive

```

## Summary

- **Configuration**: Define regex patterns in [`config.yaml`](https://github.com/asciimoo/hister/blob/main/config.yaml) under `sensitive_content_patterns` as defined in [`config/config.go`](https://github.com/asciimoo/hister/blob/main/config/config.go).
- **Compilation**: The indexer in [`server/indexer/indexer.go`](https://github.com/asciimoo/hister/blob/main/server/indexer/indexer.go) joins patterns into a single compiled regex for efficient matching.
- **Detection**: [`server/document/document.go`](https://github.com/asciimoo/hister/blob/main/server/document/document.go) runs `ProcessWithSensitivePatternContext` to check document HTML against the pattern.
- **Blocking**: Matches return `ErrSensitiveContent`, and [`server/endpoints.go`](https://github.com/asciimoo/hister/blob/main/server/endpoints.go) logs the rejection.
- **Overrides**: Use `--allow-sensitive` or `--exclude-sensitive` flags from [`cmd/index.go`](https://github.com/asciimoo/hister/blob/main/cmd/index.go) and [`cmd/root.go`](https://github.com/asciimoo/hister/blob/main/cmd/root.go) to bypass checks when necessary.

## Frequently Asked Questions

### How does Hister compile multiple sensitive content patterns?

Hister aggregates all pattern strings from the configuration map into a slice, then joins them with the OR operator (`|`) inside a single capture group. The resulting string is compiled via `regexp.MustCompile` and stored in the `idx.sensitivePattern` field, allowing one regex evaluation per document instead of multiple sequential checks.

### Can I bypass sensitive content detection for specific documents?

Yes. Individual documents can set `skip_sensitive_check` to `true` to bypass the regex evaluation. Alternatively, run the CLI with `--allow-sensitive` to disable the check entirely for the current indexing operation, or use `--exclude-sensitive` during re‑indexing to skip validation of existing records.

### What error does Hister return when sensitive content is detected?

When a document matches a configured pattern, the `ProcessWithSensitivePatternContext` function returns the error `ErrSensitiveContent`. This error propagates through the HTTP handlers in [`server/endpoints.go`](https://github.com/asciimoo/hister/blob/main/server/endpoints.go), which log a warning message ("rejected document: sensitive content") and exclude the document from the index.