How to Configure Hister for Sensitive Content Detection and Blocking

Hister blocks documents that match user‑defined regular expressions via the sensitive_content_patterns map in config.yaml, returning ErrSensitiveContent unless you override the check with the --allow‑sensitive flag.

Hister is a Go‑based indexing engine that can automatically detect and exclude sensitive content from your search index. By configuring regular expression patterns in the configuration file, you prevent documents containing credit card numbers, API keys, or other PII from being indexed. This guide explains how to set up sensitive content detection using the actual source implementation in asciimoo/hister.

Define Sensitive Content Patterns in Configuration

The configuration structure in config/config.go declares the SensitiveContentPatterns field as a map of string keys to regular expression values.

SensitiveContentPatterns map[string]string `yaml:"sensitive_content_patterns"`

Map keys serve as human‑readable identifiers (e.g., credit_card, ssn), while values contain the raw regex strings. During application startup, the indexer reads this map and compiles the expressions into a single optimized pattern matcher.

Pattern Compilation and Loading

The indexer handles pattern compilation in server/indexer/indexer.go (lines 331‑339). It aggregates all user‑defined patterns into a single alternation group to improve matching performance.

sp := make([]string, 0, len(cfg.SensitiveContentPatterns))
for _, p := range cfg.SensitiveContentPatterns {
    sp = append(sp, p)
}
sensitivePattern := regexp.MustCompile(fmt.Sprintf("(%s)", strings.Join(sp, "|")))
idx.sensitivePattern = sensitivePattern

This approach creates one *regexp.Regexp instance that matches any of the provided patterns, reducing memory overhead and CPU usage during document processing.

Document Processing and Detection Logic

When Hister processes a document, the ProcessWithSensitivePatternContext function in server/document/document.go evaluates the compiled regex against the document’s HTML content (or extracted text for remote files).

if !d.SkipSensitiveCheck && sensitivePattern != nil && sensitivePattern.MatchString(d.HTML) {
    log.Debug().Msg("Sensitive content detected")
    return ErrSensitiveContent
}

Documents return the error ErrSensitiveContent immediately upon matching, preventing them from entering the index. The check respects a per‑document skip_sensitive_check flag that allows specific documents to bypass filtering when necessary.

CLI Overrides and API Handling

By default, the HTTP handlers in server/endpoints.go propagate rejection errors to clients and log the event:

log.Warn().Str("URL", d.URL).Msg("rejected document: sensitive content")

Hister provides two command‑line flags to override detection behavior:

• --allow-sensitive – Defined in cmd/index.go (line 422), this flag forces indexing even when content matches sensitive patterns.
• --exclude-sensitive – Defined in cmd/root.go (line 361), this flag skips checks during re‑indexing operations.

Use these flags when you need to temporarily index documents that would otherwise be blocked by your security policies.

Practical Configuration Examples

Create a config.yaml that targets common sensitive data formats:

sensitive_content_patterns:
  credit_card: '\b(?:\d[ -]*?){13,16}\b'
  ssn: '\b\d{3}-\d{2}-\d{4}\b'
  api_key: '(?i)apikey\s*[:=]\s*[A-Za-z0-9]{32}'

Run the indexer with detection enabled (default behavior):

hister index

Force indexing of documents that match sensitive patterns:

hister index --allow-sensitive

Skip sensitive checks while re‑indexing existing data:

hister reindex --exclude-sensitive

Summary

• Configuration: Define regex patterns in config.yaml under sensitive_content_patterns as defined in config/config.go.
• Compilation: The indexer in server/indexer/indexer.go joins patterns into a single compiled regex for efficient matching.
• Detection: server/document/document.go runs ProcessWithSensitivePatternContext to check document HTML against the pattern.
• Blocking: Matches return ErrSensitiveContent, and server/endpoints.go logs the rejection.
• Overrides: Use --allow-sensitive or --exclude-sensitive flags from cmd/index.go and cmd/root.go to bypass checks when necessary.

Frequently Asked Questions

How does Hister compile multiple sensitive content patterns?

Hister aggregates all pattern strings from the configuration map into a slice, then joins them with the OR operator (|) inside a single capture group. The resulting string is compiled via regexp.MustCompile and stored in the idx.sensitivePattern field, allowing one regex evaluation per document instead of multiple sequential checks.

Can I bypass sensitive content detection for specific documents?

Yes. Individual documents can set skip_sensitive_check to true to bypass the regex evaluation. Alternatively, run the CLI with --allow-sensitive to disable the check entirely for the current indexing operation, or use --exclude-sensitive during re‑indexing to skip validation of existing records.

What error does Hister return when sensitive content is detected?

When a document matches a configured pattern, the ProcessWithSensitivePatternContext function returns the error ErrSensitiveContent. This error propagates through the HTTP handlers in server/endpoints.go, which log a warning message ("rejected document: sensitive content") and exclude the document from the index.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}