# How Hister Manages Sensitive Content Detection: Regex Patterns and Configuration

> Learn how Hister detects sensitive content using regex patterns. Explore its runtime configuration and integration into the indexing pipeline for efficient scanning.

- Repository: [Adam Tauber/hister](https://github.com/asciimoo/hister)
- Tags: internals
- Published: 2026-09-01

---

**Hister detects sensitive content using configurable regular expressions compiled at runtime, checking documents during the indexing pipeline unless explicitly skipped via API flags or CLI options.**

Hister is an open-source search engine that implements content filtering at the indexing layer to prevent sensitive material from entering the search index. The system uses a regex-based detection pipeline defined in the core document processing logic, allowing administrators to customize patterns through configuration files or runtime API calls. Understanding how Hister manages sensitive content detection is essential for operators handling confidential data or compliance-sensitive workloads.

## Pattern Definition and Configuration

### Default Patterns and Package-Level Storage

In [`server/document/document.go`](https://github.com/asciimoo/hister/blob/main/server/document/document.go) (lines 55-58), Hister defines a package-level variable `sensitiveContentRe` that stores the compiled regular expression used for detection. This variable serves as the default pattern when no user configuration is provided.

The `SetSensitiveContentPattern` function (lines 75-79) allows other parts of the program—or test suites—to replace the default pattern at runtime by accepting a `*regexp.Regexp` parameter and assigning it to the package variable.

### Loading User-Defined Patterns

The `Indexer` struct in [`server/indexer/indexer.go`](https://github.com/asciimoo/hister/blob/main/server/indexer/indexer.go) (lines 68-71) loads user-provided patterns from the configuration file via the `SensitiveContentPatterns` setting. During initialization, the system compiles these patterns into a single regex stored in the `sensitivePattern` field, which the indexer propagates to all document processing routines.

## Runtime Detection Flow

### Document Processing Logic

When a document is processed, the engine checks content against the compiled pattern in two locations within [`server/document/document.go`](https://github.com/asciimoo/hister/blob/main/server/document/document.go). For standard HTML documents, the check occurs at lines 155-158. For extracted text from remote or local files, the verification happens at lines 199-202.

If the document's `SkipSensitiveCheck` field is false and the pattern matches the content, processing immediately aborts with the sentinel error `ErrSensitiveContent`, preventing the document from entering the index.

### Indexer Integration and Propagation

The `Indexer` ensures consistent enforcement by forwarding the compiled `sensitivePattern` to each document's processing routine. In [`server/indexer/indexer.go`](https://github.com/asciimoo/hister/blob/main/server/indexer/indexer.go) (lines 989-992), the indexer passes the pattern to the document processing pipeline, maintaining a centralized filtering mechanism across all indexing operations.

## Bypassing the Sensitive Content Filter

### Per-Document Exceptions via API

Users can bypass sensitive-content filtering for individual documents by setting the `skip_sensitive_check` field to `true` in the API payload. This sets the `SkipSensitiveCheck` boolean on the `Document` struct, causing the processing logic to skip the regex verification entirely.

### CLI Override Flags

For bulk operations, the `--allow-sensitive` flag (or `-x` shorthand when reindexing) defined in [`cmd/index.go`](https://github.com/asciimoo/hister/blob/main/cmd/index.go) (lines 45-46) disables checks for the entire indexing run. When this flag is active, the indexer ignores the sensitive content pattern for all documents in the operation.

## Logging and Observability

When detection triggers a rejection, [`server/endpoints.go`](https://github.com/asciimoo/hister/blob/main/server/endpoints.go) (line 777) emits a warning log entry that includes the document URL. This provides administrators with audit trails for tracking which documents were excluded from the index due to pattern matches.

## Practical Implementation Examples

### Setting a Custom Pattern Programmatically

You can override the default detection pattern at runtime using the document package:

```go
import (
    "regexp"
    "github.com/asciimoo/hister/server/document"
)

// Block any occurrence of the word "secret" (case-insensitive)
re := regexp.MustCompile(`(?i)secret`)
document.SetSensitiveContentPattern(re)

```

### Indexing with CLI Flag to Ignore Sensitive Checks

```bash

# Index a directory while allowing documents that match the sensitive pattern

hister index /path/to/data --allow-sensitive

```

### Skipping the Check via API

```json
POST /api/documents
{
  "url": "https://example.com/confidential",
  "html": "<html>...</html>",
  "skip_sensitive_check": true
}

```

### Inspecting Rejection Logs

When a document is rejected, the logs contain:

```

2024-09-01T12:34:56Z WARN  document=... URL="https://example.com/confidential" msg="rejected document: sensitive content"

```

### Key Files in the Detection Pipeline

- **[`server/document/document.go`](https://github.com/asciimoo/hister/blob/main/server/document/document.go)** – Defines the `Document` struct, `ErrSensitiveContent` error, `sensitiveContentRe` variable, `SetSensitiveContentPattern` function, and the core processing logic that performs HTML and text validation.

- **[`server/indexer/indexer.go`](https://github.com/asciimoo/hister/blob/main/server/indexer/indexer.go)** – Holds the compiled `sensitivePattern` and wires it into the document processing pipeline via the `Indexer` struct initialization.

- **[`cmd/index.go`](https://github.com/asciimoo/hister/blob/main/cmd/index.go)** – Implements the `--allow-sensitive` flag that disables the check for indexing operations.

- **[`server/endpoints.go`](https://github.com/asciimoo/hister/blob/main/server/endpoints.go)** – Emits log warnings when documents are dropped due to sensitive content detection.

## Summary

- **Pattern Storage**: Hister stores compiled regex patterns in `sensitiveContentRe` (package-level) and `sensitivePattern` (indexer-level), loaded from the `SensitiveContentPatterns` configuration.
- **Detection Point**: The system checks content in [`server/document/document.go`](https://github.com/asciimoo/hister/blob/main/server/document/document.go) during processing, comparing HTML or extracted text against the compiled pattern.
- **Blocking Mechanism**: Matches return `ErrSensitiveContent`, which aborts indexing and prevents the document from entering the search index.
- **Override Options**: Use the `skip_sensitive_check` API field for individual documents or the `--allow-sensitive` CLI flag for bulk operations.
- **Audit Trail**: Rejections are logged with document URLs in [`server/endpoints.go`](https://github.com/asciimoo/hister/blob/main/server/endpoints.go) for monitoring and compliance tracking.

## Frequently Asked Questions

### How do I configure custom sensitive content patterns in Hister?

Add the `SensitiveContentPatterns` array to your Hister configuration file (YAML or TOML). The indexer compiles these patterns into the `sensitivePattern` regex during initialization in [`server/indexer/indexer.go`](https://github.com/asciimoo/hister/blob/main/server/indexer/indexer.go) (lines 68-71), replacing or augmenting the default package-level pattern.

### Can I disable sensitive content checking for specific documents only?

Yes. Set `"skip_sensitive_check": true` in the JSON payload when submitting documents via the API. This sets the `SkipSensitiveCheck` field on the `Document` struct, causing the processing logic in [`server/document/document.go`](https://github.com/asciimoo/hister/blob/main/server/document/document.go) to bypass the regex verification while still processing the document normally.

### What happens when Hister detects sensitive content in a document?

The system returns `ErrSensitiveContent`, a sentinel error defined in [`server/document/document.go`](https://github.com/asciimoo/hister/blob/main/server/document/document.go). This error aborts the document processing routine immediately, preventing the document from being indexed, and triggers a warning log entry in [`server/endpoints.go`](https://github.com/asciimoo/hister/blob/main/server/endpoints.go) that includes the document URL.

### Is it possible to bulk-index documents while ignoring sensitive content filters?

Yes. Use the `--allow-sensitive` command-line flag (or `-x` shortcut when running reindex operations) implemented in [`cmd/index.go`](https://github.com/asciimoo/hister/blob/main/cmd/index.go) (lines 45-46). This instructs the indexer to ignore the sensitive content pattern for the entire operation, allowing all documents to be indexed regardless of regex matches.