How Hister Manages Sensitive Content Detection: Regex Patterns and Configuration

Hister detects sensitive content using configurable regular expressions compiled at runtime, checking documents during the indexing pipeline unless explicitly skipped via API flags or CLI options.

Hister is an open-source search engine that implements content filtering at the indexing layer to prevent sensitive material from entering the search index. The system uses a regex-based detection pipeline defined in the core document processing logic, allowing administrators to customize patterns through configuration files or runtime API calls. Understanding how Hister manages sensitive content detection is essential for operators handling confidential data or compliance-sensitive workloads.

Pattern Definition and Configuration

Default Patterns and Package-Level Storage

In server/document/document.go (lines 55-58), Hister defines a package-level variable sensitiveContentRe that stores the compiled regular expression used for detection. This variable serves as the default pattern when no user configuration is provided.

The SetSensitiveContentPattern function (lines 75-79) allows other parts of the program—or test suites—to replace the default pattern at runtime by accepting a *regexp.Regexp parameter and assigning it to the package variable.

Loading User-Defined Patterns

The Indexer struct in server/indexer/indexer.go (lines 68-71) loads user-provided patterns from the configuration file via the SensitiveContentPatterns setting. During initialization, the system compiles these patterns into a single regex stored in the sensitivePattern field, which the indexer propagates to all document processing routines.

Runtime Detection Flow

Document Processing Logic

When a document is processed, the engine checks content against the compiled pattern in two locations within server/document/document.go. For standard HTML documents, the check occurs at lines 155-158. For extracted text from remote or local files, the verification happens at lines 199-202.

If the document's SkipSensitiveCheck field is false and the pattern matches the content, processing immediately aborts with the sentinel error ErrSensitiveContent, preventing the document from entering the index.

Indexer Integration and Propagation

The Indexer ensures consistent enforcement by forwarding the compiled sensitivePattern to each document's processing routine. In server/indexer/indexer.go (lines 989-992), the indexer passes the pattern to the document processing pipeline, maintaining a centralized filtering mechanism across all indexing operations.

Bypassing the Sensitive Content Filter

Per-Document Exceptions via API

Users can bypass sensitive-content filtering for individual documents by setting the skip_sensitive_check field to true in the API payload. This sets the SkipSensitiveCheck boolean on the Document struct, causing the processing logic to skip the regex verification entirely.

CLI Override Flags

For bulk operations, the --allow-sensitive flag (or -x shorthand when reindexing) defined in cmd/index.go (lines 45-46) disables checks for the entire indexing run. When this flag is active, the indexer ignores the sensitive content pattern for all documents in the operation.

Logging and Observability

When detection triggers a rejection, server/endpoints.go (line 777) emits a warning log entry that includes the document URL. This provides administrators with audit trails for tracking which documents were excluded from the index due to pattern matches.

Practical Implementation Examples

Setting a Custom Pattern Programmatically

You can override the default detection pattern at runtime using the document package:

import (
    "regexp"
    "github.com/asciimoo/hister/server/document"
)

// Block any occurrence of the word "secret" (case-insensitive)
re := regexp.MustCompile(`(?i)secret`)
document.SetSensitiveContentPattern(re)

Indexing with CLI Flag to Ignore Sensitive Checks

# Index a directory while allowing documents that match the sensitive pattern

hister index /path/to/data --allow-sensitive

Skipping the Check via API

POST /api/documents
{
  "url": "https://example.com/confidential",
  "html": "<html>...</html>",
  "skip_sensitive_check": true
}

Inspecting Rejection Logs

When a document is rejected, the logs contain:

2024-09-01T12:34:56Z WARN  document=... URL="https://example.com/confidential" msg="rejected document: sensitive content"

Key Files in the Detection Pipeline

• server/document/document.go – Defines the Document struct, ErrSensitiveContent error, sensitiveContentRe variable, SetSensitiveContentPattern function, and the core processing logic that performs HTML and text validation.

• server/indexer/indexer.go – Holds the compiled sensitivePattern and wires it into the document processing pipeline via the Indexer struct initialization.

• cmd/index.go – Implements the --allow-sensitive flag that disables the check for indexing operations.

• server/endpoints.go – Emits log warnings when documents are dropped due to sensitive content detection.

Summary

• Pattern Storage: Hister stores compiled regex patterns in sensitiveContentRe (package-level) and sensitivePattern (indexer-level), loaded from the SensitiveContentPatterns configuration.
• Detection Point: The system checks content in server/document/document.go during processing, comparing HTML or extracted text against the compiled pattern.
• Blocking Mechanism: Matches return ErrSensitiveContent, which aborts indexing and prevents the document from entering the search index.
• Override Options: Use the skip_sensitive_check API field for individual documents or the --allow-sensitive CLI flag for bulk operations.
• Audit Trail: Rejections are logged with document URLs in server/endpoints.go for monitoring and compliance tracking.

Frequently Asked Questions

How do I configure custom sensitive content patterns in Hister?

Add the SensitiveContentPatterns array to your Hister configuration file (YAML or TOML). The indexer compiles these patterns into the sensitivePattern regex during initialization in server/indexer/indexer.go (lines 68-71), replacing or augmenting the default package-level pattern.

Can I disable sensitive content checking for specific documents only?

Yes. Set "skip_sensitive_check": true in the JSON payload when submitting documents via the API. This sets the SkipSensitiveCheck field on the Document struct, causing the processing logic in server/document/document.go to bypass the regex verification while still processing the document normally.

What happens when Hister detects sensitive content in a document?

The system returns ErrSensitiveContent, a sentinel error defined in server/document/document.go. This error aborts the document processing routine immediately, preventing the document from being indexed, and triggers a warning log entry in server/endpoints.go that includes the document URL.

Is it possible to bulk-index documents while ignoring sensitive content filters?

Yes. Use the --allow-sensitive command-line flag (or -x shortcut when running reindex operations) implemented in cmd/index.go (lines 45-46). This instructs the indexer to ignore the sensitive content pattern for the entire operation, allowing all documents to be indexed regardless of regex matches.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}