Implementing PII De-identification with shift_dates Redaction in OpenMed

The shift_dates method in OpenMed shifts detected DATE entities by a configurable number of days while preserving original formatting and language-specific month names, enabling HIPAA-compliant temporal de-identification without destroying chronological utility for downstream analysis.

OpenMed provides a robust PII de-identification pipeline that supports multiple redaction strategies for sensitive clinical data. When implementing PII de-identification with shift_dates redaction, you transform exact temporal information by offsetting dates rather than masking them, preserving essential timeline relationships for research while removing identifiable specific dates.

How the Pipeline Works

The de-identification process follows a three-stage architecture: entity detection, method dispatch, and date transformation.

Entity Detection and Unified Output

The pipeline begins with a privacy-filter model (such as openai/privacy-filter or multilingual equivalents) that identifies PII spans in clinical text. Detection results are standardized across backends (MLX and Torch) through the PrivacyFilterTorchPipeline class in openmed/torch/privacy_filter.py. Each detected entity returns a dictionary containing entity_group, start, end, score, and word keys, ensuring consistent handling regardless of the underlying inference engine.

The Deidentify Entry Point

The core orchestration occurs in the deidentify function exposed from openmed.core.pii. This function iterates over detected entities and dispatches to specific redaction helpers based on the chosen method:

• mask: Replaces spans with [MASKED] placeholders
• redact: Replaces spans with generic [REDACTED] tags
• shift_dates: Transforms DATE entities by applying a day offset while preserving format

The method validates conflicting parameters—setting shift_dates=False while providing date_shift_days raises a ValueError as verified in tests/unit/test_pii.py.

Date-Shifting Implementation Details

The temporal transformation logic resides in two private helpers within openmed/core/pii.py:

_shift_date (line 1175): Parses date strings and applies offsets while maintaining input style. It first attempts localized month-name parsing via _parse_localized_month_date, then falls back to dateutil when available, or finally to _shift_date_basic for pure-regex handling.

_shift_date_basic (line 1389): A regex-driven parser supporting US (MM/DD/YYYY), European (DD/MM/YYYY), German (DD.MM.YYYY), ISO formats, and month-name variants. The keep_year flag enables partial de-identification by mutating only month and day components while preserving the year.

When parsing fails, the pipeline returns "[DATE_SHIFTED]" as a safe fallback placeholder.

Language Support and Localization

Multilingual date handling relies on the LANGUAGE_MONTH_NAMES mapping defined in openmed/core/pii_i18n.py. This lookup table enables accurate shifting of dates containing localized month names in Dutch, Hindi, Telugu, Portuguese, and other languages without requiring the heavyweight dateutil dependency, ensuring culturally consistent output across clinical narratives.

Practical Implementation Examples

Basic Usage with the High-Level API

The simplest implementation uses the deidentify function with the shift_dates method:

from openmed.core import deidentify

text = "Patient was admitted on 01/15/2020 and discharged on 02/20/2020."
result = deidentify(
    text,
    method="shift_dates",
    date_shift_days=30,
    keep_year=True,
)

print(result.redacted_text)

# Output: "Patient was admitted on 02/14/2020 and discharged on 03/21/2020."

This call path flows through openmed/core/pii.py → _redact_entity → _shift_date.

Direct Date String Manipulation

For custom pipelines requiring single-date processing, use the internal _shift_date helper directly:

from openmed.core.pii import _shift_date

original = "15 janvier 2020"  # French format

shifted = _shift_date(original, 30, lang="fr")
print(shifted)  # Output: "14 février 2020"

REST API Integration

Deploy the functionality via the FastAPI service defined in openmed/service/app.py:

import requests

payload = {
    "text": "Patient arrived on 2020-01-15.",
    "method": "shift_dates",
    "date_shift_days": 30,
}
resp = requests.post("http://localhost:8000/pii/deidentify", json=payload)
print(resp.json()["redacted_text"])

# Output: "Patient arrived on 2020-02-14."

Batch Processing via CLI

Process large datasets using the built-in command-line interface:

openmed pii deidentify \
    --input-file notes.txt \
    --output-file deid_notes.txt \
    --method shift_dates \
    --date-shift-days 30

The CLI invokes the same deidentify function, ensuring consistency across interfaces.

Key Source Files and Architecture

Understanding the codebase structure helps when extending or debugging the shift_dates functionality:

• openmed/core/pii.py: Contains the deidentify function, _shift_date (line 1175), and _shift_date_basic (line 1389) implementations
• openmed/core/pii_i18n.py: Houses LANGUAGE_MONTH_NAMES for multilingual month resolution
• openmed/torch/privacy_filter.py: Provides PrivacyFilterTorchPipeline for unified entity detection across backends
• tests/unit/test_pii.py: Validates method behavior, alias handling, and error conditions
• openmed/service/app.py: FastAPI router exposing the REST endpoint

Summary

• PII de-identification with shift_dates redaction modifies DATE content by applying configurable day offsets rather than masking or removing temporal data
• The implementation preserves original formatting and language-specific month names through the LANGUAGE_MONTH_NAMES mapping in openmed/core/pii_i18n.py
• Core logic resides in openmed/core/pii.py via _shift_date and _shift_date_basic, supporting the keep_year flag for partial de-identification
• Failed parsing falls back to "[DATE_SHIFTED]" placeholder to prevent data leakage
• Available through Python API, REST service (openmed/service/app.py), and CLI with consistent parameter validation

Frequently Asked Questions

What happens when shift_dates cannot parse a date format?

When the parser encounters unsupported formats, it returns the string "[DATE_SHIFTED]" as a safe fallback. This prevents original date exposure while indicating that temporal transformation was attempted. The regex-based _shift_date_basic handles most numeric formats (US, European, ISO), while localized month names require entries in LANGUAGE_MONTH_NAMES.

Does shift_dates work with multilingual clinical notes?

Yes. The implementation resolves localized month names through LANGUAGE_MONTH_NAMES in openmed/core/pii_i18n.py, supporting languages including Dutch, Hindi, Telugu, and Portuguese. This allows accurate shifting of dates written in local languages without requiring the dateutil library dependency.

How does keep_year affect the de-identification process?

Setting keep_year=True restricts transformation to month and day components only, leaving the year unchanged. This provides partial de-identification suitable for studies requiring temporal ordering across years while reducing re-identification risk from specific dates. The flag is processed in _shift_date_basic at line 1389 of openmed/core/pii.py.

Can I combine shift_dates with other redaction methods?

No, the method parameter accepts a single strategy. However, you can implement hybrid approaches by running the pipeline twice—first with shift_dates to preserve temporal relationships, then with mask or redact for other entity types. The pipeline validates parameters and raises ValueError if you attempt conflicting configurations like setting date_shift_days while using method="mask".

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}