Semantica Text Normalization Module: 8 Core Capabilities Explained

Semantica's text normalization module converts raw, unstructured text into clean, canonical strings through a pipeline of deterministic transformations including Unicode normalization, case folding, whitespace handling, punctuation removal, accent stripping, and optional tokenization with stop-word filtering.

Semantica, an open-source semantic processing framework developed by semantica-agi, provides robust text normalization utilities that prepare textual input for downstream AI applications. The semantica/normalize/ package implements a pipeline-style architecture where each transformation step produces predictable, auditable output. Understanding these capabilities is essential for developers building reliable entity matching, vector search, and knowledge graph systems.

Unicode Normalization

The foundation of Semantica's text processing starts with Unicode standardization. The _normalize_unicode function in semantica/normalize/text_normalizer.py (line 30) converts all characters to NFKC form (Normalization Form KC: Compatibility Composition).

This ensures that visually identical symbols—such as full-width "Ａ" and half-width "A", or ligature "fi" and separate "fi"—map to the same code points. Without this step, string comparisons would fail on semantically equivalent characters.

from semantica.normalize import text_normalizer as tn

# NFKC normalization handles compatibility characters

clean = tn.normalize("Ｃａｆé")  # Full-width + combined accent

# → "café" (standard ASCII-compatible representation)

Case Folding

Case sensitivity is a common source of matching failures. The _lowercase function (line 45) performs aggressive case folding to create case-insensitive representations.

clean = tn.normalize("NASDAQ: AAPL")

# → "nasdaq aapl"

This transformation applies Unicode-aware lowercasing, properly handling special cases like the Turkish "İ" to "i" mapping.

Whitespace Normalization

The _normalize_whitespace function (line 60) implements three critical whitespace rules:

• Collapse consecutive whitespace (spaces, tabs, newlines) into single spaces
• Trim leading and trailing whitespace
• Normalize line-break variants (CR, LF, CRLF) to spaces

clean = tn.normalize("  Multiple   \t  spaces\nand lines  ")

# → "multiple spaces and lines"

Punctuation Sanitization

The _remove_punctuation function (line 78) strips or replaces punctuation symbols while preserving meaningful token boundaries. This includes commas, periods, brackets, quotation marks, and Unicode punctuation categories.

clean = tn.normalize("Hello, World! (Test)")

# → "hello world test"

Optional configuration allows preserving specific punctuation (like hyphens in compound words) when semantic meaning depends on it.

Accent Stripping

After Unicode normalization, the _strip_accents function (line 92) removes diacritical marks by decomposing characters and filtering combining marks. This enables language-agnostic matching across accented variants.

clean = tn.normalize("naïve café résumé")

# → "naive cafe resume"

This transformation is particularly valuable for cross-lingual entity resolution where "München" and "Munchen" should match.

Tokenization with Optional Dependencies

The tokenise function (line 110) provides a pluggable tokenization architecture:

• Default: Simple whitespace splitting (no dependencies)
• spaCy option: Full linguistic tokenization when spacy is installed

# Simple whitespace tokenization (default)

tokens = tn.normalize(
    "The quick, brown-fox jumps!",
    tokenizer="whitespace"
)

# → "the quick brown-fox jumps"

# spaCy linguistic tokenization (optional dependency)

tokens = tn.normalize(
    "The quick, brown-fox jumps!",
    tokenizer="spacy"
)

# → ["the", "quick", ",", "brown", "-", "fox", "jumps", "!"]

Stop-Word Filtering

When nltk or spacy extras are present, the _remove_stopwords function (line 128) filters common noise words. This reduces dimensionality for downstream vectorization.

# With stop-word removal enabled

clean = tn.normalize(
    "The quick brown fox",
    remove_stopwords=True
)

# → "quick brown fox"

Stop-word lists are configurable per language and domain.

Integration with Specialized Normalizers

The text normalization pipeline integrates with three specialized modules:

Module	File	Capability
Number normalizer	semantica/normalize/number_normalizer.py	Standardizes numeric formats, strips currency symbols, normalizes thousand/decimal separators
Date normalizer	semantica/normalize/date_normalizer.py	Parses varied date strings to ISO-8601 (YYYY-MM-DD)
Entity normalizer	semantica/normalize/entity_normalizer.py	Applies domain-specific rules like "Inc." → "Inc" and URI handling

Number Normalization Examples

from semantica.normalize import number_normalizer as nn

nn.normalize("1,234.56")      # → "1234.56"

nn.normalize("$5,000.00")     # → "5000.00"

nn.normalize("€1.234,56")     # → "1234.56" (European format)

Date Normalization Examples

from semantica.normalize import date_normalizer as dn

dn.normalize("02/14/2023")          # → "2023-02-14"

dn.normalize("14 Feb 2023")         # → "2023-02-14"

dn.normalize("2023‑02‑14T15:30Z")   # → "2023-02-14"

Provenance Tracking

Every normalization step records its operation in a provenance object defined in semantica/normalize/normalize_provenance.py. This enables:

• Audit trails: Reconstruct how any string was transformed
• Debugging: Identify which step introduced unexpected changes
• Reproducibility: Document exact normalization parameters

result = tn.normalize("Hello!", track_provenance=True)
print(result.provenance.steps)

# → [UnicodeNormalization, Lowercase, WhitespaceNormalization, PunctuationRemoval]

Summary

Semantica's text normalization module provides eight core capabilities for preparing raw text:

• Unicode NFKC normalization for character standardization
• Case folding for case-insensitive matching
• Whitespace collapse and trimming for clean formatting
• Punctuation removal with configurable preservation
• Accent stripping for cross-lingual compatibility
• Pluggable tokenization (whitespace or spaCy-based)
• Optional stop-word filtering with nltk/spacy integration
• Provenance tracking for full transformation auditability

These capabilities are validated through comprehensive test coverage in tests/normalize/test_text_normalizer.py, tests/normalize/test_number_normalizer.py, and tests/normalize/test_date_normalizer.py.

Frequently Asked Questions

How does Semantica handle different Unicode normalization forms?

Semantica uses NFKC (Normalization Form KC) exclusively, as implemented in _normalize_unicode at line 30 of text_normalizer.py. NFKC provides the most aggressive compatibility decomposition, ensuring that compatibility characters (like superscripts, subscripts, and full-width forms) map to their standard equivalents. This design prioritizes matching reliability over visual fidelity.

Can I preserve specific punctuation during normalization?

Yes. The _remove_punctuation function accepts optional parameters to preserve character classes. Pass preserve_hyphens=True to keep compound word structure, or provide a custom preserve_chars set. These parameters are exposed through the main normalize() function's configuration dictionary.

What dependencies are required for full functionality?

The core text normalizer requires only Python's standard library (unicodedata for Unicode operations). Optional dependencies unlock advanced features: spacy enables linguistic tokenization, nltk provides alternative stop-word lists, and dateparser or python-dateutil enhance date recognition in date_normalizer.py. Install with pip install semantica[all] for complete capabilities.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}