# Semantica Text Normalization Module: 8 Core Capabilities Explained

> Discover Semantica's text normalization module capabilities. Learn how it transforms raw text into clean, canonical strings with 8 core functions like Unicode normalization, case folding, and more.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: deep-dive
- Published: 2026-09-05

---

**Semantica's text normalization module converts raw, unstructured text into clean, canonical strings through a pipeline of deterministic transformations including Unicode normalization, case folding, whitespace handling, punctuation removal, accent stripping, and optional tokenization with stop-word filtering.**

Semantica, an open-source semantic processing framework developed by semantica-agi, provides robust text normalization utilities that prepare textual input for downstream AI applications. The `semantica/normalize/` package implements a pipeline-style architecture where each transformation step produces predictable, auditable output. Understanding these capabilities is essential for developers building reliable entity matching, vector search, and knowledge graph systems.

## Unicode Normalization

The foundation of Semantica's text processing starts with **Unicode standardization**. The `_normalize_unicode` function in [`semantica/normalize/text_normalizer.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/normalize/text_normalizer.py) (line 30) converts all characters to **NFKC form** (Normalization Form KC: Compatibility Composition).

This ensures that visually identical symbols—such as full-width "Ａ" and half-width "A", or ligature "fi" and separate "fi"—map to the same code points. Without this step, string comparisons would fail on semantically equivalent characters.

```python
from semantica.normalize import text_normalizer as tn

# NFKC normalization handles compatibility characters

clean = tn.normalize("Ｃａｆé")  # Full-width + combined accent

# → "café" (standard ASCII-compatible representation)

```

## Case Folding

Case sensitivity is a common source of matching failures. The `_lowercase` function (line 45) performs **aggressive case folding** to create case-insensitive representations.

```python
clean = tn.normalize("NASDAQ: AAPL")

# → "nasdaq aapl"

```

This transformation applies Unicode-aware lowercasing, properly handling special cases like the Turkish "İ" to "i" mapping.

## Whitespace Normalization

The `_normalize_whitespace` function (line 60) implements three critical whitespace rules:

- **Collapse** consecutive whitespace (spaces, tabs, newlines) into single spaces
- **Trim** leading and trailing whitespace
- **Normalize** line-break variants (CR, LF, CRLF) to spaces

```python
clean = tn.normalize("  Multiple   \t  spaces\nand lines  ")

# → "multiple spaces and lines"

```

## Punctuation Sanitization

The `_remove_punctuation` function (line 78) strips or replaces punctuation symbols while preserving meaningful token boundaries. This includes commas, periods, brackets, quotation marks, and Unicode punctuation categories.

```python
clean = tn.normalize("Hello, World! (Test)")

# → "hello world test"

```

Optional configuration allows preserving specific punctuation (like hyphens in compound words) when semantic meaning depends on it.

## Accent Stripping

After Unicode normalization, the `_strip_accents` function (line 92) removes **diacritical marks** by decomposing characters and filtering combining marks. This enables language-agnostic matching across accented variants.

```python
clean = tn.normalize("naïve café résumé")

# → "naive cafe resume"

```

This transformation is particularly valuable for cross-lingual entity resolution where "München" and "Munchen" should match.

## Tokenization with Optional Dependencies

The `tokenise` function (line 110) provides a **pluggable tokenization architecture**:

- **Default**: Simple whitespace splitting (no dependencies)
- **spaCy option**: Full linguistic tokenization when `spacy` is installed

```python

# Simple whitespace tokenization (default)

tokens = tn.normalize(
    "The quick, brown-fox jumps!",
    tokenizer="whitespace"
)

# → "the quick brown-fox jumps"

# spaCy linguistic tokenization (optional dependency)

tokens = tn.normalize(
    "The quick, brown-fox jumps!",
    tokenizer="spacy"
)

# → ["the", "quick", ",", "brown", "-", "fox", "jumps", "!"]

```

## Stop-Word Filtering

When **nltk** or **spacy** extras are present, the `_remove_stopwords` function (line 128) filters common noise words. This reduces dimensionality for downstream vectorization.

```python

# With stop-word removal enabled

clean = tn.normalize(
    "The quick brown fox",
    remove_stopwords=True
)

# → "quick brown fox"

```

Stop-word lists are configurable per language and domain.

## Integration with Specialized Normalizers

The text normalization pipeline integrates with three specialized modules:

| Module | File | Capability |
|--------|------|------------|
| Number normalizer | [`semantica/normalize/number_normalizer.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/normalize/number_normalizer.py) | Standardizes numeric formats, strips currency symbols, normalizes thousand/decimal separators |
| Date normalizer | [`semantica/normalize/date_normalizer.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/normalize/date_normalizer.py) | Parses varied date strings to ISO-8601 (`YYYY-MM-DD`) |
| Entity normalizer | [`semantica/normalize/entity_normalizer.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/normalize/entity_normalizer.py) | Applies domain-specific rules like "Inc." → "Inc" and URI handling |

### Number Normalization Examples

```python
from semantica.normalize import number_normalizer as nn

nn.normalize("1,234.56")      # → "1234.56"

nn.normalize("$5,000.00")     # → "5000.00"

nn.normalize("€1.234,56")     # → "1234.56" (European format)

```

### Date Normalization Examples

```python
from semantica.normalize import date_normalizer as dn

dn.normalize("02/14/2023")          # → "2023-02-14"

dn.normalize("14 Feb 2023")         # → "2023-02-14"

dn.normalize("2023‑02‑14T15:30Z")   # → "2023-02-14"

```

## Provenance Tracking

Every normalization step records its operation in a **provenance object** defined in [`semantica/normalize/normalize_provenance.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/normalize/normalize_provenance.py). This enables:

- **Audit trails**: Reconstruct how any string was transformed
- **Debugging**: Identify which step introduced unexpected changes
- **Reproducibility**: Document exact normalization parameters

```python
result = tn.normalize("Hello!", track_provenance=True)
print(result.provenance.steps)

# → [UnicodeNormalization, Lowercase, WhitespaceNormalization, PunctuationRemoval]

```

## Summary

Semantica's text normalization module provides eight core capabilities for preparing raw text:

- **Unicode NFKC normalization** for character standardization
- **Case folding** for case-insensitive matching
- **Whitespace collapse and trimming** for clean formatting
- **Punctuation removal** with configurable preservation
- **Accent stripping** for cross-lingual compatibility
- **Pluggable tokenization** (whitespace or spaCy-based)
- **Optional stop-word filtering** with nltk/spacy integration
- **Provenance tracking** for full transformation auditability

These capabilities are validated through comprehensive test coverage in [`tests/normalize/test_text_normalizer.py`](https://github.com/semantica-agi/semantica/blob/main/tests/normalize/test_text_normalizer.py), [`tests/normalize/test_number_normalizer.py`](https://github.com/semantica-agi/semantica/blob/main/tests/normalize/test_number_normalizer.py), and [`tests/normalize/test_date_normalizer.py`](https://github.com/semantica-agi/semantica/blob/main/tests/normalize/test_date_normalizer.py).

## Frequently Asked Questions

### How does Semantica handle different Unicode normalization forms?

Semantica uses **NFKC (Normalization Form KC)** exclusively, as implemented in `_normalize_unicode` at line 30 of [`text_normalizer.py`](https://github.com/semantica-agi/semantica/blob/main/text_normalizer.py). NFKC provides the most aggressive compatibility decomposition, ensuring that compatibility characters (like superscripts, subscripts, and full-width forms) map to their standard equivalents. This design prioritizes matching reliability over visual fidelity.

### Can I preserve specific punctuation during normalization?

Yes. The `_remove_punctuation` function accepts optional parameters to preserve character classes. Pass `preserve_hyphens=True` to keep compound word structure, or provide a custom `preserve_chars` set. These parameters are exposed through the main `normalize()` function's configuration dictionary.

### What dependencies are required for full functionality?

The core text normalizer requires only Python's standard library (`unicodedata` for Unicode operations). **Optional dependencies** unlock advanced features: `spacy` enables linguistic tokenization, `nltk` provides alternative stop-word lists, and `dateparser` or `python-dateutil` enhance date recognition in [`date_normalizer.py`](https://github.com/semantica-agi/semantica/blob/main/date_normalizer.py). Install with `pip install semantica[all]` for complete capabilities.