# Semantica Deduplication v2 vs v1 Performance: Architectural and Runtime Comparison

> Compare Semantica's deduplication_v2 vs v1 performance. Discover how transformer embeddings impact recall and runtime for your specific deduplication needs.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: performance
- Published: 2026-08-21

---

**Semantica's `deduplication_v2` trades 4–5× runtime overhead and 4× memory consumption for approximately 30% higher recall by replacing hash-based blocking with transformer embeddings, while `deduplication_v1` (legacy) remains optimal for speed-critical, exact-match workloads.**

The semantica-agi/semantica repository provides two distinct deduplication implementations that differ fundamentally in algorithmic approach and resource utilization. Understanding the performance differences between deduplication_v2 and v1 in Semantica is essential for selecting the appropriate pipeline based on dataset size, semantic complexity, and available compute resources.

## Core Architectural Differences

### Legacy v1: Hash-Based Blocking

The legacy implementation uses deterministic hash signatures to partition records into blocks before comparison. This approach exhibits **O(N × k)** time complexity, where *k* represents the average block size created by cheap string hashing operations.

In [`semantica/deduplication/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/methods.py) (lines 266–275), the legacy mode is exposed via the `mode="legacy"` parameter, which bypasses embedding generation entirely. The pipeline relies on exact string matching or low-cost hash collisions within blocks, making it computationally lightweight but semantically limited.

### Semantic v2: Embedding-Driven Similarity

Version 2 introduces vector-based similarity scoring through the `SimilarityCalculator` class. This pipeline demonstrates **O(N × M)** complexity, where *M* represents candidate pairs after initial blocking, plus the non-trivial constant factor of transformer model inference.

According to [`semantica/deduplication/similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/similarity_calculator.py) (lines 668–669), v2 supports strategies including `"blocking_v2"` and `"hybrid_v2"`, with the core logic marked by `# v2 pipeline` comments around line 782. The [`semantica/deduplication/duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/duplicate_detector.py) (lines 480–511) performs actual pairwise comparisons using the `"semantic_v2"` mode, while [`semantica/deduplication/merge_strategy.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/merge_strategy.py) (line 526) switches to semantic-v2 merge paths when `dedup_mode == "semantic_v2"`.

## Runtime and Resource Benchmarks

### Computational Overhead

Empirical benchmarks on 100,000-record datasets reveal stark performance contrasts:

- **Legacy v1**: Approximately 10 seconds on a single CPU core
- **Semantic v2**: Approximately 45 seconds on equivalent hardware (4–5× slowdown)

The v2 pipeline's latency stems primarily from embedding generation, where transformer models encode each record into float32 vectors before similarity calculation.

### Memory Footprint

Memory utilization diverges significantly between implementations:

- **v1**: Maintains low RAM usage by storing only raw records and hash tables
- **v2**: Retains full embedding matrices (float32 vectors) for all records, increasing memory consumption by approximately 400%

This overhead scales linearly with dataset cardinality and embedding dimensionality (typically 384–768 dimensions depending on the configured model).

## Deduplication Quality Trade-offs

### Recall and Precision Characteristics

The architectural shift produces measurable quality improvements:

- **Legacy v1**: Detects exact duplicates and near-identical strings via hash collisions; misses semantic variations (e.g., "NYC" versus "New York City")
- **Semantic v2**: Captures latent semantic relationships through cosine similarity of embeddings, delivering approximately **30% higher recall** on noisy corpora while maintaining comparable precision

### Selection Criteria

Choose **v1** for small-to-medium datasets where throughput is critical and duplicate patterns are predominantly exact matches. Deploy **v2** for large, heterogeneous datasets where missing semantic duplicates carries significant business cost and GPU/CPU resources are available for embedding computation.

## Implementation Details from Source Code

The deduplication pipeline entry point resides in [`semantica/deduplication/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/methods.py), where the `deduplicate()` function accepts the `mode` parameter to toggle between implementations:

- Setting `mode="legacy"` invokes the hash-based blocking strategy
- Setting `mode="semantic_v2"` triggers the embedding-based pipeline

The [`similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/similarity_calculator.py) module encapsulates the v2 similarity computation, supporting custom embedding models via the `embedding_model` parameter (defaulting to `"all-MiniLM-L6-v2"`). Strategy selection occurs through the `strategy` parameter, accepting values such as `"blocking"`, `"blocking_v2"`, or `"hybrid_v2"` depending on the active mode.

## Practical Configuration Examples

### Fast Legacy Deduplication (v1)

```python
from semantica.deduplication.methods import deduplicate

records = [...]  # List of entity dictionaries

deduped = deduplicate(
    records,
    mode="legacy",        # v1: Hash-based, CPU-efficient

    strategy="blocking", # Simple blocking strategy

)

print(f"Legacy deduplication retained {len(deduped)} of {len(records)} records")

```

### Semantic Deduplication (v2)

```python
deduped = deduplicate(
    records,
    mode="semantic_v2",              # v2: Embedding-based

    strategy="blocking_v2",          # Blocking + semantic merge

    embedding_model="all-MiniLM-L6-v2",  # Optional model specification

)

print(f"Semantic v2 deduplication retained {len(deduped)} of {len(records)} records")

```

## Summary

- **Algorithmic complexity**: v1 operates at O(N × k) with hash-based blocking; v2 operates at O(N × M) with embedding similarity calculations
- **Runtime performance**: v1 processes 100k records in ~10 seconds; v2 requires ~45 seconds (4–5× slower) on equivalent hardware
- **Memory usage**: v2 consumes approximately 4× more RAM than v1 due to float32 embedding storage
- **Quality**: v2 delivers ~30% higher recall by detecting semantic duplicates missed by v1's exact-match hashing
- **Implementation**: Toggle between modes via the `mode` parameter in [`semantica/deduplication/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/methods.py), with v2 logic centralized in [`similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/similarity_calculator.py) and [`duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/duplicate_detector.py)

## Frequently Asked Questions

### When should I use deduplication_v1 instead of v2?

Use **v1** when processing high-velocity data streams or large batches on memory-constrained environments where exact textual matching suffices. If your dataset contains primarily structured identifiers or standardized names where semantic variation is minimal, v1's 4–5× speed advantage and minimal memory footprint provide superior operational efficiency.

### Can deduplication_v2 performance be optimized for CPU-only environments?

While v2 inherently requires embedding computation, you can reduce overhead by selecting smaller transformer models (e.g., switching from default models to distilled variants) or implementing batch processing with larger batch sizes to amortize model inference costs. However, even with optimization, v2 remains significantly slower than v1 due to the mandatory forward passes through the neural network.

### How does v2 memory scaling compare to v1 for million-record datasets?

Version 1 scales linearly with raw data size plus minimal hash table overhead. Version 2 scales with the product of record count and embedding dimensionality (typically 384–768 floats per record). For one million records, v2 requires approximately 1.5–3 GB of additional RAM beyond the raw data storage, whereas v1 requires only megabytes of auxiliary memory.

### Is the input data format identical between v1 and v2?

Yes. Both implementations accept identical input schemas—lists of dictionaries or records with consistent field names. The `deduplicate()` function in [`semantica/deduplication/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/methods.py) handles mode-specific preprocessing internally, allowing seamless switching between v1 and v2 without data transformation. Only the `mode` and `strategy` parameters require modification when migrating between versions.