# How to Perform Text Deduplication with Marin-dupekit: 3 Methods Explained

> Learn to perform text deduplication with Marin-dupekit. Discover three distributed modes Exact Paragraph Exact Document and Fuzzy Document for petabyte-scale corpora using the Zephyr execution engine.

- Repository: [The Marin Project/marin](https://github.com/marin-community/marin)
- Tags: how-to-guide
- Published: 2026-09-10

---

**Marin-dupekit provides three distributed deduplication modes—Exact Paragraph, Exact Document, and Fuzzy Document—that leverage the Rust-based dupekit library to remove duplicates from petabyte-scale text corpora using Zephyr's execution engine.**

Marin-dupekit is the deduplication framework within the `marin-community/marin` repository designed for large-scale text processing pipelines. It offers deterministic exact matching and probabilistic fuzzy matching capabilities built on top of the high-performance **dupekit** Rust library. Whether cleaning web crawl data or deduplicating document uploads, Marin-dupekit integrates with Zephyr's distributed engine to process Parquet and JSONL shards at scale.

## Understanding the Three Deduplication Modes

Marin-dupekit implements three distinct strategies for identifying redundant content:

- **Exact Paragraph Deduplication**: Removes duplicated paragraphs within individual documents by hashing each paragraph and retaining the first occurrence. This strict, deterministic approach is implemented in `dedup_exact_paragraph` and is ideal when you need to eliminate repeated sections inside long texts.

- **Exact Document Deduplication**: Detects completely identical documents by computing a 128-bit Xxh3 hash of the entire text field. The `dedup_exact_document` function identifies verbatim duplicates across your dataset, making it suitable for removing duplicate file uploads.

- **Fuzzy Document Deduplication**: Finds near-duplicate documents using a MinHash and Locality-Sensitive Hashing (LSH) pipeline. This probabilistic method groups semantically similar texts through connected-components graph analysis, implemented across `compute_minhash_attrs` and `compute_fuzzy_dups_attrs`.

## Core Architecture and Transformations

All deduplication pipelines in Marin-dupekit rely on low-level **dupekit** transformations from the Rust library:

1. **Text Normalization**: `CleanText` lowercases input, strips punctuation, and collapses whitespace to ensure consistent hashing.

2. **Segmentation**: `SplitParagraphs` breaks documents into discrete paragraph records for granular deduplication.

3. **Signature Generation**: 
   - `Hash` produces 128-bit Xxh3 hashes for exact matching scenarios.
   - `MinHash` creates signature vectors using configurable permutations, while `MinHashLSH` buckets these signatures into bands to enable efficient similarity search.

The pipelines execute on Zephyr's distributed engine, where `Dataset` objects load shards and apply `flat_map` or `group_by` operations. Throughout execution, Zephyr counters (`counters.pipeline.update_counter`) track metrics such as processed documents, empty signatures, and duplicate totals. Jobs also initialize Weights & Biases runs via `_init_wandb` for metric surfacing and provenance tracking.

## Exact Deduplication Methods

For scenarios requiring deterministic de-duplication, Marin-dupekit provides two exact modes in [`exact.py`](https://github.com/marin-community/marin/blob/main/exact.py).

### Exact Paragraph Deduplication

The `dedup_exact_paragraph` function processes documents paragraph-by-paragraph, keeping only the first occurrence of each unique paragraph hash.

```python
from marin.processing.classification.deduplication.exact import dedup_exact_paragraph

result = dedup_exact_paragraph(
    input_paths="gs://my-bucket/raw-data/",
    output_path="gs://my-bucket/deduped/paragraphs/",
    text_field="text",
)
print(result)   # dict with total/dups/unique counters

```

*Implementation reference*: [[`exact.py`](https://github.com/marin-community/marin/blob/main/exact.py)](https://github.com/marin-community/marin/blob/main/lib/marin/src/marin/processing/classification/deduplication/exact.py#L75-L74).

### Exact Document Deduplication

Use `dedup_exact_document` to identify completely identical files across your dataset by hashing the entire text content.

```python
from marin.processing.classification.deduplication.exact import dedup_exact_document

result = dedup_exact_document(
    input_paths=["gs://my-bucket/raw-data/"],
    output_path="gs://my-bucket/deduped/documents/",
    text_field="text",
    max_parallelism=8,
)
print(result)   # dict with total/dups/unique counters

```

*Implementation reference*: [[`exact.py`](https://github.com/marin-community/marin/blob/main/exact.py)](https://github.com/marin-community/marin/blob/main/lib/marin/src/marin/processing/classification/deduplication/exact.py#L77-L86).

## Fuzzy Document Deduplication Pipeline

Fuzzy matching requires a two-stage process to handle the computational complexity of similarity detection across large corpora.

### Step 1: Computing MinHash Signatures

First, generate MinHash attributes using `compute_minhash_attrs` in [`fuzzy_minhash.py`](https://github.com/marin-community/marin/blob/main/fuzzy_minhash.py). This step creates signature vectors and LSH buckets for each document.

```python
from marin.processing.classification.deduplication.fuzzy_minhash import compute_minhash_attrs

minhash_attrs = compute_minhash_attrs(
    source=NormalizedData("gs://my-bucket/normalized/"),
    output_path="gs://my-bucket/minhash-attrs/",
    num_perms=286,
    num_bands=26,
    ngram_size=5,
    text_cap_chars=500_000,
    seed=42,
)

```

Key parameters include `num_perms` (hash permutations), which must be divisible by `num_bands` (LSH bands). The `text_cap_chars` parameter optionally truncates very long documents to improve performance.

*Implementation reference*: [[`fuzzy_minhash.py`](https://github.com/marin-community/marin/blob/main/fuzzy_minhash.py)](https://github.com/marin-community/marin/blob/main/lib/marin/src/marin/processing/classification/deduplication/fuzzy_minhash.py#L41-L56).

### Step 2: Detecting Fuzzy Duplicates

Consume the MinHash attributes to compute duplicate clusters using connected-components analysis.

```python
from marin.processing.classification.deduplication.fuzzy_dups import compute_fuzzy_dups_attrs

fuzzy_attrs = compute_fuzzy_dups_attrs(
    inputs=[minhash_attrs],
    output_path="gs://my-bucket/fuzzy-dups/",
    max_parallelism=12,
)

```

This produces `FuzzyDupsAttrData` artifacts containing duplicate markers co-partitioned with your original data.

*Implementation reference*: [[`fuzzy_dups.py`](https://github.com/marin-community/marin/blob/main/fuzzy_dups.py)](https://github.com/marin-community/marin/blob/main/lib/marin/src/marin/processing/classification/deduplication/fuzzy_dups.py#L24-L30).

## Working with Deduplication Artifacts

Both exact and fuzzy modes produce co-partitioned Parquet artifacts that enable downstream processing without reshuffling data.

Exact modes output files containing `id`, `is_dup`, and optionally `span` columns. Fuzzy modes generate `MinHashAttrData` (from [`fuzzy_minhash.py`](https://github.com/marin-community/marin/blob/main/fuzzy_minhash.py)) and `FuzzyDupsAttrData` (from [`fuzzy_dups.py`](https://github.com/marin-community/marin/blob/main/fuzzy_dups.py)) objects that expose the `attr_dir` path to per-shard Parquet files.

Load results using the artifact API:

```python
from marin.execution.artifact import read_artifact

mh_attrs = read_artifact(step="minhash-step", artifact_type=MinHashAttrData)
fd_attrs = read_artifact(step="fuzzy-step", artifact_type=FuzzyDupsAttrData)

```

These Pydantic models provide type-safe access to deduplication metadata and file locations.

## Key Source Files

The deduplication system spans four primary modules in `lib/marin/src/marin/processing/classification/deduplication/`:

- **[`fuzzy_minhash.py`](https://github.com/marin-community/marin/blob/main/fuzzy_minhash.py)**: Implements the MinHash signature and LSH bucketing pipeline.
- **[`fuzzy_dups.py`](https://github.com/marin-community/marin/blob/main/fuzzy_dups.py)**: Executes global connected-components graph analysis on MinHash attributes.
- **[`exact.py`](https://github.com/marin-community/marin/blob/main/exact.py)**: Contains `dedup_exact_paragraph` and `dedup_exact_document` hash-based pipelines.
- **[`dedup_commons.py`](https://github.com/marin-community/marin/blob/main/dedup_commons.py)**: Provides shared utilities including WandB initialization, batch loading, and counter management.

## Summary

- Marin-dupekit offers **three deduplication strategies**: Exact Paragraph, Exact Document, and Fuzzy Document, each optimized for different redundancy patterns.
- **Exact modes** use 128-bit Xxh3 hashing for deterministic duplicate detection at paragraph or document granularity.
- **Fuzzy mode** employs a two-stage MinHash and LSH pipeline to identify near-duplicates through probabilistic similarity hashing.
- All pipelines run on Zephyr's distributed engine, producing co-partitioned Parquet artifacts via `MinHashAttrData` and `FuzzyDupsAttrData` models.
- Implementation files are located in `lib/marin/src/marin/processing/classification/deduplication/`, with [`exact.py`](https://github.com/marin-community/marin/blob/main/exact.py) handling hash-based deduplication and [`fuzzy_minhash.py`](https://github.com/marin-community/marin/blob/main/fuzzy_minhash.py)/[`fuzzy_dups.py`](https://github.com/marin-community/marin/blob/main/fuzzy_dups.py) handling similarity clustering.

## Frequently Asked Questions

### What is the difference between exact and fuzzy deduplication in Marin-dupekit?

Exact deduplication uses cryptographic hashing (Xxh3) to identify byte-for-byte identical paragraphs or documents, making it deterministic and suitable for removing verbatim copies. Fuzzy deduplication uses MinHash signatures and LSH bucketing to detect near-duplicates where text may have slight variations, making it appropriate for identifying plagiarized or reformatted content that exact matching would miss.

### How does the MinHash LSH pipeline work for fuzzy matching?

The pipeline first computes MinHash signatures using `compute_minhash_attrs`, which generates fixed-length vectors representing document content. These signatures are then divided into bands (`num_bands`) and hashed into buckets using `MinHashLSH`. Documents sharing buckets are candidate pairs, which `compute_fuzzy_dups_attrs` resolves into duplicate clusters via connected-components graph analysis, efficiently grouping similar documents without pairwise comparisons of the entire corpus.

### Can I configure the sensitivity of fuzzy deduplication?

Yes, sensitivity is controlled through the `num_perms` (permutations) and `num_bands` parameters in `compute_minhash_attrs`. Increasing `num_perms` improves signature accuracy but requires more computation. The `num_bands` value must divide `num_perms` evenly and determines the LSH collision probability—fewer bands increase recall (find more duplicates) but may reduce precision, while more bands require higher similarity for matching.

### Where are the deduplication results stored in Marin-dupekit?

Results are written to the `output_path` specified in each function call as co-partitioned Parquet files. Exact modes produce simple Parquet files with duplicate flags, while fuzzy modes generate `MinHashAttrData` and `FuzzyDupsAttrData` artifacts. These Pydantic models expose the `attr_dir` attribute pointing to the directory containing per-shard Parquet files, which can be loaded via `read_artifact` or accessed directly for downstream verification steps.