# Understanding the Filtering Capabilities of olmOCR: A Technical Deep Dive

> Explore olmOCR's advanced filtering system for high-quality PDF and document data. Learn about PDF pre-filtering, text rules, rotation checks, and language scoring for robust training data.

- Repository: [Ai2/olmocr](https://github.com/allenai/olmocr)
- Tags: deep-dive
- Published: 2026-07-02

---

**olmOCR implements a multi-layered filtering system that eliminates low-quality PDFs and document samples through PDF-level pre-filtering, sample-level text rule validation, rotation integrity checks, and optional language-model coherency scoring to ensure only high-quality training data enters the pipeline.**

The `allenai/olmocr` repository provides a robust open-source framework for preparing OCR training datasets, with comprehensive **filtering capabilities of olmOCR** designed to remove spam, malformed content, and improperly rendered documents before they reach the model. These filters operate at distinct pipeline stages, from raw PDF ingestion through final sample selection, ensuring data integrity through both heuristic rules and neural validation.

## PDF-Level Pre-Filtering with PdfFilter

Before any image rendering or text extraction occurs, the `PdfFilter` class in [`olmocr/filter/filter.py`](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py) performs aggressive pre-filtering on raw PDF files. This component evaluates document metadata and initial text samples to discard files unlikely to yield useful training data.

### Form and Spam Detection

The filter first identifies interactive documents that require special handling. The `_is_form` method detects PDFs containing interactive form fields, which are typically excluded from standard training corpora. For content quality, `_is_download_spam` scans the first five pages of extracted text for SEO-heavy terms such as "download," "free," "ebook," and "viagra," computing a spam score that must remain below a configurable `download_spam_threshold` (default 0.004) to pass validation.

### Language and Character Validation

olmOCR uses the **Lingua** library for language detection, retaining only PDFs matching the `languages_to_keep` set (defaulting to English). The filter also applies an alphabetic-character sanity check: documents containing fewer than 200 characters or with less than 50% alphabetic content are flagged as potentially OCR-damaged but retained for further inspection, while the `filter_out_pdf` method returns `True` for documents meeting exclusion criteria.

```python
from olmocr.filter.filter import PdfFilter
from lingua import Language

pdf_filter = PdfFilter(
    languages_to_keep={Language.ENGLISH, None},   # keep English or undetectable language

    apply_form_check=True,
    apply_download_spam_check=True,
    download_spam_threshold=0.004,
)

should_discard = pdf_filter.filter_out_pdf("/tmp/suspicious_document.pdf")
print("Filtered out?", should_discard)      # True → discard, False → keep

```

## Sample-Level Text Rule Filtering

After PDFs pass initial screening and are rendered to images, the `DatasetTextRuleFilter` class in [`olmocr/train/dataloader.py`](https://github.com/allenai/olmocr/blob/main/olmocr/train/dataloader.py) (lines 31–130) applies granular text-content validation. This filter examines extracted natural text and structural markup to remove samples with specific formatting issues that complicate training.

### Table and Math Content Validation

The filter rejects documents containing **markdown tables** (detected via `_contains_markdown_table`) and **malformed HTML tables** (validated through `_extract_and_validate_html_tables`). For mathematical content, `_validate_math_equations` identifies unrenderable math equations, while `_contains_math_symbols` filters standalone mathematical symbols (e.g., ∈, ⊂, ∑, ∏) appearing outside of table contexts.

### LaTeX Structure Checks

olmOCR specifically targets LaTeX formatting that may confuse text-based models. The `_contains_latex_tables` method detects table environments, and `_contains_latex_formatting_outside_math` removes samples containing formatting commands like `\textit{}` or `\textbf{}` when they appear outside of proper math blocks.

```python
from olmocr.train.dataloader import DatasetTextRuleFilter, FilterOutRotatedDocuments

sample = {
    "image": pil_image,                  # PIL.Image instance

    "page_data": page_response,           # contains natural_text, rotation flags, etc.

}

# Filter rotated pages first

rot_filter = FilterOutRotatedDocuments()
sample = rot_filter(sample)   # returns None if rotation is invalid

# Apply text-rule filtering

if sample is not None:
    txt_filter = DatasetTextRuleFilter()
    sample = txt_filter(sample)   # returns None if any formatting rule matches

```

## Rotation and Document Integrity Checks

The `FilterOutRotatedDocuments` class, also located in [`olmocr/train/dataloader.py`](https://github.com/allenai/olmocr/blob/main/olmocr/train/dataloader.py) (lines 95–115), handles physical document integrity. This filter inspects page metadata for the `is_rotation_valid` flag and `rotation_correction` values. Any sample requiring non-zero rotation correction or flagged with invalid rotation metadata is discarded, preventing artificially rotated or misaligned page images from entering the training set.

## Coherency Scoring for Quality Assurance

For optional fine-grained quality assessment, [`olmocr/filter/coherency.py`](https://github.com/allenai/olmocr/blob/main/olmocr/filter/coherency.py) provides the `get_document_coherency` function. This utility loads the **SmolLM-135M** language model to compute a fluency score by splitting text into token-limited chunks and calculating the average log-likelihood per token. Higher scores indicate more coherent, natural text, which the anchor-prompt selection logic can use to prefer higher-quality document options during prompt engineering.

```python
from olmocr.filter.coherency import get_document_coherency

text = "This is a well-written paragraph about OCR data pipelines."
score = get_document_coherency(text)
print(f"Coherency score: {score:.2f}")

```

## Summary

- **PDF-level filtering** occurs in [`olmocr/filter/filter.py`](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py) via the `PdfFilter` class, removing forms, spam, non-English documents, and character-deficient files before processing.
- **Sample-level rules** in [`olmocr/train/dataloader.py`](https://github.com/allenai/olmocr/blob/main/olmocr/train/dataloader.py) via `DatasetTextRuleFilter` eliminate markdown tables, malformed HTML, invalid math equations, and improper LaTeX formatting.
- **Rotation validation** through `FilterOutRotatedDocuments` ensures only properly aligned page images proceed.
- **Coherency scoring** using SmolLM-135M in [`olmocr/filter/coherency.py`](https://github.com/allenai/olmocr/blob/main/olmocr/filter/coherency.py) provides optional neural quality metrics for advanced filtering scenarios.

## Frequently Asked Questions

### How does olmOCR detect spam PDFs before processing?

olmOCT scans the first five pages of extracted text for high-frequency SEO terms like "download," "free," "ebook," and "viagra" using the `_is_download_spam` method in `PdfFilter`. Documents exceeding the configurable `download_spam_threshold` are automatically filtered out.

### What types of mathematical content does olmOCR filter out?

The `DatasetTextRuleFilter` removes samples containing unrenderable math equations, standalone mathematical symbols outside tables (such as ∈ or ∑), and LaTeX table environments, ensuring only clean mathematical markup enters the training data.

### Can olmOCR handle non-English documents?

By default, olmOCR keeps only English-language PDFs using the Lingua library's detection via `languages_to_keep` in `PdfFilter`. However, this set is configurable, allowing inclusion of other languages by modifying the filter initialization parameters.

### How does the coherency scorer improve training data quality?

The `get_document_coherency` function computes a fluency score using SmolLM-135M by evaluating token log-likelihoods across text chunks. This allows the pipeline to preferentially select documents with higher linguistic coherence during anchor-prompt generation, elevating overall dataset quality.