Understanding the Filtering Capabilities of olmOCR: A Technical Deep Dive

olmOCR implements a multi-layered filtering system that eliminates low-quality PDFs and document samples through PDF-level pre-filtering, sample-level text rule validation, rotation integrity checks, and optional language-model coherency scoring to ensure only high-quality training data enters the pipeline.

The allenai/olmocr repository provides a robust open-source framework for preparing OCR training datasets, with comprehensive filtering capabilities of olmOCR designed to remove spam, malformed content, and improperly rendered documents before they reach the model. These filters operate at distinct pipeline stages, from raw PDF ingestion through final sample selection, ensuring data integrity through both heuristic rules and neural validation.

PDF-Level Pre-Filtering with PdfFilter

Before any image rendering or text extraction occurs, the PdfFilter class in olmocr/filter/filter.py performs aggressive pre-filtering on raw PDF files. This component evaluates document metadata and initial text samples to discard files unlikely to yield useful training data.

Form and Spam Detection

The filter first identifies interactive documents that require special handling. The _is_form method detects PDFs containing interactive form fields, which are typically excluded from standard training corpora. For content quality, _is_download_spam scans the first five pages of extracted text for SEO-heavy terms such as "download," "free," "ebook," and "viagra," computing a spam score that must remain below a configurable download_spam_threshold (default 0.004) to pass validation.

Language and Character Validation

olmOCR uses the Lingua library for language detection, retaining only PDFs matching the languages_to_keep set (defaulting to English). The filter also applies an alphabetic-character sanity check: documents containing fewer than 200 characters or with less than 50% alphabetic content are flagged as potentially OCR-damaged but retained for further inspection, while the filter_out_pdf method returns True for documents meeting exclusion criteria.

from olmocr.filter.filter import PdfFilter
from lingua import Language

pdf_filter = PdfFilter(
    languages_to_keep={Language.ENGLISH, None},   # keep English or undetectable language

    apply_form_check=True,
    apply_download_spam_check=True,
    download_spam_threshold=0.004,
)

should_discard = pdf_filter.filter_out_pdf("/tmp/suspicious_document.pdf")
print("Filtered out?", should_discard)      # True → discard, False → keep

Sample-Level Text Rule Filtering

After PDFs pass initial screening and are rendered to images, the DatasetTextRuleFilter class in olmocr/train/dataloader.py (lines 31–130) applies granular text-content validation. This filter examines extracted natural text and structural markup to remove samples with specific formatting issues that complicate training.

Table and Math Content Validation

The filter rejects documents containing markdown tables (detected via _contains_markdown_table) and malformed HTML tables (validated through _extract_and_validate_html_tables). For mathematical content, _validate_math_equations identifies unrenderable math equations, while _contains_math_symbols filters standalone mathematical symbols (e.g., ∈, ⊂, ∑, ∏) appearing outside of table contexts.

LaTeX Structure Checks

olmOCR specifically targets LaTeX formatting that may confuse text-based models. The _contains_latex_tables method detects table environments, and _contains_latex_formatting_outside_math removes samples containing formatting commands like \textit{} or \textbf{} when they appear outside of proper math blocks.

from olmocr.train.dataloader import DatasetTextRuleFilter, FilterOutRotatedDocuments

sample = {
    "image": pil_image,                  # PIL.Image instance

    "page_data": page_response,           # contains natural_text, rotation flags, etc.

}

# Filter rotated pages first

rot_filter = FilterOutRotatedDocuments()
sample = rot_filter(sample)   # returns None if rotation is invalid

# Apply text-rule filtering

if sample is not None:
    txt_filter = DatasetTextRuleFilter()
    sample = txt_filter(sample)   # returns None if any formatting rule matches

Rotation and Document Integrity Checks

The FilterOutRotatedDocuments class, also located in olmocr/train/dataloader.py (lines 95–115), handles physical document integrity. This filter inspects page metadata for the is_rotation_valid flag and rotation_correction values. Any sample requiring non-zero rotation correction or flagged with invalid rotation metadata is discarded, preventing artificially rotated or misaligned page images from entering the training set.

Coherency Scoring for Quality Assurance

For optional fine-grained quality assessment, olmocr/filter/coherency.py provides the get_document_coherency function. This utility loads the SmolLM-135M language model to compute a fluency score by splitting text into token-limited chunks and calculating the average log-likelihood per token. Higher scores indicate more coherent, natural text, which the anchor-prompt selection logic can use to prefer higher-quality document options during prompt engineering.

from olmocr.filter.coherency import get_document_coherency

text = "This is a well-written paragraph about OCR data pipelines."
score = get_document_coherency(text)
print(f"Coherency score: {score:.2f}")

Summary

• PDF-level filtering occurs in olmocr/filter/filter.py via the PdfFilter class, removing forms, spam, non-English documents, and character-deficient files before processing.
• Sample-level rules in olmocr/train/dataloader.py via DatasetTextRuleFilter eliminate markdown tables, malformed HTML, invalid math equations, and improper LaTeX formatting.
• Rotation validation through FilterOutRotatedDocuments ensures only properly aligned page images proceed.
• Coherency scoring using SmolLM-135M in olmocr/filter/coherency.py provides optional neural quality metrics for advanced filtering scenarios.

Frequently Asked Questions

How does olmOCR detect spam PDFs before processing?

olmOCT scans the first five pages of extracted text for high-frequency SEO terms like "download," "free," "ebook," and "viagra" using the _is_download_spam method in PdfFilter. Documents exceeding the configurable download_spam_threshold are automatically filtered out.

What types of mathematical content does olmOCR filter out?

The DatasetTextRuleFilter removes samples containing unrenderable math equations, standalone mathematical symbols outside tables (such as ∈ or ∑), and LaTeX table environments, ensuring only clean mathematical markup enters the training data.

Can olmOCR handle non-English documents?

By default, olmOCR keeps only English-language PDFs using the Lingua library's detection via languages_to_keep in PdfFilter. However, this set is configurable, allowing inclusion of other languages by modifying the filter initialization parameters.

How does the coherency scorer improve training data quality?

The get_document_coherency function computes a fluency score using SmolLM-135M by evaluating token log-likelihoods across text chunks. This allows the pipeline to preferentially select documents with higher linguistic coherence during anchor-prompt generation, elevating overall dataset quality.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}