Complete PDF Filtering Pipeline in OlmOCR: Language, Form, and Spam Detection

The OlmOCR PDF filtering pipeline sequentially checks for fillable forms, validates English language using Lingua, and detects SEO spam through keyword density ratios before processing documents.

The allenai/olmocr repository implements a robust filtering system to discard low-quality PDFs before expensive OCR processing. The complete PDF filtering pipeline examines documents for fillable forms, non-English content, and download-spam characteristics using a sequential validation strategy defined in the core filter module.

Architecture and Entry Points

The filtering logic resides in olmocr/filter/filter.py within the PdfFilter class. The main processing entry point is olmocr/pipeline.py, specifically the process_single_pdf function, which invokes filter_out_pdf when the --apply_filter flag is enabled. This design ensures that only high-quality documents proceed to downstream OCR and LLM inference stages.

When initialized, the pipeline creates a cached filter instance using a lambda function that configures default parameters:

get_pdf_filter = cache(
    lambda: PdfFilter(
        languages_to_keep={Language.ENGLISH, None},
        apply_download_spam_check=True,
        apply_form_check=True,
    )
)

During document processing, the filter is applied conditionally:

if args.apply_filter and get_pdf_filter().filter_out_pdf(local_pdf_path):
    logger.info(f"Filtering out pdf {pdf_orig_path}")
    return None

The Three-Stage Filtering Process

The filter_out_pdf method executes three independent checks in sequence. If any check triggers a discard condition, the method returns True and the PDF is excluded from further processing.

Form Detection Using PyPDF

The first validation step identifies fillable PDF forms. The _is_form method (lines 29-33 of filter.py) utilizes pypdf.PdfReader.get_form_text_fields() to detect interactive form fields.

If the method returns any form fields and apply_form_check is enabled (default True), the PDF is immediately filtered out. This prevents structured data entry forms from polluting text-training corpora.

Language Detection with Lingua

The second stage validates document language using the Lingua library. The pipeline extracts text from the first five pages using pdftotext via subprocess, then passes the content to lingua.LanguageDetectorBuilder (lines 101-105).

The PDF is filtered out if the detected language is not present in languages_to_keep. The default configuration accepts only English (Language.ENGLISH) and undetermined languages (None), effectively removing non-English documents from English-centric training datasets.

Download-Spam (SEO) Detection

The final check identifies SEO-optimized spam documents using the _is_download_spam method (lines 35-62). This algorithm counts occurrences of spammy keywords—such as download, pdf, ebook, and viagra—and calculates the ratio of spam score to total word count.

If the ratio exceeds download_spam_threshold (default 0.004), the document is classified as download spam and discarded. This threshold catches keyword-stuffed pages while preserving legitimate academic or technical documents.

Safety Guards and Edge Cases

Beyond the three main filters, the pipeline implements protective logic to handle corrupted or atypical PDFs:

• PDF Readability (lines 65-76): If PdfReader cannot open or parse the file due to corruption or encryption, the document is filtered out immediately.
• Minimum Text Length (line 93): PDFs containing fewer than 200 characters are explicitly kept rather than filtered, as insufficient data exists to make reliable language or spam determinations.
• Alphabetic Density (lines 97-99): Documents where fewer than 50% of characters are alphabetic are preserved. This safety measure prevents the loss of heavily OCR-dependent scanned documents or image-heavy PDFs where text extraction yields limited alphabetic content.

Configuration and Customization

You can instantiate and configure the filter for standalone use or custom pipelines:

Basic usage with default settings:

from olmocr.filter import PdfFilter

pdf_filter = PdfFilter()
if pdf_filter.filter_out_pdf("/path/to/document.pdf"):
    print("PDF rejected")
else:
    print("PDF accepted")

Disabling form detection:

pdf_filter = PdfFilter(apply_form_check=False)

Supporting multiple languages:

from lingua import Language

pdf_filter = PdfFilter(
    languages_to_keep={Language.ENGLISH, Language.SPANISH, Language.FRENCH}
)

Summary

• The form detection stage uses PyPDF to identify fillable forms and exclude structured data entry documents.
• Language validation extracts the first five pages and uses Lingua to enforce English-language requirements by default.
• Spam detection calculates keyword density ratios to eliminate SEO-optimized download spam with a 0.004 threshold.
• Safety guards preserve short or low-alphabetic PDFs to avoid discarding valid OCR-scanned documents.
• The pipeline integrates through olmocr/pipeline.py with cached filter instances to optimize performance across large document batches.

Frequently Asked Questions

How does OlmOCR detect fillable forms in PDFs?

OlmOCR detects fillable forms using the _is_form method in olmocr/filter/filter.py (lines 29-33), which calls pypdf.PdfReader.get_form_text_fields(). If this method returns any interactive form fields, the PDF is classified as a form and filtered out unless apply_form_check is explicitly disabled during PdfFilter initialization.

What languages does the OlmOCR filter support by default?

By default, the filter accepts only English and undetermined languages (None). The default languages_to_keep parameter is set to {Language.ENGLISH, None} using the Lingua language enum. You can customize this set during instantiation to include additional languages such as Spanish, French, or German.

What is the default spam threshold and how does it work?

The default download_spam_threshold is 0.004 (0.4%). The _is_download_spam method counts occurrences of specific spam keywords—including "download," "pdf," "ebook," and "viagra"—and divides this count by the total word count. If the resulting ratio exceeds 0.004, the document is classified as SEO spam and filtered out to prevent low-quality content from entering training datasets.

Why does OlmOCR keep PDFs with very little text instead of filtering them out?

PDFs with fewer than 200 characters or less than 50% alphabetic density are intentionally kept as safety guards (lines 93 and 97-99 of filter.py). This prevents the pipeline from discarding valid scanned documents or image-heavy academic papers where OCR extraction produces limited text, ensuring that potentially valuable visual content isn't prematurely eliminated due to extraction limitations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}