How olmOCR Detects and Filters SEO Spam and Form-Based PDFs: A Technical Deep Dive

olmOCR uses a three-stage PdfFilter class—checking for interactive forms, validating English language content via the lingua library, and scoring against a curated SEO word list—to remove low-quality PDFs before they reach the OCR pipeline.

The allenai/olmocr repository implements a robust pre-processing filter that prevents SEO spam, download spam, and interactive forms from wasting OCR compute cycles. When you pass the --apply_filter flag to the pipeline, each PDF undergoes rigorous validation in olmocr/filter/filter.py before any text recognition occurs.

The Three-Stage PDF Filtering Pipeline

The PdfFilter class applies checks sequentially in the filter_out_pdf() method. A PDF must survive all three stages to proceed to OCR processing.

Stage 1: Interactive Form Detection

First, the filter checks for interactive PDF forms. The _is_form method (lines 29‑33 in olmocr/filter/filter.py) inspects the PDF structure for form fields. If the document contains interactive elements, it is immediately rejected. This prevents processing of fillable templates, tax forms, and survey documents that typically contain minimal usable text content.

Stage 2: English Language Validation

Next, the filter validates language using the lingua library. The pipeline extracts text from the first five pages using pdftotext, then performs language detection on the extracted content. The document passes only if the detected language is English or if the content is undecodable (returning None). This check appears at lines 101‑105 in olmocr/filter/filter.py, ensuring the OCR model processes primarily English-language academic and literary content.

Stage 3: SEO Spam Scoring Algorithm

The final check targets SEO spam and download spam through the _is_download_spam method (lines 35‑63). This algorithm:

1. Normalizes the extracted text to lowercase
2. Counts occurrences of curated SEO trigger words (download, pdf, epub, free, casino, viagra, etc.)
3. Computes a spam score as the ratio of SEO words to total word count

The default threshold is 0.004 (0.4% of words). If the proportion exceeds this threshold, the PDF is classified as spam and filtered out.

# Logic from _is_download_spam (olmocr/filter/filter.py)

seo_score = sum(word_counts[w] for w in seo_words if w in word_counts)
return (seo_score / total_words) > self.download_spam_threshold

Implementation Details: The PdfFilter Class

The PdfFilter class centralizes all filtering logic in olmocr/filter/filter.py. It accepts configurable parameters including languages_to_keep, apply_form_check, and download_spam_threshold.

Key configuration defaults:

• Language set: {Language.ENGLISH, None}
• Spam threshold: 0.004 (configurable via constructor)
• Form checking: Enabled via apply_form_check=True

The class uses a vocabulary-based detection approach rather than machine learning, making it lightweight and deterministic. The SEO word list targets common spam patterns found in free ebook sites, casino advertisements, and pharmaceutical spam PDFs.

Command-Line and Programmatic Usage

You can activate filtering via the command line in olmocr/pipeline.py (see lines 1226‑1228 where parser.add_argument("--apply_filter", ...) is defined):

python -m olmocr.pipeline workspace_path \
    --pdfs s3://my-bucket/papers/*.pdf \
    --apply_filter

For custom integrations, instantiate PdfFilter directly:

from olmocr.filter.filter import PdfFilter
from lingua import Language

# Configure filter for English content with strict spam detection

filter = PdfFilter(
    languages_to_keep={Language.ENGLISH, None},
    apply_form_check=True,
    apply_download_spam_check=True,
    download_spam_threshold=0.004,
)

pdf_path = "/tmp/document.pdf"
if filter.filter_out_pdf(pdf_path):
    print("PDF rejected: contains forms, non-English text, or SEO spam")
else:
    print("PDF accepted: passes all quality filters")

To debug spam scoring during development:

def inspect_spam_score(text: str):
    filter = PdfFilter()
    # Access internal method for diagnostics

    return filter._is_download_spam(text)

sample = "Free download PDF ebook file available now"
print(f"Spam detected: {inspect_spam_score(sample)}")

Summary

• olmOCR implements a PdfFilter class in olmocr/filter/filter.py that runs before the main OCR pipeline when --apply_filter is enabled.
• Three sequential checks identify interactive forms (via _is_form), validate English language content using lingua on the first five pages, and calculate an SEO spam ratio via _is_download_spam.
• Spam scoring uses a curated word list and a default threshold of 0.004 (0.4% of total words) to classify documents as SEO spam.
• Both CLI and Python API support configurable thresholds and check toggles, allowing integration into custom document processing workflows.

Frequently Asked Questions

What specific words trigger the SEO spam detection in olmOCR?

The _is_download_spam method checks against a curated set of SEO trigger words including download, pdf, epub, free, casino, viagra, and similar terms associated with spam content. The exact word list is embedded in the PdfFilter implementation in olmocr/filter/filter.py. The algorithm calculates what percentage of the total document words consist of these terms, rejecting documents exceeding the 0.4% threshold.

How does olmOCR detect the language of a PDF?

The filter extracts text from the first five pages using pdftotext, then passes the content to the lingua language detection library. The document passes only if identified as English or if the language is undetectable (returns None). This occurs in the language validation section of olmocr/filter/filter.py (lines 101‑105), ensuring the pipeline processes primarily English academic and literary content.

Can I adjust the spam detection sensitivity?

Yes. When instantiating PdfFilter programmatically, pass a custom download_spam_threshold value to the constructor. The default is 0.004 (0.4%), but you can increase it for stricter filtering or decrease it to allow more documents through. This parameter directly controls the ratio calculation in _is_download_spam without modifying the source code.

Where is the PDF filter logic located in the repository?

The core filtering logic resides in olmocr/filter/filter.py, which implements the PdfFilter class containing _is_form, _is_download_spam, and the main filter_out_pdf entry point. The command-line integration that exposes the --apply_filter flag is located in olmocr/pipeline.py around lines 1226‑1228.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}