How olmOCR Detects and Filters SEO Spam PDFs

olmOCR removes low-quality documents before processing by running the PdfFilter class that performs sequential checks for interactive forms, English language content, and SEO keyword density exceeding the 0.4% threshold.

olmOCR implements a dedicated PDF filtering system to prevent SEO spam and download spam from entering the OCR pipeline. When processing documents with the pipeline.py script from the allenai/olmocr repository, activating the --apply_filter argument at lines 1226-1228 triggers a pre-processing stage that analyzes PDFs using the PdfFilter class defined in olmocr/filter/filter.py. This filter applies three distinct validation layers to ensure only high-quality, content-rich documents proceed to recognition.

The Three-Stage PDF Filter Architecture

The PdfFilter class implements a strict sequential validation pipeline. Each PDF undergoes three specific checks before being accepted for OCR processing.

Form Detection

The filter first checks for interactive form fields using the _is_form method. If a PDF contains form elements, it is immediately classified as non-academic content and filtered out. This check is implemented in lines 29-33 of olmocr/filter/filter.py.

English Language Validation

Next, the filter validates the document language. The first five pages are extracted using pdftotext and analyzed with the lingua language detection library. According to lines 101-105 of the filter implementation, documents must detect as English (or return None for undecodable text) to pass. Non-English documents are rejected at this stage.

SEO Keyword Density Scoring

The final and most critical step for spam detection occurs in the _is_download_spam method (lines 35-63). This method normalizes extracted text and calculates a spam score based on a curated set of SEO keywords including download, pdf, epub, free, casino, and viagra.

The scoring algorithm computes:

seo_score = sum(word_counts[w] for w in seo_words if w in word_counts)
is_spam = (seo_score / total_words) > self.download_spam_threshold

By default, the download_spam_threshold is set to 0.004 (0.4% of total words). Any document exceeding this ratio of SEO terminology is classified as download spam and removed from the pipeline.

Running the Filter: Command-Line and Programmatic Usage

Activate the filter when running the olmOCR pipeline by passing the --apply_filter flag defined in olmocr/pipeline.py:

python -m olmocr.pipeline workspace_path \
    --pdfs s3://my-bucket/papers/*.pdf \
    --apply_filter

For custom processing workflows, instantiate the PdfFilter class directly:

from olmocr.filter.filter import PdfFilter
from lingua import Language

filter = PdfFilter(
    languages_to_keep={Language.ENGLISH, None},
    apply_form_check=True,
    apply_download_spam_check=True,
    download_spam_threshold=0.004,
)

pdf_path = "/tmp/example.pdf"
if filter.filter_out_pdf(pdf_path):
    print("PDF removed – likely SEO spam or a form")
else:
    print("PDF kept – passes all filters")

Debugging and Customizing Spam Detection

To inspect why a specific document was flagged, expose the private _is_download_spam method for debugging purposes:

def debug_spam_score(text: str):
    filter = PdfFilter()
    return filter._is_download_spam(text)

sample_text = "Free download PDF ebook file ..."
print("Is spam?:", debug_spam_score(sample_text))

You can adjust the sensitivity by modifying the download_spam_threshold parameter. Lower values (e.g., 0.002) create stricter filtering, while higher values (e.g., 0.01) allow more documents through.

Summary

• olmOCR uses a dedicated PdfFilter class in olmocr/filter/filter.py to pre-screen PDFs before OCR processing when the --apply_filter flag is active in olmocr/pipeline.py.
• The filter applies three sequential checks: interactive form detection (_is_form), English language validation (lines 101-105), and SEO keyword density analysis (_is_download_spam).
• Documents are flagged as spam if SEO keywords exceed 0.4% of total words (threshold 0.004), using a curated word list including terms like download, casino, and viagra.
• Only documents passing all three stages—non-form, English-language, and low spam score—proceed to the main OCR pipeline.

Frequently Asked Questions

What is the default SEO spam threshold in olmOCR?

The default download_spam_threshold is 0.004, meaning documents where SEO keywords constitute more than 0.4% of the total word count are filtered out. This value is configurable via the PdfFilter constructor parameter when using the class programmatically.

How does olmOCR handle non-English PDFs?

olmOCR extracts text from the first five pages using pdftotext and uses the lingua library to detect language. According to lines 101-105 in olmocr/filter/filter.py, only documents detected as English or those returning None (undecodable text) pass through. All other languages are rejected at the filter stage.

Can I disable specific checks while keeping the filter active?

Yes. The PdfFilter class accepts boolean parameters apply_form_check and apply_download_spam_check. Set either to False when instantiating the filter to skip that specific validation while retaining others. However, the command-line interface in pipeline.py applies all checks when --apply_filter is used.

Where does olmOCR define the SEO keyword list?

The curated list of SEO spam terms—including words like download, pdf, epub, free, casino, and viagra—is referenced within the _is_download_spam method in olmocr/filter/filter.py (lines 35-63). This word set is used to calculate the density score that determines whether a document is classified as spam.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}