# How olmOCR Detects and Filters SEO Spam and Form-Based PDFs: A Technical Deep Dive

> Discover how olmOCR uses its PdfFilter class to detect and filter SEO spam and form-based PDFs, ensuring quality text extraction. Learn more about this technical deep dive into the allenai/olmocr repository.

- Repository: [Ai2/olmocr](https://github.com/allenai/olmocr)
- Tags: deep-dive
- Published: 2026-07-05

---

**olmOCR uses a three-stage `PdfFilter` class—checking for interactive forms, validating English language content via the *lingua* library, and scoring against a curated SEO word list—to remove low-quality PDFs before they reach the OCR pipeline.**

The `allenai/olmocr` repository implements a robust pre-processing filter that prevents SEO spam, download spam, and interactive forms from wasting OCR compute cycles. When you pass the `--apply_filter` flag to the pipeline, each PDF undergoes rigorous validation in [`olmocr/filter/filter.py`](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py) before any text recognition occurs.

## The Three-Stage PDF Filtering Pipeline

The `PdfFilter` class applies checks sequentially in the `filter_out_pdf()` method. A PDF must survive all three stages to proceed to OCR processing.

### Stage 1: Interactive Form Detection

First, the filter checks for interactive PDF forms. The `_is_form` method (lines 29‑33 in [`olmocr/filter/filter.py`](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py)) inspects the PDF structure for form fields. If the document contains interactive elements, it is immediately rejected. This prevents processing of fillable templates, tax forms, and survey documents that typically contain minimal usable text content.

### Stage 2: English Language Validation

Next, the filter validates language using the **lingua** library. The pipeline extracts text from the first five pages using `pdftotext`, then performs language detection on the extracted content. The document passes only if the detected language is English or if the content is undecodable (returning `None`). This check appears at lines 101‑105 in [`olmocr/filter/filter.py`](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py), ensuring the OCR model processes primarily English-language academic and literary content.

### Stage 3: SEO Spam Scoring Algorithm

The final check targets SEO spam and download spam through the `_is_download_spam` method (lines 35‑63). This algorithm:

1. Normalizes the extracted text to lowercase
2. Counts occurrences of curated SEO trigger words (*download, pdf, epub, free, casino, viagra*, etc.)
3. Computes a spam score as the ratio of SEO words to total word count

The default threshold is `0.004` (0.4% of words). If the proportion exceeds this threshold, the PDF is classified as spam and filtered out.

```python

# Logic from _is_download_spam (olmocr/filter/filter.py)

seo_score = sum(word_counts[w] for w in seo_words if w in word_counts)
return (seo_score / total_words) > self.download_spam_threshold

```

## Implementation Details: The PdfFilter Class

The `PdfFilter` class centralizes all filtering logic in [`olmocr/filter/filter.py`](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py). It accepts configurable parameters including `languages_to_keep`, `apply_form_check`, and `download_spam_threshold`.

Key configuration defaults:
- **Language set**: `{Language.ENGLISH, None}`
- **Spam threshold**: `0.004` (configurable via constructor)
- **Form checking**: Enabled via `apply_form_check=True`

The class uses a vocabulary-based detection approach rather than machine learning, making it lightweight and deterministic. The SEO word list targets common spam patterns found in free ebook sites, casino advertisements, and pharmaceutical spam PDFs.

## Command-Line and Programmatic Usage

You can activate filtering via the command line in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) (see lines 1226‑1228 where `parser.add_argument("--apply_filter", ...)` is defined):

```bash
python -m olmocr.pipeline workspace_path \
    --pdfs s3://my-bucket/papers/*.pdf \
    --apply_filter

```

For custom integrations, instantiate `PdfFilter` directly:

```python
from olmocr.filter.filter import PdfFilter
from lingua import Language

# Configure filter for English content with strict spam detection

filter = PdfFilter(
    languages_to_keep={Language.ENGLISH, None},
    apply_form_check=True,
    apply_download_spam_check=True,
    download_spam_threshold=0.004,
)

pdf_path = "/tmp/document.pdf"
if filter.filter_out_pdf(pdf_path):
    print("PDF rejected: contains forms, non-English text, or SEO spam")
else:
    print("PDF accepted: passes all quality filters")

```

To debug spam scoring during development:

```python
def inspect_spam_score(text: str):
    filter = PdfFilter()
    # Access internal method for diagnostics

    return filter._is_download_spam(text)

sample = "Free download PDF ebook file available now"
print(f"Spam detected: {inspect_spam_score(sample)}")

```

## Summary

- **olmOCR** implements a `PdfFilter` class in [`olmocr/filter/filter.py`](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py) that runs before the main OCR pipeline when `--apply_filter` is enabled.
- **Three sequential checks** identify interactive forms (via `_is_form`), validate English language content using *lingua* on the first five pages, and calculate an SEO spam ratio via `_is_download_spam`.
- **Spam scoring** uses a curated word list and a default threshold of `0.004` (0.4% of total words) to classify documents as SEO spam.
- **Both CLI and Python API** support configurable thresholds and check toggles, allowing integration into custom document processing workflows.

## Frequently Asked Questions

### What specific words trigger the SEO spam detection in olmOCR?

The `_is_download_spam` method checks against a curated set of SEO trigger words including *download, pdf, epub, free, casino, viagra*, and similar terms associated with spam content. The exact word list is embedded in the `PdfFilter` implementation in [`olmocr/filter/filter.py`](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py). The algorithm calculates what percentage of the total document words consist of these terms, rejecting documents exceeding the 0.4% threshold.

### How does olmOCR detect the language of a PDF?

The filter extracts text from the first five pages using `pdftotext`, then passes the content to the *lingua* language detection library. The document passes only if identified as English or if the language is undetectable (returns `None`). This occurs in the language validation section of [`olmocr/filter/filter.py`](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py) (lines 101‑105), ensuring the pipeline processes primarily English academic and literary content.

### Can I adjust the spam detection sensitivity?

Yes. When instantiating `PdfFilter` programmatically, pass a custom `download_spam_threshold` value to the constructor. The default is `0.004` (0.4%), but you can increase it for stricter filtering or decrease it to allow more documents through. This parameter directly controls the ratio calculation in `_is_download_spam` without modifying the source code.

### Where is the PDF filter logic located in the repository?

The core filtering logic resides in [`olmocr/filter/filter.py`](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py), which implements the `PdfFilter` class containing `_is_form`, `_is_download_spam`, and the main `filter_out_pdf` entry point. The command-line integration that exposes the `--apply_filter` flag is located in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) around lines 1226‑1228.