# How olmOCR Detects and Filters SEO Spam PDFs

> Discover how olmOCR detects and filters SEO spam PDFs using PdfFilter for interactive forms, English content, and keyword density checks. Improve document quality effortlessly.

- Repository: [Ai2/olmocr](https://github.com/allenai/olmocr)
- Tags: how-to-guide
- Published: 2026-07-02

---

**olmOCR removes low-quality documents before processing by running the `PdfFilter` class that performs sequential checks for interactive forms, English language content, and SEO keyword density exceeding the 0.4% threshold.**

olmOCR implements a dedicated PDF filtering system to prevent SEO spam and download spam from entering the OCR pipeline. When processing documents with the [`pipeline.py`](https://github.com/allenai/olmocr/blob/main/pipeline.py) script from the allenai/olmocr repository, activating the `--apply_filter` argument at lines 1226-1228 triggers a pre-processing stage that analyzes PDFs using the `PdfFilter` class defined in [`olmocr/filter/filter.py`](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py). This filter applies three distinct validation layers to ensure only high-quality, content-rich documents proceed to recognition.

## The Three-Stage PDF Filter Architecture

The `PdfFilter` class implements a strict sequential validation pipeline. Each PDF undergoes three specific checks before being accepted for OCR processing.

### Form Detection

The filter first checks for interactive form fields using the `_is_form` method. If a PDF contains form elements, it is immediately classified as non-academic content and filtered out. This check is implemented in lines 29-33 of [`olmocr/filter/filter.py`](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py).

### English Language Validation

Next, the filter validates the document language. The first five pages are extracted using `pdftotext` and analyzed with the *lingua* language detection library. According to lines 101-105 of the filter implementation, documents must detect as English (or return `None` for undecodable text) to pass. Non-English documents are rejected at this stage.

### SEO Keyword Density Scoring

The final and most critical step for spam detection occurs in the `_is_download_spam` method (lines 35-63). This method normalizes extracted text and calculates a spam score based on a curated set of SEO keywords including *download*, *pdf*, *epub*, *free*, *casino*, and *viagra*.

The scoring algorithm computes:

```python
seo_score = sum(word_counts[w] for w in seo_words if w in word_counts)
is_spam = (seo_score / total_words) > self.download_spam_threshold

```

By default, the `download_spam_threshold` is set to **0.004** (0.4% of total words). Any document exceeding this ratio of SEO terminology is classified as download spam and removed from the pipeline.

## Running the Filter: Command-Line and Programmatic Usage

Activate the filter when running the olmOCR pipeline by passing the `--apply_filter` flag defined in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py):

```bash
python -m olmocr.pipeline workspace_path \
    --pdfs s3://my-bucket/papers/*.pdf \
    --apply_filter

```

For custom processing workflows, instantiate the `PdfFilter` class directly:

```python
from olmocr.filter.filter import PdfFilter
from lingua import Language

filter = PdfFilter(
    languages_to_keep={Language.ENGLISH, None},
    apply_form_check=True,
    apply_download_spam_check=True,
    download_spam_threshold=0.004,
)

pdf_path = "/tmp/example.pdf"
if filter.filter_out_pdf(pdf_path):
    print("PDF removed – likely SEO spam or a form")
else:
    print("PDF kept – passes all filters")

```

## Debugging and Customizing Spam Detection

To inspect why a specific document was flagged, expose the private `_is_download_spam` method for debugging purposes:

```python
def debug_spam_score(text: str):
    filter = PdfFilter()
    return filter._is_download_spam(text)

sample_text = "Free download PDF ebook file ..."
print("Is spam?:", debug_spam_score(sample_text))

```

You can adjust the sensitivity by modifying the `download_spam_threshold` parameter. Lower values (e.g., `0.002`) create stricter filtering, while higher values (e.g., `0.01`) allow more documents through.

## Summary

- **olmOCR** uses a dedicated `PdfFilter` class in [`olmocr/filter/filter.py`](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py) to pre-screen PDFs before OCR processing when the `--apply_filter` flag is active in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py).
- The filter applies three sequential checks: interactive form detection (`_is_form`), English language validation (lines 101-105), and SEO keyword density analysis (`_is_download_spam`).
- Documents are flagged as spam if SEO keywords exceed **0.4%** of total words (threshold `0.004`), using a curated word list including terms like *download*, *casino*, and *viagra*.
- Only documents passing all three stages—non-form, English-language, and low spam score—proceed to the main OCR pipeline.

## Frequently Asked Questions

### What is the default SEO spam threshold in olmOCR?

The default `download_spam_threshold` is **0.004**, meaning documents where SEO keywords constitute more than 0.4% of the total word count are filtered out. This value is configurable via the `PdfFilter` constructor parameter when using the class programmatically.

### How does olmOCR handle non-English PDFs?

olmOCR extracts text from the first five pages using `pdftotext` and uses the *lingua* library to detect language. According to lines 101-105 in [`olmocr/filter/filter.py`](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py), only documents detected as English or those returning `None` (undecodable text) pass through. All other languages are rejected at the filter stage.

### Can I disable specific checks while keeping the filter active?

Yes. The `PdfFilter` class accepts boolean parameters `apply_form_check` and `apply_download_spam_check`. Set either to `False` when instantiating the filter to skip that specific validation while retaining others. However, the command-line interface in [`pipeline.py`](https://github.com/allenai/olmocr/blob/main/pipeline.py) applies all checks when `--apply_filter` is used.

### Where does olmOCR define the SEO keyword list?

The curated list of SEO spam terms—including words like *download*, *pdf*, *epub*, *free*, *casino*, and *viagra*—is referenced within the `_is_download_spam` method in [`olmocr/filter/filter.py`](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py) (lines 35-63). This word set is used to calculate the density score that determines whether a document is classified as spam.