# Complete PDF Filtering Pipeline in OlmOCR: Language, Form, and Spam Detection

> Explore the OlmOCR PDF filtering pipeline for efficient language, form, and spam detection to streamline your document processing. Optimize your workflow today.

- Repository: [Ai2/olmocr](https://github.com/allenai/olmocr)
- Tags: how-to-guide
- Published: 2026-07-06

---

**The OlmOCR PDF filtering pipeline sequentially checks for fillable forms, validates English language using Lingua, and detects SEO spam through keyword density ratios before processing documents.**

The `allenai/olmocr` repository implements a robust filtering system to discard low-quality PDFs before expensive OCR processing. The complete PDF filtering pipeline examines documents for fillable forms, non-English content, and download-spam characteristics using a sequential validation strategy defined in the core filter module.

## Architecture and Entry Points

The filtering logic resides in **[`olmocr/filter/filter.py`](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py)** within the `PdfFilter` class. The main processing entry point is **[`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py)**, specifically the `process_single_pdf` function, which invokes `filter_out_pdf` when the `--apply_filter` flag is enabled. This design ensures that only high-quality documents proceed to downstream OCR and LLM inference stages.

When initialized, the pipeline creates a cached filter instance using a lambda function that configures default parameters:

```python
get_pdf_filter = cache(
    lambda: PdfFilter(
        languages_to_keep={Language.ENGLISH, None},
        apply_download_spam_check=True,
        apply_form_check=True,
    )
)

```

During document processing, the filter is applied conditionally:

```python
if args.apply_filter and get_pdf_filter().filter_out_pdf(local_pdf_path):
    logger.info(f"Filtering out pdf {pdf_orig_path}")
    return None

```

## The Three-Stage Filtering Process

The `filter_out_pdf` method executes three independent checks in sequence. If any check triggers a discard condition, the method returns `True` and the PDF is excluded from further processing.

### Form Detection Using PyPDF

The first validation step identifies fillable PDF forms. The `_is_form` method (lines 29-33 of [`filter.py`](https://github.com/allenai/olmocr/blob/main/filter.py)) utilizes **`pypdf.PdfReader.get_form_text_fields()`** to detect interactive form fields.

If the method returns any form fields and `apply_form_check` is enabled (default `True`), the PDF is immediately filtered out. This prevents structured data entry forms from polluting text-training corpora.

### Language Detection with Lingua

The second stage validates document language using the Lingua library. The pipeline extracts text from the first five pages using `pdftotext` via subprocess, then passes the content to **`lingua.LanguageDetectorBuilder`** (lines 101-105).

The PDF is filtered out if the detected language is not present in `languages_to_keep`. The default configuration accepts only **English** (`Language.ENGLISH`) and undetermined languages (`None`), effectively removing non-English documents from English-centric training datasets.

### Download-Spam (SEO) Detection

The final check identifies SEO-optimized spam documents using the `_is_download_spam` method (lines 35-62). This algorithm counts occurrences of spammy keywords—such as *download*, *pdf*, *ebook*, and *viagra*—and calculates the ratio of spam score to total word count.

If the ratio exceeds `download_spam_threshold` (default **0.004**), the document is classified as download spam and discarded. This threshold catches keyword-stuffed pages while preserving legitimate academic or technical documents.

## Safety Guards and Edge Cases

Beyond the three main filters, the pipeline implements protective logic to handle corrupted or atypical PDFs:

- **PDF Readability** (lines 65-76): If `PdfReader` cannot open or parse the file due to corruption or encryption, the document is filtered out immediately.
- **Minimum Text Length** (line 93): PDFs containing fewer than **200 characters** are explicitly kept rather than filtered, as insufficient data exists to make reliable language or spam determinations.
- **Alphabetic Density** (lines 97-99): Documents where fewer than **50%** of characters are alphabetic are preserved. This safety measure prevents the loss of heavily OCR-dependent scanned documents or image-heavy PDFs where text extraction yields limited alphabetic content.

## Configuration and Customization

You can instantiate and configure the filter for standalone use or custom pipelines:

**Basic usage with default settings:**

```python
from olmocr.filter import PdfFilter

pdf_filter = PdfFilter()
if pdf_filter.filter_out_pdf("/path/to/document.pdf"):
    print("PDF rejected")
else:
    print("PDF accepted")

```

**Disabling form detection:**

```python
pdf_filter = PdfFilter(apply_form_check=False)

```

**Supporting multiple languages:**

```python
from lingua import Language

pdf_filter = PdfFilter(
    languages_to_keep={Language.ENGLISH, Language.SPANISH, Language.FRENCH}
)

```

## Summary

- The **form detection** stage uses PyPDF to identify fillable forms and exclude structured data entry documents.
- **Language validation** extracts the first five pages and uses Lingua to enforce English-language requirements by default.
- **Spam detection** calculates keyword density ratios to eliminate SEO-optimized download spam with a 0.004 threshold.
- **Safety guards** preserve short or low-alphabetic PDFs to avoid discarding valid OCR-scanned documents.
- The pipeline integrates through [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) with cached filter instances to optimize performance across large document batches.

## Frequently Asked Questions

### How does OlmOCR detect fillable forms in PDFs?

OlmOCR detects fillable forms using the `_is_form` method in [`olmocr/filter/filter.py`](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py) (lines 29-33), which calls `pypdf.PdfReader.get_form_text_fields()`. If this method returns any interactive form fields, the PDF is classified as a form and filtered out unless `apply_form_check` is explicitly disabled during `PdfFilter` initialization.

### What languages does the OlmOCR filter support by default?

By default, the filter accepts only **English** and **undetermined** languages (`None`). The default `languages_to_keep` parameter is set to `{Language.ENGLISH, None}` using the Lingua language enum. You can customize this set during instantiation to include additional languages such as Spanish, French, or German.

### What is the default spam threshold and how does it work?

The default `download_spam_threshold` is **0.004** (0.4%). The `_is_download_spam` method counts occurrences of specific spam keywords—including "download," "pdf," "ebook," and "viagra"—and divides this count by the total word count. If the resulting ratio exceeds 0.004, the document is classified as SEO spam and filtered out to prevent low-quality content from entering training datasets.

### Why does OlmOCR keep PDFs with very little text instead of filtering them out?

PDFs with fewer than 200 characters or less than 50% alphabetic density are intentionally kept as safety guards (lines 93 and 97-99 of [`filter.py`](https://github.com/allenai/olmocr/blob/main/filter.py)). This prevents the pipeline from discarding valid scanned documents or image-heavy academic papers where OCR extraction produces limited text, ensuring that potentially valuable visual content isn't prematurely eliminated due to extraction limitations.