# How to Remove Headers and Footers Using olmOCR: Detection and Enforcement Guide

> Learn to remove headers and footers with olmOCR. Discover how Gemini LLM detection and TextPresenceTest assertions ensure clean OCR extraction. Get started today.

- Repository: [Ai2/olmocr](https://github.com/allenai/olmocr)
- Tags: how-to-guide
- Published: 2026-07-02

---

**olmOCR removes headers and footers by first detecting them with a Gemini LLM in [`mine_headers_footers.py`](https://github.com/allenai/olmocr/blob/main/mine_headers_footers.py), then enforcing their absence during OCR extraction using `TextPresenceTest` assertions.**

The allenai/olmocr repository treats header and footer removal as a validation workflow rather than a preprocessing step. By combining vision-based detection with text-presence testing, you can ensure extracted documents contain only the primary content without repetitive page metadata.

## The Two-Step Process for Header and Footer Removal

olmOCR implements header and footer removal through detection followed by strict enforcement. This approach separates the computer vision task of locating metadata from the natural language processing task of extraction.

### Step 1: Detect Headers and Footers with Gemini

The detection phase uses [`olmocr/bench/miners/mine_headers_footers.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/miners/mine_headers_footers.py) to analyze PDFs. The script renders a random page from each PDF to a PNG image using `render_pdf_to_base64png` from [`olmocr/data/renderpdf.py`](https://github.com/allenai/olmocr/blob/main/olmocr/data/renderpdf.py), then submits the image to the Gemini API.

The LLM returns JSON arrays labeled `"headers"` and `"footers"` containing detected text strings. The script filters out strings shorter than four characters to avoid false positives:

```python
header_footer_text = [x for x in header_footer_text if len(x.strip()) > 3]

```

Before processing, the script filters obvious junk documents (forms, spam, non-English PDFs) using `PdfFilter()` from [`olmocr/filter/filter.py`](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py).

### Step 2: Enforce Absence with TextPresence Tests

For every detected string, the miner creates a `TextPresenceTest` defined in [`olmocr/bench/tests.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/tests.py). These tests use `type="absent"`, which asserts that the OCR output must **not** contain the specific header or footer text.

The script saves these tests to `header_footer_tests.jsonl` in JSON Lines format. When the OCR pipeline runs later, it loads this file and validates that extracted text excludes all recorded metadata strings.

## Running the Header and Footer Detection Pipeline

To generate header and footer tests for your PDF corpus stored on S3, execute the miner script:

```bash
python olmocr/bench/miners/mine_headers_footers.py \
    --input_list s3_paths.txt \
    --output_dir ./output \
    --api_key $GEMINI_API_KEY \
    --temp_dir /tmp/mine_hdr_ftr

```

This command processes each PDF listed in [`s3_paths.txt`](https://github.com/allenai/olmocr/blob/main/s3_paths.txt), downloading files temporarily via `download_pdf_from_s3()`. The output file `./output/header_footer_tests.jsonl` contains entries formatted as:

```json
{"id":"doc123_pg5_header_00","pdf":"doc123_pg5.pdf","page":1,"type":"absent","text":"Conference Proceedings","max_diffs":0}

```

Each JSON object represents a constraint: the OCR text must not contain the specified string, with `max_diffs: 0` indicating zero tolerance for partial matches.

## Integrating Tests into the OCR Pipeline

After generating the test file, pass it to the main OCR pipeline to enforce header and footer removal:

```python
from olmocr.pipeline import run_ocr

run_ocr(
    input_pdf="./output/pdf/your_doc_pg5.pdf",
    tests_path="./output/header_footer_tests.jsonl",
    # additional pipeline arguments

)

```

When [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) processes pages, it runs the `TextPresenceTest` checks. If the extracted text contains any header or footer strings recorded in the JSONL file, the test fails. You can configure the pipeline to handle failures by re-extracting the page (e.g., after cropping the PDF), flagging the document for manual review, or discarding the page entirely.

## Key Files and Implementation Details

Understanding the source code structure helps customize the removal workflow:

- **[`olmocr/bench/miners/mine_headers_footers.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/miners/mine_headers_footers.py)**: Primary detection script that interfaces with Gemini and generates `TextPresenceTest` objects.
- **[`olmocr/bench/tests.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/tests.py)**: Defines `TextPresenceTest` and the `save_tests()` function for serializing constraints to JSONL.
- **[`olmocr/data/renderpdf.py`](https://github.com/allenai/olmocr/blob/main/olmocr/data/renderpdf.py)**: Provides `render_pdf_to_base64png()` for converting PDF pages to base64-encoded PNGs suitable for LLM vision inputs.
- **[`olmocr/filter/filter.py`](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py)**: Contains `PdfFilter()` and `filter_out_pdf()` for pre-screening documents before detection.
- **[`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py)**: Orchestrates OCR processing and executes the header/footer absence checks.
- **[`olmocr/bench/miners/check_headers_footers.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/miners/check_headers_footers.py)**: Utility for verifying header/footer expressions against specific PDF pages during evaluation.

## Summary

- **olmOCR uses a two-stage approach**: Gemini-based detection in [`mine_headers_footers.py`](https://github.com/allenai/olmocr/blob/main/mine_headers_footers.py) followed by assertion-based enforcement via `TextPresenceTest`.
- **Detection filters short strings**: Only text strings longer than three characters are preserved to reduce false positives.
- **Tests are stored as JSONL**: The `header_footer_tests.jsonl` file contains absent-type constraints that the OCR pipeline validates against during extraction.
- **Pipeline integration is automatic**: Pass the test file path to `run_ocr()` or [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) to enforce metadata removal without manual intervention.

## Frequently Asked Questions

### How does olmOCR detect headers and footers automatically?

olmOCR samples a random page from each PDF and renders it as a PNG image using `render_pdf_to_base64png()`. It sends this image to the Gemini LLM with a JSON schema requesting `"headers"` and `"footers"` arrays. The LLM identifies repetitive metadata text visible in the page margins, which the script extracts and filters for quality.

### What happens when header text is detected during the OCR pipeline?

When the pipeline encounters text that matches a recorded header or footer string in `header_footer_tests.jsonl`, the `TextPresenceTest` fails because the constraint type is `"absent"`. Depending on your configuration in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py), this can trigger automatic re-extraction with adjusted cropping parameters, flag the document for manual cleaning, or filter the page from the output dataset.

### Can I adjust the minimum length for detected header strings?

Yes. In [`olmocr/bench/miners/mine_headers_footers.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/miners/mine_headers_footers.py), modify the list comprehension that filters the Gemini output:

```python
header_footer_text = [x for x in header_footer_text if len(x.strip()) > 3]

```

Change the threshold `3` to your desired minimum character count before the script creates `TextPresenceTest` objects.

### Does the detection process support local PDF files or only S3 storage?

The [`mine_headers_footers.py`](https://github.com/allenai/olmocr/blob/main/mine_headers_footers.py) script accepts an `--input_list` argument pointing to a text file of paths. While the source analysis emphasizes S3 paths using `download_pdf_from_s3()`, you can adapt the script to work with local filesystem paths by modifying the download logic or using the `--temp_dir` parameter for staging local files prior to processing.