How to Remove Headers and Footers Using olmOCR: Detection and Enforcement Guide

olmOCR removes headers and footers by first detecting them with a Gemini LLM in mine_headers_footers.py, then enforcing their absence during OCR extraction using TextPresenceTest assertions.

The allenai/olmocr repository treats header and footer removal as a validation workflow rather than a preprocessing step. By combining vision-based detection with text-presence testing, you can ensure extracted documents contain only the primary content without repetitive page metadata.

The Two-Step Process for Header and Footer Removal

olmOCR implements header and footer removal through detection followed by strict enforcement. This approach separates the computer vision task of locating metadata from the natural language processing task of extraction.

Step 1: Detect Headers and Footers with Gemini

The detection phase uses olmocr/bench/miners/mine_headers_footers.py to analyze PDFs. The script renders a random page from each PDF to a PNG image using render_pdf_to_base64png from olmocr/data/renderpdf.py, then submits the image to the Gemini API.

The LLM returns JSON arrays labeled "headers" and "footers" containing detected text strings. The script filters out strings shorter than four characters to avoid false positives:

header_footer_text = [x for x in header_footer_text if len(x.strip()) > 3]

Before processing, the script filters obvious junk documents (forms, spam, non-English PDFs) using PdfFilter() from olmocr/filter/filter.py.

Step 2: Enforce Absence with TextPresence Tests

For every detected string, the miner creates a TextPresenceTest defined in olmocr/bench/tests.py. These tests use type="absent", which asserts that the OCR output must not contain the specific header or footer text.

The script saves these tests to header_footer_tests.jsonl in JSON Lines format. When the OCR pipeline runs later, it loads this file and validates that extracted text excludes all recorded metadata strings.

Running the Header and Footer Detection Pipeline

To generate header and footer tests for your PDF corpus stored on S3, execute the miner script:

python olmocr/bench/miners/mine_headers_footers.py \
    --input_list s3_paths.txt \
    --output_dir ./output \
    --api_key $GEMINI_API_KEY \
    --temp_dir /tmp/mine_hdr_ftr

This command processes each PDF listed in s3_paths.txt, downloading files temporarily via download_pdf_from_s3(). The output file ./output/header_footer_tests.jsonl contains entries formatted as:

{"id":"doc123_pg5_header_00","pdf":"doc123_pg5.pdf","page":1,"type":"absent","text":"Conference Proceedings","max_diffs":0}

Each JSON object represents a constraint: the OCR text must not contain the specified string, with max_diffs: 0 indicating zero tolerance for partial matches.

Integrating Tests into the OCR Pipeline

After generating the test file, pass it to the main OCR pipeline to enforce header and footer removal:

from olmocr.pipeline import run_ocr

run_ocr(
    input_pdf="./output/pdf/your_doc_pg5.pdf",
    tests_path="./output/header_footer_tests.jsonl",
    # additional pipeline arguments

)

When olmocr/pipeline.py processes pages, it runs the TextPresenceTest checks. If the extracted text contains any header or footer strings recorded in the JSONL file, the test fails. You can configure the pipeline to handle failures by re-extracting the page (e.g., after cropping the PDF), flagging the document for manual review, or discarding the page entirely.

Key Files and Implementation Details

Understanding the source code structure helps customize the removal workflow:

• olmocr/bench/miners/mine_headers_footers.py: Primary detection script that interfaces with Gemini and generates TextPresenceTest objects.
• olmocr/bench/tests.py: Defines TextPresenceTest and the save_tests() function for serializing constraints to JSONL.
• olmocr/data/renderpdf.py: Provides render_pdf_to_base64png() for converting PDF pages to base64-encoded PNGs suitable for LLM vision inputs.
• olmocr/filter/filter.py: Contains PdfFilter() and filter_out_pdf() for pre-screening documents before detection.
• olmocr/pipeline.py: Orchestrates OCR processing and executes the header/footer absence checks.
• olmocr/bench/miners/check_headers_footers.py: Utility for verifying header/footer expressions against specific PDF pages during evaluation.

Summary

• olmOCR uses a two-stage approach: Gemini-based detection in mine_headers_footers.py followed by assertion-based enforcement via TextPresenceTest.
• Detection filters short strings: Only text strings longer than three characters are preserved to reduce false positives.
• Tests are stored as JSONL: The header_footer_tests.jsonl file contains absent-type constraints that the OCR pipeline validates against during extraction.
• Pipeline integration is automatic: Pass the test file path to run_ocr() or olmocr/pipeline.py to enforce metadata removal without manual intervention.

Frequently Asked Questions

How does olmOCR detect headers and footers automatically?

olmOCR samples a random page from each PDF and renders it as a PNG image using render_pdf_to_base64png(). It sends this image to the Gemini LLM with a JSON schema requesting "headers" and "footers" arrays. The LLM identifies repetitive metadata text visible in the page margins, which the script extracts and filters for quality.

What happens when header text is detected during the OCR pipeline?

When the pipeline encounters text that matches a recorded header or footer string in header_footer_tests.jsonl, the TextPresenceTest fails because the constraint type is "absent". Depending on your configuration in olmocr/pipeline.py, this can trigger automatic re-extraction with adjusted cropping parameters, flag the document for manual cleaning, or filter the page from the output dataset.

Can I adjust the minimum length for detected header strings?

Yes. In olmocr/bench/miners/mine_headers_footers.py, modify the list comprehension that filters the Gemini output:

header_footer_text = [x for x in header_footer_text if len(x.strip()) > 3]

Change the threshold 3 to your desired minimum character count before the script creates TextPresenceTest objects.

Does the detection process support local PDF files or only S3 storage?

The mine_headers_footers.py script accepts an --input_list argument pointing to a text file of paths. While the source analysis emphasizes S3 paths using download_pdf_from_s3(), you can adapt the script to work with local filesystem paths by modifying the download logic or using the --temp_dir parameter for staging local files prior to processing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}