Dolma Document Format Specification: How OlmOCR Structures PDF Output

The Dolma document format specification defines a JSON schema where OlmOCR stores OCR-extracted text alongside metadata and page-level attributes, producing Dolma-compatible documents with fields like id, text, source, metadata, and attributes.

The OlmOCR pipeline transforms PDF documents into structured JSON following the Dolma document format specification. This lightweight schema enables downstream Dolma tooling to index, evaluate, and process OCR content efficiently. According to the allenai/olmocr source code, the implementation generates SHA-1 based identifiers and captures both global document properties and granular page-level attributes.

Core Schema of the Dolma Document Format

The Dolma document format specification requires a JSON object with six top-level fields. In olmocr/pipeline.py (lines 602-651), the build_dolma_document function constructs this object during PDF processing.

Required Top-Level Fields

Every Dolma document output by OlmOCR contains these mandatory fields:

• id – A stable identifier generated as the SHA-1 hash of the document text using hashlib.sha1(document_text.encode()).hexdigest().
• text – Concatenated plain text extracted from all PDF pages.
• source – Producer identifier string, set to "olmocr" in the main pipeline or "s2pdf" in legacy implementations.
• added – ISO date string (YYYY-MM-DD) indicating when the document entered the system.
• created – ISO date string representing the document creation timestamp.

Metadata Dictionary

The metadata field (assembled in olmocr/pipeline.py lines 623-632) contains provenance and processing statistics:

• Source-File – Original PDF path (local filesystem, S3, or tarball location).
• olmocr-version – Version string imported from olmocr/version.py.
• pdf-total-pages – Total page count processed from the source PDF.
• total-input-tokens – Aggregated input tokens from all page-level OCR calls.
• total-output-tokens – Aggregated output tokens generated by the model.
• total-fallback-pages – Count of pages that triggered simplified OCR fallback paths.

Attributes for Page-Level Granularity

The attributes dictionary provides fine-grained PDF-specific data required by downstream Dolma processes:

• pdf_page_numbers – Character span mappings formatted as [start_char, end_char, page_number] lists, enabling text-to-page coordinate tracking.
• primary_language – Detected language code for each processed page.
• is_rotation_valid / rotation_correction – Boolean and numeric rotation diagnostics.
• is_table / is_diagram – Boolean classification flags for document structure analysis.

Implementation in OlmOCR Source Code

OlmOCR implements the Dolma document format specification through two primary functions depending on the pipeline stage and compatibility requirements.

Legacy Helper: mk_dolma_doc

In olmocr/datatypes.py (lines 14-33), the PdfOutput.mk_dolma_doc method provides a minimal implementation for older S2PDF-style pipelines. This helper sets source to "s2pdf" and includes basic metadata fields (Source-File, pdf-pages, pdf-total-pages) while accepting additional kwargs for extensibility.

Main Pipeline Function: build_dolma_document

The primary conversion occurs in build_dolma_document within olmocr/pipeline.py (lines 602-651). This function:

1. Computes the SHA-1 hash for the id field.
2. Aggregates text from all page results into the text field.
3. Populates comprehensive metadata including token counts and version info from olmocr/version.py.
4. Constructs the attributes object with per-page character spans and detection flags.

Generating Dolma Documents

Using the Legacy Helper

For backward compatibility with S2PDF workflows:

from olmocr.datatypes import PdfOutput

pdf = PdfOutput(
    path="s3://my-bucket/documents/example.pdf",
    text="Full OCR text extracted from PDF...",
    total_pdf_pages=10,
    processed_pdf_pages=10,
)
dolma_json = pdf.mk_dolma_doc(custom_key="custom_value")
print(dolma_json)  # JSON string conforming to Dolma schema

Standard Pipeline Output

The typical OlmOCR workflow uses build_dolma_document directly:

from olmocr.pipeline import build_dolma_document
import json

# page_results contains OCR responses from the pipeline

dolma_doc = build_dolma_document(
    pdf_orig_path="s3://my-bucket/documents/example.pdf",
    page_results=page_results,
)

print(json.dumps(dolma_doc, indent=2))

Both approaches output JSON objects adhering to the Dolma document format specification, ready for ingestion by Dolma indexing and evaluation tools.

Summary

• The Dolma document format specification defines a JSON schema with id, text, source, added, created, metadata, and attributes fields.
• OlmOCR generates id values using SHA-1 hashes of the document text for stability across processing runs.
• The metadata dictionary tracks provenance via Source-File, version info from olmocr/version.py, and processing statistics like token counts and fallback page frequencies.
• Page-level character mappings and classification flags reside in the attributes object, enabling precise text-to-PDF coordinate tracking for downstream analysis.
• Implementation spans olmocr/datatypes.py for legacy support and olmocr/pipeline.py (lines 602-651) for full-featured conversion with rich attributes.

Frequently Asked Questions

What makes a JSON document "Dolma-compatible"?

A Dolma-compatible document must include the core fields id, text, source, added, and created, with metadata containing provenance information. OlmOCR extends this baseline by adding the attributes dictionary for PDF-specific data like page numbers and rotation corrections, ensuring compatibility with Dolma's downstream filtering and indexing pipelines.

How is the document ID generated in OlmOCR?

OlmOCR computes the id field using SHA-1 hashing of the UTF-8 encoded document text via hashlib.sha1(document_text.encode()).hexdigest(). This approach, implemented in olmocr/pipeline.py, guarantees that identical OCR output produces identical identifiers across different processing runs.

What processing statistics does the metadata contain?

The metadata object includes total-input-tokens and total-output-tokens aggregating LLM usage across all pages, total-fallback-pages counting simplified OCR paths taken, pdf-total-pages for page counts, and olmocr-version tracking which software version processed the file. These fields enable cost analysis and quality auditing of OCR operations.

Can I extend the Dolma document with custom fields?

Yes. The legacy mk_dolma_doc method in olmocr/datatypes.py accepts arbitrary kwargs that merge into the metadata dictionary. While the main build_dolma_document function follows a stricter schema for consistency, you can modify the source in olmocr/pipeline.py (lines 602-651) to inject additional custom metadata fields as needed for specialized downstream applications.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}