# Dolma Document Format Specification: How OlmOCR Structures PDF Output

> Explore the Dolma document format specification used by OlmOCR to structure PDF output. Learn how Dolma JSON stores OCR text and metadata for documents.

- Repository: [Ai2/olmocr](https://github.com/allenai/olmocr)
- Tags: api-reference
- Published: 2026-07-06

---

**The Dolma document format specification defines a JSON schema where OlmOCR stores OCR-extracted text alongside metadata and page-level attributes, producing Dolma-compatible documents with fields like `id`, `text`, `source`, `metadata`, and `attributes`.**

The OlmOCR pipeline transforms PDF documents into structured JSON following the Dolma document format specification. This lightweight schema enables downstream Dolma tooling to index, evaluate, and process OCR content efficiently. According to the allenai/olmocr source code, the implementation generates SHA-1 based identifiers and captures both global document properties and granular page-level attributes.

## Core Schema of the Dolma Document Format

The Dolma document format specification requires a JSON object with six top-level fields. In [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) (lines 602-651), the `build_dolma_document` function constructs this object during PDF processing.

### Required Top-Level Fields

Every Dolma document output by OlmOCR contains these mandatory fields:

- **`id`** – A stable identifier generated as the SHA-1 hash of the document text using `hashlib.sha1(document_text.encode()).hexdigest()`.
- **`text`** – Concatenated plain text extracted from all PDF pages.
- **`source`** – Producer identifier string, set to `"olmocr"` in the main pipeline or `"s2pdf"` in legacy implementations.
- **`added`** – ISO date string (`YYYY-MM-DD`) indicating when the document entered the system.
- **`created`** – ISO date string representing the document creation timestamp.

### Metadata Dictionary

The `metadata` field (assembled in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) lines 623-632) contains provenance and processing statistics:

- **`Source-File`** – Original PDF path (local filesystem, S3, or tarball location).
- **`olmocr-version`** – Version string imported from [`olmocr/version.py`](https://github.com/allenai/olmocr/blob/main/olmocr/version.py).
- **`pdf-total-pages`** – Total page count processed from the source PDF.
- **`total-input-tokens`** – Aggregated input tokens from all page-level OCR calls.
- **`total-output-tokens`** – Aggregated output tokens generated by the model.
- **`total-fallback-pages`** – Count of pages that triggered simplified OCR fallback paths.

### Attributes for Page-Level Granularity

The `attributes` dictionary provides fine-grained PDF-specific data required by downstream Dolma processes:

- **`pdf_page_numbers`** – Character span mappings formatted as `[start_char, end_char, page_number]` lists, enabling text-to-page coordinate tracking.
- **`primary_language`** – Detected language code for each processed page.
- **`is_rotation_valid`** / **`rotation_correction`** – Boolean and numeric rotation diagnostics.
- **`is_table`** / **`is_diagram`** – Boolean classification flags for document structure analysis.

## Implementation in OlmOCR Source Code

OlmOCR implements the Dolma document format specification through two primary functions depending on the pipeline stage and compatibility requirements.

### Legacy Helper: mk_dolma_doc

In [`olmocr/datatypes.py`](https://github.com/allenai/olmocr/blob/main/olmocr/datatypes.py) (lines 14-33), the `PdfOutput.mk_dolma_doc` method provides a minimal implementation for older S2PDF-style pipelines. This helper sets `source` to `"s2pdf"` and includes basic metadata fields (`Source-File`, `pdf-pages`, `pdf-total-pages`) while accepting additional kwargs for extensibility.

### Main Pipeline Function: build_dolma_document

The primary conversion occurs in `build_dolma_document` within [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) (lines 602-651). This function:
1. Computes the SHA-1 hash for the `id` field.
2. Aggregates text from all page results into the `text` field.
3. Populates comprehensive metadata including token counts and version info from [`olmocr/version.py`](https://github.com/allenai/olmocr/blob/main/olmocr/version.py).
4. Constructs the `attributes` object with per-page character spans and detection flags.

## Generating Dolma Documents

### Using the Legacy Helper

For backward compatibility with S2PDF workflows:

```python
from olmocr.datatypes import PdfOutput

pdf = PdfOutput(
    path="s3://my-bucket/documents/example.pdf",
    text="Full OCR text extracted from PDF...",
    total_pdf_pages=10,
    processed_pdf_pages=10,
)
dolma_json = pdf.mk_dolma_doc(custom_key="custom_value")
print(dolma_json)  # JSON string conforming to Dolma schema

```

### Standard Pipeline Output

The typical OlmOCR workflow uses `build_dolma_document` directly:

```python
from olmocr.pipeline import build_dolma_document
import json

# page_results contains OCR responses from the pipeline

dolma_doc = build_dolma_document(
    pdf_orig_path="s3://my-bucket/documents/example.pdf",
    page_results=page_results,
)

print(json.dumps(dolma_doc, indent=2))

```

Both approaches output JSON objects adhering to the Dolma document format specification, ready for ingestion by Dolma indexing and evaluation tools.

## Summary

- The Dolma document format specification defines a JSON schema with `id`, `text`, `source`, `added`, `created`, `metadata`, and `attributes` fields.
- OlmOCR generates `id` values using SHA-1 hashes of the document text for stability across processing runs.
- The `metadata` dictionary tracks provenance via `Source-File`, version info from [`olmocr/version.py`](https://github.com/allenai/olmocr/blob/main/olmocr/version.py), and processing statistics like token counts and fallback page frequencies.
- Page-level character mappings and classification flags reside in the `attributes` object, enabling precise text-to-PDF coordinate tracking for downstream analysis.
- Implementation spans [`olmocr/datatypes.py`](https://github.com/allenai/olmocr/blob/main/olmocr/datatypes.py) for legacy support and [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) (lines 602-651) for full-featured conversion with rich attributes.

## Frequently Asked Questions

### What makes a JSON document "Dolma-compatible"?

A Dolma-compatible document must include the core fields `id`, `text`, `source`, `added`, and `created`, with `metadata` containing provenance information. OlmOCR extends this baseline by adding the `attributes` dictionary for PDF-specific data like page numbers and rotation corrections, ensuring compatibility with Dolma's downstream filtering and indexing pipelines.

### How is the document ID generated in OlmOCR?

OlmOCR computes the `id` field using SHA-1 hashing of the UTF-8 encoded document text via `hashlib.sha1(document_text.encode()).hexdigest()`. This approach, implemented in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py), guarantees that identical OCR output produces identical identifiers across different processing runs.

### What processing statistics does the metadata contain?

The `metadata` object includes `total-input-tokens` and `total-output-tokens` aggregating LLM usage across all pages, `total-fallback-pages` counting simplified OCR paths taken, `pdf-total-pages` for page counts, and `olmocr-version` tracking which software version processed the file. These fields enable cost analysis and quality auditing of OCR operations.

### Can I extend the Dolma document with custom fields?

Yes. The legacy `mk_dolma_doc` method in [`olmocr/datatypes.py`](https://github.com/allenai/olmocr/blob/main/olmocr/datatypes.py) accepts arbitrary kwargs that merge into the `metadata` dictionary. While the main `build_dolma_document` function follows a stricter schema for consistency, you can modify the source in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) (lines 602-651) to inject additional custom metadata fields as needed for specialized downstream applications.