# How to Fine-Tune olmOCR Models with SFT: A Complete Guide

> Learn to fine-tune olmOCR models using SFT. This guide covers data prep, YAML config setup, and training execution with optional LoRA for efficient parameter training.

- Repository: [Ai2/olmocr](https://github.com/allenai/olmocr)
- Tags: tutorial
- Published: 2026-07-02

---

**Fine-tuning olmOCR models with supervised fine-tuning (SFT) requires preparing paired markdown and single-page PDF documents, configuring a YAML training file through the `Config` class, and executing the training loop from [`olmocr/train/train.py`](https://github.com/allenai/olmocr/blob/main/olmocr/train/train.py) with optional LoRA adapters for parameter-efficient training.**

The olmOCR library from the Allen Institute for AI provides a production-ready framework to fine-tune vision-language models on OCR tasks using supervised fine-tuning (SFT). This guide covers the complete workflow—from dataset preparation to checkpoint management—based on the actual implementation in the `allenai/olmocr` repository.

## Architecture of the olmOCR SFT Pipeline

The SFT system is built around three core components that handle configuration, data processing, and training orchestration.

### Configuration Management ([`olmocr/train/config.py`](https://github.com/allenai/olmocr/blob/main/olmocr/train/config.py))

The `Config` class in [`olmocr/train/config.py`](https://github.com/allenai/olmocr/blob/main/olmocr/train/config.py) serves as the entry point for all training runs. It parses YAML configuration files into typed dataclasses (`ModelConfig`, `DatasetConfig`, `TrainingConfig`) using `Config.from_yaml`, resolving OmegaConf interpolations and validating file paths. The configuration defines model parameters (including `use_lora` for adapter training), dataset directories, and hyperparameters like `learning_rate` and `gradient_accumulation_steps`.

### Data Loading and Pipeline Processing ([`olmocr/train/dataloader.py`](https://github.com/allenai/olmocr/blob/main/olmocr/train/dataloader.py))

The `BaseMarkdownPDFDataset` class recursively scans the `root_dir` for `.md` files, validating that each markdown file has a corresponding single-page PDF. During `__getitem__`, samples pass through a configurable pipeline of dataclass steps:

- **`FrontMatterParser`** – Extracts metadata into a `PageResponse` object.
- **`PDFRenderer`** – Calls `render_pdf_to_base64png` from [`olmocr/data/renderpdf.py`](https://github.com/allenai/olmocr/blob/main/olmocr/data/renderpdf.py) to rasterize pages into PIL images.
- **`StaticLengthDocumentAnchoring`** – Uses `get_anchor_text` from [`olmocr/prompts/anchor.py`](https://github.com/allenai/olmocr/blob/main/olmocr/prompts/anchor.py) to extract fixed-length anchor text for document context.
- **`FinetuningPrompt`** – Invokes `build_finetuning_prompt` from [`olmocr/prompts/prompts.py`](https://github.com/allenai/olmocr/blob/main/olmocr/prompts/prompts.py) to construct the instruction template combining anchor text and markdown formatting requests.
- **`InstructUserMessages`** – Arranges images and text into chat-message dictionaries compatible with Qwen-VL chat templates.
- **`Tokenizer`** – Generates `input_ids`, `attention_mask`, `labels`, and `pixel_values` for the vision-language model.

### Model Preparation and Training Loop ([`olmocr/train/train.py`](https://github.com/allenai/olmocr/blob/main/olmocr/train/train.py))

The `main` function in [`olmocr/train/train.py`](https://github.com/allenai/olmocr/blob/main/olmocr/train/train.py) loads the appropriate model class—`Qwen2VLForConditionalGeneration`, `Qwen2_5_VLForConditionalGeneration`, or `Qwen3VLForConditionalGeneration`—based on the config. If `use_lora` is enabled, `prepare_lora_model` wraps the base model with PEFT LoRA adapters. The training loop supports mixed-precision training via `torch.amp.autocast`, gradient checkpointing, and automatic checkpoint resumption through `load_checkpoint` and `save_checkpoint` utilities.

## Dataset Preparation for SFT

To fine-tune olmOCR models with SFT, organize your data as paired files: a single-page PDF and a markdown file containing the ground-truth OCR text. Place these pairs in a directory structure accessible to the training script. The `BaseMarkdownPDFDataset` validates that each PDF contains exactly one page during initialization, running parallel validation via `ProcessPoolExecutor` to ensure corpus integrity.

## Step-by-Step Guide to Fine-Tune olmOCR with SFT

### Step 1: Structure Your Dataset

Create a directory containing matched pairs:

```bash
/data/olmocr/train/
  ├── document1.md
  ├── document1.pdf
  ├── document2.md
  └── document2.pdf

```

Ensure each markdown file contains the clean text output corresponding to its PDF page.

### Step 2: Create the Training Configuration

Use the `Config` API to generate a YAML file:

```bash
python -c "from olmocr.train.config import create_default_config; \
c = create_default_config(); \
c.model.name = 'Qwen/Qwen2.5-VL-7B-Instruct'; \
c.model.use_lora = True; \
c.dataset.train = [{'root_dir': '/data/olmocr/train', 'pipeline': []}]; \
c.dataset.eval = [{'root_dir': '/data/olmocr/eval', 'pipeline': []}]; \
c.training.num_train_epochs = 3; \
c.training.per_device_train_batch_size = 2; \
c.training.gradient_accumulation_steps = 4; \
c.training.learning_rate = 2e-5; \
c.training.save_steps = 500; \
c.run_name = 'my-olmocr-sft-run'; \
c.to_yaml('configs/my_sft.yaml')"

```

This configuration enables **LoRA** for parameter-efficient fine-tuning, sets a batch size of 2 with gradient accumulation of 4 steps for effective batch size of 8, and saves checkpoints every 500 steps.

### Step 3: Launch the Training Run

Execute the training loop:

```bash
python -m olmocr.train.train --config configs/my_sft.yaml

```

The script automatically detects the latest checkpoint in `output_dir/<run_name>/checkpoint-<step>` if resuming, loads the optimizer and scheduler states via `load_checkpoint`, and begins training. Enable WandB logging by setting the `WANDB_PROJECT` environment variable or configuring `project_name` in the YAML.

### Step 4: Monitor and Resume Training

The training loop logs training loss and learning rate at each step, evaluates on validation sets every `eval_steps` via `evaluate_model`, and saves checkpoints respecting `save_total_limit` to automatically remove older checkpoints. To resume from an interruption, simply rerun the same command—the `load_checkpoint` logic restores `global_step`, `samples_seen`, and `best_metric` from the most recent checkpoint directory.

## Advanced Training Features

- **LoRA Integration**: Set `use_lora: true` in the model configuration to inject adapters via `prepare_lora_model`, reducing trainable parameters while maintaining performance.
- **Torch Compile**: Enable `torch_compile` in the config to use the `inductor` backend for additional speedups.
- **Custom Pipeline Steps**: Modify the `pipeline` list in dataset configurations to customize text extraction, anchoring strategies, or prompt formatting using `create_train_dataloader` for per-epoch shuffling.

## Summary

- **Fine-tuning olmOCR models with SFT** requires paired markdown and single-page PDF files processed through the `BaseMarkdownPDFDataset` pipeline.
- The **configuration system** in [`olmocr/train/config.py`](https://github.com/allenai/olmocr/blob/main/olmocr/train/config.py) validates YAML files into typed dataclasses for model, dataset, and training parameters.
- **LoRA support** via `prepare_lora_model` enables efficient fine-tuning of Qwen-VL architectures without full parameter updates.
- **Automatic checkpointing** through `save_checkpoint` and `load_checkpoint` supports resumption and maintains `save_total_limit` retention policies.
- The training loop supports **mixed-precision training**, **gradient accumulation**, and **WandB logging** for production-grade OCR model development.

## Frequently Asked Questions

### What data format is required to fine-tune olmOCR models with SFT?

OlmOCR requires single-page PDF files paired with markdown (.md) files containing the ground-truth text. The `BaseMarkdownPDFDataset` validates that each PDF has exactly one page and that the markdown file exists, processing these pairs through the configurable pipeline in [`olmocr/train/dataloader.py`](https://github.com/allenai/olmocr/blob/main/olmocr/train/dataloader.py).

### How do I enable LoRA for parameter-efficient fine-tuning?

Set `use_lora: true` in the model configuration section of your YAML file. The `prepare_lora_model` function in [`olmocr/train/train.py`](https://github.com/allenai/olmocr/blob/main/olmocr/train/train.py) automatically wraps the base model (e.g., `Qwen2_5_VLForConditionalGeneration`) with PEFT LoRA adapters. Ensure you have installed the `peft` library (`pip install peft`) before training.

### Can I resume training from a checkpoint?

Yes. If checkpoints exist in `output_dir/<run_name>/checkpoint-<step>`, the training script automatically calls `load_checkpoint` to restore the model state, optimizer, scheduler, and training metrics. Simply rerun the original command to resume from the latest checkpoint; the script updates `global_step` and `samples_seen` accordingly.

### Which vision-language models are supported for SFT?

As implemented in `allenai/olmocr`, the training script supports `Qwen2VLForConditionalGeneration`, `Qwen2_5_VLForConditionalGeneration`, and `Qwen3VLForConditionalGeneration`. Specify the model name in the configuration's `model.name` field using Hugging Face model identifiers (e.g., `Qwen/Qwen2.5-VL-7B-Instruct`).