How to Fine-Tune olmOCR Models with SFT: A Complete Guide

Fine-tuning olmOCR models with supervised fine-tuning (SFT) requires preparing paired markdown and single-page PDF documents, configuring a YAML training file through the Config class, and executing the training loop from olmocr/train/train.py with optional LoRA adapters for parameter-efficient training.

The olmOCR library from the Allen Institute for AI provides a production-ready framework to fine-tune vision-language models on OCR tasks using supervised fine-tuning (SFT). This guide covers the complete workflow—from dataset preparation to checkpoint management—based on the actual implementation in the allenai/olmocr repository.

Architecture of the olmOCR SFT Pipeline

The SFT system is built around three core components that handle configuration, data processing, and training orchestration.

Configuration Management (olmocr/train/config.py)

The Config class in olmocr/train/config.py serves as the entry point for all training runs. It parses YAML configuration files into typed dataclasses (ModelConfig, DatasetConfig, TrainingConfig) using Config.from_yaml, resolving OmegaConf interpolations and validating file paths. The configuration defines model parameters (including use_lora for adapter training), dataset directories, and hyperparameters like learning_rate and gradient_accumulation_steps.

Data Loading and Pipeline Processing (olmocr/train/dataloader.py)

The BaseMarkdownPDFDataset class recursively scans the root_dir for .md files, validating that each markdown file has a corresponding single-page PDF. During __getitem__, samples pass through a configurable pipeline of dataclass steps:

• FrontMatterParser – Extracts metadata into a PageResponse object.
• PDFRenderer – Calls render_pdf_to_base64png from olmocr/data/renderpdf.py to rasterize pages into PIL images.
• StaticLengthDocumentAnchoring – Uses get_anchor_text from olmocr/prompts/anchor.py to extract fixed-length anchor text for document context.
• FinetuningPrompt – Invokes build_finetuning_prompt from olmocr/prompts/prompts.py to construct the instruction template combining anchor text and markdown formatting requests.
• InstructUserMessages – Arranges images and text into chat-message dictionaries compatible with Qwen-VL chat templates.
• Tokenizer – Generates input_ids, attention_mask, labels, and pixel_values for the vision-language model.

Model Preparation and Training Loop (olmocr/train/train.py)

The main function in olmocr/train/train.py loads the appropriate model class—Qwen2VLForConditionalGeneration, Qwen2_5_VLForConditionalGeneration, or Qwen3VLForConditionalGeneration—based on the config. If use_lora is enabled, prepare_lora_model wraps the base model with PEFT LoRA adapters. The training loop supports mixed-precision training via torch.amp.autocast, gradient checkpointing, and automatic checkpoint resumption through load_checkpoint and save_checkpoint utilities.

Dataset Preparation for SFT

To fine-tune olmOCR models with SFT, organize your data as paired files: a single-page PDF and a markdown file containing the ground-truth OCR text. Place these pairs in a directory structure accessible to the training script. The BaseMarkdownPDFDataset validates that each PDF contains exactly one page during initialization, running parallel validation via ProcessPoolExecutor to ensure corpus integrity.

Step-by-Step Guide to Fine-Tune olmOCR with SFT

Step 1: Structure Your Dataset

Create a directory containing matched pairs:

/data/olmocr/train/
  ├── document1.md
  ├── document1.pdf
  ├── document2.md
  └── document2.pdf

Ensure each markdown file contains the clean text output corresponding to its PDF page.

Step 2: Create the Training Configuration

Use the Config API to generate a YAML file:

python -c "from olmocr.train.config import create_default_config; \
c = create_default_config(); \
c.model.name = 'Qwen/Qwen2.5-VL-7B-Instruct'; \
c.model.use_lora = True; \
c.dataset.train = [{'root_dir': '/data/olmocr/train', 'pipeline': []}]; \
c.dataset.eval = [{'root_dir': '/data/olmocr/eval', 'pipeline': []}]; \
c.training.num_train_epochs = 3; \
c.training.per_device_train_batch_size = 2; \
c.training.gradient_accumulation_steps = 4; \
c.training.learning_rate = 2e-5; \
c.training.save_steps = 500; \
c.run_name = 'my-olmocr-sft-run'; \
c.to_yaml('configs/my_sft.yaml')"

This configuration enables LoRA for parameter-efficient fine-tuning, sets a batch size of 2 with gradient accumulation of 4 steps for effective batch size of 8, and saves checkpoints every 500 steps.

Step 3: Launch the Training Run

Execute the training loop:

python -m olmocr.train.train --config configs/my_sft.yaml

The script automatically detects the latest checkpoint in output_dir/<run_name>/checkpoint-<step> if resuming, loads the optimizer and scheduler states via load_checkpoint, and begins training. Enable WandB logging by setting the WANDB_PROJECT environment variable or configuring project_name in the YAML.

Step 4: Monitor and Resume Training

The training loop logs training loss and learning rate at each step, evaluates on validation sets every eval_steps via evaluate_model, and saves checkpoints respecting save_total_limit to automatically remove older checkpoints. To resume from an interruption, simply rerun the same command—the load_checkpoint logic restores global_step, samples_seen, and best_metric from the most recent checkpoint directory.

Advanced Training Features

• LoRA Integration: Set use_lora: true in the model configuration to inject adapters via prepare_lora_model, reducing trainable parameters while maintaining performance.
• Torch Compile: Enable torch_compile in the config to use the inductor backend for additional speedups.
• Custom Pipeline Steps: Modify the pipeline list in dataset configurations to customize text extraction, anchoring strategies, or prompt formatting using create_train_dataloader for per-epoch shuffling.

Summary

• Fine-tuning olmOCR models with SFT requires paired markdown and single-page PDF files processed through the BaseMarkdownPDFDataset pipeline.
• The configuration system in olmocr/train/config.py validates YAML files into typed dataclasses for model, dataset, and training parameters.
• LoRA support via prepare_lora_model enables efficient fine-tuning of Qwen-VL architectures without full parameter updates.
• Automatic checkpointing through save_checkpoint and load_checkpoint supports resumption and maintains save_total_limit retention policies.
• The training loop supports mixed-precision training, gradient accumulation, and WandB logging for production-grade OCR model development.

Frequently Asked Questions

What data format is required to fine-tune olmOCR models with SFT?

OlmOCR requires single-page PDF files paired with markdown (.md) files containing the ground-truth text. The BaseMarkdownPDFDataset validates that each PDF has exactly one page and that the markdown file exists, processing these pairs through the configurable pipeline in olmocr/train/dataloader.py.

How do I enable LoRA for parameter-efficient fine-tuning?

Set use_lora: true in the model configuration section of your YAML file. The prepare_lora_model function in olmocr/train/train.py automatically wraps the base model (e.g., Qwen2_5_VLForConditionalGeneration) with PEFT LoRA adapters. Ensure you have installed the peft library (pip install peft) before training.

Can I resume training from a checkpoint?

Yes. If checkpoints exist in output_dir/<run_name>/checkpoint-<step>, the training script automatically calls load_checkpoint to restore the model state, optimizer, scheduler, and training metrics. Simply rerun the original command to resume from the latest checkpoint; the script updates global_step and samples_seen accordingly.

Which vision-language models are supported for SFT?

As implemented in allenai/olmocr, the training script supports Qwen2VLForConditionalGeneration, Qwen2_5_VLForConditionalGeneration, and Qwen3VLForConditionalGeneration. Specify the model name in the configuration's model.name field using Hugging Face model identifiers (e.g., Qwen/Qwen2.5-VL-7B-Instruct).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}