What Models Are Supported for SFT Training in olmOCR?

OlmOCR supports three Qwen vision-language model families for SFT training—Qwen3-VL, Qwen2.5-VL, and Qwen2-VL—automatically selecting the appropriate HuggingFace *ForConditionalGeneration class based on the model identifier string in your configuration.

The allenai/olmocr repository provides a complete pipeline for supervised fine-tuning (SFT) of vision-language models on OCR tasks. The training system inspects the name field defined in the ModelConfig dataclass to instantiate the correct architecture, allowing seamless switching between compatible Qwen VL models without modifying the core training logic.

Supported Vision-Language Model Families

The SFT training pipeline recognizes specific identifier patterns that map to distinct HuggingFace model classes. Each family offers different capabilities for document understanding and text extraction tasks.

Qwen3-VL

Models containing qwen3-vl in their identifier instantiate Qwen3VLForConditionalGeneration. This represents the latest generation of Qwen's vision-language architecture with enhanced capabilities.

model:
  name: "Qwen/Qwen3-VL-7B-Instruct"

Qwen2.5-VL (Including OlmOCR Checkpoints)

The qwen2.5-vl substring or the specific olmocr-2-7b-1025 identifier triggers Qwen2_5_VLForConditionalGeneration. This is the default model family used by the repository, with Qwen/Qwen2.5-VL-7B-Instruct serving as the standard checkpoint defined in olmocr/train/config.py at lines 177-184.

This family includes the official OlmOCR pretrained weights when specified by the olmocr-2-7b-1025 identifier.

Qwen2-VL

Models matching qwen2-vl utilize Qwen2VLForConditionalGeneration. This architecture remains fully supported for backwards compatibility and specific deployment scenarios requiring this earlier generation.

model:
  name: "Qwen/Qwen2-VL-7B-Instruct"

How Model Selection Works

The dispatch logic resides in olmocr/train/train.py at lines 330-343. The training script performs lowercase string matching on the model identifier to determine the correct class:

if "qwen3-vl" in config.model.name.lower():
    model_class = Qwen3VLForConditionalGeneration
elif "qwen2.5-vl" in config.model.name.lower() or "olmocr-2-7b-1025" in config.model.name.lower():
    model_class = Qwen2_5_VLForConditionalGeneration
elif "qwen2-vl" in config.model.name.lower():
    model_class = Qwen2VLForConditionalGeneration

This conditional block ensures that any HuggingFace model identifier containing these substrings loads with the architecture-specific *ForConditionalGeneration class required for vision-language tasks.

Configuring Models for SFT Training

You customize the training model by setting the name field within the ModelConfig section. The rest of the pipeline—including data loading, LoRA wrapping in olmocr/train/train.py, and checkpoint management in olmocr/train/prepare_checkpoint.py—adapts automatically to the selected architecture.

YAML Configuration Example

Configure a Qwen2.5-VL model with Flash Attention 2 using the following structure:

model:
  name: "Qwen/Qwen2.5-VL-7B-Instruct"
  trust_remote_code: false
  load_in_8bit: false
  load_in_4bit: false
  device_map: auto
  torch_dtype: auto
  use_flash_attention: true
  attn_implementation: flash_attention_2
  use_lora: false

Python Implementation

While the training script handles instantiation automatically, you can manually load any supported model using the corresponding Transformers class:

from transformers import Qwen2VLForConditionalGeneration
import torch

# Example: Loading Qwen2-VL manually

model = Qwen2VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2-VL-7B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=False,
    attn_implementation="flash_attention_2",
)

Key Files in the SFT Pipeline

Understanding these source files helps when extending the training system:

• olmocr/train/config.py — Defines the ModelConfig dataclass (lines 177-184) where the model identifier and loading parameters are specified.
• olmocr/train/train.py — Core training script implementing the model dispatch logic (lines 330-343) that selects the appropriate *ForConditionalGeneration class.
• olmocr/train/dataloader.py — Provides the data collator compatible with all supported vision-language models.
• olmocr/train/prepare_checkpoint.py — Handles checkpoint serialization and loading for the selected model architecture.

Summary

• OlmOCR supports three model families for SFT training: Qwen3-VL, Qwen2.5-VL (including the olmocr-2-7b-1025 checkpoint), and Qwen2-VL.
• Model selection occurs automatically based on string matching in olmocr/train/train.py, dispatching to the correct HuggingFace *ForConditionalGeneration class.
• The default model is Qwen/Qwen2.5-VL-7B-Instruct, configured in olmocr/train/config.py.
• Compatible identifiers follow HuggingFace naming conventions and must contain the specific substrings (qwen3-vl, qwen2.5-vl, qwen2-vl, or olmocr-2-7b-1025) for the dispatch logic to recognize them.

Frequently Asked Questions

Can I use custom or fine-tuned Qwen models with olmOCR's SFT pipeline?

Yes. Any model identifier containing qwen3-vl, qwen2.5-vl, qwen2-vl, or olmocr-2-7b-1025 will trigger the appropriate model class instantiation. Upload your custom checkpoint to HuggingFace and reference it in the model.name configuration field to use it for SFT training.

Does the SFT training support LoRA for parameter-efficient fine-tuning?

Yes. The training pipeline supports LoRA (Low-Rank Adaptation) regardless of which Qwen VL model you select. Set use_lora: true in your ModelConfig to wrap the selected vision-language model with LoRA adapters before training begins.

What happens if I specify a model that doesn't match the supported identifiers?

The training script will fail to identify a model class and will not proceed. The dispatch logic in olmocr/train/train.py explicitly checks for the supported identifier fragments; any model name failing these checks will not instantiate a valid *ForConditionalGeneration class, preventing the SFT process from starting.

Can I change the model architecture after starting a training run?

No, you must modify the configuration before launching the script. The model class is instantiated once at the beginning of training in olmocr/train/train.py. While you can resume training from checkpoints using the same architecture, switching to a different model family (e.g., from Qwen2-VL to Qwen3-VL) requires starting a new training session with the updated configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}