Configuring Vision-Language Model Training with Qwen2.5-VL in AReaL

AReaL configures Qwen2.5-VL training through a modular YAML file that binds together SGLang or vLLM backends, tensor-parallel sharding, and GRPO optimization across distributed RPC services.

AReaL (Asynchronous Reinforcement Learning) is a modular open-source framework that separates data ingestion, rollout generation, and policy optimization into independent RPC services. When configuring vision-language model training with Qwen2.5-VL, the system leverages specialized model adapters and asynchronous allocation strategies to handle multimodal inputs at scale.

Model Architecture and Adapter Implementation

AReaL provides a generic transformers-based VLM interface specifically designed for Qwen2.5-VL architectures within the Archon model library.

Qwen2Model and ModelArgs Configuration

The core model definition resides in areal/experimental/models/archon/qwen2/. The Qwen2ModelArgs dataclass in model/args.py specifies hyper-parameters including hidden size, token limits, and multimodal attention configurations. The Qwen2Model class in model/model.py constructs the transformer backbone and explicitly registers the visual encoder and multimodal attention layers required for processing image-text inputs.

Checkpoint Conversion

The Qwen2StateDictAdapter located in model/state_dict_adapter.py handles conversion between Hugging Face checkpoint formats and AReaL's internal Archon format. This adapter ensures that pretrained Qwen2.5-VL weights load correctly into the distributed training infrastructure without manual tensor reshaping.

Distributed Parallelization Strategies

The parallelization engine in areal/experimental/models/archon/qwen2/infra/parallelize.py applies Tensor Parallelism, FSDP2, and Context-Parallel sharding strategies automatically. Notably, because the Qwen2 architecture does not contain Q/K normalization layers, the paralleliser skips those specific transformation steps at line 171, optimizing memory usage for vision-language workloads.

Backend Configuration: SGLang vs vLLM

AReaL supports two inference backends for Qwen2.5-VL rollouts, configured via mutually exclusive YAML sections.

SGLang Multimodal Setup

SGLang provides batched multimodal token generation with static memory allocation. Configure this backend using the sglang: section in your YAML, specifying max_running_requests and context_length parameters. The system launches an SGLang RPC server on each GPU as defined in actor.scheduling_spec, enabling high-throughput image-text prompt processing.

vLLM Configuration

Alternatively, vLLM can serve the model by setting enable_multimodal: true in the vllm: configuration block. While vLLM offers dynamic batching, it requires explicit multimodal flag activation to process visual inputs correctly during rollout generation.

YAML Configuration Breakdown

The complete training pipeline is defined in examples/vlm_npu/qwen2_5_vl_3b_geometry3k_grpo.yaml, which orchestrates cluster topology, model parameters, and optimization settings.

Cluster Topology and Allocation

The cluster section (lines 10-12) defines n_nodes and n_gpus_per_node for distributed execution. The allocation_mode field specifies asynchronous allocation strategies such as vllm:d4p1t1+d4p1t1, determining how rollout workers and trainers map to available hardware.

Actor and Optimization Settings

The actor block (lines 42-69) configures the Qwen2.5-VL 3B checkpoint path (Qwen/Qwen2.5-VL-3B-Instruct), dtype precision (bfloat16), and gradient checkpointing. For vision-language models with large hidden states, enabling gradient_checkpointing: true is essential to maintain GPU memory stability during the GRPO policy optimization loop.

Dataset and Evaluation Configuration

The train_dataset and valid_dataset fields point to multimodal datasets such as hiyouga/geometry3k. The evaluator section configures held-out evaluation cycles after each epoch, with results logged via the stats_logger (WandB disabled by default in the example configuration).

Training Workflow Execution

The AReaL training loop separates rollout generation from policy optimization via lightweight RPC communication.

Rollout Generation via RPC

AReaL launches an SGLang RPC server on each GPU through actor.scheduling_spec (lines 80-84). These servers receive image-text prompts, execute the multimodal encoder, and stream generated tokens back to the trainer. The rollout worker respects max_concurrent_rollouts (line 25) to prevent memory overflow during batch processing.

GRPO Training Loop

The trainer consumes rollout trajectories through the WeightUpdateMeta protocol, computing advantage estimates and performing asynchronous weight updates. The configuration uses actor.eps_clip and ppo_n_minibatches to control the GRPO optimizer behavior. This asynchronous approach allows the visual encoder and language model to update weights without blocking rollout generation.

Evaluation and Logging

After each epoch, the evaluator runs the model on held-out Geometry-3K data. The saver and recover sections manage checkpoint persistence and fault tolerance, ensuring that multimodal training can resume from interruptions without losing visual encoder states.

Deployment Examples

Single Node Training

Execute the Qwen2.5-VL experiment on a single node with 8 GPUs:

python -m areal.infra.rpc.rpc_server &
python -m areal.main \
  --config examples/vlm_npu/qwen2_5_vl_3b_geometry3k_grpo.yaml \
  scheduler.type=local

Ray Cluster Scaling

For distributed training across 2 nodes with 8 GPUs each:

python -m areal.main \
  --config examples/vlm_npu/qwen2_5_vl_3b_geometry3k_grpo.yaml \
  cluster.n_nodes=2 cluster.n_gpus_per_node=8 \
  scheduler.type=ray

Programmatic Model Access

Load the model directly for custom preprocessing or debugging:

from areal.experimental.models.archon.qwen2.model import Qwen2Model, Qwen2ModelArgs

args = Qwen2ModelArgs.from_pretrained(
    "Qwen/Qwen2.5-VL-3B-Instruct",
    dtype="bfloat16",
    max_tokens_per_mb=4096,
)
model = Qwen2Model(args)
model.to("cuda")

SGLang Client Integration

Send multimodal prompts to a running RPC server:

import asyncio
from sglang import SGLangClient

async def generate():
    client = SGLangClient("http://localhost:21001")
    resp = await client.generate(
        prompt={"image_path": "sample.jpg", "text": "Describe the scene."},
        max_new_tokens=128,
        temperature=1.0,
    )
    print(resp["text"])

asyncio.run(generate())

Summary

• AReaL separates VLM training into modular RPC services for data ingestion, rollout generation, and GRPO optimization.
• The Qwen2.5-VL adapter in areal/experimental/models/archon/qwen2/ handles model construction, checkpoint conversion, and parallelization without Q/K normalization layers.
• Configure training via the YAML file at examples/vlm_npu/qwen2_5_vl_3b_geometry3k_grpo.yaml, selecting either SGLang or vLLM backends with multimodal flags enabled.
• Enable gradient checkpointing and bfloat16 precision to manage memory when processing high-resolution visual inputs.
• Launch distributed training using Ray clusters or local schedulers while maintaining asynchronous weight updates through the WeightUpdateMeta protocol.

Frequently Asked Questions

What is the difference between SGLang and vLLM backends in AReaL for Qwen2.5-VL?

SGLang offers static memory allocation and optimized batched multimodal generation, making it the preferred choice for high-throughput vision-language rollouts. vLLM provides dynamic batching but requires explicitly setting enable_multimodal: true to process image inputs correctly. Both backends communicate with the trainer via RPC, but SGLang typically delivers better latency consistency for visual encoders according to the AReaL source code.

How does AReaL handle checkpoint conversion for Qwen2.5-VL models?

The Qwen2StateDictAdapter in areal/experimental/models/archon/qwen2/model/state_dict_adapter.py automatically converts Hugging Face checkpoint formats into Archon's internal tensor layout. This adapter preserves visual encoder weights and multimodal attention parameters during the conversion process, eliminating manual tensor reshaping when loading Qwen/Qwen2.5-VL-3B-Instruct or similar checkpoints.

Why does the parallelization code skip Q/K normalization for Qwen2 architectures?

The paralleliser in areal/experimental/models/archon/qwen2/infra/parallelize.py skips Q/K normalization layers at line 171 because the Qwen2 architecture does not implement these specific normalization steps. This optimization reduces unnecessary computation during tensor-parallel and FSDP2 sharding, lowering memory overhead when distributing vision-language models across multiple GPUs.

Can I train Qwen2.5-VL on custom datasets other than Geometry-3K?

Yes. Modify the train_dataset and valid_dataset fields in the YAML configuration to point to any Hugging Face dataset containing image-text pairs. Ensure the dataset follows the multimodal format expected by the SGLang or vLLM backend, with image paths and text prompts properly structured. The actor configuration remains compatible with any vision-language dataset without requiring changes to the core model adapter code.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}