# Configuring Vision-Language Model Training with Qwen2.5-VL in AReaL

> Learn to configure Qwen2.5-VL training in AReaL using a modular YAML file. Integrate SGLang vLLM backends, tensor-parallel sharding & GRPO optimization.

- Repository: [inclusionAI/areal](https://github.com/inclusionai/areal)
- Tags: tutorial
- Published: 2026-03-04

---

**AReaL configures Qwen2.5-VL training through a modular YAML file that binds together SGLang or vLLM backends, tensor-parallel sharding, and GRPO optimization across distributed RPC services.**

AReaL (Asynchronous Reinforcement Learning) is a modular open-source framework that separates data ingestion, rollout generation, and policy optimization into independent RPC services. When configuring vision-language model training with Qwen2.5-VL, the system leverages specialized model adapters and asynchronous allocation strategies to handle multimodal inputs at scale.

## Model Architecture and Adapter Implementation

AReaL provides a generic transformers-based VLM interface specifically designed for Qwen2.5-VL architectures within the Archon model library.

### Qwen2Model and ModelArgs Configuration

The core model definition resides in `areal/experimental/models/archon/qwen2/`. The `Qwen2ModelArgs` dataclass in [`model/args.py`](https://github.com/inclusionai/areal/blob/main/model/args.py) specifies hyper-parameters including hidden size, token limits, and multimodal attention configurations. The `Qwen2Model` class in [`model/model.py`](https://github.com/inclusionai/areal/blob/main/model/model.py) constructs the transformer backbone and explicitly registers the **visual encoder** and **multimodal attention** layers required for processing image-text inputs.

### Checkpoint Conversion

The `Qwen2StateDictAdapter` located in [`model/state_dict_adapter.py`](https://github.com/inclusionai/areal/blob/main/model/state_dict_adapter.py) handles conversion between Hugging Face checkpoint formats and AReaL's internal Archon format. This adapter ensures that pretrained Qwen2.5-VL weights load correctly into the distributed training infrastructure without manual tensor reshaping.

## Distributed Parallelization Strategies

The parallelization engine in [`areal/experimental/models/archon/qwen2/infra/parallelize.py`](https://github.com/inclusionai/areal/blob/main/areal/experimental/models/archon/qwen2/infra/parallelize.py) applies **Tensor Parallelism**, **FSDP2**, and **Context-Parallel** sharding strategies automatically. Notably, because the Qwen2 architecture does not contain Q/K normalization layers, the paralleliser skips those specific transformation steps at line 171, optimizing memory usage for vision-language workloads.

## Backend Configuration: SGLang vs vLLM

AReaL supports two inference backends for Qwen2.5-VL rollouts, configured via mutually exclusive YAML sections.

### SGLang Multimodal Setup

**SGLang** provides batched multimodal token generation with static memory allocation. Configure this backend using the `sglang:` section in your YAML, specifying `max_running_requests` and `context_length` parameters. The system launches an SGLang RPC server on each GPU as defined in `actor.scheduling_spec`, enabling high-throughput image-text prompt processing.

### vLLM Configuration

Alternatively, **vLLM** can serve the model by setting `enable_multimodal: true` in the `vllm:` configuration block. While vLLM offers dynamic batching, it requires explicit multimodal flag activation to process visual inputs correctly during rollout generation.

## YAML Configuration Breakdown

The complete training pipeline is defined in [`examples/vlm_npu/qwen2_5_vl_3b_geometry3k_grpo.yaml`](https://github.com/inclusionai/areal/blob/main/examples/vlm_npu/qwen2_5_vl_3b_geometry3k_grpo.yaml), which orchestrates cluster topology, model parameters, and optimization settings.

### Cluster Topology and Allocation

The `cluster` section (lines 10-12) defines `n_nodes` and `n_gpus_per_node` for distributed execution. The `allocation_mode` field specifies asynchronous allocation strategies such as `vllm:d4p1t1+d4p1t1`, determining how rollout workers and trainers map to available hardware.

### Actor and Optimization Settings

The `actor` block (lines 42-69) configures the Qwen2.5-VL 3B checkpoint path (`Qwen/Qwen2.5-VL-3B-Instruct`), dtype precision (**bfloat16**), and gradient checkpointing. For vision-language models with large hidden states, enabling `gradient_checkpointing: true` is essential to maintain GPU memory stability during the **GRPO** policy optimization loop.

### Dataset and Evaluation Configuration

The `train_dataset` and `valid_dataset` fields point to multimodal datasets such as `hiyouga/geometry3k`. The `evaluator` section configures held-out evaluation cycles after each epoch, with results logged via the `stats_logger` (WandB disabled by default in the example configuration).

## Training Workflow Execution

The AReaL training loop separates rollout generation from policy optimization via lightweight RPC communication.

### Rollout Generation via RPC

AReaL launches an **SGLang RPC server** on each GPU through `actor.scheduling_spec` (lines 80-84). These servers receive image-text prompts, execute the multimodal encoder, and stream generated tokens back to the trainer. The rollout worker respects `max_concurrent_rollouts` (line 25) to prevent memory overflow during batch processing.

### GRPO Training Loop

The trainer consumes rollout trajectories through the `WeightUpdateMeta` protocol, computing advantage estimates and performing **asynchronous weight updates**. The configuration uses `actor.eps_clip` and `ppo_n_minibatches` to control the GRPO optimizer behavior. This asynchronous approach allows the visual encoder and language model to update weights without blocking rollout generation.

### Evaluation and Logging

After each epoch, the `evaluator` runs the model on held-out Geometry-3K data. The `saver` and `recover` sections manage checkpoint persistence and fault tolerance, ensuring that multimodal training can resume from interruptions without losing visual encoder states.

## Deployment Examples

### Single Node Training

Execute the Qwen2.5-VL experiment on a single node with 8 GPUs:

```bash
python -m areal.infra.rpc.rpc_server &
python -m areal.main \
  --config examples/vlm_npu/qwen2_5_vl_3b_geometry3k_grpo.yaml \
  scheduler.type=local

```

### Ray Cluster Scaling

For distributed training across 2 nodes with 8 GPUs each:

```bash
python -m areal.main \
  --config examples/vlm_npu/qwen2_5_vl_3b_geometry3k_grpo.yaml \
  cluster.n_nodes=2 cluster.n_gpus_per_node=8 \
  scheduler.type=ray

```

### Programmatic Model Access

Load the model directly for custom preprocessing or debugging:

```python
from areal.experimental.models.archon.qwen2.model import Qwen2Model, Qwen2ModelArgs

args = Qwen2ModelArgs.from_pretrained(
    "Qwen/Qwen2.5-VL-3B-Instruct",
    dtype="bfloat16",
    max_tokens_per_mb=4096,
)
model = Qwen2Model(args)
model.to("cuda")

```

### SGLang Client Integration

Send multimodal prompts to a running RPC server:

```python
import asyncio
from sglang import SGLangClient

async def generate():
    client = SGLangClient("http://localhost:21001")
    resp = await client.generate(
        prompt={"image_path": "sample.jpg", "text": "Describe the scene."},
        max_new_tokens=128,
        temperature=1.0,
    )
    print(resp["text"])

asyncio.run(generate())

```

## Summary

- **AReaL** separates VLM training into modular RPC services for data ingestion, rollout generation, and GRPO optimization.
- The **Qwen2.5-VL** adapter in `areal/experimental/models/archon/qwen2/` handles model construction, checkpoint conversion, and parallelization without Q/K normalization layers.
- Configure training via the **YAML file** at [`examples/vlm_npu/qwen2_5_vl_3b_geometry3k_grpo.yaml`](https://github.com/inclusionai/areal/blob/main/examples/vlm_npu/qwen2_5_vl_3b_geometry3k_grpo.yaml), selecting either **SGLang** or **vLLM** backends with multimodal flags enabled.
- Enable **gradient checkpointing** and **bfloat16** precision to manage memory when processing high-resolution visual inputs.
- Launch distributed training using **Ray clusters** or local schedulers while maintaining asynchronous weight updates through the `WeightUpdateMeta` protocol.

## Frequently Asked Questions

### What is the difference between SGLang and vLLM backends in AReaL for Qwen2.5-VL?

**SGLang** offers static memory allocation and optimized batched multimodal generation, making it the preferred choice for high-throughput vision-language rollouts. **vLLM** provides dynamic batching but requires explicitly setting `enable_multimodal: true` to process image inputs correctly. Both backends communicate with the trainer via RPC, but SGLang typically delivers better latency consistency for visual encoders according to the AReaL source code.

### How does AReaL handle checkpoint conversion for Qwen2.5-VL models?

The `Qwen2StateDictAdapter` in [`areal/experimental/models/archon/qwen2/model/state_dict_adapter.py`](https://github.com/inclusionai/areal/blob/main/areal/experimental/models/archon/qwen2/model/state_dict_adapter.py) automatically converts Hugging Face checkpoint formats into Archon's internal tensor layout. This adapter preserves visual encoder weights and multimodal attention parameters during the conversion process, eliminating manual tensor reshaping when loading `Qwen/Qwen2.5-VL-3B-Instruct` or similar checkpoints.

### Why does the parallelization code skip Q/K normalization for Qwen2 architectures?

The paralleliser in [`areal/experimental/models/archon/qwen2/infra/parallelize.py`](https://github.com/inclusionai/areal/blob/main/areal/experimental/models/archon/qwen2/infra/parallelize.py) skips Q/K normalization layers at line 171 because the Qwen2 architecture does not implement these specific normalization steps. This optimization reduces unnecessary computation during tensor-parallel and FSDP2 sharding, lowering memory overhead when distributing vision-language models across multiple GPUs.

### Can I train Qwen2.5-VL on custom datasets other than Geometry-3K?

Yes. Modify the `train_dataset` and `valid_dataset` fields in the YAML configuration to point to any Hugging Face dataset containing image-text pairs. Ensure the dataset follows the multimodal format expected by the **SGLang** or **vLLM** backend, with image paths and text prompts properly structured. The `actor` configuration remains compatible with any vision-language dataset without requiring changes to the core model adapter code.