# Setting Up Huawei Ascend NPU Training Infrastructure for AReaL: A Complete Guide

> Set up Huawei Ascend NPU training infrastructure for AReaL easily. Learn how to leverage Docker, CANN, vLLM-Ascend, and SLURM for GPU-like deep learning workflows.

- Repository: [inclusionAI/areal](https://github.com/inclusionai/areal)
- Tags: how-to-guide
- Published: 2026-03-04

---

**AReaL supports Huawei Ascend NPUs through Docker-based containerization, pre-built images with CANN and vLLM-Ascend, and SLURM scheduler integration that maps `CUDA_VISIBLE_DEVICES` to `ASCEND_RT_VISIBLE_DEVICES` for seamless GPU-like training workflows.**

Setting up Huawei Ascend NPU training infrastructure for large language model workloads requires specific hardware provisioning, containerized runtime environments, and scheduler configurations. The AReaL (Asynchronous Reinforcement Learning) framework provides native support for Ascend NPUs through a dedicated workflow that mirrors its CUDA-based GPU handling. This guide covers the complete setup process based on the inclusionai/areal source code, from hardware prerequisites to multi-node distributed training.

## Hardware and System Prerequisites

Before deploying AReaL on Ascend NPUs, ensure your cluster nodes meet the minimum specifications outlined in [`docs/tutorial/installation_npu.md`](https://github.com/inclusionai/areal/blob/main/docs/tutorial/installation_npu.md) (lines 5-14). Each training node must expose **16 Ascend NPUs**, **64 CPU cores**, and **≥1 TB of RAM**. Additionally, high-speed RoCE (RDMA over Converged Ethernet) networking is required for both single-node and multi-node experiments to handle the communication overhead of asynchronous reinforcement learning. These specifications apply uniformly whether you are running a standalone experiment or scaling across a distributed cluster.

## Pre-built Docker Images for Ascend NPUs

AReaL provides specialized Docker images that embed the complete Ascend software stack, eliminating manual CANN installation. According to [`docs/tutorial/installation_npu.md`](https://github.com/inclusionai/areal/blob/main/docs/tutorial/installation_npu.md) (lines 24-28), the images are hosted on Huawei's Swan repository and tagged by chip generation:

- `swr.cn-north-9.myhuaweicloud.com/areal/areal_npu:v0.5.0-a2` for Ascend A2 chips
- `swr.cn-north-9.myhuaweicloud.com/areal/areal_npu:v0.5.0-a3` for Ascend A3 chips

These containers include the **Ascend CANN stack**, the **vLLM-Ascend plugin**, and all Python dependencies required for inference and training. The Docker entrypoint mounts host drivers and device nodes to enable direct NPU control from within the containerized environment.

### Mounting NPU Devices and Drivers

To launch the container with full hardware access as specified in [`docs/tutorial/installation_npu.md`](https://github.com/inclusionai/areal/blob/main/docs/tutorial/installation_npu.md) (lines 55-86), execute the following command, which mounts all 16 NPU device nodes (`/dev/davinci0` through `/dev/davinci15`), the driver manager, and system libraries:

```bash

# Select the image matching your hardware (A2 or A3)

IMAGE=swr.cn-north-9.myhuaweicloud.com/areal/areal_npu:v0.5.0-a3
WORK_DIR=$HOME/areal_workspace

docker pull $IMAGE
docker run -itd --cap-add=SYS_PTRACE --net=host \
  --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
  --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
  --device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \
  --device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \
  --device=/dev/davinci_manager --device=/dev/devmm_svm --device=/dev/hisi_hdc \
  --shm-size=1200g \
  -v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
  -v /usr/local/dcmi:/usr/local/dcmi \
  -v /etc/ascend_install.info:/etc/ascend_install.info \
  -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
  -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
  -v /var/log/npu/:/usr/slog \
  -v ${WORK_DIR}:/workspace \
  --privileged=true \
  --name areal_npu $IMAGE /bin/bash

```

This configuration mounts the host's `npu-smi` utility, DCMI interfaces, and Ascend driver directories into the container, enabling full NPU visibility and management.

## Installing AReaL with NPU Dependencies

Once inside the running container, install AReaL from the `ascend` branch to obtain NPU-specific code paths. The installation uses `uv` to resolve dependencies including Ascend-specific wheels:

```bash
git clone https://github.com/inclusionAI/AReaL
cd AReaL
git checkout ascend
uv pip install -r pyproject.toml --extra all_npu

```

The `--extra all_npu` flag pulls the necessary Ascend-specific dependencies, including optimized kernels and the vLLM-Ascend backend, ensuring compatibility with the hardware mounted in the previous step.

## Launching Single-Node Training Jobs

With the environment configured, you can run training scripts unchanged from their GPU equivalents. For Vision-Language Model (VLM) training, AReaL provides ready-to-use shell scripts in the `examples/vlm_npu/` directory. Execute the single-node Geometry3K GRPO training script:

```bash

# Inside the container at /workspace/AReaL

bash examples/vlm_npu/qwen2_5_vl_3b_geometry3k_grpo.sh

```

This script configures the **Qwen2.5-VL-3B** model for Reinforcement Learning with Group Relative Policy Optimization (GRPO) on a single node with 16 NPUs. For mathematical reasoning tasks, reference the configuration in [`examples/math/gsm8k_grpo_npu.yaml`](https://github.com/inclusionai/areal/blob/main/examples/math/gsm8k_grpo_npu.yaml), which specifies NPU-specific resource allocations.

## Multi-Node Training: SLURM and Ray Integration

AReaL supports distributed Ascend NPU training through both SLURM and Ray cluster managers, abstracting the underlying hardware differences to provide a GPU-like experience.

### SLURM Scheduler Integration

When submitting jobs via SLURM, the framework automatically handles device visibility. As implemented in [`areal/infra/scheduler/slurm.py`](https://github.com/inclusionai/areal/blob/main/areal/infra/scheduler/slurm.py) (lines 98-101), the scheduler wrapper injects the **ASCEND_RT_VISIBLE_DEVICES** environment variable alongside standard CUDA handling:

```bash

# Automatically executed by the SLURM wrapper

export ASCEND_RT_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES

```

This ensures each SLURM task sees only the NPU devices allocated to it, mirroring the behavior of `CUDA_VISIBLE_DEVICES` for GPU jobs. Submit multi-node jobs using standard `sbatch` commands; the scheduler builds a Bash command that exports both variables before running the user script or Ray worker inside the container.

### Ray Cluster Setup for Distributed Training

For environments using Ray, start the head node on your primary server:

```bash

# On Node 0 (head)

ray start --head

```

Then connect worker nodes using the head node's IP address:

```bash

# On Worker Nodes

RAY_HEAD_IP=10.0.0.1  # Replace with actual head node IP

ray start --address $RAY_HEAD_IP

```

After cluster initialization, run any AReaL training script with the `--n_nodes=N` argument. The AReaL launcher detects Ray resources and distributes the workload automatically across the Ascend NPUs, as described in [`docs/tutorial/installation_npu.md`](https://github.com/inclusionai/areal/blob/main/docs/tutorial/installation_npu.md) (lines 6-20). The same scripts used for GPU clusters—such as [`examples/vlm_npu/qwen2_5_vl_3b_virl39k_grpo_multinode.sh`](https://github.com/inclusionai/areal/blob/main/examples/vlm_npu/qwen2_5_vl_3b_virl39k_grpo_multinode.sh)—execute unchanged on Ascend hardware.

## Summary

- **Hardware requirements**: Each node requires 16 Ascend NPUs, 64 CPU cores, ≥1 TB RAM, and RoCE networking per [`docs/tutorial/installation_npu.md`](https://github.com/inclusionai/areal/blob/main/docs/tutorial/installation_npu.md).
- **Container images**: Use pre-built images from `swr.cn-north-9.myhuaweicloud.com/areal/areal_npu` (v0.5.0-a2 or v0.5.0-a3) with embedded CANN and vLLM-Ascend stacks.
- **Device access**: Mount all 16 `/dev/davinci*` devices, driver directories, and system libraries when starting containers.
- **Installation**: Checkout the `ascend` branch and install with `uv pip install --extra all_npu`.
- **Scheduler integration**: The SLURM wrapper in [`areal/infra/scheduler/slurm.py`](https://github.com/inclusionai/areal/blob/main/areal/infra/scheduler/slurm.py) automatically sets `ASCEND_RT_VISIBLE_DEVICES` based on `CUDA_VISIBLE_DEVICES`.
- **Execution**: Run single-node scripts like [`examples/vlm_npu/qwen2_5_vl_3b_geometry3k_grpo.sh`](https://github.com/inclusionai/areal/blob/main/examples/vlm_npu/qwen2_5_vl_3b_geometry3k_grpo.sh) or scale via Ray/SLURM using standard AReaL commands.

## Frequently Asked Questions

### What hardware specifications are required for Ascend NPU training with AReaL?

Each cluster node must provide 16 Ascend NPUs, 64 CPU cores, and at least 1 TB of system memory, along with high-speed RoCE networking. These requirements, documented in [`docs/tutorial/installation_npu.md`](https://github.com/inclusionai/areal/blob/main/docs/tutorial/installation_npu.md), apply to both single-node and distributed multi-node configurations to ensure sufficient compute and bandwidth for asynchronous RL workloads.

### How do AReaL's Ascend Docker images differ from standard GPU containers?

Unlike standard CUDA-based images, AReaL's Ascend images (hosted on Huawei's Swan repository) bundle the **Ascend CANN stack** and **vLLM-Ascend plugin** specifically for Huawei silicon. The images tagged `v0.5.0-a2` and `v0.5.0-a3` target Ascend A2 and A3 chips respectively, and require mounting specific host device nodes (`/dev/davinci*`) and driver directories that differ from NVIDIA GPU device bindings.

### Why does AReaL use both `CUDA_VISIBLE_DEVICES` and `ASCEND_RT_VISIBLE_DEVICES`?

The AReaL SLURM scheduler wrapper maintains compatibility with existing GPU-centric infrastructure by reading `CUDA_VISIBLE_DEVICES` and exporting it as `ASCEND_RT_VISIBLE_DEVICES` for Ascend hardware. As shown in [`areal/infra/scheduler/slurm.py`](https://github.com/inclusionai/areal/blob/main/areal/infra/scheduler/slurm.py) (lines 98-101), this mapping allows the same job submission scripts and resource allocation logic to function transparently across both NVIDIA and Huawei hardware without code changes.

### Can existing GPU training scripts run unchanged on Ascend NPUs?

Yes. AReaL's architecture abstracts hardware-specific implementations, allowing scripts like [`examples/math/gsm8k_rl.py`](https://github.com/inclusionai/areal/blob/main/examples/math/gsm8k_rl.py) and `examples/vlm_npu/*.sh` to execute on Ascend NPUs without modification. The framework detects the underlying runtime (CUDA vs. CANN) and adjusts backend operations automatically, provided the environment is properly configured with the Ascend Docker images and the `ASCEND_RT_VISIBLE_DEVICES` variable is set.