How to Use ODS with AMD GPUs: Automated ROCm Setup and Configuration Guide

ODS automatically detects AMD hardware during installation by scanning /sys/class/drm for vendor ID 0x1002, then configures the ROCm runtime and mounts GPU device nodes into Docker containers.

The Osmantic/ODS repository streamlines GPU-accelerated inference by automatically configuring AMD ROCm support without manual Docker tuning. When you use ODS with AMD GPUs, the installer queries sysfs interfaces to classify your hardware as APU, discrete, or mixed, then applies the appropriate container overlays and environment variables.

Prerequisites for AMD GPU Support

ODS requires a ROCm-compatible kernel and specific device nodes to enable GPU acceleration.

Kernel and Driver Requirements

You need Linux kernel 6.0 or newer with the amdgpu and amdkfd modules loaded. Verify and load these modules:

sudo modprobe amdgpu
sudo modprobe amdkfd

Install the ROCm development stack:

sudo apt update && sudo apt install -y rocm-dkms rocm-dev

Device Node Verification

Before running the installer, confirm the kernel created the required device nodes:

ls -l /dev/kfd /dev/dri/renderD*

These nodes must exist on the host filesystem. If you are running inside an LXD or LXC container, the installer will trigger show_amd_gpu_device_guidance() and fall back to CPU mode until you expose these devices to the container.

How ODS Detects AMD Hardware

The detection logic resides in ods/installers/lib/detection.sh. The detect_gpu() function iterates through GPU vendors and identifies AMD hardware by reading vendor IDs from /sys/class/drm.

When the script encounters vendor ID 0x1002, it performs the following classification using ods/installers/lib/amd-topo.sh:

• APU: Unified memory architecture with shared system RAM (measured via mem_info_gtt_total)
• Discrete: Dedicated VRAM detected via mem_info_vram_total
• Mixed: Hybrid configurations containing both integrated and discrete AMD GPUs

The script sets GPU_BACKEND="amd" and GPU_MEMORY_TYPE based on these readings, enabling ROCm-specific configuration paths.

Installation and Configuration Process

To use ODS with AMD GPUs, execute the standard installer after satisfying the kernel prerequisites:

curl -fsSL https://raw.githubusercontent.com/Osmantic/ODS/main/install.sh | bash

The installer invokes ods/installers/phases/02-detection.sh, which writes the detected configuration to your .env file:

GPU_BACKEND=amd

The installer then merges the AMD-specific Docker Compose overlay. The resulting orchestration command becomes:

docker compose -f ods/docker-compose.base.yml \
               -f ods/docker-compose.amd.yml up -d

The ods/docker-compose.amd.yml overlay substitutes standard container images for ROCm-enabled variants and mounts /dev/kfd and /dev/dri/renderD* into service containers like llama-server.

Verifying AMD Backend Configuration

After installation, confirm successful GPU detection using the diagnostic utility:

ods doctor | grep GPU_BACKEND

You should see GPU_BACKEND=amd in the output, confirming that the stack uses ROCm rather than CUDA or CPU fallbacks.

Verify GPU accessibility from within running containers:

docker exec -it llama-server bash -c "lspci | grep -i amd"
docker exec -it llama-server bash -c "cat /sys/class/drm/card0/device/mem_info_vram_total"

Troubleshooting AMD GPU Detection

If hardware detection fails, the installer falls back to CPU mode and logs warnings via amd_gpu_missing_runtime_devices().

Missing Device Nodes

When /dev/kfd or /dev/dri/renderD* are inaccessible, the detection script cannot proceed. This commonly occurs in unprivileged containers or when kernel modules are not loaded. Execute modprobe amdgpu and modprobe amdkfd on the host, then restart the installer.

Hardware Classification Limitations

While ODS fully validates AMD Strix Halo APUs with unified memory, discrete RDNA GPUs may be classified as "mixed" mode. These cards function but receive limited validation compared to APU configurations. Consult ods/docs/HARDWARE-GUIDE.md for the current support matrix.

Forcing AMD Backend Mode

For testing or recovery scenarios, manually specify the backend before running detection:

export GPU_BACKEND=amd
ods doctor

This forces ods/installers/phases/02-detection.sh to apply ROCm configuration regardless of automated scan results.

Summary

• ODS detects AMD GPUs automatically in ods/installers/lib/detection.sh by scanning /sys/class/drm for vendor ID 0x1002 and reading mem_info_vram_total and mem_info_gtt_total
• The installer sets GPU_BACKEND=amd and applies docker-compose.amd.yml to enable ROCm container images and device mounts
• Prerequisites include kernel modules amdgpu and amdkfd, plus visible device nodes at /dev/kfd and /dev/dri/renderD*
• Run ods doctor to verify backend configuration and inspect detection logs
• Containerized environments may require host-level device passthrough or will fall back to CPU mode with guidance from show_amd_gpu_device_guidance()

Frequently Asked Questions

Does ODS support discrete AMD GPUs or only APUs?

ODS provides full validation for AMD Strix Halo APUs with unified memory. Discrete RDNA GPUs operate but may be classified as "mixed" mode by ods/installers/lib/detection.sh, resulting in limited validation compared to APU configurations. Check ods/docs/HARDWARE-GUIDE.md for specific model support.

Why does ODS fall back to CPU mode on my AMD system?

This occurs when the installer cannot access /dev/kfd or /dev/dri/renderD*. The amd_gpu_missing_runtime_devices() function logs warnings when running inside unprivileged LXC/LXD containers or when the amdgpu module is unloaded. Load the kernel modules on the host and ensure device nodes are visible before running the installer.

How do I switch from CUDA to AMD ROCm after installing ODS?

Run export GPU_BACKEND=amd followed by ods doctor to force redetection. The installer phase in ods/installers/phases/02-detection.sh will regenerate your .env file and apply the docker-compose.amd.yml overlay during the next stack deployment.

Are Docker flags required to expose AMD GPUs to containers?

No manual flags are necessary. When GPU_BACKEND=amd is set, the docker-compose.amd.yml overlay automatically configures device mounts for /dev/kfd and render nodes, and selects ROCm-enabled container images without additional runtime parameters.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}