How to Enable the Vulkan Backend for GPU Inference in llama.cpp

Enable GPU acceleration by compiling llama.cpp with -DGGML_VULKAN=ON, installing the LunarG Vulkan SDK, and selecting your GPU at runtime with the --device Vulkan0 flag.

The llama.cpp project provides a high-performance Vulkan backend for vendor-neutral GPU inference across Linux, Windows, and macOS. This backend leverages the ggml tensor engine to offload computations to any GPU supporting Vulkan 1.2 or higher, offering an alternative to CUDA or Metal. This guide covers the complete workflow from SDK installation to runtime device selection, referencing the actual source implementation in the ggml-org/llama.cpp repository.

Install the Vulkan SDK

Before building, install the LunarG Vulkan SDK to provide headers, libraries, and the Vulkan loader required by the backend.

Linux Installation

wget -qO- https://sdk.lunarg.com/sdk/download/latest/linux/vulkan-sdk.tar.xz | tar -xJ
export VULKAN_SDK=$PWD/1.x.x.x/x86_64
export PATH=$VULKAN_SDK/bin:$PATH
export LD_LIBRARY_PATH=$VULKAN_SDK/lib:$LD_LIBRARY_PATH
vulkaninfo  # Verify installation

Windows Installation

Download and run the installer from LunarG, then add C:\VulkanSDK\<version>\Bin to your system PATH and set the VULKAN_SDK environment variable to the SDK root directory.

Build llama.cpp with Vulkan Support

Configure the build with the GGML_VULKAN CMake option defined in ggml/CMakeLists.txt at lines 218-224.

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
mkdir build && cd build
cmake .. -DGGML_VULKAN=ON -DGGML_CUDA=OFF -DGGML_METAL=OFF
make -j$(nproc)

This command enables the ggml_vulkan source files and links against the Vulkan loader (Vulkan::Vulkan) as specified in ggml-config.cmake.in at lines 74-76. Disabling other backends with -DGGML_CUDA=OFF and -DGGML_METAL=OFF ensures you test the Vulkan path explicitly, though you can enable multiple backends simultaneously.

Select the Vulkan Device at Runtime

Use the --list-devices flag to enumerate available backends, then target a specific GPU with --device.

List Available Devices

./main --list-devices

Example output:

Device 0: Vulkan0 (Vulkan 1.3, driver XYZ)
Device 1: CPU0 (ggml CPU backend)

Run Inference on Vulkan

./main -m models/7B/ggml-model-q4_0.gguf \
       -p "Explain quantum computing briefly." \
       --device Vulkan0 \
       -n 128

The device selection logic is implemented in llama-bench.cpp at lines 407-638, which parses the --device argument and maps it to the corresponding backend. This flag works across all llama.cpp tools including llama-cli, llama-bench, and llama-server.

Deploy with Docker

Pre-built images with Vulkan support are documented in docs/docker.md at lines 28-30.

Image	Description
ghcr.io/ggml-org/llama.cpp:full-vulkan	Full build with all backends
ghcr.io/ggml-org/llama.cpp:light-vulkan	CPU + Vulkan only
ghcr.io/ggml-org/llama.cpp:server-vulkan	Server-oriented build with Vulkan

Run with GPU passthrough on Linux:

docker run -it --rm \
   -v "$(pwd):/app:Z" \
   --device /dev/dri/renderD128:/dev/dri/renderD128 \
   ghcr.io/ggml-org/llama.cpp:full-vulkan \
   ./main -m /app/models/7B/ggml-model-q4_0.gguf \
          -p "Write a haiku about spring." \
          --device Vulkan0 -n 64

Programmatic Control via C++ API

Force Vulkan selection in code by setting the device field in llama_context_params:

#include "llama.h"

int main() {
    llama_model *model = llama_load_model_from_file(
        "models/7B/ggml-model-q4_0.gguf", nullptr);
    
    llama_context_params params = llama_context_default_params();
    params.device = "Vulkan0";  // Same syntax as CLI --device flag
    
    llama_context *ctx = llama_new_context_with_model(model, params);
    
    // Inference loop...
    
    llama_free_context(ctx);
    llama_free_model(model);
    return 0;
}

Troubleshooting and Debugging

To disable Vulkan temporarily for debugging, set the environment variable checked in ggml/src/ggml-backend-reg.cpp at lines 123-127:

export GGML_DISABLE_VULKAN=1
./main --list-devices  # Vulkan devices will not appear

If vulkaninfo fails or devices do not appear, verify that VULKAN_SDK points to the correct directory and that your GPU drivers support Vulkan 1.2 or higher.

Summary

• Compile with -DGGML_VULKAN=ON to enable the backend in ggml/CMakeLists.txt at lines 218-224.
• Install the LunarG Vulkan SDK and configure the VULKAN_SDK environment variable.
• Select your GPU at runtime using --device Vulkan0 or the C++ API params.device field.
• Debug by setting GGML_DISABLE_VULKAN=1 to hide the backend registration in ggml/src/ggml-backend-reg.cpp.

Frequently Asked Questions

What CMake flag enables Vulkan support in llama.cpp?

Set -DGGML_VULKAN=ON during the CMake configuration step. This option is defined in ggml/CMakeLists.txt at lines 218-224 and links the Vulkan loader to the build.

How do I verify that Vulkan is available before running inference?

Run ./main --list-devices to enumerate all detected backends. If the Vulkan SDK is properly installed and GGML_DISABLE_VULKAN is not set, you should see entries like Vulkan0 in the output.

Can I use Vulkan alongside CUDA or Metal?

Yes, llama.cpp supports multiple backends simultaneously. However, you typically select one device per session using --device. To force Vulkan specifically, disable other backends at build time with -DGGML_CUDA=OFF -DGGML_METAL=OFF or simply choose the Vulkan device at runtime.

Why does my Vulkan device not appear in the device list?

Ensure the LunarG Vulkan SDK is installed and the VULKAN_SDK environment variable points to the correct directory. Also verify that GGML_DISABLE_VULKAN is not set, as this variable blocks backend registration in ggml/src/ggml-backend-reg.cpp at lines 123-127.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}