How Project N.O.M.A.D. Automatically Detects and Utilizes NVIDIA GPUs for AI Acceleration

Yes, Project N.O.M.A.D. automatically detects NVIDIA GPUs by querying the Docker daemon for the NVIDIA runtime and executing nvidia-smi inside the Ollama container to verify GPU accessibility.

Project N.O.M.A.D. (Neural Operations Management and Deployment) is an open-source AI assistant platform that streamlines running large language models via Ollama in Docker containers. One of its key features is the ability to automatically detect and utilize NVIDIA GPUs for hardware-accelerated inference, eliminating manual configuration for supported hardware.

How Project N.O.M.A.D. Detects NVIDIA GPUs

The detection mechanism operates in two distinct stages to ensure accurate hardware identification and container compatibility.

Stage 1: Docker Runtime Detection via _detectGPUType()

The system first queries the Docker daemon to determine if the NVIDIA Container Toolkit is properly installed. In admin/app/services/docker_service.ts, the _detectGPUType() method inspects the runtime configuration:

const dockerInfo = await this.docker.info()
const runtimes = dockerInfo.Runtimes || {}
if ('nvidia' in runtimes) {
  // NVIDIA runtime detected → GPU is potentially usable
}

If the nvidia runtime is present, the system confirms that the host NVIDIA driver and container toolkit are installed, allowing GPU pass-through to containers. If the runtime is missing but lspci detects an NVIDIA device, the code flags toolkitMissing so the UI can advise installing the NVIDIA Container Toolkit.

Stage 2: In-Container Verification with getNvidiaSmiInfo()

Once the runtime is confirmed, the system verifies actual GPU accessibility inside the Ollama container. Located in admin/app/services/system_service.ts, the getNvidiaSmiInfo() method executes nvidia-smi within the container and parses the CSV output:

if (!graphics.controllers || graphics.controllers.length === 0) {
  const runtimes = dockerInfo.Runtimes || {}
  if ('nvidia' in runtimes) {
    gpuHealth.hasNvidiaRuntime = true
    const nvidiaInfo = await this.getNvidiaSmiInfo()
    if (Array.isArray(nvidiaInfo)) {
      graphics.controllers = nvidiaInfo.map(gpu => ({
        model: gpu.model,
        vendor: gpu.vendor,
        vram: gpu.vram,
      }))
      gpuHealth.status = 'ok'
      gpuHealth.ollamaGpuAccessible = true
    }
  }
}

The getNvidiaSmiInfo() method runs nvidia-smi --query-gpu=name,memory.total inside the Ollama container, strips non-printable characters, splits the CSV rows, and returns a clean array of {vendor, model, vram} objects.

Handling Detection Failures and Edge Cases

The system includes comprehensive error handling for various failure scenarios:

• Ollama container not running: getNvidiaSmiInfo() returns 'OLLAMA_NOT_FOUND' and sets gpuHealth.status to 'ollama_not_installed'.
• NVIDIA runtime present but nvidia-smi errors: Returns 'BAD_RESPONSE' with gpuHealth.status set to 'passthrough_failed'.
• GPU present but NVIDIA Container Toolkit missing: _detectGPUType() returns {type: 'none', toolkitMissing: true}, allowing the UI to surface installation warnings.
• No GPU detected: gpuHealth.type is set to 'none' with gpuHealth.status as 'ok' (indicating no GPU is required).

Implementation Examples

Checking GPU Status via API

You can expose GPU detection status through an API endpoint using the SystemService:

import SystemService from '#app/services/system_service'

// Example: an API route that returns GPU status
router.get('/api/gpu-status', async ({ response }) => {
  const sysInfo = await SystemService.getSystemInfo()
  if (!sysInfo) {
    return response.internalServerError({ error: 'Unable to collect system info' })
  }

  const { gpuHealth, graphics } = sysInfo
  response.ok({
    detected: gpuHealth.status,
    hasNvidiaRuntime: gpuHealth.hasNvidiaRuntime ?? false,
    ollamaAccessible: gpuHealth.ollamaGpuAccessible ?? false,
    gpus: graphics.controllers?.map(c => ({
      model: c.model,
      vendor: c.vendor,
      vram: c.vram,
    })) ?? [],
  })
})

Example response when an NVIDIA GPU is available:

{
  "detected": "ok",
  "hasNvidiaRuntime": true,
  "ollamaAccessible": true,
  "gpus": [
    { "model": "GeForce RTX 3080", "vendor": "NVIDIA", "vram": 10240 }
  ]
}

Displaying GPU Information in the Frontend

Use this React component to display GPU status in your UI:

import { useEffect, useState } from 'react'
import axios from 'axios'

export function GpuInfo() {
  const [info, setInfo] = useState<any>(null)

  useEffect(() => {
    axios.get('/api/gpu-status')
      .then(r => setInfo(r.data))
      .catch(() => setInfo({ error: 'Failed to fetch GPU info' }))
  }, [])

  if (!info) return <p>Loading…</p>
  if (info.error) return <p>{info.error}</p>

  return (
    <div>
      <h3>GPU Detection</h3>
      <p>Status: {info.detected}</p>
      {info.gpus.length > 0 ? (
        <ul>
          {info.gpus.map((g: any, i: number) => (
            <li key={i}>{g.vendor} {g.model} – {g.vram} MiB</li>
          ))}
        </ul>
      ) : (
        <p>No GPU detected.</p>
      )}
    </div>
  )
}

Manual GPU Detection in Scripts

For administrative scripts or debugging, invoke the low-level helper directly:

import SystemService from '#app/services/system_service'

async function listNvidiaGpus() {
  const sys = new SystemService(/* DockerService instance */)
  const info = await sys.getNvidiaSmiInfo()
  console.log('Raw NVIDIA info:', info)
}
listNvidiaGpus()

Key Source Files and Functions

The GPU detection system is implemented across several critical files in the admin directory:

• admin/app/services/docker_service.ts – Contains _detectGPUType() which queries the Docker daemon for NVIDIA runtimes and performs host-level GPU detection via lspci.
• admin/app/services/system_service.ts – Houses getNvidiaSmiInfo() for executing nvidia-smi inside containers and the GPU fallback logic within getSystemInfo() that orchestrates the full detection pipeline.
• admin/types/system.ts – Defines the GpuHealthStatus interface specifying the shape of GPU health reports including status, hasNvidiaRuntime, and ollamaGpuAccessible fields.
• admin/constants/service_names.ts – Stores the OLLAMA constant used to identify the target container for GPU queries.

Summary

• Project N.O.M.A.D. automatically detects NVIDIA GPUs through a two-stage process involving Docker runtime inspection and in-container nvidia-smi execution.
• The _detectGPUType() method in docker_service.ts verifies NVIDIA Container Toolkit installation by checking for the nvidia runtime in Docker daemon info.
• getNvidiaSmiInfo() in system_service.ts executes GPU queries inside the Ollama container, parsing CSV output to extract model names, vendors, and VRAM capacity.
• Comprehensive error handling covers missing containers, toolkit installation failures, and passthrough errors, surfacing detailed status via the gpuHealth object.
• GPU information is exposed through the system API, enabling frontend components to conditionally enable GPU-accelerated AI workloads based on hardware availability.

Frequently Asked Questions

Does Project N.O.M.A.D. support AMD GPUs?

Currently, Project N.O.M.A.D. primarily implements automated detection for NVIDIA GPUs through the NVIDIA Container Toolkit and nvidia-smi. While the _detectGPUType() method in docker_service.ts contains logic to identify AMD GPUs via lspci, the automated container integration and GPU health reporting are optimized for NVIDIA hardware. AMD GPU support would require similar runtime detection mechanisms for ROCm or other AMD container toolkits.

What are the requirements for NVIDIA GPU detection?

For Project N.O.M.A.D. to automatically detect and utilize an NVIDIA GPU, the host system must have the NVIDIA driver installed, the NVIDIA Container Toolkit configured, and the Docker daemon must expose the nvidia runtime. The Ollama container must also be running, as the system executes nvidia-smi inside this specific container to verify GPU accessibility. If the toolkit is missing but a GPU is present, the system flags toolkitMissing in the health status.

How can I troubleshoot GPU detection failures?

If Project N.O.M.A.D. fails to detect your NVIDIA GPU, check the gpuHealth field in the system information API response. Status values like 'ollama_not_installed' indicate the Ollama container isn't running, while 'passthrough_failed' suggests the NVIDIA runtime is present but nvidia-smi returned an error. Verify that docker info shows the nvidia runtime and that nvidia-smi works on the host. The _detectGPUType() method in docker_service.ts also logs when the toolkit is missing but hardware is detected.

Can I use Project N.O.M.A.D. without a GPU?

Yes, Project N.O.M.A.D. functions entirely on CPU-based inference when no GPU is detected. The gpuHealth object will report type: 'none' and status: 'ok', indicating that while no GPU is present, the system is operating normally. The Ollama service will automatically fall back to CPU execution for model inference, though performance will be significantly slower compared to GPU acceleration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}