ODS Hardware Tier Default Models: CPU, AMD, NVIDIA, and Apple Silicon

Open Data Stack (ODS) automatically selects default LLM models based on four hardware tiers—assigning llama2-7b to CPU and Apple Silicon, llama2-13b to AMD GPUs, and mixtral-8x7b to NVIDIA GPUs—via JSON configuration files in ods/config/backends/.

When deploying local language models, understanding the default models for each ODS hardware tier ensures optimal performance without manual tuning. The Osmantic/ODS repository uses a tier-detection system that probes your hardware during installation and maps detected capabilities to pre-configured model defaults stored in backend-specific JSON files.

Hardware Tier Overview and Default Models

ODS categorizes host machines into four distinct hardware classes. Each tier maps to a specific backend JSON file that defines the default model and available alternatives.

Hardware Tier	Detection Method	Default Model	Memory Profile
CPU	No GPU detected	llama2-7b (GGUF)	Optimized for ≥4GB RAM
AMD	ROCm/AMD GPU detected	llama2-13b (GGUF)	Balanced quality and VRAM usage
NVIDIA	CUDA GPU detected	mixtral-8x7b (GGUF)	High-performance multi-GPU
Apple	Apple Silicon (M-series) detected	llama2-7b (GGUF)	Neural Engine optimized

The first entry in the models array within each backend JSON determines the default. For example, in ods/config/backends/nvidia.json, the array position [0] specifies mixtral-8x7b as the automatic selection for NVIDIA hardware.

Backend Configuration Files

ODS stores tier definitions in ods/config/backends/. Each file contains a models array where the first element serves as the default for that hardware class.

CPU Tier Configuration

For CPU-only machines, ods/config/backends/cpu.json specifies lightweight models that run efficiently without GPU acceleration:

{
  "backend": "cpu",
  "models": [
    "llama2-7b",
    "phi-2",
    "tinyllama"
  ],
  "format": "gguf"
}

AMD GPU Tier Configuration

AMD GPU hosts use ods/config/backends/amd.json, defaulting to a mid-size model that leverages ROCm capabilities:

{
  "backend": "amd",
  "models": [
    "llama2-13b",
    "llama2-7b",
    "mistral-7b"
  ],
  "format": "gguf"
}

NVIDIA GPU Tier Configuration

High-performance NVIDIA systems reference ods/config/backends/nvidia.json, which prioritizes the Mixtral mixture-of-experts model:

{
  "backend": "nvidia",
  "models": [
    "mixtral-8x7b",
    "llama2-13b",
    "codellama-13b"
  ],
  "format": "gguf"
}

Apple Silicon Tier Configuration

Apple devices use ods/config/backends/apple.json, sharing the lightweight default with CPU tiers but optimized for the Apple Neural Engine:

{
  "backend": "apple",
  "models": [
    "llama2-7b",
    "phi-2",
    "mistral-7b"
  ],
  "format": "gguf"
}

Hardware Detection and Model Selection

ODS determines your hardware tier through a detection pipeline involving two critical shell scripts in the installer library.

Detection Mechanism

The ods/installers/lib/detection.sh script probes system hardware using nvidia-smi, roc-smi, and sysctl commands. It exports the ODS_BACKEND environment variable as cpu, amd, nvidia, or apple based on detected capabilities.

Subsequently, ods/installers/lib/tier-map.sh reads ODS_BACKEND and loads the corresponding JSON from ods/config/backends/${ODS_BACKEND}.json. This script extracts the first model from the models array and sets it as the active default for the installation session.

Programmatically Accessing Default Models

You can retrieve the current hardware tier's default model programmatically using the CLI or Python SDK.

Bash CLI Method

Query the active backend and parse the default model using standard shell tools:

#!/bin/bash

# Retrieve detected backend (cpu, amd, nvidia, apple)

backend=$(ods-cli env get ODS_BACKEND)

# Extract first model from tier configuration

default_model=$(jq -r '.models[0]' \
  "/opt/ods/ods/config/backends/${backend}.json")

echo "Active hardware tier: ${backend}"
echo "Default model: ${default_model}"

Python Implementation

For dashboard extensions or API services, use the following pattern implemented in ods/extensions/services/dashboard-api/helpers.py:

import json
import os
from pathlib import Path

def get_default_model() -> str:
    """
    Retrieves the default model for the current ODS hardware tier.
    Returns the first entry from the backend's models array.
    """
    backend = os.getenv("ODS_BACKEND", "cpu")
    config_path = (
        Path(__file__).parents[3] / "ods" / "config" / "backends" / f"{backend}.json"
    )
    
    with config_path.open() as f:
        data = json.load(f)
    
    # First element in models array is the default

    return data["models"][0]

# Usage

print(f"Default model for current tier: {get_default_model()}")

Customizing Default Models

To change the default model for a specific tier, reorder the models array in the corresponding backend JSON file so your preferred model appears first.

Using jq to modify the NVIDIA tier default:

# Promote llama3-8b to default while preserving other models

jq '.models = ["llama3-8b"] + [.models[] | select(. != "llama3-8b")]' \
  ods/config/backends/nvidia.json > tmp.json \
  && mv tmp.json ods/config/backends/nvidia.json

Alternatively, create a custom model-library.json in your ODS root directory. This file overrides the standard backend configurations without modifying the core repository files, allowing you to persist custom defaults across updates.

Summary

• ODS hardware tiers are defined in ods/config/backends/ with separate JSON files for cpu, amd, nvidia, and apple backends.
• Default models are determined by the first element in each backend's models array: llama2-7b (CPU/Apple), llama2-13b (AMD), and mixtral-8x7b (NVIDIA).
• Hardware detection occurs via ods/installers/lib/detection.sh, which sets the ODS_BACKEND environment variable consumed by ods/installers/lib/tier-map.sh.
• Programmatic access requires reading the backend JSON and extracting the zeroth index of the models list using jq or Python's json module.
• Customization involves reordering the JSON array or providing a model-library.json override file.

Frequently Asked Questions

How does ODS determine which hardware tier to use?

ODS runs ods/installers/lib/detection.sh during installation, which checks for NVIDIA GPUs via nvidia-smi, AMD GPUs via roc-smi, and Apple Silicon via sysctl. The script sets the ODS_BACKEND environment variable to cpu, amd, nvidia, or apple, which tier-map.sh then uses to select the appropriate configuration file from ods/config/backends/.

Can I override the default model without modifying JSON files?

Yes. Set the ODS_DEFAULT_MODEL environment variable before running the installer or dashboard services. When present, this variable takes precedence over the first element in the backend's models array, though the JSON configuration remains unchanged on disk.

What model format does ODS use for hardware tier defaults?

All default models specified in ods/config/backends/*.json use the GGUF format (Georgi Gerganov Universal Format). This quantized format provides efficient inference across CPU and GPU backends while maintaining compatibility with the llama.cpp inference engine used by ODS services.

Where are the hardware detection scripts located?

The detection logic resides in ods/installers/lib/detection.sh, while the tier-to-model mapping logic is implemented in ods/installers/lib/tier-map.sh. These scripts are invoked during the ods install process and by the dashboard API helpers when initializing model serving endpoints.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}