# How to Configure Multi-GPU and Device Mapping for Heretic: A Complete Guide

> Configure multi-GPU and device mapping for Heretic using device_map and max_memory in TOML, CLI, or env variables. Optimize your performance with this complete guide.

- Repository: [Philipp Emanuel Weidmann/heretic](https://github.com/p-e-w/heretic)
- Tags: how-to-guide
- Published: 2026-02-19

---

**Configure multi-GPU and device mapping for Heretic by setting `device_map` and `max_memory` in the `Settings` model, accessible via TOML config files, CLI flags, or environment variables.**

Heretic, the open-source evaluation framework hosted at `p-e-w/heretic`, supports distributed inference across multiple GPUs and accelerators through tight integration with Hugging Face Accelerate. By configuring **device mapping** (which layers reside on which GPU) and **memory limits** (per-device VRAM caps), you can optimize throughput for models that exceed single-GPU memory or balance workloads across heterogeneous hardware.

## Understanding Device Mapping and Memory Management in Heretic

### The Settings Model

All multi-GPU configuration in Heretic flows through the `Settings` class defined in [`src/heretic/config.py`](https://github.com/p-e-w/heretic/blob/main/src/heretic/config.py). Two fields control device placement:

- **`device_map`** (lines 103-106): Accepts `"auto"` to let Accelerate calculate optimal placement, or an explicit dictionary mapping layer indices to device IDs (e.g., `{"0": 0, "1": 1}`).
- **`max_memory`** (lines 108-111): Optional per-device memory constraints formatted as strings (e.g., `{"0": "20GB", "1": "22GB"}`). Accelerate uses these limits when partitioning model layers.

### How Accelerate Handles Device Placement

When Heretic loads a model, it passes your `device_map` and `max_memory` values directly to `accelerate.init_empty_weights()` and `accelerate.load_checkpoint_and_dispatch()`. This happens transparently—no manual tensor movement required. The startup logic in [`src/heretic/main.py`](https://github.com/p-e-w/heretic/blob/main/src/heretic/main.py) (lines 176-188) enumerates available CUDA, XPU, MLU, SDAA, MUSA, and MPS devices, printing their names and VRAM to help you construct valid device maps.

## Configuration Methods for Multi-GPU Setups

Heretic supports three hierarchical configuration layers: environment variables (lowest priority), TOML config files, and CLI flags (highest priority).

### Using a TOML Configuration File

Create a [`config.toml`](https://github.com/p-e-w/heretic/blob/main/config.toml) in the repository root for persistent multi-GPU settings:

```toml

# config.toml

device_map = { "0" = 0, "1" = 1 }
max_memory = { "0" = "22GB", "1" = "14GB" }

```

This configuration splits layers between GPU 0 and GPU 1 while capping memory usage at 22 GB and 14 GB respectively. The `Settings.settings_customise_sources` method in [`src/heretic/config.py`](https://github.com/p-e-w/heretic/blob/main/src/heretic/config.py) automatically loads this file on startup.

### Command-Line Interface Overrides

Override config files or set device mapping ad-hoc using JSON-formatted CLI arguments:

```bash
heretic run \
  --device_map '{"0":0,"1":1}' \
  --max_memory '{"0":"22GB","1":"14GB"}'

```

Heretic parses these strings and injects them into the `Settings` model before model initialization.

### Environment Variable Configuration

For containerized deployments or quick experiments, use `HERETIC_` prefixed environment variables:

```bash
export HERETIC_DEVICE_MAP='{"0":0,"1":1}'
export HERETIC_MAX_MEMORY='{"0":"22GB","1":"14GB"}'
heretic run

```

Environment variables are processed after CLI flags but before TOML files, allowing flexible layering of configurations.

## Practical Multi-GPU Configuration Examples

### Balancing Unequal GPUs

Consider a system with two NVIDIA GPUs: GPU 0 has 24 GB VRAM, GPU 1 has 16 GB VRAM. To maximize throughput without out-of-memory errors:

```toml
device_map = { "0" = 0, "1" = 1 }
max_memory = { "0" = "22GB", "1" = "14GB" }

```

The `max_memory` values leave headroom for activation tensors and system overhead. Heretic's utility functions in [`src/heretic/utils.py`](https://github.com/p-e-w/heretic/blob/main/src/heretic/utils.py) (specifically `get_device_stats()`) report actual VRAM utilization during runtime to help you fine-tune these caps.

### Extending to Three or More GPUs

Adding a third GPU requires only extending the dictionaries:

```toml
device_map = { "0" = 0, "1" = 1, "2" = 2 }
max_memory = { "0" = "22GB", "1" = "14GB", "2" = "16GB" }

```

Heretic automatically detects the additional device during startup (as implemented in [`src/heretic/main.py`](https://github.com/p-e-w/heretic/blob/main/src/heretic/main.py) lines 176-188) and validates that your device map indices exist in the enumerated hardware list.

## Verifying Your Device Mapping Configuration

After configuring, confirm correct placement by observing the startup logs. Heretic prints enumerated devices in [`src/heretic/main.py`](https://github.com/p-e-w/heretic/blob/main/src/heretic/main.py) (lines 176-188), showing:

```

Detected devices:
  cuda:0: NVIDIA GeForce RTX 3090 (24GB)
  cuda:1: NVIDIA GeForce RTX 3080 (16GB)

```

If your `device_map` references non-existent indices, Heretic raises a validation error before loading weights, preventing silent CPU fallback or out-of-memory crashes.

## Summary

- **Configure multi-GPU and device mapping for Heretic** through the `Settings` model in [`src/heretic/config.py`](https://github.com/p-e-w/heretic/blob/main/src/heretic/config.py), specifically the `device_map` and `max_memory` fields.
- Use **TOML files** for persistent configurations, **CLI flags** for ad-hoc overrides, or **environment variables** for containerized deployments.
- Reference **actual GPU indices** detected by the startup logic in [`src/heretic/main.py`](https://github.com/p-e-w/heretic/blob/main/src/heretic/main.py) (lines 176-188) when building explicit device maps.
- Set **conservative `max_memory` values** to leave headroom for activations and system overhead, especially when balancing heterogeneous GPUs.

## Frequently Asked Questions

### What is the difference between device_map and max_memory in Heretic?

The `device_map` parameter controls **which layers are placed on which device**, accepting either `"auto"` for automatic distribution or a dictionary mapping layer names to device indices. The `max_memory` parameter sets **per-device VRAM caps** (e.g., `{"0": "20GB"}`), instructing Accelerate to respect these limits when partitioning the model. Use `device_map` for explicit placement control and `max_memory` to prevent out-of-memory errors on GPUs with limited VRAM.

### Can I use automatic device mapping instead of manual configuration?

Yes. Set `device_map = "auto"` in your [`config.toml`](https://github.com/p-e-w/heretic/blob/main/config.toml) or pass `--device_map '"auto"'` via CLI. When set to `"auto"`, Heretic delegates placement decisions to Hugging Face Accelerate, which analyzes layer sizes and available VRAM to generate an optimal mapping. This is ideal for homogeneous multi-GPU setups where you want maximum memory utilization without manual tuning.

### How do I check if Heretic detected my GPUs correctly?

Heretic enumerates all available accelerators during startup in [`src/heretic/main.py`](https://github.com/p-e-w/heretic/blob/main/src/heretic/main.py) (lines 176-188). The startup log displays a "Detected devices" list showing device indices (e.g., `cuda:0`), model names, and VRAM capacity. If a GPU is missing from this list, verify PyTorch installation (`torch.cuda.is_available()`) and driver compatibility. Heretic validates `device_map` indices against this detected list, raising errors for invalid references before model loading begins.

### What file contains the device detection logic in Heretic?

The device detection and reporting logic resides in [`src/heretic/main.py`](https://github.com/p-e-w/heretic/blob/main/src/heretic/main.py), specifically lines 176-188. This code iterates through supported backends (CUDA, XPU, MLU, SDAA, MUSA, MPS), retrieves device properties, and prints a formatted summary of available hardware. For runtime VRAM statistics aggregation, see [`src/heretic/utils.py`](https://github.com/p-e-w/heretic/blob/main/src/heretic/utils.py), which contains `get_device_stats()` used by the CLI to display memory utilization during inference.