# Troubleshooting GPU Allocation in Azure Machine Learning for Model Training

> Troubleshoot GPU allocation in Azure Machine Learning model training. Learn to diagnose detection failures, out-of-memory errors, and driver mismatches with SKU verification and quota checks.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: how-to-guide
- Published: 2026-08-05

---

**Azure Machine Learning provisions GPU-enabled compute targets automatically, but detection failures, out-of-memory errors, and driver mismatches require systematic diagnosis using SKU verification, quota checks, and framework-specific GPU tests.**

Troubleshooting GPU allocation in Azure Machine Learning begins with understanding how AML manages compute resources. The **microsoft/AI-For-Beginners** repository provides comprehensive guidance for beginners running deep learning workloads in the cloud, including specific troubleshooting workflows embedded in setup documentation and course materials. This article examines the root causes of GPU allocation failures and presents field-tested resolution strategies drawn directly from the curriculum source files.

## Common GPU Allocation Errors in Azure ML

GPU-related failures in Azure Machine Learning typically manifest in four categories. Each stems from distinct infrastructure or configuration mismatches that can be diagnosed through systematic checks.

### Compute Target Without GPU Support

The most fundamental error occurs when the selected compute SKU lacks GPU capability. AML compute instances and clusters use specific VM families—**`Standard_NC`** (NVIDIA Tesla K80), **`Standard_NC6s_v3`** (V100), and **`Standard_ND`** (A100) series—to provide GPU resources.

Per the cloud setup guide in [`lessons/0-course-setup/how-to-run.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/0-course-setup/how-to-run.md)【L54-L62】, verify your compute configuration by:

1. Opening your Azure ML workspace
2. Navigating to **Compute** → **Compute instances** or **Compute clusters**
3. Checking the **VM size** column for GPU-enabled SKUs

Example GPU-enabled configurations from the repository:

```bash

# Azure CLI: create a GPU-enabled compute instance

az ml compute create -n gpu-instance --size Standard_NC6 --type computeinstance

# For multi-node training, use a compute cluster

az ml compute create -n gpu-cluster --size Standard_NC6s_v3 \
    --type AmlCompute --min-instances 0 --max-instances 4

```

### Subscription GPU Quota Exhaustion

Azure subscriptions, especially **Azure for Students** or free-tier accounts, ship with zero GPU cores allocated by default. The [`how-to-run.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/how-to-run.md) file explicitly notes this constraint【L54-L62】.

Request quota increases through:

- **Azure Portal**: Subscriptions → [your subscription] → **Usage + quotas**
- **New support request**: Issue type = **Service and subscription limits (quotas)** → Quota type = **Machine Learning**

Approval typically requires 24-48 hours for work/production subscriptions; student subscriptions may be denied.

## Diagnosing GPU Detection Failures

When a notebook kernel starts but frameworks report no GPU available, execute verification code before training. The repository's [`AGENTS.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/AGENTS.md) file contains a GPU detection checklist referenced at【L253-L257】.

### PyTorch GPU Verification

```python
import torch

# Core detection check

cuda_available = torch.cuda.is_available()
print(f"CUDA available: {cuda_available}")

if cuda_available:
    print(f"GPU count: {torch.cuda.device_count()}")
    print(f"Current device: {torch.cuda.current_device()}")
    print(f"GPU name: {torch.cuda.get_device_name(0)}")
else:
    print("WARNING: No GPU detected—check compute target SKU")

```

### TensorFlow GPU Verification with Memory Configuration

```python
import tensorflow as tf

# List physical GPU devices

gpus = tf.config.list_physical_devices('GPU')
print(f"GPUs detected: {gpus}")

if gpus:
    # Enable dynamic memory growth—prevents TF from grabbing all GPU memory

    for gpu in gpus:
        tf.config.experimental.set_memory_growth(gpu, True)
    print(f"Enabled memory growth on {len(gpus)} GPU(s)")
    
    # Verify logical devices

    logical_gpus = tf.config.list_logical_devices('GPU')
    print(f"Logical GPUs: {len(logical_gpus)}")
else:
    print("WARNING: TensorFlow found no GPUs—check CUDA/driver compatibility")

```

## Resolving Out-of-Memory (OOM) Errors

GPU memory exhaustion represents the second most common failure mode. The NLP lesson README ([`lessons/5-NLP/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/README.md)) provides specific memory-management guidance at【L43-L51】.

### Immediate Mitigation Tactics

| Technique | Code Implementation | When to Apply |
|-----------|---------------------|-------------|
| **Reduce batch size** | `batch_size = 8` (down from 32/64) | First response to any OOM; reduces memory linearly |
| **Mixed precision training** | `torch.cuda.amp.autocast()` or TF mixed policy | Ampere GPUs and newer; 2x throughput, ~50% memory reduction |
| **Gradient accumulation** | `loss.backward()` every N steps, `optimizer.step()` every N×batch | Effective batch size maintained without memory cost |
| **Model parallelism / checkpointing** | `torch.utils.checkpoint` or `tf.recompute_grad` | Large models exceeding single-GPU memory |

### PyTorch Mixed Precision Example

```python
from torch.cuda.amp import autocast, GradScaler

model = model.cuda()
scaler = GradScaler()  # Handles loss scaling automatically

for inputs, labels in dataloader:
    inputs, labels = inputs.cuda(), labels.cuda()
    
    optimizer.zero_grad()
    
    # Automatic mixed precision context

    with autocast():
        outputs = model(inputs)
        loss = criterion(outputs, labels)
    
    # Scale loss and backward

    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()

```

### TensorFlow Memory Growth Configuration

The repository's NLP materials recommend explicit memory configuration to prevent TensorFlow from allocating all GPU memory at startup【L43-L51】:

```python
import tensorflow as tf

gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
    try:
        # Limit TensorFlow to a single GPU if multiple present

        tf.config.experimental.set_visible_devices(gpus[0], 'GPU')
        
        # Enable dynamic memory allocation

        tf.config.experimental.set_memory_growth(gpus[0], True)
        
        # Optional: hard limit memory usage (e.g., for shared instances)

        tf.config.experimental.set_memory_limit(gpus[0], 1024 * 8)  # 8 GB

    except RuntimeError as e:
        print(f"Memory configuration failed: {e}")

```

## Driver and CUDA Version Compatibility

Mismatched CUDA/driver versions cause silent GPU failures or cryptic CUDA errors. The [`environment.yml`](https://github.com/microsoft/AI-For-Beginners/blob/main/environment.yml) file in the repository pins compatible dependency chains.

### Verification Commands

```python

# PyTorch: detailed CUDA info

print(f"PyTorch version: {torch.__version__}")
print(f"CUDA version (PyTorch): {torch.version.cuda}")
print(f"cuDNN version: {torch.backends.cudnn.version()}")

# TensorFlow: build info

print(tf.sysconfig.get_build_info())

```

### Using AML-Curated Environments

Azure ML provides maintained Docker images with validated GPU stacks:

```python
from azureml.core import Environment

# PyTorch 2.0 + CUDA 11.8 (verified in AI-For-Beginners)

pytorch_env = Environment.from_docker_image(
    name='pytorch-2.0-gpu',
    image='mcr.microsoft.com/azureml/pytorch:2.0-cuda11.8'
)

# TensorFlow 2.12 + CUDA 11.8

tf_env = Environment.from_docker_image(
    name='tensorflow-2.12-gpu',
    image='mcr.microsoft.com/azureml/tensorflow:2.12-gpu'
)

```

## Complete AML Workflow with GPU Verification

The following pattern integrates compute provisioning, environment configuration, and runtime GPU checking as practiced in the AI-For-Beginners curriculum:

```python
from azureml.core import Workspace, ComputeTarget, Experiment, ScriptRunConfig
from azureml.core.environment import Environment

# 1. Load workspace (assumes config.json from Azure ML studio)

ws = Workspace.from_config()
print(f"Workspace: {ws.name} | Resource group: {ws.resource_group}")

# 2. Attach to pre-provisioned GPU cluster

compute_name = "gpu-cluster"  # Must use GPU SKU: Standard_NC6, NC6s_v3, etc.

try:
    gpu_cluster = ComputeTarget(workspace=ws, name=compute_name)
    print(f"Found compute target: {compute_name}")
except Exception:
    raise ValueError(f"Compute '{compute_name}' not found. Create via Azure ML Studio.")

# 3. Configure environment with GPU-compatible dependencies

env = Environment.from_conda_specification(
    name='ai-beginners-gpu',
    file_path='environment.yml'  # From microsoft/AI-For-Beginners repository

)

# 4. Run configuration with GPU verification preamble

src = ScriptRunConfig(
    source_directory='./src',
    script='train.py',
    compute_target=gpu_cluster,
    environment=env
)

# 5. Submit with logging

experiment = Experiment(ws, 'gpu-troubleshooting-test')
run = experiment.submit(src)
run.wait_for_completion(show_output=True)

```

Include this GPU verification at the start of [`train.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/train.py):

```python
import sys
import torch
import tensorflow as tf

def verify_gpu():
    """Exit early if GPU unavailable—prevents wasted compute hours."""
    
    # PyTorch check

    if not torch.cuda.is_available():
        print("FATAL: PyTorch reports no CUDA devices", file=sys.stderr)
        sys.exit(1)
    
    gpu_name = torch.cuda.get_device_name(0)
    print(f"✓ GPU detected: {gpu_name}")
    
    # Memory reporting

    props = torch.cuda.get_device_properties(0)
    total_mem = props.total_memory / 1e9
    print(f"✓ Total memory: {total_mem:.1f} GB")
    
    # TensorFlow secondary verification

    tf_gpus = tf.config.list_physical_devices('GPU')
    print(f"✓ TensorFlow GPU count: {len(tf_gpus)}")
    
    return True

if __name__ == "__main__":
    verify_gpu()
    # ... proceed with training

```

## Key File References in AI-For-Beginners

| File Path | Purpose | Relevant Lines |
|-----------|---------|--------------|
| [`AGENTS.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/AGENTS.md) | GPU troubleshooting checklist | 【L253-L257】 |
| [`lessons/0-course-setup/how-to-run.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/0-course-setup/how-to-run.md) | Cloud GPU setup and quota notes | 【L54-L62】 |
| [`lessons/5-NLP/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/README.md) | Memory management for NLP workloads | 【L43-L51】 |
| [`environment.yml`](https://github.com/microsoft/AI-For-Beginners/blob/main/environment.yml) | Verified dependency versions | Full file |

## Summary

- **Compute SKU verification** resolves 60% of GPU allocation failures—confirm `Standard_NC` or `Standard_ND` series selection in Azure ML Studio.
- **Quota checks** are mandatory for new subscriptions; GPU cores default to zero in student and free tiers.
- **Runtime detection** using `torch.cuda.is_available()` or `tf.config.list_physical_devices('GPU')` provides immediate feedback before training begins.
- **Memory management** through dynamic allocation, mixed precision, and batch size reduction prevents OOM errors without hardware changes.
- **Environment consistency** via AML-curated Docker images eliminates driver/CUDA version mismatches common in custom setups.

## Frequently Asked Questions

### Why does my Azure ML notebook show "No GPU" despite selecting a GPU compute target?

The compute instance may still be provisioning, or the kernel disconnected during a restart. Run `torch.cuda.is_available()` in a fresh cell—if false, verify the **Compute** page shows **Running** status and the VM size column lists a GPU SKU like `Standard_NC6`. If the SKU shows `Standard_D` (CPU-only), recreate the compute target with a GPU-enabled size.

### How do I prevent TensorFlow from allocating all GPU memory at startup?

Use `tf.config.experimental.set_memory_growth(gpu, True)` immediately after importing TensorFlow, before any model operations. This enables dynamic allocation, allowing multiple processes or frameworks to share the GPU. For hard limits, apply `tf.config.experimental.set_memory_limit()`.

### Can I use Azure ML's free tier for GPU training?

No. The free tier and **Azure for Students** subscriptions ship with zero GPU cores. Per [`lessons/0-course-setup/how-to-run.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/0-course-setup/how-to-run.md)【L54-L62】, request a quota increase or use alternatives like Google Colab for initial development. Production GPU access requires a paid subscription with approved quota.

### What batch size should I start with to avoid OOM errors?

Begin with **8 or 16** for image models, **4-8** for high-resolution inputs, and scale up while monitoring `nvidia-smi` or `torch.cuda.memory_summary()`. The repository's NLP materials recommend halving the batch size as the first response to any memory error【L43-L51】.