# Common Failure Modes for Hugging Face Training Jobs: A Complete Troubleshooting Guide

> Troubleshoot common Hugging Face training job failures like OOM errors and dataset issues. Learn solutions for gradient checkpointing, streaming, and hub limits.

- Repository: [Hugging Face/skills](https://github.com/huggingface/skills)
- Tags: how-to-guide
- Published: 2026-03-08

---

**The most frequent training failures stem from GPU out-of-memory errors, dataset loading misconfigurations, and Hugging Face Hub rate limits, all of which can be resolved through gradient checkpointing, streaming datasets, and cached authentication tokens.**

Training jobs on the Hugging Face platform involve complex interactions between distributed compute, large datasets, and the Hugging Face Hub. Understanding these **common failure modes for Hugging Face training jobs** is essential for maintaining high availability and minimizing downtime. This guide draws directly from the `huggingface/skills` repository to provide actionable fixes for the seven most critical failure categories.

## 1. Out-of-Memory (OOM) Errors

OOM errors are the most common reason for training interruptions. They manifest on both GPU and CPU depending on where the bottleneck occurs.

### GPU OOM Errors

**Symptom:** `CUDA out of memory` or `RuntimeError: CUDA error: out of memory`.

**Root Cause:** The model parameters, activation buffers, or optimizer states exceed the VRAM of the selected GPU.

**Resolution:**

- Reduce `per_device_train_batch_size` or enable gradient accumulation via `gradient_accumulation_steps`.
- Activate **gradient checkpointing** by calling `model.gradient_checkpointing_enable()` to trade compute for memory.
- Upgrade to a GPU with more VRAM; consult the hardware recommendations in [`skills/hugging-face-jobs/references/hardware_guide.md`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/references/hardware_guide.md).

### CPU OOM Errors

**Symptom:** `MemoryError` during data preprocessing.

**Root Cause:** Large tokenized datasets are fully materialized in RAM instead of being streamed.

**Resolution:**

- Use `datasets.set_format("torch", columns=["input_ids"], output_all_columns=False)` to load tensors lazily.
- Enable streaming mode for massive datasets: `load_dataset(..., streaming=True)`.

## 2. Dataset-Related Errors

### DatasetNotFoundError and Permission Issues

**Symptom:** `DatasetNotFoundError` or `FileNotFoundError` when loading from the Hub.

**Root Cause:** The dataset identifier is misspelled, the repository is private, or the authentication token lacks read permissions.

**Resolution:**

- Verify the exact Hub identifier (e.g., `datasets.load_dataset("username/dataset_name")`).
- Provide a valid read-only token; see the authentication guidance in [`skills/hugging-face-jobs/references/troubleshooting.md`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/references/troubleshooting.md).

### Schema Mismatches and Column Errors

**Symptom:** Unexpected column types or missing fields during training.

**Root Cause:** The dataset schema changed after the job started.

**Resolution:**

- Pin a specific dataset revision: `revision="v1.0.0"`.
- Add a pre-processing validation step that asserts expected column names before training begins.

## 3. Token-Usage and Rate-Limiting

### 429 Too Many Requests

**Symptom:** `429 Too Many Requests` from the Hub API.

**Root Cause:** The job exceeds the request quota, often by repeatedly pulling large checkpoints.

**Resolution:**

- Cache model files locally using `hf_hub_download(..., cache_dir=...)`.
- Reduce `push_to_hub` frequency; trigger pushes only at epoch end or after a set number of steps. See the throttling guidelines in [`skills/hugging-face-jobs/references/token_usage.md`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/references/token_usage.md).

### Authentication Failures

**Symptom:** `InvalidToken` or permission denied errors.

**Root Cause:** The `HF_TOKEN` is expired or lacks the `write` scope.

**Resolution:**

- Regenerate the token from the Hub UI and export it as `HF_TOKEN` before launching the job.

## 4. Hardware Mismatch and Driver Issues

### CUDA Availability Errors

**Symptom:** `"torch.cuda.is_available()" returns False` on a GPU node.

**Root Cause:** The node’s CUDA drivers are missing or incompatible with the installed PyTorch version.

**Resolution:**

- Ensure the Docker image uses the same CUDA version as the node; refer to the Docker base images listed in the repository.
- Reinstall PyTorch with the matching CUDA wheel: `pip install torch==<version>+cu116`.

### cuDNN Compatibility Errors

**Symptom:** `RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR`.

**Root Cause:** Incompatible cuDNN library version.

**Resolution:**

- Align cuDNN (e.g., cuDNN 8.x) with the PyTorch build. The compatibility matrix is documented in the Hugging Face trainer docs.

## 5. Network and Connectivity Problems

### Stalled Logs and Timeouts

**Symptom:** Stalled logs or job killed after a timeout.

**Root Cause:** The VM loses internet connectivity, common on spot-instance clusters.

**Resolution:**

- Enable automatic retries via `retry_strategy` in the trainer configuration.
- Use a checkpointing callback to persist progress frequently.

### SSL Certificate Verification Failures

**Symptom:** `SSL: CERTIFICATE_VERIFY_FAILED` when pulling from the Hub.

**Root Cause:** Corporate firewalls intercept TLS traffic.

**Resolution:**

- Set `HF_ENDPOINT` to a reachable mirror.
- Configure a custom CA bundle using `REQUESTS_CA_BUNDLE`.

## 6. Version Incompatibilities

**Symptom:** `ImportError` or unexpected behavior after upgrading `transformers`.

**Root Cause:** The job’s [`requirements.txt`](https://github.com/huggingface/skills/blob/main/requirements.txt) pins an older version while the code uses newer APIs, or API semantics changed (e.g., `Trainer` arguments).

**Resolution:**

- Pin compatible versions of `transformers`, `datasets`, and `accelerate`; see the **requirements.txt** used by the Hugging Face jobs scripts.
- Review the changelog in the Hugging Face release notes and adjust the training script accordingly.

## 7. Checkpoint and Hub Saving Issues

### Disk Space Exhaustion

**Symptom:** `OSError: [Errno 28] No space left on device` when saving checkpoints.

**Root Cause:** The job’s allocated storage is exhausted (default is 10 GB).

**Resolution:**

- Reduce `save_total_limit` or increase the attached volume size.
- Off-load checkpoints to the Hub using `push_to_hub`; see the guidance in [`skills/hugging-face-jobs/references/hub_saving.md`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/references/hub_saving.md).

### Checkpoint Corruption on Pre-emption

**Symptom:** Checkpoint corruption after a pre-empted spot node.

**Root Cause:** Incomplete writes due to sudden termination.

**Resolution:**

- Enable `accelerate`’s `resume_from_checkpoint` flag.
- Keep a copy in a persistent bucket (e.g., S3).

## Practical Code Examples for Resilient Training

The `huggingface/skills` repository provides concrete implementations to mitigate these failures. Below are adapted snippets from [`skills/hugging-face-jobs/scripts/generate-responses.py`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/scripts/generate-responses.py) and [`skills/hugging-face-jobs/scripts/cot-self-instruct.py`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/scripts/cot-self-instruct.py).

### Guarding Against OOM with Gradient Checkpointing

```python
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, Trainer, TrainingArguments

model_name = "facebook/opt-6.7b"
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
model.gradient_checkpointing_enable()   # <-- reduces VRAM use

tokenizer = AutoTokenizer.from_pretrained(model_name)

training_args = TrainingArguments(
    output_dir="./results",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    fp16=True,                     # mixed precision

    logging_steps=50,
    save_total_limit=2,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
)
trainer.train()

```

*Key reference*: the **hardware guide** and **troubleshooting** docs explain why `gradient_checkpointing_enable()` helps with large models.

### Streaming Large Datasets to Avoid CPU OOM

```python
from datasets import load_dataset

# Stream instead of downloading the full dataset

dataset = load_dataset("bigscience-data/PG-19", split="train", streaming=True)

def tokenize_fn(batch):
    return tokenizer(batch["text"], truncation=True, padding="max_length", max_length=512)

tokenized_dataset = dataset.map(tokenize_fn, batched=True, remove_columns=["text"])

```

*Key reference*: [`skills/hugging-face-jobs/references/troubleshooting.md`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/references/troubleshooting.md) discusses streaming as a remedy for dataset-size-related crashes.

### Automatic Retries for Hub Operations

```python
from huggingface_hub import HfApi, HfFolder, Repository

api = HfApi()
repo = Repository(
    local_dir="./model",
    clone_from="username/my-finetuned-model",
    token=HfFolder.get_token(),
    repo_type="model",
    # Retry policy (default 3 attempts, exponential back-off)

    git_user="my-bot",
    git_email="bot@hf.co"
)

# Push checkpoint after each epoch

repo.push_to_hub(commit_message="Add epoch checkpoint")

```

*Key reference*: token-usage guidance in [`skills/hugging-face-jobs/references/token_usage.md`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/references/token_usage.md) outlines best practices for throttling Hub calls.

### Handling Spot-Instance Pre-emption

```python
import os
from pathlib import Path
from accelerate import Accelerator

accelerator = Accelerator()
if accelerator.state.deepspeed_plugin is not None:
    # DeepSpeed automatically checkpoints on pre-emption

    pass
else:
    # Manual checkpoint

    checkpoint_dir = Path("./ckpt")
    checkpoint_dir.mkdir(parents=True, exist_ok=True)
    trainer.save_model(checkpoint_dir)

```

*Key reference*: the **hardware guide** and [`skills/hugging-face-jobs/references/hub_saving.md`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/references/hub_saving.md) detail checkpoint persistence strategies.

## Key Troubleshooting Resources in huggingface/skills

The following files in the `huggingface/skills` repository provide definitive reference material for diagnosing and resolving training failures:

| File | Purpose | Direct Link |
|------|---------|-------------|
| [`skills/hugging-face-jobs/references/troubleshooting.md`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/references/troubleshooting.md) | Consolidated list of known failure modes, debugging tips, and common error messages. | [troubleshooting.md](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/references/troubleshooting.md) |
| [`skills/hugging-face-jobs/references/token_usage.md`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/references/token_usage.md) | Describes HF token quotas, rate-limiting, and best practices for Hub interactions. | [token_usage.md](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/references/token_usage.md) |
| [`skills/hugging-face-jobs/references/hardware_guide.md`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/references/hardware_guide.md) | Recommendations for GPU/CPU specs, CUDA/cuDNN compatibility, and spot-instance considerations. | [hardware_guide.md](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/references/hardware_guide.md) |
| [`skills/hugging-face-jobs/references/hub_saving.md`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/references/hub_saving.md) | Strategies for checkpoint management, pushing to the Hub, and avoiding storage-related errors. | [hub_saving.md](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/references/hub_saving.md) |
| [`skills/hugging-face-jobs/scripts/generate-responses.py`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/scripts/generate-responses.py) | Example script showing a typical training loop and how to capture logs for debugging. | [generate-responses.py](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/scripts/generate-responses.py) |
| [`skills/hugging-face-jobs/scripts/cot-self-instruct.py`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/scripts/cot-self-instruct.py) | Demonstrates self-instructed fine-tuning; includes checkpoint callbacks useful for pre-emption handling. | [cot-self-instruct.py](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/scripts/cot-self-instruct.py) |

## Summary

Resilient Hugging Face training requires proactive handling of resource constraints, data validation, and network reliability. The key takeaways for troubleshooting **common failure modes for Hugging Face training jobs** include:

- **Mitigate OOM errors** by enabling `gradient_checkpointing_enable()` and using `streaming=True` for large datasets.
- **Prevent dataset loading failures** by pinning dataset revisions and validating authentication tokens against private repositories.
- **Avoid rate limiting** by caching Hub downloads with `hf_hub_download(..., cache_dir=...)` and throttling `push_to_hub` calls.
- **Ensure hardware compatibility** by aligning CUDA/cuDNN versions with PyTorch wheels as specified in the hardware guide.
- **Handle pre-emption** by implementing `accelerate` checkpointing callbacks and saving to persistent storage.

## Frequently Asked Questions

### What causes `CUDA out of memory` errors during Hugging Face training?

GPU OOM errors occur when the model parameters, activation buffers, or optimizer states exceed the available VRAM. According to the `huggingface/skills` source code, you can resolve this by reducing `per_device_train_batch_size`, enabling `gradient_accumulation_steps`, or activating `model.gradient_checkpointing_enable()` to trade computation for memory efficiency.

### How do I fix `DatasetNotFoundError` when loading from the Hub?

This error typically indicates a misspelled repository name, a private dataset without proper authentication, or an expired token. The troubleshooting guide in [`skills/hugging-face-jobs/references/troubleshooting.md`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-jobs/references/troubleshooting.md) recommends verifying the exact Hub identifier and ensuring your `HF_TOKEN` environment variable contains a valid read-only or write-capable token.

### Why does my training job stall with `429 Too Many Requests`?

The Hugging Face Hub imposes rate limits on API calls. Jobs that repeatedly download large checkpoints or call `push_to_hub` too frequently trigger throttling. As implemented in `huggingface/skills`, you should cache files locally using `hf_hub_download(..., cache_dir=...)` and limit Hub pushes to epoch boundaries or specific step intervals.

### How do I prevent checkpoint corruption on spot instances?

Spot instances can terminate without warning, causing incomplete checkpoint writes. The `huggingface/skills` repository recommends using the `accelerate` library’s `resume_from_checkpoint` flag and manually saving checkpoints to persistent storage (e.g., S3) using `trainer.save_model()` within pre-emption handlers.