# How Herbie Implements Error Handling and Retry Logic for Downloads

> Learn how Herbie handles download errors with URL validation and explicit exceptions. Discover its user-delegated retry approach for robust download management.

- Repository: [Brian Blaylock/herbie](https://github.com/blaylockbk/herbie)
- Tags: how-to-guide
- Published: 2026-02-26

---

**Herbie detects download failures through proactive URL validation and explicit exception raising, but delegates retry responsibility to the user rather than implementing automatic retry loops.**

The Herbie library ([blaylockbk/herbie](https://github.com/blaylockbk/herbie)) provides a Python interface for downloading meteorological GRIB2 data from various model archives. Understanding how Herbie handles network failures and download errors is essential for building resilient data pipelines. This article examines the error detection mechanisms and retry strategies implemented in the Herbie source code, specifically within [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) and related modules.

## Proactive Error Detection Before Download

Herbie emphasizes **error detection** over recovery, implementing multiple validation layers to identify unavailable resources before initiating full downloads.

### URL Validation with HEAD Requests

The `_check_grib` method in [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) (lines 23-45) performs pre-flight validation by sending a `HEAD` request to the prospective GRIB file URL. It inspects the `Content-Length` header to confirm file availability:

```python

# Conceptual implementation based on source analysis

def _check_grib(self, url):
    response = requests.head(url)
    if response.status_code == 200 and 'Content-Length' in response.headers:
        return True
    return False

```

If the check fails, the method returns `False` and the file is considered unavailable, preventing wasted bandwidth on doomed requests.

### Index File Availability Checks

For subset downloads, `_check_idx` (lines 46-84 in [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py)) iterates over possible inventory suffixes, wrapping each request in a `try/except` block. It returns the first reachable `.idx` URL while suppressing non-critical errors:

```python

# Simplified logic from source

def _check_idx(self, url):
    for suffix in ['.idx', '.grb2.idx']:
        try:
            idx_url = url + suffix
            response = requests.head(idx_url)
            if response.status_code == 200:
                return idx_url
        except requests.RequestException:
            continue
    return None

```

When `verbose=True`, errors are printed to stderr, but the process continues rather than aborting, allowing fallback to alternative index formats.

### Pando Archive Connectivity

Before contacting the Pando archive, `_ping_pando` (lines 15-22) performs a lightweight `HEAD` request. Any exception is caught silently, allowing the download to proceed regardless of ping success, implementing a "fail-open" strategy for this optional connectivity check.

## Download Error Handling Strategies

Once validation passes, Herbie implements distinct error handling patterns for full-file versus subset downloads.

### Full-File Download Exceptions

The `download_with_requests` function (located in [`src/herbie/fast.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/fast.py), lines 67-78) streams files using `requests.get(..., stream=True)`. It explicitly raises `HTTPError` via `response.raise_for_status()`:

```python
import requests
from requests import HTTPError

def download_with_requests(url, outfile):
    response = requests.get(url, stream=True)
    try:
        response.raise_for_status()
    except HTTPError as e:
        # Exception propagates to caller

        raise
    # Stream content to file...

```

If the request fails, the exception propagates unmodified to the caller, who must handle the failure or retry.

### Subset Download Error Management

Within `Herbie.download`, the inner `subset` function implements granular error handling for partial data retrieval. It catches `IOError` and `requests.RequestException`, logs a message when `verbose=True`, and re-raises a `RuntimeError`:

```python

# Conceptual representation of subset error handling

def subset(url):
    try:
        response = requests.get(url, timeout=30)
        response.raise_for_status()
        # Process subset...

    except (IOError, requests.RequestException) as e:
        if verbose:
            print(f"Subset download failed: {e}")
        raise RuntimeError(f"Failed to download subset: {e}")

```

Notably, **no automatic retry is performed**; the caller must invoke `download` again to attempt recovery.

### Missing File Handling

The `Herbie.download` method accepts an `errors` parameter controlling behavior when `self.grib` or `self.idx` is `None`:

- **`errors="warn"`**: Returns early with a warning message, returning `None`
- **`errors="raise"`**: Raises a `ValueError` immediately

This allows pipelines to choose between fail-fast and fail-tolerant behaviors.

## Retry Logic Implementation

Herbie deliberately **does not implement automatic retry loops** with exponential back-off or circuit breakers. Instead, it follows a "detect and delegate" philosophy:

1. **No built-in retries**: The library relies on `requests` default behavior (single attempt) and immediate exception raising
2. **Caller responsibility**: Users must wrap `Herbie.download` calls in their own retry mechanisms
3. **Timeout protection**: Subset downloads use a 30-second timeout (`requests.get(..., timeout=30)`), after which `requests.Timeout` is caught by the generic `RequestException` handler

For production pipelines requiring resilience, implement a wrapper using libraries like `tenacity` or a simple loop:

```python
import time
from herbie import Herbie

def robust_download(attempts=3, delay=5, **herbie_kwargs):
    for attempt in range(1, attempts + 1):
        try:
            h = Herbie(**herbie_kwargs)
            return h.download()
        except Exception as exc:
            if attempt == attempts:
                raise
            print(f"Attempt {attempt} failed ({exc}); retrying in {delay}s...")
            time.sleep(delay)

# Usage

path = robust_download(date="2024-04-01", model="gfs", fxx=12)

```

## Practical Code Examples

### Basic Download with Error Warnings

```python
from herbie import Herbie

h = Herbie(date="2024-04-01", model="hrrr", fxx=6)

# Returns None if file unavailable, prints warning

local_path = h.download(errors="warn")
print(local_path)

```

### Strict Error Handling

```python
h = Herbie(date="2024-04-01", model="gfs", fxx=0)
try:
    # Raises ValueError immediately if GRIB/IDX missing

    path = h.download(errors="raise")
except ValueError as e:
    print(f"Download aborted: {e}")

```

### Subset Download with Exception Handling

```python
h = Herbie(date="2024-04-01", model="gfs", fxx=0)
try:
    # 30-second timeout applied internally

    subset_path = h.download(search=":TMP:2 m:", errors="raise")
except RuntimeError as e:
    print(f"Subset failed: {e}")

```

## Summary

- **Proactive validation**: Herbie validates URLs via `HEAD` requests in `_check_grib`, `_check_idx`, and `_ping_pando` before downloading
- **Explicit exceptions**: Download failures raise `HTTPError`, `RuntimeError`, or `ValueError` rather than returning silent failures
- **No automatic retries**: The library does not implement retry loops; callers must wrap `Herbie.download` in their own retry logic
- **Configurable failure modes**: The `errors` parameter allows choosing between warning (`"warn"`) and exception (`"raise"`) behaviors when files are missing
- **Timeout protection**: Subset downloads enforce a 30-second timeout to prevent hanging connections

## Frequently Asked Questions

### Does Herbie automatically retry failed downloads?

No, Herbie does not implement automatic retry logic with exponential back-off or circuit breakers. The library performs single-attempt downloads using the `requests` library and raises exceptions immediately upon failure. Users must implement their own retry wrappers around `Herbie.download` if resilience is required.

### What exceptions does Herbie raise when a download fails?

Herbie raises several exception types depending on the failure mode. `ValueError` is raised when `errors="raise"` is set and the GRIB or index file is unavailable. `RuntimeError` is raised when subset downloads fail due to network issues or IO problems. `requests.HTTPError` is raised when the HTTP response status indicates a server error during full-file downloads.

### How does Herbie validate that a file exists before downloading?

Before initiating downloads, Herbie performs proactive validation through `HEAD` request checks. The `_check_grib` method in [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) validates the GRIB file URL and `Content-Length` header, while `_check_idx` tests potential index file suffixes. These checks prevent wasted bandwidth on unavailable resources.

### Can I configure how long Herbie waits before timing out a download?

Subset downloads enforce a default 30-second timeout via the `timeout` parameter in `requests.get`. However, full-file downloads using `download_with_requests` do not specify an explicit timeout by default, relying on the `requests` library defaults. Users cannot directly configure these timeouts through the `Herbie.download` API without implementing custom download wrappers.