How Herbie Implements Error Handling and Retry Logic for Downloads

Herbie detects download failures through proactive URL validation and explicit exception raising, but delegates retry responsibility to the user rather than implementing automatic retry loops.

The Herbie library (blaylockbk/herbie) provides a Python interface for downloading meteorological GRIB2 data from various model archives. Understanding how Herbie handles network failures and download errors is essential for building resilient data pipelines. This article examines the error detection mechanisms and retry strategies implemented in the Herbie source code, specifically within src/herbie/core.py and related modules.

Proactive Error Detection Before Download

Herbie emphasizes error detection over recovery, implementing multiple validation layers to identify unavailable resources before initiating full downloads.

URL Validation with HEAD Requests

The _check_grib method in src/herbie/core.py (lines 23-45) performs pre-flight validation by sending a HEAD request to the prospective GRIB file URL. It inspects the Content-Length header to confirm file availability:

# Conceptual implementation based on source analysis

def _check_grib(self, url):
    response = requests.head(url)
    if response.status_code == 200 and 'Content-Length' in response.headers:
        return True
    return False

If the check fails, the method returns False and the file is considered unavailable, preventing wasted bandwidth on doomed requests.

Index File Availability Checks

For subset downloads, _check_idx (lines 46-84 in src/herbie/core.py) iterates over possible inventory suffixes, wrapping each request in a try/except block. It returns the first reachable .idx URL while suppressing non-critical errors:

# Simplified logic from source

def _check_idx(self, url):
    for suffix in ['.idx', '.grb2.idx']:
        try:
            idx_url = url + suffix
            response = requests.head(idx_url)
            if response.status_code == 200:
                return idx_url
        except requests.RequestException:
            continue
    return None

When verbose=True, errors are printed to stderr, but the process continues rather than aborting, allowing fallback to alternative index formats.

Pando Archive Connectivity

Before contacting the Pando archive, _ping_pando (lines 15-22) performs a lightweight HEAD request. Any exception is caught silently, allowing the download to proceed regardless of ping success, implementing a "fail-open" strategy for this optional connectivity check.

Download Error Handling Strategies

Once validation passes, Herbie implements distinct error handling patterns for full-file versus subset downloads.

Full-File Download Exceptions

The download_with_requests function (located in src/herbie/fast.py, lines 67-78) streams files using requests.get(..., stream=True). It explicitly raises HTTPError via response.raise_for_status():

import requests
from requests import HTTPError

def download_with_requests(url, outfile):
    response = requests.get(url, stream=True)
    try:
        response.raise_for_status()
    except HTTPError as e:
        # Exception propagates to caller

        raise
    # Stream content to file...

If the request fails, the exception propagates unmodified to the caller, who must handle the failure or retry.

Subset Download Error Management

Within Herbie.download, the inner subset function implements granular error handling for partial data retrieval. It catches IOError and requests.RequestException, logs a message when verbose=True, and re-raises a RuntimeError:

# Conceptual representation of subset error handling

def subset(url):
    try:
        response = requests.get(url, timeout=30)
        response.raise_for_status()
        # Process subset...

    except (IOError, requests.RequestException) as e:
        if verbose:
            print(f"Subset download failed: {e}")
        raise RuntimeError(f"Failed to download subset: {e}")

Notably, no automatic retry is performed; the caller must invoke download again to attempt recovery.

Missing File Handling

The Herbie.download method accepts an errors parameter controlling behavior when self.grib or self.idx is None:

• errors="warn": Returns early with a warning message, returning None
• errors="raise": Raises a ValueError immediately

This allows pipelines to choose between fail-fast and fail-tolerant behaviors.

Retry Logic Implementation

Herbie deliberately does not implement automatic retry loops with exponential back-off or circuit breakers. Instead, it follows a "detect and delegate" philosophy:

1. No built-in retries: The library relies on requests default behavior (single attempt) and immediate exception raising
2. Caller responsibility: Users must wrap Herbie.download calls in their own retry mechanisms
3. Timeout protection: Subset downloads use a 30-second timeout (requests.get(..., timeout=30)), after which requests.Timeout is caught by the generic RequestException handler

For production pipelines requiring resilience, implement a wrapper using libraries like tenacity or a simple loop:

import time
from herbie import Herbie

def robust_download(attempts=3, delay=5, **herbie_kwargs):
    for attempt in range(1, attempts + 1):
        try:
            h = Herbie(**herbie_kwargs)
            return h.download()
        except Exception as exc:
            if attempt == attempts:
                raise
            print(f"Attempt {attempt} failed ({exc}); retrying in {delay}s...")
            time.sleep(delay)

# Usage

path = robust_download(date="2024-04-01", model="gfs", fxx=12)

Practical Code Examples

Basic Download with Error Warnings

from herbie import Herbie

h = Herbie(date="2024-04-01", model="hrrr", fxx=6)

# Returns None if file unavailable, prints warning

local_path = h.download(errors="warn")
print(local_path)

Strict Error Handling

h = Herbie(date="2024-04-01", model="gfs", fxx=0)
try:
    # Raises ValueError immediately if GRIB/IDX missing

    path = h.download(errors="raise")
except ValueError as e:
    print(f"Download aborted: {e}")

Subset Download with Exception Handling

h = Herbie(date="2024-04-01", model="gfs", fxx=0)
try:
    # 30-second timeout applied internally

    subset_path = h.download(search=":TMP:2 m:", errors="raise")
except RuntimeError as e:
    print(f"Subset failed: {e}")

Summary

• Proactive validation: Herbie validates URLs via HEAD requests in _check_grib, _check_idx, and _ping_pando before downloading
• Explicit exceptions: Download failures raise HTTPError, RuntimeError, or ValueError rather than returning silent failures
• No automatic retries: The library does not implement retry loops; callers must wrap Herbie.download in their own retry logic
• Configurable failure modes: The errors parameter allows choosing between warning ("warn") and exception ("raise") behaviors when files are missing
• Timeout protection: Subset downloads enforce a 30-second timeout to prevent hanging connections

Frequently Asked Questions

Does Herbie automatically retry failed downloads?

No, Herbie does not implement automatic retry logic with exponential back-off or circuit breakers. The library performs single-attempt downloads using the requests library and raises exceptions immediately upon failure. Users must implement their own retry wrappers around Herbie.download if resilience is required.

What exceptions does Herbie raise when a download fails?

Herbie raises several exception types depending on the failure mode. ValueError is raised when errors="raise" is set and the GRIB or index file is unavailable. RuntimeError is raised when subset downloads fail due to network issues or IO problems. requests.HTTPError is raised when the HTTP response status indicates a server error during full-file downloads.

How does Herbie validate that a file exists before downloading?

Before initiating downloads, Herbie performs proactive validation through HEAD request checks. The _check_grib method in src/herbie/core.py validates the GRIB file URL and Content-Length header, while _check_idx tests potential index file suffixes. These checks prevent wasted bandwidth on unavailable resources.

Can I configure how long Herbie waits before timing out a download?

Subset downloads enforce a default 30-second timeout via the timeout parameter in requests.get. However, full-file downloads using download_with_requests do not specify an explicit timeout by default, relying on the requests library defaults. Users cannot directly configure these timeouts through the Herbie.download API without implementing custom download wrappers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}