How Herbie Handles Multi-Source Data Discovery and Download Priority

Herbie uses a configurable priority list to filter and reorder the SOURCES dictionary, trying archives in sequence until it finds the requested GRIB2 file.

Herbie is an open-source Python library designed to fetch GRIB2 model data from diverse public archives. Understanding how it performs multi-source data discovery is essential for optimizing download speeds and ensuring resilience when primary archives are unavailable.

Where Model Sources Are Defined

The SOURCES Dictionary in Model Templates

Each supported model in Herbie defines a SOURCES dictionary inside its template file located in src/herbie/models/<model>.py. This dictionary maps human-readable source names to URL patterns that resolve to concrete file paths based on the requested date, product, and forecast hour.

For example, the RAP model template in src/herbie/models/rap.py defines:

SOURCES = {
    "aws": "s3://noaa-rap-models/...",
    "nomads": "https://nomads.ncep.noaa.gov/pub/data/nccf/com/rap/prod/...",
    "pando": "https://pando-ruc.gsfc.nasa.gov/...",
}

The Herbie constructor loads these templates dynamically. As implemented in src/herbie/core.py at lines 71-78, the initialization calls getattr(model_templates, self.model).template(self) to attach the appropriate SOURCES dictionary to the instance.

How the Priority Mechanism Works

Constructor Arguments and Global Defaults

The priority behavior is controlled through the priority parameter in the Herbie class constructor. According to src/herbie/core.py at lines 58-62, the constructor stores the user-provided list:

self.priority = priority          # e.g., ["aws", "nomads"]

self.save_dir = Path(save_dir).expand()
self.overwrite = overwrite
self.verbose = verbose

If no priority is specified, Herbie falls back to the order defined in the model's SOURCES dictionary. The library-wide default priority is ["aws", "nomads"], defined in src/herbie/latest.py and read from the global configuration in src/herbie/__init__.py.

Filtering and Reordering Logic

Before searching for files, Herbie applies the priority list to filter and reorder the SOURCES dictionary. This logic appears in src/herbie/core.py within the find_grib() method:

if self.priority is not None:
    self.SOURCES = {
        key: self.SOURCES[key] for key in self.priority if key in self.SOURCES
    }

The same filtering block appears in find_idx() at lines 79-83. This dictionary comprehension both filters out sources not present in the priority list and reorders the remaining sources according to the user's specified sequence.

The Multi-Source Discovery Workflow

Local Cache Verification

Before querying remote archives, Herbie checks for existing local files. If the requested file exists in save_dir and overwrite=False, the library returns the cached path immediately without initiating network requests.

Source Iteration and Availability Checks

After applying the priority filter, Herbie iterates over the ordered SOURCES dictionary. For each source, it performs lightweight availability checks using _check_grib() and _check_idx() methods. These functions issue HTTP HEAD requests to confirm file existence without downloading the full payload.

The code loops over the ordered dictionary (insertion order is preserved in Python 3.7+). For each source:

• Pando sources receive special handling to avoid TLS handshake failures
• Azure URLs require SAS token generation via requests.get before validation
• Local paths are verified using standard Path operations
• All other sources use raw URL strings with HTTP HEAD verification

Fallback Behavior

The first source yielding a successful check stops the loop. Herbie stores the resulting URL or Path in self.grib and the source name in self.grib_source (or self.idx / self.idx_source for index files). If no sources contain the requested file, the methods return (None, None), allowing downstream code to handle the missing data gracefully.

Configuring Download Priority

Users can specify priority through three mechanisms:

Method	Implementation
Constructor argument	Herbie(date, model="gfs", priority=["google", "aws"])
Command-line interface	herbie --model gfs --priority google aws
Configuration file	Set priority = ["aws", "nomads", "google"] in ~/.config/herbie/config.cfg

Practical Code Examples

Example 1: Use Default Priority

from herbie import Herbie

# Searches AWS first, then NOMADS based on default configuration

h = Herbie("2024-04-01", model="gfs", fxx=6)
print(h.grib_source)  # Output: "aws" or "nomads"

Example 2: Prioritize Google Cloud

h = Herbie(
    "2024-04-01",
    model="gfs",
    fxx=6,
    priority=["google", "aws", "nomads"]   # Custom order

)
print(h.grib_source)  # Output: "google" if file exists there

Example 3: Inspect Filtered Sources

h = Herbie("2024-04-01", model="gfs", priority=["azure", "pando"])
print(list(h.SOURCES.keys()))

# Output: ['azure', 'pando']

Example 4: Local Cache Only

h = Herbie(
    "2024-04-01",
    model="gfs",
    fxx=6,
    overwrite=False        # Prevents re-downloading existing files

)

# Returns local path if cached, otherwise searches remote sources

print(h.grib)  # Path object pointing to local or remote file

Summary

• Herbie performs multi-source data discovery by iterating through a prioritized list of archives defined in each model's SOURCES dictionary.
• The priority parameter filters and reorders sources at runtime, allowing users to force specific download sequences via constructor arguments, CLI flags, or configuration files.
• Source availability is verified via lightweight HTTP HEAD requests in _check_grib() and _check_idx() before any data transfer begins.
• The system provides automatic fallback: if the primary source is unavailable, Herbie proceeds to the next source in the priority list until it finds the file or exhausts all options.

Frequently Asked Questions

What happens if no priority list is specified?

If the priority parameter is omitted, Herbie falls back to the order defined in the model template's SOURCES dictionary. The library-wide default is ["aws", "nomads"], defined in src/herbie/latest.py, which is used when no user configuration overrides it.

How does Herbie check if a file exists on a remote server?

Herbie uses the _check_grib() and _check_idx() methods to issue HTTP HEAD requests against candidate URLs. This lightweight check confirms file existence and accessibility without downloading the full payload. Special handling exists for Azure (SAS token generation) and Pando (TLS handshake optimization).

Can I use a local file path as a source?

Yes. If the "local" key is included in the SOURCES dictionary or priority list, Herbie verifies the file path directly on disk using standard Path operations. This is useful for working with pre-downloaded archives or private datasets that mirror public formats.

What is the default priority order in Herbie?

The default priority order is ["aws", "nomads"], which prioritizes Amazon Web Services (AWS) Open Data first, followed by NOAA's NOMADS server. This default is hardcoded in src/herbie/latest.py and can be overridden via the constructor, CLI, or configuration file.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}