# How Herbie Handles Multi-Source Data Discovery and Download Priority

> Learn how Herbie manages multi-source data discovery and download priority. Herbie uses a configurable priority list to find your GRIB2 files efficiently.

- Repository: [Brian Blaylock/herbie](https://github.com/blaylockbk/herbie)
- Tags: how-to-guide
- Published: 2026-02-26

---

**Herbie uses a configurable priority list to filter and reorder the SOURCES dictionary, trying archives in sequence until it finds the requested GRIB2 file.**

Herbie is an open-source Python library designed to fetch GRIB2 model data from diverse public archives. Understanding how it performs **multi-source data discovery** is essential for optimizing download speeds and ensuring resilience when primary archives are unavailable.

## Where Model Sources Are Defined

### The SOURCES Dictionary in Model Templates

Each supported model in Herbie defines a `SOURCES` dictionary inside its template file located in `src/herbie/models/<model>.py`. This dictionary maps human-readable source names to URL patterns that resolve to concrete file paths based on the requested date, product, and forecast hour.

For example, the RAP model template in [`src/herbie/models/rap.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/models/rap.py) defines:

```python
SOURCES = {
    "aws": "s3://noaa-rap-models/...",
    "nomads": "https://nomads.ncep.noaa.gov/pub/data/nccf/com/rap/prod/...",
    "pando": "https://pando-ruc.gsfc.nasa.gov/...",
}

```

The `Herbie` constructor loads these templates dynamically. As implemented in [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) at lines 71-78, the initialization calls `getattr(model_templates, self.model).template(self)` to attach the appropriate `SOURCES` dictionary to the instance.

## How the Priority Mechanism Works

### Constructor Arguments and Global Defaults

The priority behavior is controlled through the `priority` parameter in the `Herbie` class constructor. According to [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) at lines 58-62, the constructor stores the user-provided list:

```python
self.priority = priority          # e.g., ["aws", "nomads"]

self.save_dir = Path(save_dir).expand()
self.overwrite = overwrite
self.verbose = verbose

```

If no priority is specified, Herbie falls back to the order defined in the model's `SOURCES` dictionary. The library-wide default priority is `["aws", "nomads"]`, defined in [`src/herbie/latest.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/latest.py) and read from the global configuration in [`src/herbie/__init__.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/__init__.py).

### Filtering and Reordering Logic

Before searching for files, Herbie applies the priority list to filter and reorder the `SOURCES` dictionary. This logic appears in [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) within the `find_grib()` method:

```python
if self.priority is not None:
    self.SOURCES = {
        key: self.SOURCES[key] for key in self.priority if key in self.SOURCES
    }

```

The same filtering block appears in `find_idx()` at lines 79-83. This dictionary comprehension both **filters** out sources not present in the priority list and **reorders** the remaining sources according to the user's specified sequence.

## The Multi-Source Discovery Workflow

### Local Cache Verification

Before querying remote archives, Herbie checks for existing local files. If the requested file exists in `save_dir` and `overwrite=False`, the library returns the cached path immediately without initiating network requests.

### Source Iteration and Availability Checks

After applying the priority filter, Herbie iterates over the ordered `SOURCES` dictionary. For each source, it performs lightweight availability checks using **`_check_grib()`** and **`_check_idx()`** methods. These functions issue HTTP `HEAD` requests to confirm file existence without downloading the full payload.

The code loops over the ordered dictionary (insertion order is preserved in Python 3.7+). For each source:

- **Pando** sources receive special handling to avoid TLS handshake failures
- **Azure** URLs require SAS token generation via `requests.get` before validation
- **Local** paths are verified using standard `Path` operations
- All other sources use raw URL strings with HTTP `HEAD` verification

### Fallback Behavior

The first source yielding a successful check stops the loop. Herbie stores the resulting URL or `Path` in `self.grib` and the source name in `self.grib_source` (or `self.idx` / `self.idx_source` for index files). If no sources contain the requested file, the methods return `(None, None)`, allowing downstream code to handle the missing data gracefully.

## Configuring Download Priority

Users can specify priority through three mechanisms:

| Method | Implementation |
|--------|----------------|
| **Constructor argument** | `Herbie(date, model="gfs", priority=["google", "aws"])` |
| **Command-line interface** | `herbie --model gfs --priority google aws` |
| **Configuration file** | Set `priority = ["aws", "nomads", "google"]` in `~/.config/herbie/config.cfg` |

## Practical Code Examples

### Example 1: Use Default Priority

```python
from herbie import Herbie

# Searches AWS first, then NOMADS based on default configuration

h = Herbie("2024-04-01", model="gfs", fxx=6)
print(h.grib_source)  # Output: "aws" or "nomads"

```

### Example 2: Prioritize Google Cloud

```python
h = Herbie(
    "2024-04-01",
    model="gfs",
    fxx=6,
    priority=["google", "aws", "nomads"]   # Custom order

)
print(h.grib_source)  # Output: "google" if file exists there

```

### Example 3: Inspect Filtered Sources

```python
h = Herbie("2024-04-01", model="gfs", priority=["azure", "pando"])
print(list(h.SOURCES.keys()))

# Output: ['azure', 'pando']

```

### Example 4: Local Cache Only

```python
h = Herbie(
    "2024-04-01",
    model="gfs",
    fxx=6,
    overwrite=False        # Prevents re-downloading existing files

)

# Returns local path if cached, otherwise searches remote sources

print(h.grib)  # Path object pointing to local or remote file

```

## Summary

- **Herbie** performs **multi-source data discovery** by iterating through a prioritized list of archives defined in each model's `SOURCES` dictionary.
- The **`priority`** parameter filters and reorders sources at runtime, allowing users to force specific download sequences via constructor arguments, CLI flags, or configuration files.
- Source availability is verified via lightweight HTTP `HEAD` requests in `_check_grib()` and `_check_idx()` before any data transfer begins.
- The system provides automatic fallback: if the primary source is unavailable, Herbie proceeds to the next source in the priority list until it finds the file or exhausts all options.

## Frequently Asked Questions

### What happens if no priority list is specified?

If the `priority` parameter is omitted, Herbie falls back to the order defined in the model template's `SOURCES` dictionary. The library-wide default is `["aws", "nomads"]`, defined in [`src/herbie/latest.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/latest.py), which is used when no user configuration overrides it.

### How does Herbie check if a file exists on a remote server?

Herbie uses the `_check_grib()` and `_check_idx()` methods to issue HTTP `HEAD` requests against candidate URLs. This lightweight check confirms file existence and accessibility without downloading the full payload. Special handling exists for Azure (SAS token generation) and Pando (TLS handshake optimization).

### Can I use a local file path as a source?

Yes. If the `"local"` key is included in the `SOURCES` dictionary or priority list, Herbie verifies the file path directly on disk using standard `Path` operations. This is useful for working with pre-downloaded archives or private datasets that mirror public formats.

### What is the default priority order in Herbie?

The default priority order is `["aws", "nomads"]`, which prioritizes Amazon Web Services (AWS) Open Data first, followed by NOAA's NOMADS server. This default is hardcoded in [`src/herbie/latest.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/latest.py) and can be overridden via the constructor, CLI, or configuration file.