# FastHerbie Internal Mechanisms for Bulk and Multithreaded Operations Explained

> Discover FastHerbie internals for bulk and multithreaded operations. Learn how it uses ThreadPoolExecutor for parallel Herbie object instantiation, downloads, and xarray dataset construction.

- Repository: [Brian Blaylock/herbie](https://github.com/blaylockbk/herbie)
- Tags: internals
- Published: 2026-02-26

---

**FastHerbie orchestrates bulk GRIB2 file operations by spawning a `ThreadPoolExecutor` to instantiate multiple `Herbie` objects in parallel, then reuses the same concurrency pattern for parallel downloads and xarray dataset construction.**

FastHerbie is a high-performance orchestration layer in the `blaylockbk/herbie` repository designed for bulk meteorological data retrieval. Unlike the standard `Herbie` class which handles single files, FastHerbie manages matrices of forecast runs and lead times, leveraging Python's `concurrent.futures` to parallelize network I/O and CPU-bound parsing operations.

## How FastHerbie Manages Bulk Object Creation

### Input Validation and Task Calculation

Before spawning threads, FastHerbie validates inputs through `_validate_DATES` and `_validate_fxx` (lines [36-59](https://github.com/blaylockbk/herbie/blob/main/src/herbie/fast.py#L36-L59) in [`src/herbie/fast.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/fast.py)). These helpers coerce user input into list-like objects and compute the total task count:

```python
self.tasks = len(DATES) * len(fxx)

```

This `self.tasks` value drives thread pool sizing throughout the object's lifecycle.

### Parallel Instantiation with ThreadPoolExecutor

The constructor caps thread count at `max_threads` and launches a `ThreadPoolExecutor` (lines [90-118](https://github.com/blaylockbk/herbie/blob/main/src/herbie/fast.py#L90-L118)):

```python
threads = min(self.tasks, max_threads)
with ThreadPoolExecutor(threads) as exe:
    futures = [
        exe.submit(Herbie, date=DATE, f=f, **kwargs)
        for DATE in DATES
        for f in fxx
    ]
    for future in as_completed(futures):
        if future.exception() is None:
            self.objects.append(future.result())
        else:
            log.error(f"Exception has occured : {future.exception()}")

```

Each thread executes `Herbie(date=..., fxx=..., **kwargs)`, creating a full `Herbie` object that resolves remote GRIB2 locations and inventory indices independently.

### Error Handling and Aggregation

FastHerbie implements fail-soft behavior: exceptions during individual `Herbie` instantiation are logged via `log.error` but do not abort the bulk operation. Successful objects accumulate in `self.objects`, preserving partial results even when specific forecast runs are missing from remote servers.

## Post-Creation Bookkeeping and Sorting

### Lexicographic Sorting Strategy

After all futures complete, FastHerbie sorts the object list to ensure predictable matrix ordering (lines [119-171](https://github.com/blaylockbk/herbie/blob/main/src/herbie/fast.py#L119-L171)):

```python
self.objects.sort(key=lambda H: H.fxx)   # lead-time order

self.objects.sort(key=lambda H: H.date)  # then date order

```

This double-sort produces a lexicographic `[date, fxx]` ordering, creating a logical time-series matrix where rows represent run dates and columns represent forecast lead times.

### File Availability Slicing

FastHerbie generates two convenience slices for downstream operations:

```python
self.file_exists = [H for H in self.objects if H.grib is not None]
self.file_not_exists = [H for H in self.objects if H.grib is None]

```

These lists allow `download()` and `xarray()` methods to skip non-existent files without redundant network checks.

## Parallel Download Operations

### Thread Pool Configuration

The `download` method (lines [202-250](https://github.com/blaylockbk/herbie/blob/main/src/herbie/fast.py#L202-L250)) reuses the task-count logic to size its thread pool:

```python
def download(self, search=None, *, max_threads=20, **download_kwargs):
    threads = min(self.tasks, max_threads)
    outFiles = []
    with ThreadPoolExecutor(threads) as exe:
        futures = [
            exe.submit(H.download, search, **download_kwargs) 
            for H in self.file_exists
        ]
        for future in as_completed(futures):
            if future.exception() is None:
                outFiles.append(future.result())

```

### HTTP Range Request Handling

Each `Herbie.download` call invoked by the executor handles the actual transfer logic defined in [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py). When the `search` parameter is provided, `Herbie` uses HTTP Range requests to download only specific GRIB2 messages; otherwise, it performs full-file retrieval via `download_with_requests` from [`src/herbie/misc.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/misc.py).

## Parallel Xarray Ingestion and Dataset Merging

### Concurrent GRIB2 Reading

The `xarray` method (lines [252-300](https://github.com/blaylockbk/herbie/blob/main/src/herbie/fast.py#L252-L300)) follows the same executor pattern but returns aggregated datasets rather than file paths:

```python
def xarray(self, search, *, max_threads=None, **xarray_kwargs):
    if max_threads:
        threads = min(self.tasks, max_threads)
        with ThreadPoolExecutor(max_threads) as exe:
            futures = [
                exe.submit(H.xarray, **xarray_kwargs) 
                for H in self.file_exists
            ]
            ds_list = [future.result() for future in as_completed(futures)]
    else:
        ds_list = [H.xarray(**xarray_kwargs) for H in self.file_exists]

```

When `max_threads` is `None`, the method falls back to serial processing, which aids debugging and reduces memory pressure on resource-constrained systems.

### Hyper-cube Normalization and Concatenation

After parallel ingestion, FastHerbie normalizes the resulting "hyper-cubes" (datasets with varying level types). It groups datasets by level type, sorts by `step` and `time`, reshapes into a `[len(DATES), len(fxx)]` matrix, and concatenates using `xr.combine_nested`. This produces a unified `xarray.Dataset` with dimensions `time` (run date) and `step` (forecast lead), ready for meteorological analysis.

## Key Source Files and Architecture

| File | Purpose | Key Components |
|------|---------|----------------|
| [`src/herbie/fast.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/fast.py) | Implements `FastHerbie` class and multithreaded orchestration | `__init__`, `download`, `xarray`, `ThreadPoolExecutor` logic |
| [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) | Defines `Herbie` class for single-file operations | `download`, `xarray`, inventory resolution, URL building |
| [`src/herbie/misc.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/misc.py) | Utility functions for HTTP transfers | `download_with_requests` |
| [`src/herbie/help.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/help.py) | Search syntax documentation | Inventory search helpers |
| [`src/herbie/wgrib2.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/wgrib2.py) | External tool wrapper | Inventory file generation |

## Practical Code Examples

### Example 1: Creating a Matrix of HRRR Forecast Files

```python
from herbie.fast import FastHerbie
import pandas as pd

# 48 hourly runs ending at the most recent full hour

now = pd.Timestamp.now("utc").floor("1h")
dates = pd.date_range(end=now, periods=48, freq="1h")

fh = FastHerbie(dates, fxx=[0], model="hrrr", product="sfc")
print(f"Found {len(fh.file_exists)} GRIB files out of {len(fh)} tasks")

```

This creates **48** `Herbie` objects in parallel, validating file existence across the HRRR surface product suite.

### Example 2: Parallel Download with Thread Control

```python

# Continue from the previous FastHerbie instance `fh`

downloaded = fh.download(max_threads=20)   # up to 20 concurrent HTTP streams

print(f"Downloaded {len(downloaded)} files")

```

The `max_threads` parameter caps concurrency to prevent overwhelming local I/O or remote server rate limits.

### Example 3: Bulk Xarray Dataset Construction

```python

# Load all files into memory in parallel

ds = fh.xarray(search="TMP:2 m", max_threads=15)

# Result is a combined Dataset with time and step dimensions

if isinstance(ds, list):
    ds = ds[0]   # select first level type if multiple returned

print(ds)

```

This executes **15** concurrent `Herbie.xarray` calls, then normalizes and concatenates the results into a unified Dataset with `time` (run date) and `step` (forecast lead) dimensions.

## Summary

- **FastHerbie** acts as a multithreaded orchestration layer over the single-file `Herbie` class, located in [`src/herbie/fast.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/fast.py).
- **Bulk creation** uses `ThreadPoolExecutor` to instantiate `Herbie` objects for every combination of `DATES` and `fxx`, with automatic thread capping via `min(self.tasks, max_threads)`.
- **Error resilience** is implemented through `as_completed` iteration, logging exceptions without aborting the entire batch operation.
- **Post-processing** includes lexicographic sorting by date and lead time, plus slicing into `file_exists` and `file_not_exists` lists.
- **Parallel I/O** for downloads and xarray ingestion reuses the same executor pattern, supporting HTTP Range requests for subset downloads and hyper-cube normalization for dataset merging.

## Frequently Asked Questions

### How does FastHerbie handle failed downloads or missing files?

FastHerbie implements a fail-soft strategy during both object creation and download phases. When instantiating `Herbie` objects in bulk, exceptions caught via `future.exception()` are logged with full tracebacks but do not halt the `ThreadPoolExecutor`, allowing valid objects to accumulate in `self.objects`. During downloads, only objects present in `self.file_exists` (those with resolved GRIB URLs) are submitted to the thread pool, automatically skipping unavailable files.

### What is the difference between FastHerbie and the standard Herbie class?

The standard `Herbie` class in [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) handles single-file operations: resolving a specific GRIB2 URL for a given run date and forecast lead, downloading that file, and converting it to xarray. `FastHerbie` in [`src/herbie/fast.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/fast.py) is a bulk orchestrator that creates many `Herbie` instances across a matrix of dates and lead times, then coordinates parallel downloads and dataset merging using `ThreadPoolExecutor`.

### Can I control the number of threads used for bulk operations?

Yes, both the constructor and I/O methods expose a `max_threads` parameter. During initialization, the actual thread count is calculated as `min(self.tasks, max_threads)` to prevent spawning idle workers. The `download()` and `xarray()` methods accept their own `max_threads` arguments, allowing you to tune concurrency separately for object creation versus I/O operations based on network bandwidth or local system resources.

### How does FastHerbie merge multiple GRIB2 files into a single xarray Dataset?

After parallel ingestion via `ThreadPoolExecutor`, the `xarray()` method normalizes the returned datasets by grouping them according to level type (e.g., surface, pressure levels). It sorts each group by `step` and `time`, reshapes the data into a matrix of shape `[len(DATES), len(fxx)]`, and concatenates using `xr.combine_nested`. This produces a unified Dataset with `time` (run initialization) and `step` (forecast lead) dimensions, regardless of how many individual GRIB2 files were processed.