# Herbie Dask Integration: Distributed Computing with xarray

> Learn how to achieve distributed computing with xarray by integrating Herbie. Leverage Dask for efficient parallel computation and unlock the power of your data.

- Repository: [Brian Blaylock/herbie](https://github.com/blaylockbk/herbie)
- Tags: tutorial
- Published: 2026-02-26

---

**Herbie does not ship with built-in Dask integration, but its xarray-native architecture enables seamless distributed and parallel computation through standard Dask-xarray workflows.**

The `blaylockbk/herbie` library provides Pythonic access to numerical weather prediction data, returning pure `xarray.Dataset` objects that are immediately compatible with Dask's parallel computing engine. Because Herbie leverages cfgrib for GRIB2 decoding and returns standard xarray structures, you can integrate Dask for both lazy chunked analysis and distributed batch processing without modifying Herbie's source code.

## How Herbie Works with Dask

Herbie's Dask compatibility stems from three architectural decisions in the codebase:

- **xarray Backend**: The `Herbie.xarray()` method in [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) (lines 1226‑1249) opens GRIB2 files using cfgrib and returns a regular `xarray.Dataset`. This standard return type is the foundation for all Dask operations.
- **Lazy Chunking**: Any `xarray.Dataset` returned by Herbie can be lazily chunked using `.chunk()`, which converts underlying NumPy arrays to Dask arrays without loading data into memory.
- **Parallel Batch API**: The `FastHerbie` class in [`src/herbie/fast.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/fast.py) (lines 90‑140) already parallelizes object creation and file downloads using `ThreadPoolExecutor`, providing a ready-made structure for Dask delayed workflows.

Herbie lists **dask** as an optional dependency in [`environment.yml`](https://github.com/blaylockbk/herbie/blob/main/environment.yml) (line 17), confirming that the library is designed to support distributed computing environments.

## Method 1: Lazy Chunking with Dask Arrays

The simplest integration pattern involves loading data with Herbie, applying Dask chunks, and executing parallel computations. This approach keeps memory usage low by processing data in blocks.

```python
import dask
import herbie
import xarray as xr

# Load GRIB2 data as an xarray Dataset (eager load, but small)

ds = herbie.Herbie("2024-03-01 00:00", model="gfs").xarray("TMP:2 m", remove_grib=False)

# Convert to Dask arrays by chunking along the time dimension

ds_chunked = ds.chunk({"time": 10})

# Define lazy computation: spatial mean per time step

temp_mean = ds_chunked["TMP"].mean(dim=["lat", "lon"])

# Execute in parallel using the default Dask scheduler

result = temp_mean.compute()
print(result)

```

**Key implementation details**:

- The `.chunk()` method transforms the Dataset's underlying arrays into Dask arrays.
- Operations remain lazy until `.compute()` is called, triggering parallel execution across available cores or a distributed cluster.
- The `remove_grib=False` parameter preserves the downloaded file for subsequent reads, avoiding redundant downloads during iterative development.

## Method 2: Distributed Batch Processing with FastHerbie

For processing multiple forecast runs or ensemble members, wrap `Herbie.xarray()` calls with `dask.delayed` to distribute work across a cluster. This pattern leverages Herbie's `FastHerbie` helper for efficient object instantiation while offloading the heavy I/O to Dask workers.

```python
from dask import delayed, compute
import herbie

# Define temporal range for batch processing

dates = ["2024-03-01", "2024-03-02", "2024-03-03"]
fxx = [0, 6, 12]

# Initialize FastHerbie with multithreading for object creation

fh = herbie.FastHerbie(dates, fxx=fxx, max_threads=20, model="hrrr")

# Wrap each xarray read in a Dask delayed object

delayed_datasets = [
    delayed(H.xarray)("TMP:2 m", remove_grib=False) 
    for H in fh.file_exists
]

# Compute all datasets in parallel across the cluster

datasets = compute(*delayed_datasets, scheduler="distributed")

# Concatenate results along the forecast lead dimension

combined = xr.concat(datasets, dim="step")

```

**Implementation notes**:

- `FastHerbie` uses `ThreadPoolExecutor` (as implemented in [`src/herbie/fast.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/fast.py)) to rapidly validate file existence and create `Herbie` objects.
- Wrapping `H.xarray()` with `delayed` converts the eager I/O operation into a lazy task graph node.
- Use `scheduler="distributed"` when connected to a Dask cluster; `scheduler="threads"` suffices for local parallel execution.

## Method 3: Full Distributed Cluster Workflows

For production environments with dedicated Dask clusters, use `Client.submit()` to distribute `Herbie.xarray()` calls directly to workers. This eliminates the intermediate thread pool and allows Dask to manage all parallelism.

```python
from dask.distributed import Client
import herbie
import xarray as xr

# Connect to an existing Dask cluster

client = Client("tcp://scheduler-host:8786")

# Create FastHerbie instance for batch definition

fh = herbie.FastHerbie(["2024-03-01"], fxx=[0, 6, 12], model="gfs")

# Submit xarray reads directly to cluster workers

futures = [
    client.submit(H.xarray, "TMP:2 m", remove_grib=False) 
    for H in fh.file_exists
]

# Gather results (Dataset objects already in worker memory)

datasets = client.gather(futures)

# Perform additional Dask-native operations

combined = xr.concat(datasets, dim="step").chunk({"time": 5})
temp_std = combined["TMP"].std(dim="time")
print(temp_std.compute())

```

**Critical distinctions**:

- `client.submit()` schedules the entire `Herbie.xarray` execution on remote workers, reducing data transfer overhead.
- The resulting Datasets reside in distributed memory and can be further chunked and processed using standard xarray-Dask APIs.
- This pattern requires Dask to be installed on all worker nodes, as indicated by the optional dependency declaration in [`environment.yml`](https://github.com/blaylockbk/herbie/blob/main/environment.yml).

## Performance Considerations

Optimize Herbie Dask integration with these specific configurations:

- **Chunk Sizes**: Match Dask chunks to GRIB2 message boundaries when possible. Pass `backend_kwargs={"chunks": {"time": 1}}` to `Herbie.xarray()` for fine-grained control at load time.
- **File Cleanup**: Set `remove_grib=False` during development to avoid re-downloading data, but enable cleanup in production to prevent disk space exhaustion on worker nodes.
- **Memory Management**: The `FastHerbie` class materializes `Herbie` objects locally before distributing work; ensure the object list fits comfortably in the client process memory before submitting to the cluster.

## Summary

- Herbie returns standard `xarray.Dataset` objects from `Herbie.xarray()`, making them immediately compatible with Dask's parallel computing engine.
- No built-in Dask integration is required; simply call `.chunk()` on any Herbie-produced Dataset to enable lazy, distributed computation.
- Use `dask.delayed` with `FastHerbie` for parallel batch processing of multiple forecast times or models.
- For cluster computing, submit `Herbie.xarray()` calls directly via `Client.submit()` to distribute GRIB2 decoding across workers.
- Herbie explicitly supports Dask as an optional dependency, confirming architectural compatibility.

## Frequently Asked Questions

### Does Herbie require Dask to run?

No. Herbie operates independently as a data access library. Dask is listed as an optional dependency in [`environment.yml`](https://github.com/blaylockbk/herbie/blob/main/environment.yml) (line 17) and is only required if you want to perform distributed or parallel computations on the returned xarray Datasets.

### Can I use Dask with Herbie's FastHerbie class?

Yes. `FastHerbie` (defined in [`src/herbie/fast.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/fast.py)) uses `ThreadPoolExecutor` for local parallelism, but you can wrap its output with `dask.delayed` or `client.submit()` to distribute work across a Dask cluster. The class provides the list of `Herbie` objects needed for batch workflows.

### What file formats does Herbie use with Dask?

Herbie primarily decodes GRIB2 files using the cfgrib engine. The resulting Dataset contains Dask arrays once you apply `.chunk()`, allowing Dask to process the GRIB2 data in parallel chunks without loading entire files into memory.

### How do I chunk data when loading with Herbie?

You have two options: call `.chunk()` on the Dataset returned by `Herbie.xarray()`, or pass chunk specifications via `backend_kwargs` to cfgrib during the initial load. The first method is simpler; the second offers finer control over how GRIB2 messages are split across Dask workers.