# How to Subset GRIB2 Files by Variable Using Herbie: A Complete Guide

> Easily subset GRIB2 files by variable with Herbie using HTTP Range requests. Download only the weather variables you need, saving time and disk space. Get the complete guide.

- Repository: [Brian Blaylock/herbie](https://github.com/blaylockbk/herbie)
- Tags: how-to-guide
- Published: 2026-02-26

---

**Herbie enables efficient subsetting of GRIB2 files by variable using HTTP Range requests on inventory indexes, allowing you to download only specific weather variables without retrieving multi-gigabyte files.**

The Herbie library provides a Pythonic interface for accessing meteorological GRIB2 datasets from remote servers. When working with high-resolution weather models, GRIB2 files often exceed several gigabytes, making it impractical to download entire files just to extract a single variable like temperature or wind speed. By leveraging index files and HTTP Range requests, Herbie allows you to subset GRIB2 files by variable, transferring only the bytes that contain your data of interest.

## Why Subset GRIB2 Files by Variable?

High-resolution weather models such as the HRRR or GFS produce GRIB2 files that can range from 500 MB to over 5 GB per time step. When you only need surface temperature or 10-meter wind components, downloading the full file wastes bandwidth, storage, and processing time. Subsetting by variable reduces transfer sizes to mere kilobytes or megabytes, enabling rapid iteration and analysis workflows.

## How Herbie Subsets GRIB2 Files by Variable

Herbie implements a six-step workflow to subset GRIB2 files by variable efficiently. This process is orchestrated through the `Herbie` class in [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py).

### Step 1: Reading the Inventory Index

Herbie first creates or fetches a wgrib2/eccodes index that lists every GRIB message with its byte offsets, variable name, level, and forecast time. The `Herbie.index_as_dataframe` method builds a `pandas.DataFrame` from this index file, enabling fast in-memory filtering. This implementation resides in [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) at lines 814–882.

### Step 2: Filtering Variables with Regex

Once the index is loaded, the `Herbie.inventory(search)` method filters the DataFrame using a regular expression pattern. The method matches against a "search_this" column that concatenates relevant GRIB metadata. For example, passing `search=":TMP:2 m:"` returns only rows representing temperature at 2 meters. This filtering logic appears in [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) at lines 362–388.

### Step 3: Grouping Consecutive GRIB Messages

To minimize HTTP overhead, Herbie groups consecutive GRIB messages that appear sequentially in the original file. The subsetting logic in [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) (lines 553–560) identifies contiguous byte ranges so that a single HTTP Range request can retrieve multiple variables or levels at once, rather than issuing separate requests for each message.

### Step 4: Downloading Byte Ranges via HTTP

For each grouped range, Herbie issues an HTTP request with a `bytes=start-end` header. The server returns only the specified slice of the GRIB2 file, which Herbie writes to a temporary subset file. This download mechanism is implemented in [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) at lines 775–799, using Python's `requests` library to handle the Range headers.

### Step 5: Generating Deterministic Subset Filenames

Herbie creates subset filenames that encode the model, date, lead time, and a hash of the selected messages. This deterministic naming ensures uniqueness and enables caching, preventing redundant downloads when the same subset is requested again. The filename generation logic resides in [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) at lines 664–702.

### Step 6: Local File Fallback

If the full GRIB2 file already exists locally, Herbie bypasses the network entirely. The `subset` method detects local files and reads the required byte ranges directly from disk, falling back to HTTP Range requests only when the local file is unavailable. This logic appears in [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) at lines 730–795.

## Practical Examples: Subsetting GRIB2 by Variable

The following examples demonstrate how to subset GRIB2 files by variable using Herbie's `download` and `xarray` methods.

### Download a Single Variable

This example downloads only the 2-meter temperature field from an HRRR analysis:

```python
from herbie import Herbie

# Initialize Herbie for the desired model run

h = Herbie(
    date="2024-09-01T00:00",
    model="hrrr",
    fxx=0,                # analysis hour (no forecast lead)

    product="sfc",
)

# Download only temperature at 2 meters

# The search syntax uses colons to separate fields

subset_path = h.download(search=":TMP:2 m:")
print(f"Subset saved to: {subset_path}")

```

### Load Subset Directly as Xarray

For analysis workflows, you can load the subset directly into an xarray Dataset without writing an intermediate file:

```python

# Load U and V wind components at 10 meters

ds = h.xarray(search=":UGRD:10 m:|:VGRD:10 m:")
print(ds)

```

### Combine Multiple Variables

You can subset multiple variables in a single call by combining patterns with the pipe operator (`|`):

```python

# Fetch wind at 850 hPa and surface temperature in one operation

subset_path = h.download(
    search=":UGRD:850 mb:|:VGRD:850 mb:|:TMP:surface:"
)

```

**Important Notes on Syntax**

- The `search` parameter accepts a **regular-expression-like** string where GRIB fields (variable, level, etc.) are delimited by colons.
- Omitting `search` or setting it to `":"` downloads the full file.
- Subset files are automatically cached in `~/.config/herbie/` or your specified `save_dir`.

## Key Implementation Files

The subsetting functionality is distributed across the following source files:

| File | Responsibility |
|------|----------------|
| [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) | Core `Herbie` class implementing index handling (`index_as_dataframe`), inventory filtering (`inventory`), grouping logic, HTTP Range downloads, and subset filename generation |
| [`src/herbie/wgrib2.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/wgrib2.py) | Wrapper around the `wgrib2` utility for generating inventory files when remote indexes are unavailable |
| [`src/herbie/help.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/help.py) | Helper functions that produce searchable examples for the `search` syntax (`_search_help`) |
| [`tests/test_download_subset.py`](https://github.com/blaylockbk/herbie/blob/main/tests/test_download_subset.py) | Test suite validating the subset download path for correctness and performance |

## Summary

- **Herbie** enables efficient subsetting of GRIB2 files by variable using HTTP Range requests on inventory indexes.
- The workflow involves reading an index file (`index_as_dataframe`), filtering with regex (`inventory`), grouping consecutive messages, and issuing targeted HTTP byte-range requests.
- Subsetting reduces download sizes from gigabytes to kilobytes when you only need specific variables like temperature or wind components.
- The library automatically handles local file fallbacks and generates deterministic, cacheable filenames for subset files.

## Frequently Asked Questions

### What is the search syntax for subsetting GRIB2 variables in Herbie?

Herbie uses a colon-delimited format similar to wgrib2 conventions. You specify fields like variable name, level, and forecast time separated by colons (e.g., `:TMP:2 m:` for 2-meter temperature). You can combine multiple patterns using the pipe operator `|` to subset several variables in one call. The `search` parameter accepts regular expressions, allowing flexible matching against the GRIB inventory.

### How does Herbie handle non-consecutive GRIB messages when subsetting?

Herbie automatically groups consecutive GRIB messages that appear sequentially in the original file into single download chunks. For non-consecutive messages, it issues separate HTTP Range requests for each distinct byte range. This optimization minimizes the number of HTTP requests while ensuring you only download the specific messages containing your variables, regardless of their position in the original file.

### Can I subset GRIB2 files if I already have the full file downloaded locally?

Yes. When you call `download()` or `xarray()` with a search parameter, Herbie first checks if the full GRIB2 file exists in your local cache. If found, it reads the required byte ranges directly from the local file instead of making HTTP requests, effectively performing a local subset operation. This fallback mechanism is implemented in [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) within the subsetting logic.

### What is the performance benefit of subsetting GRIB2 files by variable?

Subsetting can reduce data transfer volumes by orders of magnitude. A typical high-resolution weather model GRIB2 file may exceed 5 GB, while a single variable like 2-meter temperature might occupy only 50-100 KB. By downloading only the specific byte ranges containing your variables via HTTP Range requests, you reduce download times from minutes to seconds and minimize local storage requirements.