Extending Herbie to Support New or Custom Weather Models: A Complete Guide

To extend Herbie, you must create a model-template class that defines the attributes DESCRIPTION, DETAILS, PRODUCTS, SOURCES, and LOCALFILE, then expose it either by importing it in herbie/models/__init__.py for public use or by registering it via the herbie.plugins entry point for private packages.

Extending Herbie to support new or custom weather models requires implementing a standardized template interface that teaches the core engine how to construct remote URLs, local filenames, and inventory indices. Whether you are adding support for a public meteorological dataset or integrating a proprietary internal model, Herbie's modular architecture in the blaylockbk/herbie repository allows seamless integration without modifying the core download logic. This guide covers the architectural patterns, required attributes, and registration mechanisms necessary to make your custom model work with the Herbie API.

How Herbie Discovers and Loads Model Templates

Herbie discovers weather-model data through specialized model-template classes that populate a Herbie instance with metadata at runtime. When a user calls Herbie(model='my_model'), the core engine in src/herbie/core.py normalizes the model name to lowercase and retrieves the corresponding template from the model_templates namespace.

The Template Loading Flow

The instantiation process follows a strict sequence defined in src/herbie/core.py:

1. Argument parsing: Herbie.__init__ computes self.date and self.fxx before calling template(self).
2. Template execution: The method getattr(model_templates, self.model).template(self) populates required attributes on the instance.
3. Validation: The _validate method checks that PRODUCTS, sources, and dates are correctly configured.
4. File discovery: find_grib() scans self.SOURCES (respecting priority) while find_idx() uses self.IDX_SUFFIX to locate inventory files.

Public Templates vs. Plugins

Herbie supports two distinct methods for registering templates:

• Public templates: Python classes placed in src/herbie/models/<model>.py and imported in src/herbie/models/__init__.py. These ship with the core package and use lowercase class names.
• Plugin templates: Classes distributed in separate pip-installable packages that declare an entry point in the herbie.plugins group. These are auto-discovered at import time via importlib.metadata.entry_points() (lines 45-52 of src/herbie/models/__init__.py).

Creating a Public Model Template

Public templates reside in the src/herbie/models/ directory and become part of the core herbie package. Each template must define a class with a template(self) method that receives the Herbie instance and attaches required attributes.

Required Template Attributes

The following five attributes are mandatory for every template:

Attribute	Purpose	Example
DESCRIPTION	Human-readable model summary	"MyModel – Global 0.5° forecast"
DETAILS	Dictionary of documentation links	{"model description": "https://example.com/docs"}
PRODUCTS	Mapping of product IDs to descriptions	{"0p5": "0.5-degree grid"}
SOURCES	Dictionary mapping source names to URL templates	{"aws": "https://bucket.s3.amazonaws.com/{pattern}"}
LOCALFILE	Local cache filename format	f"{self.get_remoteFileName}"

Minimal Public Template Implementation

Create a new file at src/herbie/models/my_model.py:

# src/herbie/models/my_model.py

"""Herbie template for a fictional “MyModel” (global 0.5° grid)."""

from datetime import datetime


class my_model:
    """MyModel – a simple demonstration model."""
    def template(self):
        # Required attributes

        self.DESCRIPTION = "MyModel – Example global 0.5° forecast"
        self.DETAILS = {
            "model description": "https://example.com/my_model",
            "data source": "https://data.example.com/my_model",
        }

        self.PRODUCTS = {
            "0p5": "0.5‑degree grid, all fields",
            "0p25": "0.25‑degree grid, high‑resolution fields",
        }

        # Remote locations with placeholders for date, product, and forecast hour

        post_root = f"my_model.{self.date:%Y%m%d/%H}/{self.product}.f{self.fxx:03d}"
        self.SOURCES = {
            "aws": f"https://my-model-pds.s3.amazonaws.com/{post_root}",
            "nomads": f"https://nomads.example.com/{post_root}",
        }

        # Local cache filename

        self.LOCALFILE = f"{self.get_remoteFileName}"

        # Optional: only needed if index file differs from default .grib2.idx

        # self.IDX_SUFFIX = [".idx"]

        # self.IDX_STYLE = "wgrib2"

After creating the file, expose the template by adding an import to src/herbie/models/__init__.py:

# src/herbie/models/__init__.py

...
from .my_model import *

The class name (my_model) must be lowercase to match the model='my_model' argument used when instantiating Herbie.

Packaging a Herbie Plugin for Private Models

For proprietary or experimental models that should not be committed to the public repository, create a separate Python package that registers an entry point. This approach keeps the core repository clean while allowing Herbie(model='MyModel') to function identically.

Entry Point Configuration

In your plugin package's setup.cfg (or pyproject.toml), declare the entry point:

[options.entry_points]
herbie.plugins =
    mymodel = myherbie_plugin.mymodel:MyModel

Plugin Class Implementation

Unlike public templates, plugin classes can use standard Python naming conventions (PascalCase). Create myherbie_plugin/mymodel.py:

"""Plugin template for MyModel (private data source)."""

from datetime import datetime


class MyModel:
    """Same structure as a public template; name can be any valid Python class."""
    def template(self):
        self.DESCRIPTION = "MyModel – Private data source"
        self.DETAILS = {"private repo": "https://intranet.example.com/my_model"}
        self.PRODUCTS = {"full": "Full model output"}
        
        post_root = f"my_model.{self.date:%Y%m%d/%H}/full.f{self.fxx:03d}"
        self.SOURCES = {"local": f"/mnt/private/my_model/{post_root}"}
        self.LOCALFILE = f"{self.get_remoteFileName}"

Install the package with pip install .. When herbie is imported, the plugin discovery loop in src/herbie/models/__init__.py (lines 45-52) automatically loads the class and prints a confirmation: Herbie: Added model "MyModel" from myherbie_plugin.

Configuring Index File Handling

Remote GRIB2 files require inventory indices for subsetting. Herbie uses two optional template attributes to locate these:

• IDX_SUFFIX: List of file extensions to search for (default: [".grib2.idx"]). Override if your archive uses .idx, .grb2.idx, or similar.
• IDX_STYLE: Parser format for the index file, either "wgrib2" (default) or "eccodes". This determines how index_as_dataframe() in src/herbie/core.py parses the inventory.

Set these attributes in the template() method only if your model deviates from the standard wgrib2-style index files.

Testing and Validating Your Template

Before submitting a public template or distributing a plugin, verify functionality with a minimal test:

from herbie import Herbie

# Test instantiation

H = Herbie('2023-01-01 00:00', model='my_model', product='0p5', fxx=6)

# Verify remote discovery

assert H.grib is not None, "GRIB URL not found"

# Verify inventory parsing

df = H.inventory()
assert not df.empty, "Inventory is empty"

# Verify download and subsetting

H.download()
ds = H.xarray('TMP:2 m')

Add similar assertions to the repository's test suite in tests/ to prevent regressions when the core engine updates.

Summary

Extending Herbie to support new or custom weather models requires implementing a standardized interface:

• Create a template class with DESCRIPTION, DETAILS, PRODUCTS, SOURCES, and LOCALFILE attributes that define how to locate and cache GRIB2 files.
• Register publicly by placing the module in src/herbie/models/ and importing it in __init__.py, or distribute privately via a pip package using the herbie.plugins entry point.
• Follow naming conventions: Use lowercase class names for public templates, predictable URL patterns with {self.date} and {self.fxx} placeholders, and override IDX_SUFFIX/IDX_STYLE only when index files differ from the wgrib2 standard.
• Validate functionality by testing Herbie instantiation, inventory(), download(), and xarray() methods against real data.

Frequently Asked Questions

What attributes are absolutely required for a Herbie model template?

Every template must define five attributes in the template(self) method: DESCRIPTION (string summary), DETAILS (dictionary of documentation links), PRODUCTS (dictionary mapping product IDs to descriptions), SOURCES (dictionary of URL templates with placeholders), and LOCALFILE (string defining the local cache path, typically f"{self.get_remoteFileName}"). The core engine in src/herbie/core.py raises errors during _validate() if any of these are missing.

How do I add a custom model without modifying the core Herbie repository?

Package your template as a standalone Python package and register it using Python entry points. Add [options.entry_points] to your setup.cfg with the group herbie.plugins pointing to your template class. When installed, src/herbie/models/__init__.py discovers the plugin automatically at import time via importlib.metadata.entry_points(group="herbie.plugins"), making the model available without changing the core source code.

What is the difference between IDX_STYLE "wgrib2" and "eccodes"?

IDX_STYLE determines how Herbie parses the index file associated with GRIB2 data. The default "wgrib2" style parses inventory files generated by the wgrib2 utility, commonly used by NOAA and AWS open data. The "eccodes" style parses indices generated by ECMWF's ecCodes library, which uses different formatting conventions. Set this attribute in your template only if your data source provides eccCodes-style index files; otherwise, the default wgrib2 parser handles most standard GRIB2 inventories.

How does Herbie validate that my template is working correctly?

The Herbie class validates templates in three stages: first, it checks that the model name exists in the model_templates namespace (loaded from src/herbie/models/__init__.py); second, the _validate() method confirms that the requested product exists in self.PRODUCTS and that the date/fxx combination produces valid URLs; third, find_grib() and find_idx() verify that remote files are reachable. If any stage fails, Herbie raises descriptive errors indicating whether the issue is template configuration, network reachability, or file availability.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}