Herbie Caching Mechanism: How Downloaded Files Are Managed and Stored

Herbie uses a filesystem-based deduplication strategy rather than a traditional cache subsystem, skipping downloads when files already exist in the configured save directory unless explicitly overwritten.

The blaylockbk/herbie Python package simplifies downloading meteorological GRIB2 files from various models. Understanding Herbie's caching mechanism is essential for managing disk space and avoiding redundant downloads, as the library relies on deterministic file paths and simple existence checks to manage its local data store.

Filesystem-Based Deduplication Strategy

Herbie does not implement a complex cache invalidation system. Instead, it treats the local filesystem as the authoritative store, using predictable directory structures to determine if data already exists.

Default Save Directory Configuration

The root location for all cached files is controlled by the global configuration key config["default"]["save_dir"]. By default, this resolves to ~/data, though you can override it by setting the HERBIE_SAVE_DIR environment variable before importing the library. This configuration is loaded during module initialization in src/herbie/__init__.py (lines 86–98).

Deterministic File Naming Convention

To enable reliable cache hits, Herbie constructs deterministic file paths using the pattern:

{save_dir}/{model}/{YYYYMMDD}/{localFileName}

The localFileName is derived from the model's specific template, ensuring that identical model runs always map to identical filesystem locations. This logic is implemented in src/herbie/core.py (lines 650–652).

Download Cache Control and Overwrite Behavior

The Herbie.download() method implements the core caching logic through file existence checks and overwrite flags.

File Existence Checks

When download() is invoked, Herbie first checks if the target file already exists on disk. If the file is present and the instance's overwrite attribute is False, the method skips the network request entirely and returns the existing file path immediately. This check occurs in src/herbie/core.py (lines 502–506).

Global and Per-Call Overwrite Settings

You can control cache behavior at two levels:

1. Global configuration: Set overwrite = false in the Herbie config file to enforce cache-first behavior across all instances.
2. Per-call override: Pass overwrite=True to the Herbie constructor or the download() method to force a fresh download regardless of cache status.

The parameter resolution logic is found in src/herbie/core.py (lines 1152–1155).

In-Memory and Auxiliary Caching

Beyond the primary file store, Herbie implements secondary caching mechanisms for expensive computational operations.

GRIB Index Caching with cached_property

After downloading a GRIB2 file, Herbie generates an inventory index to enable subsetting by variable. This index parsing is computationally expensive, so the library caches the resulting DataFrame using functools.cached_property via the index_as_dataframe attribute. This ensures the GRIB inventory is parsed only once per file instance, with subsequent accesses returning the cached result. The implementation is in src/herbie/core.py (lines 723–727).

BallTree Spatial Index Caching

For the pick_points helper functionality, Herbie can build a BallTree spatial index to accelerate nearest-neighbor lookups on model grids. When use_cached_tree=True (the default), the serialized tree is stored as a pickle file under:

{cache_dir}/BallTree/{tree_name}_{grid_size}.pkl

On subsequent calls with matching parameters, Herbie loads the existing tree rather than rebuilding it. To force reconstruction and overwrite the cache, pass use_cached_tree="replant". This logic resides in src/herbie/pick_points.py (lines 150–179 and 167–174).

Summary

• Herbie implements a filesystem-based deduplication strategy rather than a traditional cache, storing files under ~/data (configurable via HERBIE_SAVE_DIR).
• The deterministic path structure ({model}/{date}/{filename}) ensures identical model runs map to the same location.
• Downloads are skipped when files exist and overwrite=False, with control available globally or per-call.
• In-memory caching via cached_property prevents redundant GRIB index parsing.
• Optional BallTree spatial index caching accelerates repeated geospatial queries by persisting serialized trees to disk.

Frequently Asked Questions

How do I change the default cache directory in Herbie?

Set the HERBIE_SAVE_DIR environment variable before importing the library, or modify the save_dir key in the Herbie configuration file. The path is resolved during module initialization in src/herbie/__init__.py.

Will Herbie re-download files if I run the same request twice?

No, unless you explicitly set overwrite=True. By default, Herbie checks if the file exists in the save directory and skips the download if present, returning the cached file path immediately.

What is the BallTree cache used for in Herbie?

The BallTree cache stores serialized spatial indexes used by the pick_points helper to accelerate nearest-neighbor searches on model grids. It is saved as a pickle file under {cache_dir}/BallTree/ and reused across Python sessions unless use_cached_tree="replant" is specified.

How does Herbie handle GRIB file indexing performance?

Herbie caches the parsed GRIB inventory as a pandas DataFrame using Python's functools.cached_property decorator. This ensures the expensive index parsing operation occurs only once per file instance, with subsequent accesses returning the cached DataFrame from memory.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}