# Herbie Caching Mechanism: How Downloaded Files Are Managed and Stored > Explore Herbie's filesystem caching strategy for managing downloaded files. Learn how Herbie deduplicates files and skips unnecessary downloads, saving time and storage. - Repository: [Brian Blaylock/herbie](https://github.com/blaylockbk/herbie) - Tags: internals - Published: 2026-02-26 --- **Herbie uses a filesystem-based deduplication strategy rather than a traditional cache subsystem, skipping downloads when files already exist in the configured save directory unless explicitly overwritten.** The `blaylockbk/herbie` Python package simplifies downloading meteorological GRIB2 files from various models. Understanding Herbie's caching mechanism is essential for managing disk space and avoiding redundant downloads, as the library relies on deterministic file paths and simple existence checks to manage its local data store. ## Filesystem-Based Deduplication Strategy Herbie does not implement a complex cache invalidation system. Instead, it treats the local filesystem as the authoritative store, using predictable directory structures to determine if data already exists. ### Default Save Directory Configuration The root location for all cached files is controlled by the global configuration key `config["default"]["save_dir"]`. By default, this resolves to `~/data`, though you can override it by setting the `HERBIE_SAVE_DIR` environment variable before importing the library. This configuration is loaded during module initialization in [`src/herbie/__init__.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/__init__.py) (lines 86–98). ### Deterministic File Naming Convention To enable reliable cache hits, Herbie constructs deterministic file paths using the pattern: ``` {save_dir}/{model}/{YYYYMMDD}/{localFileName} ``` The `localFileName` is derived from the model's specific template, ensuring that identical model runs always map to identical filesystem locations. This logic is implemented in [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) (lines 650–652). ## Download Cache Control and Overwrite Behavior The `Herbie.download()` method implements the core caching logic through file existence checks and overwrite flags. ### File Existence Checks When `download()` is invoked, Herbie first checks if the target file already exists on disk. If the file is present and the instance's `overwrite` attribute is `False`, the method skips the network request entirely and returns the existing file path immediately. This check occurs in [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) (lines 502–506). ### Global and Per-Call Overwrite Settings You can control cache behavior at two levels: 1. **Global configuration**: Set `overwrite = false` in the Herbie config file to enforce cache-first behavior across all instances. 2. **Per-call override**: Pass `overwrite=True` to the `Herbie` constructor or the `download()` method to force a fresh download regardless of cache status. The parameter resolution logic is found in [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) (lines 1152–1155). ## In-Memory and Auxiliary Caching Beyond the primary file store, Herbie implements secondary caching mechanisms for expensive computational operations. ### GRIB Index Caching with cached_property After downloading a GRIB2 file, Herbie generates an inventory index to enable subsetting by variable. This index parsing is computationally expensive, so the library caches the resulting DataFrame using `functools.cached_property` via the `index_as_dataframe` attribute. This ensures the GRIB inventory is parsed only once per file instance, with subsequent accesses returning the cached result. The implementation is in [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) (lines 723–727). ### BallTree Spatial Index Caching For the `pick_points` helper functionality, Herbie can build a **BallTree** spatial index to accelerate nearest-neighbor lookups on model grids. When `use_cached_tree=True` (the default), the serialized tree is stored as a pickle file under: ``` {cache_dir}/BallTree/{tree_name}_{grid_size}.pkl ``` On subsequent calls with matching parameters, Herbie loads the existing tree rather than rebuilding it. To force reconstruction and overwrite the cache, pass `use_cached_tree="replant"`. This logic resides in [`src/herbie/pick_points.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/pick_points.py) (lines 150–179 and 167–174). ## Summary - Herbie implements a **filesystem-based deduplication** strategy rather than a traditional cache, storing files under `~/data` (configurable via `HERBIE_SAVE_DIR`). - The **deterministic path structure** (`{model}/{date}/{filename}`) ensures identical model runs map to the same location. - Downloads are skipped when files exist and `overwrite=False`, with control available globally or per-call. - **In-memory caching** via `cached_property` prevents redundant GRIB index parsing. - Optional **BallTree spatial index caching** accelerates repeated geospatial queries by persisting serialized trees to disk. ## Frequently Asked Questions ### How do I change the default cache directory in Herbie? Set the `HERBIE_SAVE_DIR` environment variable before importing the library, or modify the `save_dir` key in the Herbie configuration file. The path is resolved during module initialization in [`src/herbie/__init__.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/__init__.py). ### Will Herbie re-download files if I run the same request twice? No, unless you explicitly set `overwrite=True`. By default, Herbie checks if the file exists in the save directory and skips the download if present, returning the cached file path immediately. ### What is the BallTree cache used for in Herbie? The BallTree cache stores serialized spatial indexes used by the `pick_points` helper to accelerate nearest-neighbor searches on model grids. It is saved as a pickle file under `{cache_dir}/BallTree/` and reused across Python sessions unless `use_cached_tree="replant"` is specified. ### How does Herbie handle GRIB file indexing performance? Herbie caches the parsed GRIB inventory as a pandas DataFrame using Python's `functools.cached_property` decorator. This ensures the expensive index parsing operation occurs only once per file instance, with subsequent accesses returning the cached DataFrame from memory.