# Deploying Herbie in Production Data Pipelines: A Complete Guide > Learn to deploy Herbie in production data pipelines. Our guide covers centralized storage, parallel downloads with FastHerbie, idempotency, and persistent spatial caches for efficient data processing. - Repository: [Brian Blaylock/herbie](https://github.com/blaylockbk/herbie) - Tags: how-to-guide - Published: 2026-02-26 --- **Deploying Herbie in production data pipelines requires configuring a centralized storage directory, leveraging `FastHerbie` for parallel async downloads, disabling overwrites to ensure idempotency, and persisting BallTree spatial caches to prevent redundant index recomputation.** Herbie is a Python library that abstracts the discovery, download, and local caching of Numerical Weather Prediction (NWP) model files. When transitioning from interactive notebooks to automated workflows, deploying Herbie in production data pipelines requires careful attention to deterministic file management, error handling, and resource optimization as implemented in the `blaylockbk/herbie` repository. ## Core Architecture for Production Workloads Herbie’s modular design separates concerns across distinct components that production engineers must understand: - **Configuration Layer**: Loads user settings from `~/.config/herbie/config.toml` and respects environment overrides such as `HERBIE_SAVE_DIR` and `HERBIE_CONFIG_PATH` (see [`src/herbie/__init__.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/__init__.py) lines 78-81). - **Core `Herbie` Class**: Handles model-specific URL construction, local file path resolution via `Herbie._localFilePath`, and orchestrates the download flow (see [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) lines 161-170). - **Download Engine**: Provides `download_with_requests` for synchronous fetches and `FastHerbie.download` for asynchronous bulk retrieval using `aiohttp` (see [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) line 67 and [`src/herbie/fast.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/fast.py) line 207). - **Spatial Index Cache**: Optional on-disk BallTree caches accelerate repeated geographic queries via `pick_points` and the `HerbieAccessor` interface (see [`src/herbie/pick_points.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/pick_points.py) lines 150-170). - **Accessor API**: Supplies a pandas-like `.loc[]` syntax for subsetting datasets without re-downloading raw GRIB2 files (see [`src/herbie/accessors.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/accessors.py) lines 96-105). ## Configuration and Deterministic Storage ### Pinning the Save Directory Production pipelines must specify a stable, shared storage location rather than relying on default user directories. Set `config["default"]["save_dir"]` programmatically or via the `HERBIE_SAVE_DIR` environment variable to point to a mounted NFS share, object-store bucket, or distributed filesystem. Herbie enforces a deterministic file layout: files save to `///.grib2`. This predictable structure allows downstream tasks to check for existence before triggering redundant downloads. ### Environment-Based Configuration For containerized deployments, override configuration values through environment variables rather than modifying TOML files. The library automatically ingests `HERBIE_SAVE_DIR` and `HERBIE_CONFIG_PATH` during initialization in [`src/herbie/__init__.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/__init__.py). ## Reliability and Idempotency Patterns ### Atomic File Operations The `download` method in [`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py) writes data to a temporary file and performs an atomic rename upon completion. This prevents pipeline corruption from partial downloads during network interruptions or pod evictions. ### Idempotent Downloads Set `config["default"]["overwrite"] = False` (the default behavior) to ensure that existing files are never clobbered. The `download` method returns the local file path, enabling downstream tasks to skip processing when data already exists. ### Configurable Retry Logic `download_with_requests` catches `HTTPError` exceptions and retries failed requests based on `config["default"]["download_retry"]`, which defaults to 3 attempts. For flaky or high-latency archives, increase this value to 5 or higher in your production configuration. ## Performance Optimization Strategies ### Parallel Downloads with FastHerbie For bulk retrieval of multiple forecast hours or ensemble members, instantiate `FastHerbie` from [`src/herbie/fast.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/fast.py). This subclass uses `aiohttp` to fetch files concurrently. Tune the `max_clients` parameter based on your network bandwidth and the remote server's connection limits to maximize throughput without triggering rate limiting. ### BallTree Spatial Caching Spatial queries using `pick_points` or the `HerbieAccessor` rely on BallTree indices stored under `save_dir/BallTree/_.pkl`. Reusing these caches across pipeline runs avoids O(N²) rebuilds of the spatial index. Set `use_cached_tree=True` for standard operations, and only use `use_cached_tree="replant"` when the underlying model grid definition changes. ### Resource Cleanup Temporary directories are removed automatically via `Herbie.__del__` when `cleanup=True`. However, in long-running services or Kubernetes pods, explicitly invoke `herbie.cleanup()` after batch processing to free disk space immediately rather than waiting for garbage collection. ## Security and Authentication Patterns ### Handling Restricted Archives Certain archives such as NOAA’s NCEI require `wget` or `curl` with embedded credentials. Herbie exposes raw URLs through the `_download` method, allowing you to subclass `Herbie` and override `_download` to inject authentication tokens, SSH keys, or signed URLs for secure environments. ## Observability and Maintenance ### Logging Integration Herbie emits logs through the `"herbie"` namespace accessible via `logging.getLogger("herbie")`. Forward these logs to your central observability stack such as ELK, CloudWatch, or Datadog to monitor download latency, cache hits, and retry events. ### Version Pinning for Reproducibility Record `herbie.__version__` in your pipeline metadata to ensure reproducibility across environments. The repository ships a comprehensive pytest suite under `tests/`; run this suite in CI/CD pipelines whenever upgrading the library to validate download, caching, and accessor behavior. ## Production Pipeline Example The following implementation demonstrates an automated HRRR ingestion workflow with optimized storage, parallel downloads, and spatial subsetting: ```python from pathlib import Path from herbie import Herbie, FastHerbie, config # ---------------------------------------------------------------------- # 1️⃣ Configure shared storage and reliability settings # ---------------------------------------------------------------------- config["default"]["save_dir"] = Path("/mnt/data/herbie_cache") config["default"]["overwrite"] = False config["default"]["download_retry"] = 5 # ---------------------------------------------------------------------- # 2️⃣ Define forecast hours for batch retrieval # ---------------------------------------------------------------------- forecast_hours = range(0, 13, 3) # ---------------------------------------------------------------------- # 3️⃣ Use FastHerbie for concurrent async downloads # ---------------------------------------------------------------------- fh = FastHerbie( date="2024-03-01", model="hrrr", product="sfc", fxx=forecast_hours, save_dir=config["default"]["save_dir"], overwrite=False, ) local_files = fh.download() # ---------------------------------------------------------------------- # 4️⃣ Load and subset using the accessor API # ---------------------------------------------------------------------- H = Herbie(local_files[0]) ds = H.xarray() # Subset to bounding box without re-downloading subset = ds.herbie.sel(lat=slice(30, 40), lon=slice(-100, -90)) # ---------------------------------------------------------------------- # 5️⃣ Persist processed output # ---------------------------------------------------------------------- out_path = Path("/mnt/data/processed/hrrr_subset_20240301_f00.nc") subset.to_netcdf(out_path) print(f"✅ Subset written to {out_path}") ``` ## Key Source Files for Production Review Understanding these core files is essential for debugging and extending Herbie in production: - **[`src/herbie/__init__.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/__init__.py)**: Loads user configuration and processes environment variable overrides for `HERBIE_SAVE_DIR`. - **[`src/herbie/core.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/core.py)**: Implements the `Herbie` class, URL generation, local file path logic (`Herbie._localFilePath`), and the synchronous `download` method with atomic file handling. - **[`src/herbie/fast.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/fast.py)**: Provides the `FastHerbie` subclass with asynchronous `download` capabilities for high-throughput scenarios. - **[`src/herbie/pick_points.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/pick_points.py)**: Manages BallTree creation and on-disk cache persistence for spatial queries. - **[`src/herbie/accessors.py`](https://github.com/blaylockbk/herbie/blob/main/src/herbie/accessors.py)**: Defines `HerbieAccessor` for lazy subsetting of xarray datasets via the `.herbie` property. ## Summary - **Pin a stable `save_dir`** on shared storage and configure it via environment variables or the global `config` object. - **Set `overwrite=False`** to ensure idempotent pipeline runs that skip existing files. - **Use `FastHerbie`** for bulk retrieval, tuning `max_clients` to match your network capacity. - **Persist BallTree caches** on shared storage and reuse them across runs to eliminate spatial index recomputation. - **Override `_download`** in a subclass when integrating with authenticated or restricted data archives. - **Integrate logging** with your observability platform and record `herbie.__version__` in pipeline metadata for full traceability. ## Frequently Asked Questions ### How does Herbie handle partial or interrupted downloads? Herbie prevents data corruption by writing downloads to temporary files and performing an atomic rename operation only after the file is fully written. This ensures that incomplete files never appear in the target directory, even if the process crashes or loses network connectivity mid-stream. ### Can Herbie download from password-protected or restricted weather archives? Yes. While Herbie provides standard HTTP download methods, you can subclass the `Herbie` class and override the `_download` method to inject custom authentication logic. This pattern allows integration with token-based APIs, SSH key authentication, or enterprise proxy requirements for archives like NOAA’s NCEI. ### What is the difference between `Herbie` and `FastHerbie`? `Herbie` performs synchronous downloads sequentially using `download_with_requests`, suitable for single-file retrieval. `FastHerbie` extends this functionality with asynchronous I/O via `aiohttp`, enabling concurrent downloads of multiple forecast hours or ensemble members. Use `FastHerbie` for production batch ingestion and tune the `max_clients` parameter to optimize throughput. ### How do I prevent Herbie from re-downloading files that already exist? Set the configuration value `config["default"]["overwrite"] = False` or pass `overwrite=False` to the constructor. This idempotent default checks for file existence at the deterministic path `///.grib2` and skips the download if the file is already present, saving bandwidth and processing time during pipeline reruns.