Marin Configuration Options: A Complete Guide to `config/marin.yaml`

Marin's runtime behavior is controlled by the YAML file config/marin.yaml, which defines the Iris cluster name, Google Cloud Storage bucket mappings per region, and temporary file handling with configurable TTL values.

The marin-community/marin repository orchestrates large-scale ML workloads on TPUs using the Iris scheduler. Understanding the configuration options for Marin is essential for deploying training and inference pipelines across Google Cloud regions. The canonical configuration resides in config/marin.yaml, a single source of truth that determines cluster selection, storage backends, and data lifecycle policies.

The config/marin.yaml File Structure

The canonical configuration file defines three fundamental top-level sections that all Iris-based jobs rely on. According to the marin-community/marin source code, these sections control cluster targeting, persistent storage, and temporary data handling. The file can also be extended with additional keys such as provisioning, peers, and jwt, which are consumed by components like Pulumi, Iris federation, and Finelog.

The iris Section

This section specifies the Iris cluster that will host all training, evaluation, and inference jobs. In the production configuration, this points to the TPU cluster name:

iris: marin

The value marin represents the production TPU cluster. This key is consumed by the Iris scheduler when materializing job specifications.

The data Section

The data section configures storage backend parameters, regional bucket mappings, and temporary file policies.

Storage Scheme

The scheme parameter defines the storage protocol. Currently hardcoded to gs for Google Cloud Storage:

data:
  scheme: gs

Changing this value would require updating storage-layer adapters throughout the codebase, as all bucket URLs are interpreted as gs://<bucket>/….

Regional Buckets (region_buckets)

This mapping assigns dedicated GCS buckets to each supported GCP region to minimize latency. The format follows:

region_buckets:
  us-central1: { bucket: marin-us-central1, store: gcs }
  us-east1: { bucket: marin-us-east1, store: gcs }

Each entry specifies the bucket name and store type (gcs). The Iris scheduler uses these values to materialize persistent datasets, model checkpoints, and log artifacts. Adding a new region requires only inserting a new entry in this map.

Temporary Storage (temp)

The temp subsection controls ephemeral data handling and lifecycle policies:

temp:
  bucket: marin-temp
  path: tmp
  ttl_days: [1, 2, 3, 4, 5, 6, 7, 14, 30]

The ttl_days array enumerates allowed time-to-live values for the "marin_temp_bucket" helper. Users can override the default path prefix using the MARIN_TEMP_PREFIX environment variable.

How Configuration Is Loaded

Marin uses two primary mechanisms to load and merge configuration values from YAML into Python objects.

Rigging Library Parser

The Rigging library parses config/marin.yaml via the ClusterConfig helper defined in lib/rigging/src/rigging/filesystem/cluster_config.py. This exposes the configuration as a typed Python dictionary, enabling downstream code to query specific values like cfg["data"]["region_buckets"]["us-east1"]["bucket"].

Iris CLI Overrides

The Iris CLI reads a checkout-local .marin.yaml (if present) and merges its env: section with command-line flags, as implemented in lib/iris/src/iris/cli/job.py. This allows repository-specific overrides without modifying the canonical configuration.

Practical Code Examples

The following examples demonstrate how to programmatically access Marin configuration options using standard libraries.

Loading and Inspecting Region Buckets

from pathlib import Path
import yaml

# Load the canonical configuration.

config_path = Path("config/marin.yaml")
with config_path.open() as f:
    cfg = yaml.safe_load(f)

# Access the bucket name for a given region.

region = "us-east1"
bucket_name = cfg["data"]["region_buckets"][region]["bucket"]
print(f"The bucket for {region} is: {bucket_name}")

Creating Temporary Paths with TTL

import random

# Assuming cfg is loaded as above

temp_cfg = cfg["data"]["temp"]
ttl = random.choice(temp_cfg["ttl_days"])
temp_path = f"{temp_cfg['path']}/ttl-{ttl}"
print(f"Temporary path (TTL {ttl} days): gs://{temp_cfg['bucket']}/{temp_path}")

Key Configuration Files

File	Role
config/marin.yaml	Canonical production configuration file containing all top-level sections.
lib/iris/config/marin-dev.yaml	Development-scale variant with smaller caps and isolated state.
lib/rigging/src/rigging/filesystem/cluster_config.py	Parser that converts YAML into typed ClusterConfig objects.
lib/iris/src/iris/cli/job.py	CLI entry point handling checkout-local .marin.yaml overrides.
docs/tutorials/storage-bucket.md	Documentation for bucket lifecycle and temporary storage management.

Summary

• Marin uses config/marin.yaml as the single source of truth for runtime configuration.
• The iris key specifies the target TPU cluster (e.g., marin for production).
• Regional GCS buckets are mapped under data.region_buckets to minimize latency.
• Temporary data supports configurable TTL values from 1 to 30 days.
• The Rigging library parses configurations via ClusterConfig, while the Iris CLI supports local overrides through .marin.yaml.

Frequently Asked Questions

Where is the main Marin configuration file located?

The canonical configuration resides at config/marin.yaml in the repository root. This file is parsed by the Rigging library's ClusterConfig class defined in lib/rigging/src/rigging/filesystem/cluster_config.py and defines production cluster settings, storage mappings, and temporary data policies.

How do I add a new GCP region to Marin?

Add a new entry under the data.region_buckets section in config/marin.yaml using the format <region>: { bucket: <bucket-name>, store: gcs }. The Iris scheduler will automatically recognize the new bucket for job artifact storage in that region.

What TTL values are supported for temporary storage?

The data.temp.ttl_days array specifies allowed values: [1, 2, 3, 4, 5, 6, 7, 14, 30]. These values control how long data persists in temporary buckets before automatic cleanup by the storage lifecycle manager.

Can I override configuration values for local development?

Yes. The Iris CLI reads a checkout-local .marin.yaml file and merges its env: section with command-line flags, as implemented in lib/iris/src/iris/cli/job.py. This mechanism allows developers to override paths and cluster settings without modifying the canonical config/marin.yaml.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}