Pulumi Infrastructure Pattern for Deploying Marin Clusters: Complete Implementation Guide

Marin clusters use an Infrastructure pattern in the marin-iac Pulumi project that separates cluster-scoped resources into individual stacks while maintaining shared GCP resources (IAM, networking) in a central marin stack, all backed by a unified GCS state backend and GCP KMS encryption.

The marin-community/marin repository implements a scalable Infrastructure pattern for cloud deployments. This pattern, defined in infra/pulumi, enables teams to provision independent Marin clusters while keeping critical GCP infrastructure centrally governed and secure.

Understanding the Marin Infrastructure Pattern Architecture

The pattern divides infrastructure into two distinct layers to balance isolation with shared governance.

Centralized Shared Resources in the marin Stack

At the foundation lies the marin stack defined in infra/pulumi/Pulumi.marin.yaml. This stack owns all shared GCP infrastructure including IAM grants, VPC networking, and service accounts. According to the marin-community/marin source code, any resource that must persist across cluster lifecycles lives here rather than in cluster-specific configurations.

The shared stack writes to the same backend as all other stacks: a GCS bucket at gs://marin-iac-state/ encrypted with a GCP KMS key at gcpkms://…/marin-iac-keyring/cryptoKeys/marin-iac-key.

Cluster-Scoped Stack Isolation

Individual clusters receive dedicated Pulumi stacks named Pulumi.<cluster>.yaml. Each stack resides in the infra/pulumi directory and targets specific cluster configurations such as node counts, machine types, and regional deployments. This isolation ensures that destroying one cluster's resources never impacts the shared infrastructure or other running clusters.

How to Deploy a New Marin Cluster

Deploying a cluster follows a three-phase workflow that leverages reusable components from the shared library.

Step 1: Create the Cluster Stack Configuration

Create a new stack file Pulumi.<cluster>.yaml in infra/pulumi/. Specify the runtime and cluster-specific parameters:

# infra/pulumi/Pulumi.my-cluster.yaml

name: my-cluster
runtime:
  name: python
  options:
    virtualenv: .venv
config:
  gcp:project: marin-gcp
  gcp:region: us-central1
  # cluster‑specific parameters

  cluster:nodeCount: 3
  cluster:machineType: n1-standard-4

Step 2: Reference Shared Components

In your cluster program (e.g., infra/pulumi/src/cluster/my_cluster.py), import reusable constructs from infra/pulumi/src/iac/ to ensure consistency:

# infra/pulumi/src/cluster/my_cluster.py

import pulumi
from iac.gcp.cloud_run import CloudRunService

service = CloudRunService(
    name="my-service",
    image="gcr.io/marin/my-image:latest",
    env={"ENV": "prod"},
)
pulumi.export("service_url", service.url)

To access shared resources like VPC networks from the central marin stack, use StackReference:

import pulumi
from pulumi import StackReference

# Grab shared VPC from the central 'marin' stack

marin = StackReference("marin")
vpc_id = marin.get_output("vpc_id")

Step 3: Import Existing Resources (Optional)

For resources created outside of Pulumi, use the import command documented in infra/pulumi/README.md:

# Follow the guide in infra/pulumi/README.md

pulumi import --program infra/pulumi/src/iac/gcp/iam_binding.py \
    gcp:cloudresourcemanager/ProjectIamBinding:binding my-binding \
    "projects/marin-gcp roles/editor user:example@example.com"

State Management and Security Configuration

All stacks in the marin-iac project share a unified state backend. The configuration stores state in gs://marin-iac-state/ and encrypts secrets using the KMS key marin-iac-key in the marin-iac-keyring. This centralized approach enables teams to collaborate on infrastructure while maintaining strict encryption standards across every cluster deployment.

Handling IAM Bindings and Permissions

Because the marin stack owns all IAM bindings, cluster-specific permissions require careful coordination. Any grant targeting a resource created by a cluster stack must be declared in infra/pulumi/src/iac/gcp/iam_data.yaml or an adjacent module within the shared infrastructure code.

This constraint guarantees a single source of truth for permissions across the entire deployment, preventing privilege drift between independently managed clusters. When your cluster requires new access rights, modify the central IAM data files rather than declaring bindings in cluster-scoped code.

Summary

• Marin clusters use the Infrastructure pattern in infra/pulumi to separate cluster-specific resources from shared GCP infrastructure.
• The marin stack (infra/pulumi/Pulumi.marin.yaml) centrally manages IAM, networking, and service accounts used by all clusters.
• Individual clusters use dedicated stack files (Pulumi.<cluster>.yaml) that reference reusable components from infra/pulumi/src/iac/.
• All stacks share a GCS backend (gs://marin-iac-state/) and GCP KMS encryption for state and secrets.
• IAM bindings must be declared in infra/pulumi/src/iac/gcp/iam_data.yaml to maintain centralized permission governance.

Frequently Asked Questions

What is the difference between the marin stack and individual cluster stacks?

The marin stack is the central shared stack that owns global GCP resources like IAM bindings, VPC networks, and service accounts used across all deployments. Individual cluster stacks (e.g., Pulumi.my-cluster.yaml) contain only cluster-specific configurations such as node counts and regional settings, and they reference outputs from the marin stack rather than duplicating shared infrastructure.

How does Marin manage secrets and state for Pulumi deployments?

All stacks in the marin-iac project store state in a single GCS bucket (gs://marin-iac-state/) and encrypt secrets using a dedicated GCP KMS key (marin-iac-key). This unified backend configuration ensures that every cluster stack accesses the same encrypted state store while maintaining isolation between stack-specific resources.

Can I import existing GCP resources into a Marin cluster stack?

Yes. The Infrastructure pattern supports importing pre-existing resources using standard Pulumi CLI commands. As documented in infra/pulumi/README.md, you can run pulumi import with the specific program path (e.g., infra/pulumi/src/iac/gcp/iam_binding.py) to bring existing IAM bindings or other resources under Pulumi management without recreating them.

Where should I declare new IAM permissions for a Marin cluster?

All IAM bindings must be declared in infra/pulumi/src/iac/gcp/iam_data.yaml or adjacent modules within the shared iac directory. Because the marin stack maintains exclusive ownership of IAM grants, cluster-specific programs cannot declare new permissions directly—they must reference the central IAM configuration to ensure consistent permission governance across all clusters.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}