Managing GKE Secrets with Secret Manager Integration: Configuration and Best Practices

GKE clusters can automatically sync secrets from Google Secret Manager to Kubernetes Secret objects using Workload Identity, eliminating static service-account keys and supporting automatic rotation every 120 seconds.

The google/skills repository establishes a golden-path security default that enables Secret Manager integration during cluster creation. This approach ensures workloads can consume versioned, encrypted secrets without embedding long-lived credentials in container images or configuration files.

Architecture of GKE Secret Manager Integration

The integration operates through three core components that automate secret propagation while maintaining strict security boundaries.

How the Sync Works

Google Secret Manager stores versioned secrets encrypted at rest. The GKE Secret Manager integration periodically polls for the latest secret version and automatically creates or updates corresponding Kubernetes Secret objects in the target namespace. According to the reference configuration in skills/cloud/gke-basics/references/gke-security.md, this synchronization occurs with an automatic rotation interval of 120 secondsL9-L14.

Workload Identity and Authentication

Workload Identity maps a Kubernetes Service Account (KSA) to a Google Service Account (GSA). Pods authenticate to Secret Manager via the GSA, eliminating the need for embedded service-account keys. The KSA requires a specific annotation to establish this mapping:

metadata:
  annotations:
    iam.gke.io/gcp-service-account: my-app-gsa@${PROJECT}.iam.gserviceaccount.com

Enabling Secret Manager on GKE Clusters

The Skills repository recommends enabling this feature as a Day-1 configuration. When a cluster is created, the secretManagerConfig.enabled flag is set to true by default, with automatic rotation configured at 120-second intervalsL9-L14.

For existing clusters created without this feature, enable it using the following gcloud commands referenced in skills/cloud/gke-basics/references/gke-security.md:

# Verify current state

gcloud container clusters describe $CLUSTER_NAME --region $REGION \
  --format="value(secretManagerConfig.enabled)" --quiet

# Enable Secret Manager with automatic rotation (120s)

gcloud container clusters update $CLUSTER_NAME --region $REGION \
  --enable-secret-manager \
  --secret-manager-rotation-interval=120s \
  --quiet

Syncing Secrets to Kubernetes

To consume secrets stored in Secret Manager, complete the following configuration steps:

1. Create the secret in Secret Manager (e.g., my-db-password).

2. Grant IAM permissions to the GSA using the minimal privileged role:

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member "serviceAccount:${GSA}@${PROJECT_ID}.iam.gserviceaccount.com" \
  --role "roles/secretmanager.secretAccessor" \
  --quiet

3. Annotate the Kubernetes Service Account to link it to the GSA:

kubectl annotate serviceaccount my-app-ksa \
  -n prod iam.gke.io/gcp-service-account=${GSA}@${PROJECT_ID}.iam.gserviceaccount.com

4. Reference the secret in your pod specification. The GKE integration automatically creates the Kubernetes Secret, which you can reference using standard secretKeyRef syntax:

apiVersion: v1
kind: Pod
metadata:
  name: demo-pod
  namespace: prod
spec:
  serviceAccountName: my-app-ksa
  containers:
  - name: app
    image: gcr.io/google-appengine/python
    env:
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: my-db-password
          key: latest

Security Best Practices and Access Control

Adhering to the principle of least privilege is critical when managing GKE secrets with Secret Manager integration. The configuration defined in skills/cloud/gke-basics/references/gke-security.md emphasizes the following controlsL95-L99:

• Grant only roles/secretmanager.secretAccessor to the GSA rather than broader roles
• Enable automatic rotation (120s interval) to reduce exposure windows if secrets are compromised
• Use Workload Identity exclusively instead of downloading and mounting service-account keys
• Enable Secret Manager at cluster creation to ensure consistent security posture across all workloads

Troubleshooting Common Issues

If secret rotation fails or pods cannot access secrets, verify the GSA possesses the secretmanager.versions.access permission. The CLI troubleshooting guide in skills/cloud/gke-basics/references/cli-reference.md notes that missing this specific permission is the most common cause of sync failuresL237-L242.

Check rotation status using:

gcloud container clusters describe $CLUSTER_NAME --region $REGION \
  --format="value(secretManagerConfig.enabled)" --quiet

Summary

• GKE Secret Manager integration automatically syncs versioned secrets to Kubernetes Secret objects every 120 seconds when enabled
• Workload Identity removes the need for static service-account keys by binding KSAs to GSAs
• Enable the feature at cluster creation via secretManagerConfig.enabled for the most secure default posture
• Grant minimal IAM permissions (roles/secretmanager.secretAccessor) to adhere to least-privilege principles
• Reference files: Implementation details reside in skills/cloud/gke-basics/references/gke-security.md, with troubleshooting guidance in skills/cloud/gke-basics/references/cli-reference.md

Frequently Asked Questions

How do I verify Secret Manager is enabled on my GKE cluster?

Run the command gcloud container clusters describe $CLUSTER_NAME --region $REGION --format="value(secretManagerConfig.enabled)" to check the current state. If the output is empty or false, the integration is not active and you must run the cluster update command with --enable-secret-manager to enable it.

What IAM permissions does the Google Service Account need?

The GSA requires only the roles/secretmanager.secretAccessor role, which provides the secretmanager.secrets.get and secretmanager.versions.access permissions. According to the Skills repository security guidelines, you should avoid granting broader roles like roles/secretmanager.admin to workload identities.

How often does GKE sync secrets from Secret Manager?

By default, the Skills repository configures a 120-second rotation interval. This means GKE polls Secret Manager for new secret versions every two minutes and updates the corresponding Kubernetes Secret objects automatically. You can verify this interval using the gcloud container clusters describe command.

Can I enable Secret Manager integration on an existing GKE cluster?

Yes, you can enable it on existing clusters using the gcloud container clusters update command with the --enable-secret-manager flag. However, this represents a Day-1 security change, so you should plan for a brief maintenance window or rolling node pool updates to ensure all nodes receive the updated configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}