Security Best Practices for Storing GCP Service Account Credentials in Kestra

Store Google Cloud service account keys in Kestra’s Secret Store and reference them using {{secret('GCP_CREDS')}} in plugin defaults to prevent credential leakage and enable rotation without code changes.

The DataTalksClub/data-engineering-zoomcamp repository provides reference implementations for modern data engineering workflows using Kestra. When connecting to Google Cloud Platform, understanding the security best practices for storing GCP service account credentials in Kestra ensures your DAGs remain compliant, auditable, and protected against accidental exposure in version control.

Why Hard-Coding Credentials Creates Unacceptable Risk

Embedding a raw JSON service account key directly into a flow YAML file introduces four critical attack vectors:

• Source-code leakage: A hard-coded key checked into Git exposes your cloud infrastructure to anyone with repository access.
• Rotation friction: Updating a compromised key requires editing every flow file, committing changes, and redeploying, increasing the window of vulnerability.
• Audit blindness: Static strings in YAML provide no trail of who accessed credentials or when they were used.
• Environment rigidity: A single repository cannot easily support different keys for development, staging, and production without complex templating.

The repository’s example flows demonstrate that separating sensitive credentials from workflow logic eliminates these risks entirely.

Use Kestra Secret Store for GCP Authentication

Kestra’s Secret Store encrypts sensitive values at rest and injects them at runtime. The serviceAccount property in GCP plugins (io.kestra.plugin.gcp.*) accepts a secret expression, allowing the engine to retrieve your JSON key only during task execution.

In 02-workflow-orchestration/flows/07_gcp_setup.yaml, the recommended pattern uses pluginDefaults to apply authentication settings globally:

pluginDefaults:
  - type: io.kestra.plugin.gcp
    values:
      serviceAccount: "{{secret('GCP_CREDS')}}"
      projectId: "{{kv('GCP_PROJECT_ID')}}"
      location: "{{kv('GCP_LOCATION')}}"
      bucket: "{{kv('GCP_BUCKET_NAME')}}"

This configuration ensures that all subsequent GCP tasks—such as io.kestra.plugin.gcp.gcs.CreateBucket or io.kestra.plugin.gcp.bigquery.Query—inherit the credentials without repeating sensitive data in task definitions. The flow 08_gcp_taxi.yaml at line 245 reuses this same secret reference pattern for its ETL operations.

Step-by-Step Implementation for GCP Workflows

1. Store the Service Account Key in Kestra Secrets

Navigate to the Kestra UI and create a new secret:

• Name: GCP_CREDS
• Type: Plain text or File upload
• Value: The entire contents of your downloaded service account JSON key

Alternatively, use the Kestra CLI for automation:

kestra secret create GCP_CREDS --value "$(cat path/to/key.json)"

2. Configure Plugin Defaults in Your Flow

Reference the secret in your flow YAML using the expression {{secret('GCP_CREDS')}}. Store this flow in version control without risk, as only the secret identifier appears in the file.

3. Separate Configuration from Secrets Using KV Store

For non-sensitive, environment-specific values like project IDs or regions, use the KV Store rather than secrets. This distinction allows developers to view configuration while keeping credentials opaque:

kestra kv set GCP_PROJECT_ID my-gcp-project
kestra kv set GCP_LOCATION us-central1

In the flow, reference these with {{kv('GCP_PROJECT_ID')}}. This pattern, demonstrated in 07_gcp_setup.yaml, keeps your YAML portable across environments.

Production Security Enhancements

Beyond basic secret storage, harden your Kestra deployment with these additional controls:

• Encryption at rest: Ensure your PostgreSQL backend uses pgcrypto or disk-level encryption to protect the secret database.
• Fine-grained RBAC: Restrict secret read/write permissions to specific roles (e.g., admin), preventing operators from viewing raw credentials in the UI.
• Audit logging: Enable Kestra’s audit logs to record every secret access with user identity and timestamp for compliance reporting.
• Workload Identity: For production deployments on GCP, configure Kestra to run under a Workload Identity-federated service account. This eliminates the need for JSON keys entirely; omit the serviceAccount property and let the VM or Cloud Run instance authenticate natively.

Summary

• Never embed GCP service account JSON keys directly in flow YAML files.
• Store sensitive keys in the Kestra Secret Store under a descriptive name like GCP_CREDS.
• Reference secrets using {{secret('NAME')}} syntax in pluginDefaults to apply them across all GCP tasks.
• Separate configuration (project ID, location) from credentials using the KV Store.
• Rotate keys by updating the secret value in Kestra, requiring zero code changes or redeployments.

Frequently Asked Questions

What happens if I accidentally commit a GCP service account key to Git?

Immediately revoke the key in the Google Cloud Console to prevent unauthorized access. Because Kestra separates secrets from flow definitions, moving forward you only need to store the secret identifier (e.g., GCP_CREDS) in YAML while the actual value lives encrypted in Kestra’s database.

How do I rotate a GCP service account key without stopping my Kestra flows?

Update the secret value in the Kestra UI or via the API. All flows referencing {{secret('GCP_CREDS')}} will automatically receive the new value on their next execution without requiring Git commits or redeployments.

Should I use Kestra Secrets or Workload Identity for GCP authentication?

Use Workload Identity for production workloads running on GCP infrastructure (GKE, Cloud Run, Compute Engine), as it eliminates JSON keys entirely. Use Kestra Secrets for hybrid environments, local development, or when running Kestra outside GCP, as demonstrated in the DataTalksClub repository’s setup flows.

Can I use environment variables instead of the Secret Store for local development?

While the repository’s 03-data-warehouse/extras/.env-example shows fallbacks like GOOGLE_APPLICATION_CREDENTIALS, treat this only as local convenience. For production orchestration, migrate to the Secret Store to gain encryption at rest, access logging, and centralized rotation capabilities.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}