# GKE Disaster Recovery and Backup Best Practices: A Complete Implementation Guide > Master GKE disaster recovery and backup best practices. Implement native GKE Backup, application hooks, and cross-regional restores for minimal downtime and rapid recovery. - Repository: [Google/skills](https://github.com/google/skills) - Tags: best-practices - Published: 2026-08-13 --- **GKE disaster recovery and backup best practices require enabling native GKE Backup for cluster-level protection, implementing application-aware pre-backup hooks for stateful workloads, and configuring cross-regional restore plans to achieve near-zero RPO and rapid RTO.** Implementing robust disaster recovery for Google Kubernetes Engine starts with the native backup capabilities documented in the `google/skills` repository. According to the source code analysis of [`skills/cloud/gke-backup-dr/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/gke-backup-dr/SKILL.md), production GKE deployments require automated backup plans that capture both cluster state and PersistentVolumeClaim data while supporting granular namespace-level restores. This guide covers the technical implementation details derived directly from the repository's skill files and production readiness checklists. ## Enable GKE Backup for Cluster-Level Protection GKE Backup is a native service that creates **Backup Plans** and **Restore Plans** for entire clusters or selected namespaces. As documented in lines 20-38 of [`skills/cloud/gke-backup-dr/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/gke-backup-dr/SKILL.md), the service captures the state of PersistentVolumeClaims (PVCs) through CSI snapshots and stores backup metadata in a regional backup repository. Enable the backup service on your cluster: ```bash gcloud container clusters update \ --enable-gke-backup --region --quiet ``` Create a scheduled backup plan: ```bash gcloud container backup-restore backup-plans create \ --cluster= --location= \ --schedule="0 2 * * *" ``` Trigger an on-demand backup: ```bash gcloud container backup-restore backups create \ --backup-plan= --location= --quiet ``` ## Configure Cross-Regional Restore Strategies Disaster recovery requires the ability to restore workloads to alternate regions when primary regions become unavailable. The restore workflow involves creating a restore plan targeting a different cluster, then executing the restoration. Create a restore plan for your DR cluster: ```bash gcloud container backup-restore restore-plans create \ --cluster= --location= \ --backup-plan= ``` Execute the restore operation: ```bash gcloud container backup-restore restores create \ --restore-plan= --backup= \ --location= --quiet ``` ## Protect Stateful Workloads with Pre-Backup Hooks For databases, queues, or any stateful service, complement GKE Backup with **application-aware backup hooks** that run pre-backup to quiesce the workload. According to the "pre-backup hooks" section in [`skills/cloud/gke-backup-dr/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/gke-backup-dr/SKILL.md), these hooks ensure data consistency before snapshots are taken. Example pre-backup hook for MySQL: ```yaml apiVersion: backup.gke.io/v1 kind: BackupHook metadata: name: mysql-quiesce spec: exec: command: ["/bin/sh", "-c", "mysqladmin flush-tables --user=$USER --password=$PASS"] ``` ## Secure Backups with CMEK Encryption Protect backup data using Customer-Managed Encryption Keys (CMEK) to meet compliance requirements. As implemented in lines 46-53 of [`skills/cloud/gke-backup-dr/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/gke-backup-dr/SKILL.md), GKE Backup supports encryption via the `--backup-encryption-key` flag. **Standard GKE Backup** stores snapshots in a regional bucket managed by Google with optional CMEK encryption. For large-scale workloads, consider **Backup for GKE** with **CSI Volume Snapshots** to accelerate restore times while maintaining encryption boundaries. ## Automate Backup Verification and Monitoring Regular verification prevents silent backup failures. Lines 41-45 of the skill file demonstrate how to list and describe backups for validation: ```bash gcloud container backup-restore backups list --location= gcloud container backup-restore backups describe --location= ``` Add validation steps in your CI/CD pipeline to confirm that the latest backup succeeded and that restore tests have passed. For operational monitoring, enable **Backup metrics** in Cloud Monitoring and configure alerts for failed jobs: ```bash gcloud monitoring alerts create \ --condition-display-name="GKE Backup Failure" \ --condition-filter='metric.type="gke_backup/backup_successful" AND metric.label."status"="FAILED"' \ --notification-channels= ``` ## Optimize Costs with Retention Policies and ComputeClasses Control storage costs by implementing retention policies that retain only necessary daily, weekly, or monthly backups. The [`skills/cloud/gke-productionize/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/gke-productionize/SKILL.md) file references backup configuration as a required production readiness step, emphasizing cost management alongside data protection. For compute optimization, use **Autopilot ComputeClasses** (Balanced, Scale-Out, Performance) to match workload I/O characteristics while avoiding over-provisioned nodes during recovery operations. This guidance appears in the architecture guides referenced at [`skills/cloud/google-cloud-solution-architecture/references/architecture-guides.md`](https://github.com/google/skills/blob/main/skills/cloud/google-cloud-solution-architecture/references/architecture-guides.md). ## Implement Disaster Recovery Runbooks Create operational runbooks covering specific failure scenarios. The [`skills/cloud/gke-productionize/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/gke-productionize/SKILL.md) checklist mandates running the `gke-backup-dr` skill as part of production readiness, ensuring teams document procedures for: - **Cluster-wide outage**: Restore from the latest backup plan to a new GKE cluster in the same or a different region - **Namespace failure**: Use Restore Plans targeting only the affected namespace - **Persistent volume corruption**: Trigger a snapshot-based restore of the specific PVC - **Data-center loss**: Deploy a regional GKE cluster (multi-zone) and restore backups from the secondary region ## Summary - **Enable GKE Backup** on production clusters using `gcloud container clusters update --enable-gke-backup` to activate the native backup service - **Configure Backup Plans** with cron schedules in [`skills/cloud/gke-backup-dr/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/gke-backup-dr/SKILL.md) to automate daily snapshots of PVCs and cluster state - **Implement pre-backup hooks** for stateful applications to ensure database consistency before CSI snapshots execute - **Encrypt backups** using CMEK via the `--backup-encryption-key` parameter for compliance-sensitive workloads - **Test cross-regional restores** regularly to validate disaster recovery procedures and minimize RTO during actual outages - **Monitor backup health** using Cloud Monitoring metrics and automate verification in CI/CD pipelines ## Frequently Asked Questions ### What is the difference between GKE Backup and manual persistent disk snapshots? **GKE Backup automates the orchestration of CSI snapshots across all PVCs in a cluster while capturing Kubernetes metadata, IAM bindings, and resource configurations.** Manual snapshots only capture disk state without the Kubernetes context needed for complete application restoration. According to [`skills/cloud/gke-backup-dr/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/gke-backup-dr/SKILL.md), the service stores backup metadata in regional repositories and supports namespace-granular restores that manual snapshots cannot achieve. ### How do I protect databases running on GKE during backup operations? **Implement application-aware pre-backup hooks that quiesce databases before snapshot creation.** The [`skills/cloud/gke-backup-dr/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/gke-backup-dr/SKILL.md) file provides examples using `BackupHook` resources to execute commands like `mysqladmin flush-tables` or `pg_basebackup` immediately before the CSI snapshot occurs. This ensures crash-consistent backups for transaction-heavy workloads. ### Can I restore a GKE backup to a different region for disaster recovery? **Yes, GKE Backup supports cross-regional restores by creating restore plans that target clusters in alternate regions.** As documented in the source commands, you specify a different `--location` and `--cluster` when creating the restore plan, allowing you to recover from regional outages by restoring to a DR cluster in `us-east1` from backups captured in `us-central1`. ### How do I monitor GKE backup job failures and set up alerting? **Enable the `gke_backup/backup_successful` metric in Cloud Monitoring and create alert policies for FAILED status labels.** The repository examples show using `gcloud monitoring alerts create` with filters targeting `metric.type="gke_backup/backup_successful" AND metric.label."status"="FAILED"`. Additionally, automate verification by listing backups via `gcloud container backup-restore backups list` in your CI/CD pipelines to detect configuration drift or permission issues before disasters occur.