# How to Troubleshoot GKE Workloads: A Systematic 5-Step Diagnostic Workflow

> Troubleshoot GKE workloads efficiently with a 5-step diagnostic workflow. Analyze pods, logs, and connectivity without altering production state. Learn more now.

- Repository: [Google/skills](https://github.com/google/skills)
- Tags: how-to-guide
- Published: 2026-09-05

---

**The GKE Workload Troubleshooting skill in the `google/skills` repository provides a read-only, five-step diagnostic workflow that extracts cluster context, analyzes pod status and events, retrieves application logs, verifies service connectivity, and generates GitOps-compatible manifest patches without modifying production state.**

Troubleshooting GKE workloads requires a systematic approach to distinguish infrastructure failures from application-level bugs. The `google/skills` repository offers a comprehensive, non-interactive diagnostic skill designed specifically for Google Kubernetes Engine environments. This guide explains how to troubleshoot GKE workloads using the methodology defined in [`skills/cloud/gke-workload-troubleshooting/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/gke-workload-troubleshooting/SKILL.md), following a decision-tree approach that prioritizes safety and observability.

## The 5-Step Diagnostic Workflow

The skill executes a sequential diagnostic pipeline, starting with environment discovery and ending with GitOps-based remediation proposals.

### Step 0 – Context Discovery

The workflow automatically extracts critical environment variables including `project_id`, `cluster_name`, `cluster_location`, `workload_name`, and `workload_namespace`, defaulting to `default` when unspecified. It queries local configuration using:

```bash
gcloud config get-value project
kubectl config current-context

```

These values are stored as template variables `{project_id}`, `{cluster_name}`, etc., ensuring all subsequent commands target the correct GKE environment without manual input.

### Step 1 – Pod Status and Conditions

Analyze workload selectors and pod states to classify failure types like `CrashLoopBackOff`, `Pending`, or `ContainerCreating`:

```bash
kubectl get deployment {workload_name} -n {workload_namespace} -o jsonpath='{.spec.selector.matchLabels}'
kubectl get pods -l {selector_labels} -n {workload_namespace}

```

Based on the pod phase, the skill routes to specific downstream steps. For example, an `ExitCode: 137` triggers immediate OOM investigation in Step 3, while `ImagePullBackOff` routes to event analysis in Step 2.

### Step 2 – Namespace Events

Retrieve recent scheduling, mounting, and image pulling events using time-windowed queries centered on the incident timestamp. This filters noise to a focused 1-hour window:

```bash
kubectl get events -n {workload_namespace} --sort-by='.metadata.creationTimestamp'
gcloud logging read "resource.type=\"k8s_cluster\" AND logName=\"projects/{project_id}/logs/events\"" --start-time="[timestamp]" --end-time="[timestamp]"

```

Parse for signatures like `FailedScheduling` (resource constraints), `FailedMount` (volume issues), or `BackOff` (image pull failures).

### Step 3 – Application Logs

Extract logs from both current and terminated containers to detect OOM patterns, stack traces, network timeouts, or permission errors:

```bash
kubectl logs {pod_name} -n {workload_namespace} --all-containers --tail=100
kubectl logs {pod_name} -n {workload_namespace} -p --tail=100

```

The `-p` flag retrieves logs from the previous container instance, essential for diagnosing `CrashLoopBackOff` scenarios where the current container has restarted.

### Step 4 – Service Connectivity

Verify that dependent services are reachable and that NetworkPolicies permit traffic flow:

```bash
kubectl get endpoints {target_service_name} -n {target_namespace}
kubectl get networkpolicies -n {workload_namespace} -o yaml

```

If connectivity checks fail, the skill identifies missing endpoints or restrictive policies blocking ingress/egress.

### Step 5 – GitOps Correction

Synthesize root-cause analysis into a YAML manifest patch without applying changes directly. For example, generating a memory limit increase:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
spec:
  template:
    spec:
      containers:
      - name: payment-api
        resources:
          limits:
            memory: "512Mi"   # increased from 256Mi

          requests:
            memory: "256Mi"

```

The skill never applies changes directly; it produces a diff suitable for branch creation and pull request submission, preserving CI/CD pipeline integrity.

## Practical Troubleshooting Example

Consider diagnosing a failing `payment-api` deployment in the `default` namespace. Execute this command sequence to replicate the skill's analysis:

```bash

# 1. Extract selector labels

kubectl get deployment payment-api -n default -o jsonpath='{.spec.selector.matchLabels}'

# 2. List affected pods

kubectl get pods -l app=payment-api -n default

# 3. Inspect specific pod status

kubectl get pod payment-api-7f9c9d9d5b-abcde -n default -o yaml

# 4. Query recent cluster events (last 30 minutes)

gcloud logging read \
  "resource.type=\"k8s_cluster\" AND logName=\"projects/$PROJECT/logs/events\" \
   AND jsonPayload.involvedObject.namespace=\"default\"" \
  --start-time="$(date -u -d '30 minutes ago' +%FT%TZ)" \
  --end-time="$(date -u -d '30 minutes' +%FT%TZ)" \
  --project=$PROJECT

# 5. Retrieve logs including previous terminated container

kubectl logs payment-api-7f9c9d9d5b-abcde -n default --all-containers -p --tail=200

```

## Safety and Architecture Principles

The skill operates in **non-interactive safe mode**: if `kubectl` or `gcloud` commands fail due to authentication or network issues, it falls back to presenting the exact command sequence for manual execution rather than retrying indefinitely.

**Time-windowed logging** centers queries on a 1-hour window around the identified incident, eliminating unrelated historical noise. **Decision-tree routing** uses conditional logic (e.g., "If `ExitCode: 137` → OOM → Step 3") to guide users directly to relevant diagnostics, reducing cognitive load and resolution time.

## Related Diagnostic Resources

The `google/skills` repository provides adjacent troubleshooting capabilities:

- **[`skills/cloud/gke-node-notready/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/gke-node-notready/SKILL.md)** – Handles node `NotReady`/`Unknown` conditions, often the root cause when pods remain in `Pending`
- **[`skills/cloud/gke-observability/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/gke-observability/SKILL.md)** – Configures logging and metrics sinks for deep root-cause analysis
- **[`skills/cloud/gke-upgrades/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/gke-upgrades/SKILL.md)** – Addresses version incompatibility failures following cluster maintenance
- **[`README.md`](https://github.com/google/skills/blob/main/README.md)** – Repository overview and skill installation instructions

## Summary

- The GKE Workload Troubleshooting skill provides a read-only, 5-step diagnostic workflow ([`skills/cloud/gke-workload-troubleshooting/SKILL.md`](https://github.com/google/skills/blob/main/skills/cloud/gke-workload-troubleshooting/SKILL.md)) targeting common pod failures including `CrashLoopBackOff`, `OOMKilled`, and `ImagePullBackOff`
- It automatically extracts cluster context and uses decision trees to route between pod status checks, event analysis, log retrieval, and connectivity verification
- Time-windowed logging and targeted `jsonpath` queries filter noise to the relevant incident window
- All remediation follows GitOps principles, generating manifest patches suitable for pull request submission rather than direct cluster modification

## Frequently Asked Questions

### What makes this troubleshooting approach "read-only"?

The skill never mutates cluster state. It analyzes the current configuration, identifies root causes, and generates YAML manifest diffs (e.g., memory limit adjustments or toleration additions) intended for commit via pull request, ensuring all changes undergo code review and CI/CD validation.

### How does the skill handle unreachable clusters?

If live `gcloud` or `kubectl` commands fail due to network partitions or authentication expiration, the skill enters dry-run mode and presents the exact command sequence for human operators to execute manually. This prevents endless automated retries against unavailable endpoints.

### Which pod status indicators trigger specific diagnostic paths?

The skill employs conditional routing based on pod phases: `Pending` status triggers node resource and scheduling event checks (Step 2), `CrashLoopBackOff` prioritizes previous container logs for OOM detection (Step 3), and `ImagePullBackOff` focuses on registry authentication and image pull events in namespace logs.

### Can I use this workflow for workloads outside the default namespace?

Yes. While the skill defaults to the `default` namespace, it extracts the `workload_namespace` parameter from user prompts or environment configurations, applying the `-n` flag consistently across all `kubectl` commands and Cloud Logging queries to target specific namespaces.