# How to Debug Kubernetes Pod Crash Loops with Advanced kubectl Commands and Log Analysis > Effectively debug Kubernetes pod crash loops. Learn advanced kubectl commands like describe and logs --previous, plus log analysis techniques, to quickly resolve issues. - Repository: [Arie Bregman/devops-exercises](https://github.com/bregman-arie/devops-exercises) - Tags: how-to-guide - Published: 2026-02-28 --- **To effectively debug Kubernetes pod crash loops, use `kubectl describe pod` to inspect the Last State and termination exit codes, `kubectl logs --previous` to capture output from the crashed container instance, and `kubectl get pod -o yaml` to validate probe configurations and resource limits.** A **CrashLoopBackOff** status indicates that a Kubernetes container starts, fails, and restarts repeatedly, creating a restart loop that prevents application availability. According to the **bregman-arie/devops-exercises** repository—specifically the troubleshooting guidance in [`topics/kubernetes/README.md`](https://github.com/bregman-arie/devops-exercises/blob/main/topics/kubernetes/README.md) (line 449) and the CKA preparation notes in [`topics/kubernetes/CKA.md`](https://github.com/bregman-arie/devops-exercises/blob/main/topics/kubernetes/CKA.md) (line 178)—systematic debugging requires analyzing pod events, container logs, and runtime specifications to identify the root cause. Mastering these advanced `kubectl` techniques enables you to diagnose misconfigurations, dependency failures, or resource constraints causing the crash loop. ## Understanding the CrashLoopBackOff State The **CrashLoopBackOff** state occurs when a container exits with an error before completing its startup sequence, triggering Kubernetes to back off and retry with increasing delays. As documented in [`topics/kubernetes/README.md`](https://github.com/bregman-arie/devops-exercises/blob/main/topics/kubernetes/README.md), this pattern typically signals misconfigured environment variables, missing secrets, invalid command arguments, or application runtime errors that prevent successful initialization. Recognizing this state early allows you to focus diagnostic efforts on the container's last execution rather than the current waiting state. ## Step-by-Step Debugging Workflow ### 1. Identify Pods in CrashLoopBackOff Begin by locating all affected pods across namespaces to prioritize troubleshooting efforts. ```bash kubectl get pods -A | grep CrashLoopBackOff ``` Filter by namespace if you know the target workload, or use `-A` for cluster-wide visibility. ### 2. Inspect Pod Status and Events The `kubectl describe` command reveals the **State**, **Last State**, and **Events** that pinpoint the failure mechanism. As emphasized in [`topics/kubernetes/CKA.md`](https://github.com/bregman-arie/devops-exercises/blob/main/topics/kubernetes/CKA.md) (line 178), focus on these specific fields to differentiate between configuration errors and runtime crashes. ```bash kubectl describe pod -n ``` Key fields to examine include: - **State**: Confirms the pod is in a restart loop (e.g., `Waiting`, `CrashLoopBackOff`). - **Last State**: Displays the exit code and termination reason from the previous run (e.g., `Terminated`, `OOMKilled`, exit code 137). - **Events**: Surface immediate issues such as `FailedMount`, `BackOff`, or image pull errors. ### 3. Analyze Container Logs Retrieve stdout and stderr output from both the current attempt and the previously crashed instance to capture the exact error message. ```bash # Current attempt logs kubectl logs -n # Previous (crashed) container instance logs kubectl logs -n --previous ``` For multi-container pods, specify the target container with the `-c` flag: `kubectl logs -c --previous`. ### 4. Check Init Container Failures A failing **init container** blocks the main application container from starting, often manifesting as a crash loop on the primary pod status. ```bash kubectl logs -n -c ``` If the init container exits with an error, the main container never starts, causing Kubernetes to report the pod as crashing despite the application image being valid. ### 5. Validate Runtime Configuration Export the full pod specification to inspect hidden configuration issues such as misconfigured probes, resource limits, or security contexts. ```bash kubectl get pod -n -o yaml ``` Analyze these specific sections: - **Liveness and Readiness Probes**: Verify `httpGet` paths, ports, and `initialDelaySeconds` settings that may trigger premature restarts. - **Resources**: Check `limits` and `requests` for CPU and memory; an `OOMKilled` status indicates the container exceeded its memory limit. - **ImagePullSecrets**: Confirm registry credentials are correctly mounted if the image resides in a private repository. - **SecurityContext**: Review `runAsUser`, `fsGroup`, and `privileged` settings that may violate node-level PodSecurityPolicies. ### 6. Monitor Real-Time State Changes Use the watch flag to observe rapid transitions between states that may not appear in static snapshots. ```bash kubectl get pod -n -w ``` Combine with `--output=wide` to view node assignment and identify node-affinity or taint-related scheduling issues. ### 7. Leverage Ephemeral Containers for Live Debugging When a container crashes too quickly for `kubectl exec`, attach an **ephemeral container** (requires Kubernetes 1.23+) to inspect the filesystem without modifying the original pod spec. ```bash kubectl debug -n -it --image=busybox --target= ``` This technique allows you to examine configuration files, environment variables, or network connectivity from within the pod's namespace while the main container is in a crash loop. ### 8. Check Resource Metrics for Throttling High CPU throttling or memory pressure can cause applications to fail health checks and restart repeatedly. ```bash kubectl top pod -n ``` This requires the **metrics-server** component. If CPU usage approaches the configured limit, increase the resource allocation or optimize the application code to prevent throttling-induced crashes. ### 9. Reproduce the Failure Locally If logs are inconclusive, run the container image locally to isolate infrastructure issues from application bugs. ```bash docker run --rm -it ``` Observe immediate error messages or missing dependency failures that Kubernetes may suppress or log differently. ### 10. Apply Fixes and Verify After correcting the manifest—whether fixing a probe path, adding a missing secret, or adjusting resource limits—apply the changes and verify the pod reaches the `Running` state. ```bash kubectl apply -f corrected-pod.yaml # Or for Deployments: kubectl rollout restart deployment/ -n ``` Re-run `kubectl get pods` and `kubectl logs` to confirm the restart count stabilizes at zero and the application responds to health checks. ## Summary - **CrashLoopBackOff** indicates a container is starting, crashing, and restarting repeatedly due to application or configuration errors. - **`kubectl describe pod`** reveals the **Last State** and exit codes that classify the failure type (e.g., `OOMKilled`, `Error`). - **`kubectl logs --previous`** captures the stderr/stdout output from the crashed instance before Kubernetes restarts it. - **`kubectl get pod -o yaml`** exposes probe definitions, resource limits, and security contexts that may trigger premature terminations. - **Ephemeral containers** provide shell access to crashing pods when `kubectl exec` is unavailable due to rapid container exits. - The **bregman-arie/devops-exercises** repository documents these techniques in [`topics/kubernetes/README.md`](https://github.com/bregman-arie/devops-exercises/blob/main/topics/kubernetes/README.md) (line 449) and [`topics/kubernetes/CKA.md`](https://github.com/bregman-arie/devops-exercises/blob/main/topics/kubernetes/CKA.md) (line 178) as essential CKA exam troubleshooting skills. ## Frequently Asked Questions ### What is the difference between `kubectl logs` and `kubectl describe` when debugging crash loops? **`kubectl logs`** displays the stdout and stderr output generated by the application inside the container, revealing runtime errors, stack traces, or missing dependency messages. **`kubectl describe`** queries the Kubernetes API server to show pod lifecycle events, scheduling decisions, and the **Last State** field containing the exit code and termination reason. Use logs to see what the application said before dying, and describe to see why Kubernetes thinks it died (e.g., `OOMKilled` vs. `Error`). ### How can I debug a container that exits too quickly for `kubectl exec`? Use **`kubectl debug`** to attach an ephemeral container to the existing pod namespace. This technique, available in Kubernetes 1.23+, allows you to run a debugging image (such as `busybox` or `nicolaka/netshoot`) alongside the crashing container. You can then inspect the filesystem, network configuration, or environment variables without requiring the target container to remain running. ### What does exit code 137 indicate in a Kubernetes CrashLoopBackOff? Exit code **137** (128 + 9) indicates the container received a **SIGKILL** signal, typically because it exceeded its configured **memory limit** and the Linux OOM (Out of Memory) killer terminated the process. When you see this in the `Last State` field of `kubectl describe`, increase the `resources.limits.memory` value in the pod spec or optimize the application's memory footprint to prevent the kernel from force-killing the container. ### Why does my pod show CrashLoopBackOff even when `kubectl logs` returns no output? If `kubectl logs` shows no output but `kubectl logs --previous` also shows nothing, the container likely failed during initialization before the application started logging, or the logging driver is misconfigured. Check **init container** logs with `kubectl logs -c `, as a failing init container blocks the main container and causes the crash loop. Also verify the container command and entrypoint in `kubectl get pod -o yaml` to ensure the specified binary exists in the image.