How to Debug Kubernetes Pod Crash Loops with Advanced kubectl Commands and Log Analysis

To effectively debug Kubernetes pod crash loops, use kubectl describe pod to inspect the Last State and termination exit codes, kubectl logs --previous to capture output from the crashed container instance, and kubectl get pod -o yaml to validate probe configurations and resource limits.

A CrashLoopBackOff status indicates that a Kubernetes container starts, fails, and restarts repeatedly, creating a restart loop that prevents application availability. According to the bregman-arie/devops-exercises repository—specifically the troubleshooting guidance in topics/kubernetes/README.md (line 449) and the CKA preparation notes in topics/kubernetes/CKA.md (line 178)—systematic debugging requires analyzing pod events, container logs, and runtime specifications to identify the root cause. Mastering these advanced kubectl techniques enables you to diagnose misconfigurations, dependency failures, or resource constraints causing the crash loop.

Understanding the CrashLoopBackOff State

The CrashLoopBackOff state occurs when a container exits with an error before completing its startup sequence, triggering Kubernetes to back off and retry with increasing delays. As documented in topics/kubernetes/README.md, this pattern typically signals misconfigured environment variables, missing secrets, invalid command arguments, or application runtime errors that prevent successful initialization. Recognizing this state early allows you to focus diagnostic efforts on the container's last execution rather than the current waiting state.

Step-by-Step Debugging Workflow

1. Identify Pods in CrashLoopBackOff

Begin by locating all affected pods across namespaces to prioritize troubleshooting efforts.

kubectl get pods -A | grep CrashLoopBackOff

Filter by namespace if you know the target workload, or use -A for cluster-wide visibility.

2. Inspect Pod Status and Events

The kubectl describe command reveals the State, Last State, and Events that pinpoint the failure mechanism. As emphasized in topics/kubernetes/CKA.md (line 178), focus on these specific fields to differentiate between configuration errors and runtime crashes.

kubectl describe pod <pod-name> -n <namespace>

Key fields to examine include:

• State: Confirms the pod is in a restart loop (e.g., Waiting, CrashLoopBackOff).
• Last State: Displays the exit code and termination reason from the previous run (e.g., Terminated, OOMKilled, exit code 137).
• Events: Surface immediate issues such as FailedMount, BackOff, or image pull errors.

3. Analyze Container Logs

Retrieve stdout and stderr output from both the current attempt and the previously crashed instance to capture the exact error message.

# Current attempt logs

kubectl logs <pod-name> -n <namespace>

# Previous (crashed) container instance logs

kubectl logs <pod-name> -n <namespace> --previous

For multi-container pods, specify the target container with the -c flag: kubectl logs <pod> -c <container> --previous.

4. Check Init Container Failures

A failing init container blocks the main application container from starting, often manifesting as a crash loop on the primary pod status.

kubectl logs <pod-name> -n <namespace> -c <init-container-name>

If the init container exits with an error, the main container never starts, causing Kubernetes to report the pod as crashing despite the application image being valid.

5. Validate Runtime Configuration

Export the full pod specification to inspect hidden configuration issues such as misconfigured probes, resource limits, or security contexts.

kubectl get pod <pod-name> -n <namespace> -o yaml

Analyze these specific sections:

• Liveness and Readiness Probes: Verify httpGet paths, ports, and initialDelaySeconds settings that may trigger premature restarts.
• Resources: Check limits and requests for CPU and memory; an OOMKilled status indicates the container exceeded its memory limit.
• ImagePullSecrets: Confirm registry credentials are correctly mounted if the image resides in a private repository.
• SecurityContext: Review runAsUser, fsGroup, and privileged settings that may violate node-level PodSecurityPolicies.

6. Monitor Real-Time State Changes

Use the watch flag to observe rapid transitions between states that may not appear in static snapshots.

kubectl get pod <pod-name> -n <namespace> -w

Combine with --output=wide to view node assignment and identify node-affinity or taint-related scheduling issues.

7. Leverage Ephemeral Containers for Live Debugging

When a container crashes too quickly for kubectl exec, attach an ephemeral container (requires Kubernetes 1.23+) to inspect the filesystem without modifying the original pod spec.

kubectl debug -n <namespace> <pod-name> -it --image=busybox --target=<container>

This technique allows you to examine configuration files, environment variables, or network connectivity from within the pod's namespace while the main container is in a crash loop.

8. Check Resource Metrics for Throttling

High CPU throttling or memory pressure can cause applications to fail health checks and restart repeatedly.

kubectl top pod <pod-name> -n <namespace>

This requires the metrics-server component. If CPU usage approaches the configured limit, increase the resource allocation or optimize the application code to prevent throttling-induced crashes.

9. Reproduce the Failure Locally

If logs are inconclusive, run the container image locally to isolate infrastructure issues from application bugs.

docker run --rm -it <image> <entrypoint-or-cmd>

Observe immediate error messages or missing dependency failures that Kubernetes may suppress or log differently.

10. Apply Fixes and Verify

After correcting the manifest—whether fixing a probe path, adding a missing secret, or adjusting resource limits—apply the changes and verify the pod reaches the Running state.

kubectl apply -f corrected-pod.yaml

# Or for Deployments:

kubectl rollout restart deployment/<deployment-name> -n <namespace>

Re-run kubectl get pods and kubectl logs to confirm the restart count stabilizes at zero and the application responds to health checks.

Summary

• CrashLoopBackOff indicates a container is starting, crashing, and restarting repeatedly due to application or configuration errors.
• kubectl describe pod reveals the Last State and exit codes that classify the failure type (e.g., OOMKilled, Error).
• kubectl logs --previous captures the stderr/stdout output from the crashed instance before Kubernetes restarts it.
• kubectl get pod -o yaml exposes probe definitions, resource limits, and security contexts that may trigger premature terminations.
• Ephemeral containers provide shell access to crashing pods when kubectl exec is unavailable due to rapid container exits.
• The bregman-arie/devops-exercises repository documents these techniques in topics/kubernetes/README.md (line 449) and topics/kubernetes/CKA.md (line 178) as essential CKA exam troubleshooting skills.

Frequently Asked Questions

What is the difference between kubectl logs and kubectl describe when debugging crash loops?

kubectl logs displays the stdout and stderr output generated by the application inside the container, revealing runtime errors, stack traces, or missing dependency messages. kubectl describe queries the Kubernetes API server to show pod lifecycle events, scheduling decisions, and the Last State field containing the exit code and termination reason. Use logs to see what the application said before dying, and describe to see why Kubernetes thinks it died (e.g., OOMKilled vs. Error).

How can I debug a container that exits too quickly for kubectl exec?

Use kubectl debug to attach an ephemeral container to the existing pod namespace. This technique, available in Kubernetes 1.23+, allows you to run a debugging image (such as busybox or nicolaka/netshoot) alongside the crashing container. You can then inspect the filesystem, network configuration, or environment variables without requiring the target container to remain running.

What does exit code 137 indicate in a Kubernetes CrashLoopBackOff?

Exit code 137 (128 + 9) indicates the container received a SIGKILL signal, typically because it exceeded its configured memory limit and the Linux OOM (Out of Memory) killer terminated the process. When you see this in the Last State field of kubectl describe, increase the resources.limits.memory value in the pod spec or optimize the application's memory footprint to prevent the kernel from force-killing the container.

Why does my pod show CrashLoopBackOff even when kubectl logs returns no output?

If kubectl logs shows no output but kubectl logs --previous also shows nothing, the container likely failed during initialization before the application started logging, or the logging driver is misconfigured. Check init container logs with kubectl logs <pod> -c <init-container>, as a failing init container blocks the main container and causes the crash loop. Also verify the container command and entrypoint in kubectl get pod -o yaml to ensure the specified binary exists in the image.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}