# How to Troubleshoot Kubernetes Pods That Appear Healthy but Cannot Communicate > Troubleshoot Kubernetes pods that seem healthy but fail communication. Explore CNI, Services, NetworkPolicies, and iptables for network layer issues. - Repository: [Anil Kumar/DevOps-Interview-Guide](https://github.com/litu54/DevOps-Interview-Guide) - Tags: how-to-guide - Published: 2026-08-10 --- **When Kubernetes pods show a **Running** status but cannot reach each other, the issue typically lies in the networking layer beneath the pod status, requiring systematic checks of CNI plugins, Service configurations, NetworkPolicies, and iptables rules.** This guide draws from real-world interview scenarios documented in the `litu54/DevOps-Interview-Guide` repository, specifically the troubleshooting patterns captured in [`JPMorgan/DevOps_SRE.md`](https://github.com/litu54/DevOps-Interview-Guide/blob/main/JPMorgan/DevOps_SRE.md) and [`Others/DevOps_Engineer_4.md`](https://github.com/litu54/DevOps-Interview-Guide/blob/main/Others/DevOps_Engineer_4.md). If your pods report healthy status yet fail to communicate, use this systematic approach to isolate whether the root cause is at the CNI, Service, or policy layer. ## Verify Pod-Level Connectivity First Start by confirming basic Layer 3 connectivity between the pods. As noted in [`JPMorgan/DevOps_SRE.md`](https://github.com/litu54/DevOps-Interview-Guide/blob/main/JPMorgan/DevOps_SRE.md)【L9】, healthy-looking pods can still suffer from underlying network partition issues. Check that each pod has a unique IP address in the cluster network: ```bash kubectl get pod -o wide ``` Test direct pod-to-pod communication by pinging from one container to another: ```bash kubectl exec pod-a -- ping -c 3 $(kubectl get pod pod-b -o jsonpath='{.status.podIP}') ``` If the ping fails, the problem exists at the **CNI (Container Network Interface)** level. If ping succeeds but application-level calls fail, investigate the Service abstraction or higher-level networking components. ## Inspect Service Configuration and Selectors A common cause of silent communication failures is a **Service** misconfiguration where the selector does not match the pod labels, or the `port` to `targetPort` mapping is incorrect. According to [`Others/DevOps_Engineer_4.md`](https://github.com/litu54/DevOps-Interview-Guide/blob/main/Others/DevOps_Engineer_4.md)【L32-L34】, verifying these mappings is a critical troubleshooting step. Check your Service endpoints to ensure pods are actually registered: ```bash kubectl get endpoints my-svc ``` Review the Service YAML for label alignment: ```yaml apiVersion: v1 kind: Service metadata: name: backend-svc spec: selector: app: backend # must match pod label 'app: backend' ports: - port: 80 targetPort: 8080 # must match container port ``` Test the Service cluster IP directly from a pod: ```bash kubectl exec pod-a -- curl -s : ``` ## Review NetworkPolicies for Silent Blocks **NetworkPolicies** can silently drop traffic between pods in the same namespace without generating pod-level error events. List active policies to check for overly restrictive rules: ```bash kubectl get networkpolicy -n ``` A restrictive policy might block all ingress by default. Apply a minimal allow-all policy for intra-namespace traffic to test: ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-all spec: podSelector: {} ingress: - {} ``` If communication resumes after applying this policy, incrementally restrict rules to identify the specific traffic pattern being blocked. ## Examine CNI Plugin Health and Logs Most clusters use **Calico**, **Cilium**, **Weave**, or **flannel** for pod networking. Verify the CNI pods are running in `kube-system`: ```bash kubectl get pods -n kube-system -l k8s-app=calico-node ``` Check logs for IP exhaustion or VXLAN errors: ```bash kubectl logs -n kube-system ``` Look for errors like "IP-AM exhausted" or "VXLAN failure" that would prevent proper pod IP assignment despite the pod showing as Running. ## Probe iptables and kube-proxy Rules The **kube-proxy** component programs **iptables** (or IPVS) rules for Service traffic. Missing or corrupted NAT rules can cause packets to drop silently. Inspect current iptables rules on the node: ```bash sudo iptables-save | grep -i KUBE ``` Look for `KUBE-SVC` chains corresponding to your Service. If rules appear missing or corrupted, restart kube-proxy to force rule recreation: ```bash kubectl rollout restart daemonset/kube-proxy -n kube-system ``` ## Confirm DNS Resolution with CoreDNS If pods communicate via DNS names (e.g., `my-svc.default.svc.cluster.local`), verify **CoreDNS** is healthy: ```bash kubectl get pods -n kube-system -l k8s-app=kube-dns ``` Test resolution from within a pod: ```bash kubectl exec -n kube-system -- nslookup my-svc ``` A misconfigured [`resolv.conf`](https://github.com/litu54/DevOps-Interview-Guide/blob/main/resolv.conf) or broken CoreDNS deployment manifests as "no error logs but communication fails" because applications hang waiting for DNS resolution. ## Check Node-Level Firewall Rules Node-level **iptables** or host firewall configurations can override Kubernetes networking rules. SSH into the worker node and verify no DROP rules are blocking pod CIDR ranges: ```bash sudo iptables -L -v ``` Check that the node's firewall allows traffic between pod CIDR blocks and the cluster's service network. ## Capture Packets with Diagnostic Tools Use **tcpdump** within the pod's network namespace to confirm whether packets leave the container and arrive at the destination: ```bash kubectl exec -it pod-a -- tcpdump -i any -nn port ``` Run this on both the source and destination pods simultaneously. If packets leave the source but never arrive at the destination, the issue lies in the intermediate network path (CNI, node firewall, or NetworkPolicy). ## Recreate Resources to Clear Stale State When all else fails, stale iptables entries or cached endpoints may persist despite configuration corrections. Force a fresh network setup by recreating the Service and pods: ```bash kubectl delete svc my-svc kubectl delete pod pod-a pod-b kubectl apply -f my-deployment.yaml kubectl apply -f my-svc.yaml ``` This clears any lingering state in the CNI or kube-proxy. ## Summary - **Verify basic connectivity** first with `ping` and `curl` to isolate Layer 3 vs. Layer 7 issues. - **Check Service selectors** and port mappings, as mismatches cause empty endpoint lists without obvious errors. - **Review NetworkPolicies** that may silently block intra-namespace traffic. - **Inspect CNI health** and logs for IP allocation or overlay network failures. - **Validate iptables rules** programmed by kube-proxy for Service NAT. - **Test DNS resolution** through CoreDNS when using service names. - **Use tcpdump** to confirm packet flow before concluding the issue is application-level. ## Frequently Asked Questions ### Why do my pods show Running but cannot communicate? Kubernetes marks pods as **Running** once the container starts and probes succeed, but this status does not guarantee network connectivity. The issue usually resides in the **CNI plugin**, **Service configuration**, or **NetworkPolicy** rules that operate below the pod abstraction layer. ### How do I check if a Kubernetes Service is blocking traffic? Verify the Service has active endpoints using `kubectl get endpoints `. If the endpoint list is empty, the **selector labels** do not match your pod labels, or the pods are not passing readiness probes. Also check if a **NetworkPolicy** is restricting ingress to the service endpoints. ### What is the fastest way to test pod-to-pod networking in Kubernetes? Execute a shell in one pod and ping the IP of another pod directly: `kubectl exec pod-a -- ping `. If this fails, the problem is at the CNI or node level. If it succeeds but service calls fail, investigate the **Service** and **kube-proxy** configuration.