Debugging Network Issues in K8s

Troubleshooting networking issues in docker and kubernetes is often difficult and the root cause can be in different places including the underlying networking infrastructure. I have collected some information which might be helpful.

Most importantly, is that you understand the concepts of networking in docker and kubernetes:

• Docker: Networking overview
• Kubernetes: Cluster Networking
• Kubernetes: Services, Load Balancing, and Networking

Docker images #

In order to debug issues you need the right tools. The following docker images provide a set of tools. While netshoot is my preference, the others are also very helpful:

So you can use these docker images to test dns resolution or tracing a ping

Usage on Docker #

You can start the container and then login to it

docker run -d nicolaka/netshoot --name netshoot

docker exec -it netshot /bin/bash

You can also run some commands directly rather than login to the container:

docker run -it -rm nicolaka/netshoot ping google.com

Run a command and then delete the container again:

docker run -it --rm gophernet/traceroute 8.8.8.8

Usage on Kubernetes #

You can either create a single pod - without a deployment:

kubectl run netshoot --image=nicolaka/netshoot

Or you create a deployment:

kubectl create deployment netshoot --image=nicolaka/netshoot

Then you open the console of the pod

kubectl exec -it netshoot /bin/bash

Which tool for what? #

As Linux user you might be familiar with most of the tools provided within these containers. If not, I really recommend to have a look at https://github.com/nicolaka/netshoot, which gives you a lot of details.

Kubernetes Commands #

Here is a list of helpful commands

1. Get the CoreDNS

   k get configmap -o yaml coredns -n kube-system