Troubleshooting Network Traffic with CRI-O and Kubernetes

Running immutable infra is the holy grail for many people, however there are times when you’ll need to get down in the weeds in order to troubleshoot issues. Let’s imagine a scenario; you need to verify that a pod is receiving traffic, but the image is built FROM scratch. As scratch containers are as minimal as possible, there’s no shell in the image, so there’s no way you can exec into it and hope to do anything remotely useful.

Clearly we need to break out tcpdump on the host, but how do you construct a filter to ensure you are only seeing the traffic you need? With the power of network namespaces, you don’t need to. Ignoring pods with host networking configured for now, each pod is isolated into it’s own network stack, and we can use this to our advantage. By running tcpdump inside the same namespace we know we’re only collecting the traffic for that pod.

I’ll be investigating a single replica of an Nginx ingress-controller Deployment. We’ll need the pod’s name in a moment:

$ kubectl get po | grep nginx
ingress-nginx-controller-58f5bc766d-hwc2d   1/1     Running   0              5d22h

With that, we now need to figure out the pod’s network namespace using crictl - this is done on the host machine for that pod. Because crictl is built with Kubernetes in mind, the containers are helpfully labelled with information from Kubernetes about the pod and container(s). You can use this information to filter list of pods:

$ crictl ps --label io.kubernetes.pod.name=ingress-nginx-controller-58f5bc766d-hwc2d
CONTAINER           IMAGE                                                              CREATED             STATE               NAME                ATTEMPT             POD ID
aa482cc25878a       89ed8c731a3870a99f6f7f2fb6bdaead3882ad80f977e141d68c0a7500c1a4d2   5 days ago          Running             controller          0                   d9f4429d95562

From this, we know the container’s ID is aa482cc25878a. We can then take that and use jq to find the network namespace path:

$ crictl inspect -o json aa482cc25878a  | jq -r '.info.runtimeSpec.linux.namespaces[] | select(.type=="network") | .path'
/var/run/netns/275cdae1-15ce-433a-a651-f0f46507ff25

With this path, we can use either nsenter or ip netns exec to run commands in the pod’s namespace. First with nsenter using the full path:

$ nsenter --net=/var/run/netns/275cdae1-15ce-433a-a651-f0f46507ff25 ip a s
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
45: eth0@if46: mtu 1500 qdisc noqueue state UP group default
    link/ether 92:b5:12:8f:55:a9 brd ff:ff:ff:ff:ff:ff link-netns df291683-c2fa-4fc5-ae66-6e9ab2f46944
    inet 10.0.3.179/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::90b5:12ff:fe8f:55a9/64 scope link
       valid_lft forever preferred_lft forever

And then with ip using just the namespace ID from the end of the path:

$ ip netns exec 275cdae1-15ce-433a-a651-f0f46507ff25 ip a s
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
45: eth0@if46: mtu 1500 qdisc noqueue state UP group default
    link/ether 92:b5:12:8f:55:a9 brd ff:ff:ff:ff:ff:ff link-netns df291683-c2fa-4fc5-ae66-6e9ab2f46944
    inet 10.0.3.179/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::90b5:12ff:fe8f:55a9/64 scope link
       valid_lft forever preferred_lft forever

Armed with this information, you’ll be able to utilise any network troubleshooting tools which are available on the host machine in order to debug networking at the container level.