With the 1.4 release of Kubernetes, Google have made instantiating a cluster a whole lot easier. Using Kubeadm, you can bring up a cluster with a single command on each node. A further command will create a DaemonSet which brings up a Weave mesh network between all your nodes.
As always with complex systems such as Kubernetes, there are some potential pitfalls to be aware of. Firstly, the getting started guide notes that v1.11.2 of Docker is recommended, but v1.10.3 and v1.12.1 also work well (don’t go straight for the latest release like I tried to). If you wish to have your nodes talk over a private network, you’ll also need to explicitly declare this when you init the master node, otherwise Kubernetes will default to using your primary interface/route:
root@kube1:~# kubeadm init --api-advertise-addresses=172.16.0.128 --api-external-dns-names=kube1.int.domain.com Running pre-flight checks <master/tokens> generated token: "486896.cc87489554fbb1c7" <master/pki> generated Certificate Authority key and certificate: Issuer: CN=kubernetes | Subject: CN=kubernetes | CA: true Not before: 2016-11-16 20:36:56 +0000 UTC Not After: 2026-11-14 20:36:56 +0000 UTC Public: /etc/kubernetes/pki/ca-pub.pem Private: /etc/kubernetes/pki/ca-key.pem Cert: /etc/kubernetes/pki/ca.pem <master/pki> generated API Server key and certificate: Issuer: CN=kubernetes | Subject: CN=kube-apiserver | CA: false Not before: 2016-11-16 20:36:56 +0000 UTC Not After: 2017-11-16 20:36:56 +0000 UTC Alternate Names: [172.16.0.128 10.96.0.1 kube1.int.domain.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] Public: /etc/kubernetes/pki/apiserver-pub.pem Private: /etc/kubernetes/pki/apiserver-key.pem Cert: /etc/kubernetes/pki/apiserver.pem <master/pki> generated Service Account Signing keys: Public: /etc/kubernetes/pki/sa-pub.pem Private: /etc/kubernetes/pki/sa-key.pem <master/pki> created keys and certificates in "/etc/kubernetes/pki" <util/kubeconfig> created "/etc/kubernetes/kubelet.conf" <util/kubeconfig> created "/etc/kubernetes/admin.conf" <master/apiclient> created API client configuration <master/apiclient> created API client, waiting for the control plane to become ready <master/apiclient> all control plane components are healthy after 13.060118 seconds <master/apiclient> waiting for at least one node to register and become ready <master/apiclient> first node is ready after 6.003302 seconds <master/apiclient> attempting a test deployment <master/apiclient> test deployment succeeded <master/discovery> created essential addon: kube-discovery, waiting for it to become ready <master/discovery> kube-discovery is ready after 1.003735 seconds <master/addons> created essential addon: kube-proxy <master/addons> created essential addon: kube-dns Kubernetes master initialised successfully! You can now join any number of machines by running the following on each node: kubeadm join --token=486896.cc87489554fbb1c7 172.16.0.128
With that done, you can see all the service containers up and running on the master node:
root@kube1:~# kubectl get pods --namespace=kube-system NAME READY STATUS RESTARTS AGE dummy-2088944543-whf5o 1/1 Running 0 14h etcd-kube1.domain.com 1/1 Running 0 14h kube-apiserver-kube1.domain.com 1/1 Running 0 14h kube-controller-manager-kube1.domain.com 1/1 Running 0 14h kube-discovery-1150918428-qy41j 1/1 Running 0 14h kube-dns-654381707-s92ze 1/1 ContainerCreating 0 14h kube-proxy-2fh61 1/1 Running 0 13h kube-scheduler-kube1.domain.com 1/1 Running 0 14h
It’s now time to join your slave nodes to the master:
root@kube2:~# kubeadm join --token=871c6f.af7b03c1d93ad11a 172.16.0.128 Running pre-flight checks <util/tokens> validating provided token <node/discovery> created cluster info discovery client, requesting info from "http://172.16.0.128:9898/cluster-info/v1/?token-id=871c6f" <node/discovery> cluster info object received, verifying signature using given token <node/discovery> cluster info signature and contents are valid, will use API endpoints [https://172.16.0.128:6443] <node/bootstrap> trying to connect to endpoint https://172.16.0.128:6443 <node/bootstrap> detected server version v1.4.4 <node/bootstrap> successfully established connection with endpoint https://172.16.0.128:6443 <node/csr> created API client to obtain unique certificate for this node, generating keys and certificate signing request <node/csr> received signed certificate from the API server: Issuer: CN=kubernetes | Subject: CN=system:node:kube2.domain.com | CA: false Not before: 2016-11-16 12:37:00 +0000 UTC Not After: 2017-11-16 12:37:00 +0000 UTC <node/csr> generating kubelet configuration <util/kubeconfig> created "/etc/kubernetes/kubelet.conf" Node join complete: * Certificate signing request sent to master and response received. * Kubelet informed of new secure connection details. Run 'kubectl get nodes' on the master to see this machine join
root@kube1:~# kubectl get nodes NAME STATUS AGE kube1.domain.com Ready 14h kube2.domain.com Ready 14h kube3.domain.com Ready 14h
The master now knows about the cluster, however the DNS container will be stuck creating – this is because it needs networking to enable it to spawn successfully. Kubeadm makes this easy enough:
root@kube1:/# kubectl apply -f https://git.io/weave-kube daemonset "weave-net" created
This will create a DaemonSet provided by Weaveworks which in turn pulls down all the necessary Docker containers to build the networking:
root@kube1:~# kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system dummy-2088944543-0h6ww 1/1 Running 0 1h kube-system etcd-kube1.domain.com 1/1 Running 0 1h kube-system kube-apiserver-kube1.domain.com 1/1 Running 0 1h kube-system kube-controller-manager-kube1.domain.com 1/1 Running 0 1h kube-system kube-discovery-1150918428-6hcm9 1/1 Running 0 1h kube-system kube-dns-654381707-p7ul4 3/3 Running 0 1h kube-system kube-proxy-3pchs 1/3 Running 0 1h kube-system kube-proxy-b5hx8 2/3 Running 0 1h kube-system kube-proxy-y9k33 3/3 Running 0 1h kube-system kube-scheduler-kube1.domain.com 1/1 Running 0 1h kube-system weave-net-8vbez 2/3 Running 0 1h kube-system weave-net-x1bfw 1/3 CrashLoopBackOff 15 1h kube-system weave-net-pg4s1 1/3 CrashLoopBackOff 15 1h
You can now see the DNS service container has started correctly and is able to serve requests, however the two new weave-net containers on the slave nodes will fail to start. Inspecting the logs for an affected container shows the following:
root@kube1:~# kubectl logs weave-net-pg4s1 -c weave --namespace=kube-system 2016/11/16 13:41:49 error contacting APIServer: Get https://10.96.0.1:443/api/v1/nodes: dial tcp 10.96.0.1:443: i/o timeout; trying with fallback: http://localhost:8080 2016/11/16 13:41:49 Could not get peers: Get http://localhost:8080/api/v1/nodes: dial tcp [::1]:8080: getsockopt: connection refused Failed to get peers
This is due to the local kube-proxy service on that node using the wrong interface. Fortunately this is easy enough to fix (you’ll need to ensure ‘jq’ is installed first to manipulate the json):
root@kube1:~# root@kube1:~# kubectl -n kube-system get ds -l'component=kube-proxy'-o json | jq'.items[0].spec.template.spec.containers[0].command |= .+ ["--cluster-cidr=10.32.0.0/12"]'| kubectl apply -f-&& kubectl -n kube-system delete pods -l'component=kube-proxy'daemonset "kube-proxy" configured pod "kube-proxy-3tqjn" deleted pod "kube-proxy-572au" deleted pod "kube-proxy-tsc3c" deleted
Note that the above is now incorrect for Kubernetes 1.6, you’ll need the following (thanks Mike!):
kubectl -n kube-system get ds -l 'component=kube-proxy' -o json | jq '.items[0].spec.template.spec.containers[0].command |= .+ ["--cluster-cidr=10.32.0.0/12"]' | kubectl apply -f - && kubectl -n kube-system delete pods -l 'component=kube-proxy'
This will correct the –cluster-cidr flag and then delete the containers. Kubernetes applies a restart policy of ‘always’ to the service containers, so these will be respawned:
root@kube1:~# kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system dummy-2088944543-whf5o 1/1 Running 0 28m kube-system etcd-kube1.analbeard.com 1/1 Running 0 28m kube-system kube-apiserver-kube1.analbeard.com 1/1 Running 0 28m kube-system kube-controller-manager-kube1.analbeard.com 1/1 Running 0 28m kube-system kube-discovery-1150918428-qy41j 1/1 Running 0 28m kube-system kube-dns-654381707-s92ze 3/3 Running 0 27m kube-system kube-proxy-2fh61 1/1 Running 0 48s kube-system kube-proxy-k0c64 1/1 Running 0 48s kube-system kube-proxy-xm82v 1/1 Running 0 48s kube-system kube-scheduler-kube1.analbeard.com 1/1 Running 0 28m kube-system weave-net-gzakd 2/2 Running 1 3m kube-system weave-net-x1bfw 2/2 Running 0 4m kube-system weave-net-y9k33 2/2 Running 1 57s
Success! One working cluster. It is worth bearing in mind that deploying via Kubeadm is fairly limited at the moment – it only creates a single etcd container which won’t be resilient if a node is lost. The tool is being heavily worked on however, and more functionality will be added in later releases.
Thanks, this was right on the nose for us!
One update, the label for name kube-proxy in the daemon set has changed. Here’s a version of the command to fix the kube-proxy service that worked for me with kubeadm 1.6 on Ubuntu 16.04:
kubectl -n kube-system get ds -l ‘k8s-app=kube-proxy’ -o json | jq ‘.items[0].spec.template.spec.containers[0].command |= .+ [“–cluster-cidr=10.32.0.0/12”]’ | kubectl apply -f – && kubectl -n kube-system delete pods -l ‘k8s-app=kube-proxy’
Thanks for that fix Mike, I’ve updated the post to reflect. Glad to hear someone found my ramblings useful!