All the choices

So you’re looking to start using Kubernetes, but you’re overwhelmed by the multitude of deployment options available? Judging by the length of the Picking the Right Solution section to the Kubernetes docs, it’s safe to assume that you’re not alone. Even after you’ve made it past the provisioning stage, you then need to learn how to administrate what is a very complex system. In short; Kubernetes is not easy. Naturally then, the easiest way to use Kubernetes is to let someone else look after the infrastructure for you. If you’re looking for the most hands-off turnkey solution then you should investigate Google’s Container Engine, or RedHat’s OpenShift Dedicated platform.

Whilst these solutions do make it easier to consume Kubernetes for your workloads by abstracting away the management overheads, you may well need to maintain greater control over your data. Whilst using the cloud is perfectly safe, this may not be something your security controls permit, making cloud-hosted options such as those mentioned above unviable. In these cases, unless you have a large budget to spend on something like RedHat’s on-premise Container Platform, you’ll need to look at rolling your own.

At this point, picking the right solution has become a little easier. As with all open-source projects, it’s important to judge their suitability – you don’t want to adopt an unmaintained project unless you have the time and resources to maintain it yourself. This is especially important when interacting with Kubernetes, as the development moves fast and new features are added regularly.

With all this in mind, I ended up settling on Kubespray (originally called Kargo). Kubespray is a Kubernetes incubator project, which means it is on its way to becoming a fully-fledged community project. I spend a fair bit of my own time working with Ansible, and as Kubespray is just a large set of playbooks, it was the obvious choice.

Getting started

The rest of this post assumes that your target hosts are already correctly configured for administration via Ansible – if you’re starting off with virgin nodes then you can use something like my Ansible node bootstrapping playbook. Obviously this still requires access of some fashion; in my case my hosting provider Memset gives you the ability to inject an SSH key at provisioning time. Once this is in place I can run my playbook to reconfigure the node as I see fit.

Configuring your hosts file

Now you’re ready to start, you’ll obviously need to clone the Kubespray repo. Once you’ve done this, duplicate the inventory directory and jump there. You’ll need to configure your inventory file to suit your deployment; below is mine for illustration’s sake:

The [all] group needs to contain all nodes in the cluster. If you wish to have them communicate over a different IP to the access IP then you can specify this using the ip variable. In my case, I have a VLAN between all 5 nodes and I want my internal cluster traffic to traverse this, rather than the public internet.

The nodes listed under the [kube-master] heading will become cluster masters who’s job is to modify the cluster’s state by scheduling pods etc. For HA purposes, it’s a good idea to have more than one.

Any node in the kube-node group will be available for scheduling pods onto. In a larger cluster these nodes will purely function as minions, however in my case my masters are also minions too.

Finally, the [etcd] group tells Ansible which nodes to deploy your etcd cluster onto. Kubernetes stores a lot of information in etcd so three nodes is good practise. You could run with a single etcd node, but if that goes away then the entire cluster will grind to a halt.

Global config options

Contained within your duplicated inventory directory is a group_vars directory. Assuming you’re familiar with Ansible the purpose of this directory is clear. Unless you require some edge cases, the majority of all.yml can remain as provided:

Nothing too complicated going on here; one thing to bear in mind here is that the module loading isn’t persistent. Ansible will modprobe any modules Kubernetes needs, but this won’t survive a reboot (as I discovered the hard way).

Cluster config options

The k8s-cluster.yml file requires a little more customisation. Firstly, I’d suggest disabling anonymous authentication to the API for security reasons. You then need to pick the version of Kubernetes to deploy, and set some passwords:

Next up, some networking config. As is to be expected, there are a multitude of options when it comes to setting up networking in Kubernetes. Kubespray gives you a subset of these options; I chose to go with Weave as it means I can deploy Weave Scope at a later date.

Lastly, we have some options it’s worth enabling which help visualise the state of your cluster. The Netchecker deployment places pods on all hosts which constantly verify the state of the networking by attempting to communicate with each other.


With everything configured to your liking, it’s time to deploy to your nodes. In the root of the Kubespray checkout is cluster.yml; this is what we’ll need to use. Deployment is carried out the usual Ansible way:

The run-time of this playbook is very much dependent on a number of variables, such as the number of nodes and also how much Internet-connected bandwidth they have; for my nodes, it took around 20 minutes.

Accessing your cluster

Before you can use your cluster, you need to configure access to it. During the cluster bootstrapping process, various SSL certificates will have been created which we can use to authenticate ourselves. From your master node in /etc/kubernetes/ssl you’ll need to grab a copy of the node’s admin certificate, the corresponding private key, and the CA certificate. You’ll need to place these in /home/user/.kube/ on the machine you wish to use to administrate the cluster, along with a config file pointing to them:

Finally, ensure the server: variable is a URL by which you can talk to your master node.

Verify the cluster state

At this point, you should be able to run kubectl get pods -o wide --all-namespaces (note that you may see more or less pods if you chose different options in your config).

Provided all pods are listed as running then you should have a healthy cluster. The most complete way to administrate your cluster is using kubectl, however it’s not great for at-a-glance monitoring. Fortunately, Kubernetes has you covered with their dashboard. You can deploy it straight from the git repo:

With that done, you will need to grab a terminal as you’ll use kubectl to proxy the dashboard from the cluster:

Finally, open your favourite browser and head to http://localhost:8001/ui and you should see the dashboard. Success!

Bonus round

If you went with Weave for your networking, you can also deploy Weave Scope which gives you a great deal of insight into your cluster. Unsurprisingly, we deploy on top of Kubernetes which means no need to configure anything – Weave will automatically provide the metrics to the Scope pods.

With that done, you’ll then use kubectl to forward the relevant port:

And then as for the dashboard, open up your browser and navigate to

Deploying Kubernetes on VMs with Kubespray
Tagged on:                     

Leave a Reply

Your email address will not be published. Required fields are marked *