Storing Terraform’s remote state in Minio Whilst AWS’s free S3 tier is almost certainly sufficient to store Terraform’s remote state, it may be the case that you have a requirement to keep the data on-site, or alternatively if you’re using Terraform in an air-gapped environment then you have no choice but to self-host. Enter Minio. If you’ve not used it before, the TLDR is that Minio provides an S3-compatible API in a single binary. This makes it perfect to store your Terraform state in. ...

Obtaining a wildcard LetsEncrypt cert with Ansible Earlier this year, LetsEncrypt made their wildcard x509 certificates available to the general public. Whilst this is a massive step forward over individual certificates for each domain, it does come with the overhead of having to distribute the wildcard certificate to the (possibly many) places you would use it. Ignoring that issue for now, I wrote a quick Ansible playbook which uses the dns-01 challenge method and my Memset DNS management modules (available in Ansible 2.6+) to provide the verification. ...

Lessons I learnt whilst developing modules Having now spent quite some time working on my initial Ansible modules for Memset, I’ve assembled some handy hints on areas which tripped me up at various times in my journey. It should be noted that this post is written from the point of view of someone who is not a developer and is therefore not as au fait with some of the processes mentioned as others may be. ...

A solution in need of a problem Like all good sysadmins, my personal website has been a ‘coming soon’ splash page for quite some time. According to the Wayback Machine, it’s been this way since some time in 2014. As I’m sure many can sympathise with, there are always far more interesting and shiny things to be experimenting with than building a website. One of the interesting things I like to experiment with is Kubernetes (as should be apparent from the tag cloud). Up until now, this has mostly consisted of building clusters, tweaking them and then tearing them down again. Whilst this gives me experience from the Operations side, I’m not getting the end-to-end experience a consumer of my cluster would have. ...

All the choices So you’re looking to start using Kubernetes, but you’re overwhelmed by the multitude of deployment options available? Judging by the length of the Picking the Right Solution section to the Kubernetes docs, it’s safe to assume that you’re not alone. Even after you’ve made it past the provisioning stage, you then need to learn how to administrate what is a very complex system. In short; Kubernetes is not easy. ...

When you receive a new server, there are a variety of pre-requisites required before Ansible can be used to administrate the host. Below is my own personal playbook which works for both Debian and RedHat (and derivative) systems. --- # ansible-playbook bootstrap-ansible-target.yml -b -e 'user=user' - hosts: "{{ host }}" remote_user: "{{ user | default('root') }}" gather_facts: no pre_tasks: - name: attempt to update apt's cache raw: test -e /usr/bin/apt-get && apt-get update ignore_errors: yes - name: attempt to install Python on Debian-based systems raw: test -e /usr/bin/apt-get && apt-get -y install python-simplejson python ignore_errors: yes - name: attempt to install Python on CentOS-based systems raw: test -e /usr/bin/yum && yum -y install python-simplejson python ignore_errors: yes - setup: tasks: - name: Create admin user group group: name: admin system: yes state: present - name: Ensure sudo is installed package: name: sudo state: present - name: Create Ansible user user: name: ansible shell: /bin/bash comment: "Ansible management user" home: /home/ansible createhome: yes - name: Add Ansible user to admin group user: name: ansible groups: admin append: yes - name: Add authorized key authorized_key: user: ansible state: present key: "{{ lookup('file', '/etc/ansible/.ssh/id_rsa.pub') }}" - name: Copy sudoers file command: cp -f /etc/sudoers /etc/sudoers.tmp - name: Backup sudoers file command: cp -f /etc/sudoers /etc/sudoers.bak - name: Ensure admin group can sudo lineinfile: dest: /etc/sudoers.tmp state: present regexp: '^%admin' line: '%admin ALL=(ALL) NOPASSWD: ALL' when: ansible_os_family == 'Debian' - name: Ensure admin group can sudo lineinfile: dest: /etc/sudoers.tmp state: present regexp: '^%admin' insertafter: '^root' line: '%admin ALL=(ALL) NOPASSWD: ALL' when: ansible_os_family == 'RedHat' - name: Replace sudoers file shell: visudo -q -c -f /etc/sudoers.tmp && cp -f /etc/sudoers.tmp /etc/sudoers - name: Test Ansible user's access local_action: shell ssh ansible@{{ host }} "sudo echo success" become: False register: ansible_success - name: Remove Ansible SSH key from bootstrap user's authorized keys lineinfile: path: "{{ ansible_env.HOME }}/.ssh/authorized_keys" state: absent regexp: '^ssh-rsa AAAAB3N' when: ansible_success.stdout == "success"

Following on from my previous post, I discovered rather to my dismay that although I had my nodes initially communicating over the secondary interface, the weave services (and thus my inter-pod traffic) was all going over the public interface. As these are VPSes, they have a public IP on eth0 and a VLAN IP on eth1, so it makes sense for all inter-pod traffic to stay internal. If I check the logs for one of the weave-net containers, we can see all comms are going via the 1.1.1.x IPs (for the purposes of this post they are the public IPs of each VPS): ...

With the 1.4 release of Kubernetes, Google have made instantiating a cluster a whole lot easier. Using Kubeadm, you can bring up a cluster with a single command on each node. A further command will create a DaemonSet which brings up a Weave mesh network between all your nodes. As always with complex systems such as Kubernetes, there are some potential pitfalls to be aware of. Firstly, the getting started guide notes that v1.11.2 of Docker is recommended, but v1.10.3 and v1.12.1 also work well (don’t go straight for the latest release like I tried to). If you wish to have your nodes talk over a private network, you’ll also need to explicitly declare this when you init the master node, otherwise Kubernetes will default to using your primary interface/route: ...