Howdy 👋

A postmortem around sporadic flapping nodes in my ClusterAPI management cluster. ...

A short postmortem into why a node in my Proxmox cluster went offline twice within a few days. ...

Recently, I started getting the dreaded emails from smartd on one of my Proxmox nodes. The drive was (slowly) failing and needed to be replaced. Unfortunately due to my nodes being micro form-factor(MFF), they only have a single drive. This means that I would also need to migrate the VM disks to the new drive too. As this is a homelab setup, I don’t have a ton of capacity or any fancy storage solutions, so I would need to get a little creative and replace the drive in-situ. ...

A little while back, I wrote a short piece on integrating Cilium with OPNsense using BGP. With more recent releases of Cilium, the team have introduced the Cilium BGP Control Plane (currently as a beta feature). This reworking of the BGP integration replaces the old MetalLB-based control plane and as such the older feature must first be disabled. To enable the new feature, you can either pass an argument to Cilium: --enable-bgp-control-plane=true Or if you use Helm to install Cilium then the following values are required: ...

Running immutable infra is the holy grail for many people, however there are times when you’ll need to get down in the weeds in order to troubleshoot issues. Let’s imagine a scenario; you need to verify that a pod is receiving traffic, but the image is built FROM scratch. As scratch containers are as minimal as possible, there’s no shell in the image, so there’s no way you can exec into it and hope to do anything remotely useful. ...

I’ve recently been experimenting with Hashicorp’s Consul in my home infrastructure because I want to use it to provide service discovery and automatic DNS provisioning when I create Proxmox instances with Terraform. Consul is a bit of a hefty beast to get to grips with and getting DNS lookups working when you have ACLs enabled can be a little tricky - it’s taken me a day or two of going round in circles to figure this one out. ...

Update: Cilium now has a new BGP integration so things have changed a little; see this post for more details. If (like me) you happen to follow the development of the Cilium CNI plugin for Kubernetes then you’ll have seen the recent 1.10 release which included many shiny features. One exciting addition is the ability to announce Service IPs via BGP. Running Kubernetes in a homelab environment quickly highlights that there are some aspects which are a little lacking when compared to the integration you get from the cloud provider offerings. One of the biggest limitations is the inability to create Loadbalancer services to expose ingress controllers and the like. MetalLB has been around for some years now and its aim is to improve this situation by using either ARP or BGP to announce routes to Service IPs inside your cluster(s). This means that you can create Loadbalancer Services inside your on-prem (or homelab) network, and it goes a long way towards reducing the friction of running Kubernetes outside of the cloud providers. ...

This blog is going to be about using Hashicorp’s Vault to issue short-lived certificates to use with SSH. Most guides have you using a username & password to authenticate with Vault, but I’ve chosen to delegate that to GitHub instead. I’m assuming you already have a Vault server running - I won’t be covering that in the course of this blog. You’ll also need a sufficiently-privileged Vault token, and jq installed on the machine you wish to SSH from. ...

Running Thanos without Kubernetes If you’ve been around the cloud-native world for a while, you’ll no doubt be familiar with (and quite likely already be using) Prometheus. You may however not have heard of Thanos. Put simply, Thanos takes Prometheus and makes it even more awesome. In their own words, the high-level description of Thanos is the following: Thanos is a set of components that can be composed into a highly available metric system with unlimited storage capacity, which can be added seamlessly on top of existing Prometheus deployments. ...