A little while back, I wrote a short piece on integrating Cilium with OPNsense using BGP. With more recent releases of Cilium, the team have introduced the Cilium BGP Control Plane (currently as a beta feature). This reworking of the BGP integration replaces the old MetalLB-based control plane and as such the older feature must first be disabled. To enable the new feature, you can either pass an argument to Cilium:
--enable-bgp-control-plane=true
Or if you use Helm to install Cilium then the following values are required:
bgpControlPlane:
enabled: true
Note that the OPNsense configuration described in the previous post is unchanged; it is only the configuration of Cilium which differs.
BGP configuration
Configuration of the BGP feature is somewhat different to the single bgp-config
ConfigMap used by the old MetalLB integration. Below is an example of the old configuration which we’ll translate into the new Custom Resources provided by Cilium:
apiVersion: v1
kind: ConfigMap
metadata:
name: bgp-config
namespace: cilium
data:
config.yaml: |
peers:
- peer-address: 172.25.0.1
peer-asn: 65401
my-asn: 65502
address-pools:
- name: default
protocol: bgp
addresses:
- 172.27.0.32-172.27.0.62
CiliumLoadBalancerIPPool
The address pool is now configured by way of the new CiliumLoadBalancerIPPool
Custom Resource. This CR separates out the address configuration from the BGP router configuration:
apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
name: service-pool
spec:
cidrs:
- cidr: 172.27.0.32/27
CiliumBGPPeeringPolicy
The CiliumBGPPeeringPolicy
CR is a little more complex, however most of the neighbors
section is just the defaults and so they can be ignored:
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeeringPolicy
metadata:
name: bgp-peering-policy-worker
spec:
nodeSelector:
matchLabels:
metal.sidero.dev/serverclass: worker
virtualRouters:
- localASN: 65502
serviceSelector:
matchExpressions:
- {key: "io.cilium/bgp-announce", operator: In, values: ['worker']}
neighbors:
- peerAddress: '172.25.0.1/32'
peerASN: 65401
eBGPMultihopTTL: 10
connectRetryTimeSeconds: 120
holdTimeSeconds: 90
keepAliveTimeSeconds: 30
gracefulRestart:
enabled: true
restartTimeSeconds: 120
Some sections of the peering policy require a little more explanation.
Firstly, I only have a single neighbor defined as my OPNsense firewall is the only peer in my network. This is easily translated over from the old configuration, along with the local ASN for the cluster:
virtualRouters:
- localASN: 65502
neighbors:
- peerAddress: '172.25.0.1/32'
peerASN: 65401
The new peering policy resource allows far more flexibility for managing BGP topologies. In my case, I only want to allow worker nodes to announce routes, so I’ve configured the peering policy to only apply to worker nodes. This is achieved by applying a label to those nodes which is then used in the policy:
nodeSelector:
matchLabels:
metal.sidero.dev/serverclass: worker
The final config option to note is the serviceSelector
; I do not want every Service IP to be announced so I ensure that Cilium only announces those which I want it to. In order to achieve this I label any Services which should be announced with io.cilium/bgp-announce=worker
:
serviceSelector:
matchExpressions:
- {key: "io.cilium/bgp-announce", operator: In, values: ['worker']}
I chose this scheme because I also have a peering policy which only applies to control plane nodes.
BGP topologies
Having only one upstream peer, my setup is quite simple. It is possible to create more complex topologies; for these I suggest referring to the documentation.