Building an HA k3s Control Plane on Ubuntu 24.04 LTS: Terraform, Ansible, kube-vip, and a Real Failover Drill

This is the latest iteration of my homelab automation: Terraform provisions six Ubuntu 24.04 LTS VMs on Proxmox, Ansible turns three of them into an HA k3s control plane with embedded etcd and a kube-vip floating VIP, and ArgoCD takes over GitOps management of everything above the OS the moment the cluster is up. It’s IaC end to end — nothing in this post was clicked into existence, and the two commands that matter (terraform apply, ansible-playbook site.yml) are the only manual steps in the whole build.

I’m writing it as a build log rather than an architecture essay, because the interesting decisions are the ones you only hit while actually sequencing a bootstrap — the order kube-vip has to exist in relative to k3s, which controller each of the other two join against and why it isn’t the VIP, what breaks if you get either wrong. Every command below is real, run against the repo at ~/k3s-enterprise.

Why Ubuntu 24.04 LTS, not Rocky Linux 10

#

The previous version of this automation ran a non-systemd, no-SELinux distribution — genuinely lightweight, nothing wrong with it. I set out to move this iteration onto Rocky Linux 10 on purpose, because it’s the same family of Linux I design platforms on at work: at Ebucks I run workloads on OpenShift, where SELinux enforcing and Security Context Constraints aren’t something you opt into — the platform holds you to them by default, and I wanted my own infrastructure carrying the same constraint instead of a friendlier one.

Rocky’s GenericCloud qcow2 wouldn’t boot on this Proxmox host. Not a config mistake I could fix from my side — a broken image on this particular hypervisor, and root-causing someone else’s cloud image wasn’t the problem I sat down to solve. So I swapped the template to Ubuntu 24.04 LTS and moved on. I’ll say the trade plainly rather than pretend it away: I lose the SELinux/SCC-style enforcement OpenShift gives me for free at work, and Ubuntu’s default AppArmor confinement isn’t a drop-in replacement for it. What didn’t change is firewalld, configured with explicit rules rather than a flat, everything-trusts-everything home network — the cluster subnet is trusted for etcd/kubelet/BGP/Cilium health traffic, the kube-apiserver port is opened deliberately rather than the whole host, and that posture doesn’t care which distro sits underneath it. apt replaces dnf; systemd was never in question either way.

Architecture

#

Three controllers, three workers, one floating VIP that is never a VM — kube-vip binds 172.16.10.19 to whichever controller currently holds Kubernetes-Lease-based leadership, in ARP mode. Cilium replaces kube-proxy entirely and peers eBGP with the upstream router to advertise LoadBalancer IPs, same as the rest of the stack.

Prerequisites

#

• Proxmox host with an API token (VM.Allocate, VM.Clone, VM.Config.*, Datastore.AllocateSpace, SDN.Use)
• A BGP-capable upstream router peering on 172.16.10.254, ASN 65000
• Terraform, Ansible, kubectl, helm installed locally, SSH key at ~/.ssh/id_rsa
• A free, routable IPv4 address for the control-plane VIP, on the controllers’ subnet, outside any DHCP or Cilium LB-IPAM range

1. Build the Ubuntu 24.04 LTS template

#

scripts/create-template.sh runs on the Proxmox host. It downloads the Ubuntu 24.04 LTS (Noble) cloud image straight from the official Canonical mirror, imports the disk into a fresh VM, and converts the result to a template. qemu-guest-agent isn’t baked into the image at this stage — roles/common installs and enables it via Ansible the first time the playbook touches a provisioned node:

scp scripts/create-template.sh root@proxmox:/tmp/
ssh root@proxmox bash /tmp/create-template.sh

Downloading Ubuntu 24.04 LTS cloud image...
Creating VM 9000 (ubuntu-24-04-cloudimg)...
Importing disk to fast...
Adding cloud-init drive...
Converting to template...

Done! Template 9000 (ubuntu-24-04-cloudimg) is ready.
Set template_vm_id = 9000 and ssh_user = "ubuntu" in your terraform.tfvars.

TEMPLATE_VM_ID is overridable via env var if 9000 is already taken.

2. Terraform — six VMs, three of them etcd members

#

cd infra/terraform
cp terraform.tfvars.example terraform.tfvars
$EDITOR terraform.tfvars

The variable that actually enforces the HA topology is controllers — a list, not a single object, with a Terraform validation block that refuses anything but an odd count:

variable "controllers" {
  type = list(object({
    name = string, vm_id = number, ipv4_cidr = string, ipv6_cidr = string
    cpu_cores = number, memory_mb = number, disk_gb = number
  }))
  validation {
    condition     = length(var.controllers) % 2 == 1
    error_message = "controllers must have an odd count for etcd quorum (1, 3, 5, ...)."
  }
}

3 is the practical minimum — quorum survives exactly one controller loss. terraform.tfvars sets three controllers at 172.16.10.10–.12, three workers at .21–.23, and kube_vip_address = "172.16.10.19":

ssh-add ~/.ssh/id_rsa   # bpg provider needs SSH for cloud-init
terraform init
terraform plan
terraform apply

proxmox_virtual_environment_vm.controllers["k3s-controller-01"]: Creating...
proxmox_virtual_environment_vm.controllers["k3s-controller-02"]: Creating...
proxmox_virtual_environment_vm.controllers["k3s-controller-03"]: Creating...
proxmox_virtual_environment_vm.workers["k3s-worker-01"]: Creating...
proxmox_virtual_environment_vm.workers["k3s-worker-02"]: Creating...
proxmox_virtual_environment_vm.workers["k3s-worker-03"]: Creating...

Apply complete! Resources: 6 added, 0 changed, 0 destroyed.

Outputs:
controller_ips = ["172.16.10.10", "172.16.10.11", "172.16.10.12"]
worker_ips     = ["172.16.10.21", "172.16.10.22", "172.16.10.23"]

Each controller gets one disk (OS + etcd data). Each worker gets two — vda for the OS, a raw unformatted vdb (512 GB) for DirectPV — with discard = on and iothread = true on both.

3. Ansible — bootstrap the HA control plane

#

inventory/hosts.yml and inventory/group_vars/all.yml mirror the Terraform values — kube_vip_address: "172.16.10.19", kube_vip_version: "v0.9.1", k3s_version: "v1.36.1+k3s1". site.yml runs five roles in order: common → k3s_server (all 3 controllers) → cilium → k3s_agent (all 3 workers) → argocd.

ssh-add ~/.ssh/id_rsa
ansible-playbook -i inventory/hosts.yml site.yml

Two chicken-and-egg problems, solved

#

Getting three controllers and a VIP to come up cleanly means resolving two ordering problems that don’t exist with a single controller.

Problem one — who joins etcd, and against what address. The first controller in groups['k3s_controller'] sets cluster-init: true and bootstraps the embedded etcd cluster alone. The other two must join that cluster, not start their own — but they can’t join via the VIP, because the VIP isn’t up until a controller already is:

# k3s-server-config.yaml.j2
{% if inventory_hostname == groups['k3s_controller'][0] %}
cluster-init: true
{% else %}
# Points at that node's real IP, not the VIP — the VIP is only guaranteed up
# once at least one controller has fully started, so the first join has to
# go direct.
server: "https://{{ hostvars[groups['k3s_controller'][0]]['ansible_host'] }}:{{ k3s_api_port }}"
{% endif %}

Workers, joining later via k3s_agent, don’t have this problem — by the time they come up, all three controllers are already etcd members and the VIP is a settled thing to depend on, so they join against 172.16.10.19, not any one controller’s IP.

Problem two — kube-vip has to exist before k3s does. The static pod manifest is written to /var/lib/rancher/k3s/server/manifests/kube-vip.yaml before k3sup installs k3s on each controller. k3s’s embedded deploy controller applies anything already sitting in that directory the instant the local apiserver comes up, so kube-vip starts automatically at boot on every controller and joins the leader-election Lease immediately — no separate kubectl apply that would itself need a working API endpoint to run.

cilium runs once, against controller[0], and installs with k8sServiceHost set to the VIP rather than a node — hardcoding a controller here would turn every pod’s API access into a single point of failure again, right after the control plane stopped being one.

4. Verify the cluster

#

export KUBECONFIG=~/.kube/k3s.yaml
kubectl get nodes -o wide

NAME                STATUS   ROLES                  AGE   VERSION
k3s-controller-01   Ready    control-plane,master   14m   v1.36.1+k3s1
k3s-controller-02   Ready    control-plane,master   12m   v1.36.1+k3s1
k3s-controller-03   Ready    control-plane,master   11m   v1.36.1+k3s1
k3s-worker-01       Ready    <none>                 8m    v1.36.1+k3s1
k3s-worker-02       Ready    <none>                 8m    v1.36.1+k3s1
k3s-worker-03       Ready    <none>                 7m    v1.36.1+k3s1

grep 'server:' ~/.kube/k3s.yaml

server: https://172.16.10.19:6443

The kubeconfig points at the VIP, not a node — the whole point of the exercise. cilium status --wait confirms the CNI is healthy across all six nodes:

   /¯¯\
/¯¯\__/¯¯\    Cilium:             OK
\__/¯¯\__/    Operator:           OK
/¯¯\__/¯¯\    Envoy DaemonSet:    OK
\__/¯¯\__/    Hubble Relay:       OK
   \__/       ClusterMesh:        disabled

DaemonSet   cilium   Desired: 6, Ready: 6/6, Available: 6/6

And the kube-vip Lease shows which controller is currently holding the VIP:

kubectl get lease plndr-cp-lock -n kube-system

NAME            HOLDER               AGE
plndr-cp-lock   k3s-controller-02    14m

5. Prove it: end-to-end tests, then a real failover drill

#

A cluster that’s merely Ready hasn’t proven it’s HA. Two things in this repo exist specifically to close that gap: scripts/e2e-test.sh, pure kubectl/curl smoke tests runnable from anywhere with a kubeconfig, and e2e-test.yml, an Ansible playbook that runs those checks and then actually kills a controller.

./scripts/e2e-test.sh

━━━ 1. Node readiness ━━━
  ✓ 3/3 control-plane nodes Ready
  ✓ 3/3 worker nodes Ready

━━━ 2. Control-plane HA (kube-vip) ━━━
  ✓ 3/3 kube-vip pods Running
  ✓ kubeconfig server (172.16.10.19:6443) does not match any single controller IP

━━━ 3. Cilium CNI ━━━
  ✓ Cilium DaemonSet rollout healthy
  ✓ cilium-dbg status --brief OK (from cilium-x7g2p)

━━━ 4. Workload smoke test (namespace: e2e-smoke-48213) ━━━
  ✓ test Deployment (3 replicas) reached Ready
  ✓ replicas spread across 3 distinct nodes
  ✓ in-cluster DNS resolves e2e-nginx.e2e-smoke-48213.svc.cluster.local
  ✓ pod-to-Service HTTP request succeeded (ClusterIP, kube-proxy replacement path)
  ✓ LoadBalancer Service got an IP from Cilium LB-IPAM: 172.16.1.14

━━━ Summary ━━━
  All checks passed.

Node readiness, kube-vip pod count, Cilium DaemonSet health via cilium-dbg, a 3-replica Deployment with anti-affinity checked for actual node spread (not just replica count), in-cluster DNS, Service connectivity, and LoadBalancer IPAM — all from outside the cluster, no SSH required.

The failover drill is the part that actually matters, because a design claim (“the control plane survives losing a node”) is worthless until something stops a controller and checks:

ansible-playbook -i inventory/hosts.yml e2e-test.yml

TASK [Show detected leader] ***************************************
ok: [localhost] => {
    "msg": "kube-vip leader: k3s-controller-02 (lease read succeeded)"
}

TASK [Stop k3s on k3s-controller-02] *******************************
changed: [localhost -> k3s-controller-02]

TASK [Wait for the VIP to fail over to a surviving controller] *****
FAILED - RETRYING: Wait for the VIP to fail over (18 retries left)
ok: [localhost]

TASK [Show node status during failover] ****************************
ok: [localhost] => {
    "msg": [
        "k3s-controller-01   Ready",
        "k3s-controller-02   NotReady",
        "k3s-controller-03   Ready",
        "k3s-worker-01       Ready",
        "k3s-worker-02       Ready",
        "k3s-worker-03       Ready"
    ]
}

TASK [Restart k3s on k3s-controller-02] ****************************
changed: [localhost -> k3s-controller-02]

TASK [Wait for the stopped controller to rejoin as Ready] **********
ok: [localhost]

TASK [Report result] ************************************************
ok: [localhost] => {
    "msg": [
        "Failover drill passed: k3s-controller-02 was stopped, the VIP kept serving from a surviving controller, and k3s-controller-02 rejoined cleanly."
    ]
}

It reads the real Lease holder rather than assuming controller-01, stops k3s on that node over SSH, polls the VIP until a survivor answers, checks node status mid-failover, then restarts and confirms a clean rejoin — never more than one controller down at a time, safe to run against a live cluster because it only touches its own e2e-smoke-* namespace otherwise.

GitOps: making sure ArgoCD isn’t the next single point of failure

#

An HA control plane underneath a GitOps controller that’s still a single pod is a half-finished job. argocd-values.yaml.j2 sizes ArgoCD for the same topology it’s deploying onto:

The controller staying at 1 replica isn’t an oversight — running it HA needs shard-by-cluster mode, which only earns its complexity once you’re managing more than one cluster from the same ArgoCD. Everything else that can be stateless or HA-backed without that trade-off, is.

Storage: default to safe, not fast

#

The same instinct behind the control plane shows up in the StorageClass defaults, too: don’t make the convenient thing the default if it quietly becomes a data-loss trap; make the safe thing the default and require a deliberate opt-in for the sharp tool.

nfs — backed by csi-driver-nfs , the Kubernetes CSI SIG’s driver, pointed at a plain NFS export (172.16.10.4:/export/homelab-pvc, no vendor API, no ZFS backend, nothing to stand up beyond an NFS server) — is now the cluster default:

# gitops/apps/storage/nfs-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: nfs.csi.k8s.io
parameters:
  server: 172.16.10.4
  share: /export/homelab-pvc
reclaimPolicy: Retain
volumeBindingMode: Immediate

directpv-min-io — the raw /dev/vdb disk Terraform attaches to each worker — lost its is-default-class annotation in the same change. It’s still there, still fast, still non-replicated node-local storage, but now it’s explicit opt-in only (storageClassName: directpv-min-io), reserved for the narrow set of workloads that genuinely need that speed and can tolerate losing a volume outright if that specific worker’s disk dies — MariaDB Galera, Prometheus, OpenSearch: things that are either already replicated at the application layer or fine losing local, cache-shaped data. Everything else — application config, low-I/O state, anything I’d be annoyed to lose and haven’t deliberately engineered replication for — lands on nfs unless it asks for something else by name.

reclaimPolicy: Retain on the nfs class is the other half of that same decision. Delete a PersistentVolumeClaim by accident, or let a Helm chart get uninstalled and take its PVC with it, and the underlying data isn’t destroyed — the volume just detaches, and the export sits under /export/homelab-pvc on the NFS server until someone consciously goes and cleans it up. That’s a deliberate trade against tidiness: a Retain-backed cluster accumulates stale exports over time, and I’d rather deal with that housekeeping occasionally than have a helm uninstall I ran half-asleep quietly take a database with it.

kubectl get storageclass

NAME                    PROVISIONER              RECLAIMPOLICY   VOLUMEBINDINGMODE      DEFAULT
nfs (default)           nfs.csi.k8s.io           Retain          Immediate              true
directpv-min-io         directpv-min-io          Delete          WaitForFirstConsumer   false

kubectl get pvc -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,STORAGECLASS:.spec.storageClassName

NAMESPACE    NAME                        STORAGECLASS
vaultwarden  vaultwarden-data            nfs
mariadb      mariadb-galera-0            directpv-min-io
monitoring   prometheus-data-0           directpv-min-io
gitea        gitea-shared-storage        nfs

Anyone cloning this repo just needs an NFS export reachable from the cluster subnet and two values in terraform.tfvars/group_vars/all.yml (default_storage_nfs_server, default_storage_nfs_share) — no ZFS host, no vendor REST API, no credentials to seal before the default StorageClass even works.

Summary: three Ubuntu 24.04 LTS controllers with firewalld locked down and embedded etcd, fronted by a kube-vip VIP, replace what a single node used to do — Terraform enforces the odd-count quorum requirement at plan time, Ansible resolves the two bootstrap ordering hazards that only show up once you have more than one controller, and e2e-test.yml proves the result by actually stopping a controller and watching the cluster carry on rather than just asserting it would.

Get in touch · View Resume · Connect on LinkedIn