Homelab Part 4 - k3s Automation

In Part 3 we deployed k3s manually — great for understanding what each piece does, but not something you want to repeat. This post automates the entire thing: Terraform provisions the VMs, Ansible configures them and installs k3s + Cilium, and ArgoCD bootstraps itself and takes over GitOps management of the full cluster stack.

Source: git.wagner.org.za/homelab/k3s-proxmox-automation

Architecture

#

┌──────────────────────────────────────────────────────────────────────────────┐
│  Proxmox Host                                                                │
│                                                                              │
│  ┌─────────────────┐  ┌───────────────┐  ┌───────────────┐  ┌────────────┐   │
│  │ k3s-controller  │  │ k3s-worker-01 │  │ k3s-worker-02 │  │k3s-worker  │   │
│  │ 172.16.10.1     │  │ 172.16.10.2   │  │ 172.16.10.3   │  │172.16.10.4 │   │
│  │ 8 cores / 16 GB │  │ 16c / 32 GB   │  │ 16c / 32 GB   │  │16c / 32 GB │   │
│  │ vda: 200 GB     │  │ vda: 200 GB   │  │ vda: 200 GB   │  │vda: 200 GB │   │
│  │                 │  │ vdb: 512 GB   │  │ vdb: 512 GB   │  │vdb: 512 GB │   │
│  │ k3s server      │  │ vdc: 512 GB   │  │ vdc: 512 GB   │  │vdc: 512 GB │   │
│  │ Cilium / ArgoCD │  │ k3s agent     │  │ k3s agent     │  │k3s agent   │   │
│  └────────┬────────┘  └──────┬────────┘  └──────┬────────┘  └─────┬──────┘   │
│           └───────────────-──┴──────────────────┴────────────-────┘          │
└──────────────────────────────────────────────────────────────────────────────┘

Each worker gets three disks:

• vda — OS + container images
• vdb — raw, unformatted disk for DirectPV (high-performance local storage)
• vdc — XFS-formatted, mounted at /var/lib/longhorn for Longhorn replicated storage

Tool chain

#

Prerequisites

#

• Proxmox node with the Debian 13 cloud-init template ready (see Part 2 )
• Terraform ≥ 1.9 and Ansible ≥ 2.16 installed locally
• k3sup will be installed automatically by the Ansible playbook if missing
• A Proxmox API token with VM provisioning permissions
• SSH key pair — public key goes into terraform.tfvars, private key used by Ansible

1. Terraform — Provision VMs

#

Clone the repo and initialise Terraform.

git clone https://git.wagner.org.za/homelab/k3s-proxmox-automation.git
cd k3s-proxmox-automation/infra/terraform

terraform init

Copy the example vars file and fill in your values.

cp terraform.tfvars.example terraform.tfvars

Key variables to update:

proxmox_endpoint     = "https://pve:8006"
proxmox_api_token    = "root@pam!terraform=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
proxmox_insecure     = true     # if using a self-signed cert

template_vm_id = 9000           # your Debian 13 cloud-init template
storage_pool   = "virtuals"

ipv4_gateway = "172.16.10.254"
ssh_public_key = "ssh-ed25519 AAAA..."

controller = {
  name      = "k3s-controller-01"
  vm_id     = 200
  ipv4_cidr = "172.16.10.1/24"
  ipv6_cidr = "fd00::1/64"
  cpu_cores = 8
  memory_mb = 16384
  disk_gb   = 200
}

Apply and Terraform will clone the template and provision all four VMs with cloud-init networking.

terraform plan
terraform apply

proxmox_virtual_environment_vm.controller: Creating...
proxmox_virtual_environment_vm.workers["k3s-worker-01"]: Creating...
proxmox_virtual_environment_vm.workers["k3s-worker-02"]: Creating...
proxmox_virtual_environment_vm.workers["k3s-worker-03"]: Creating...

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

Outputs:
controller_ip = "172.16.10.1"
worker_ips    = ["172.16.10.2", "172.16.10.3", "172.16.10.4"]

2. Ansible — Configure Nodes and Install k3s

#

The playbook runs five roles in order.

# site.yml
common      → all nodes    (packages, kernel, sysctl, swap, disk setup)
k3s_server  → controller   (k3sup install, kubeconfig fetch)
cilium      → controller   (Helm install, wait for CNI ready)
k3s_agent   → workers      (k3sup join)
argocd      → controller   (Helm install, app-of-apps bootstrap)

Update the inventory to match your IPs (should match terraform.tfvars).

# inventory/hosts.yml
k3s_controller:
  hosts:
    k3s-controller-01:
      ansible_host: 172.16.10.1

k3s_workers:
  hosts:
    k3s-worker-01:
      ansible_host: 172.16.10.2
    k3s-worker-02:
      ansible_host: 172.16.10.3
    k3s-worker-03:
      ansible_host: 172.16.10.4

Copy and fill in the cluster environment file — this drives all secrets and site-specific values.

cp cluster.env.example cluster.env
$EDITOR cluster.env

GITOPS_REPO_URL="https://git.wagner.org.za/homelab/k3s-proxmox-automation.git"
CLOUDFLARE_TOKEN="..."
DB_SUPERUSER_PASSWORD="..."
KEYCLOAK_ADMIN_PASSWORD="..."
GRAFANA_ADMIN_PASSWORD="..."
BLOB_STORAGE_ENDPOINT="http://192.168.1.10:9000"

Run the full playbook.

ansible-playbook site.yml

You can also run individual roles with tags.

ansible-playbook site.yml --tags common
ansible-playbook site.yml --tags k3s_server
ansible-playbook site.yml --tags cilium
ansible-playbook site.yml --tags k3s_agent
ansible-playbook site.yml --tags argocd

What each role does

#

common — runs on every node. Installs qemu-guest-agent, open-iscsi, nfs-common, xfsprogs and other required packages. Loads br_netfilter, overlay, and vxlan kernel modules. Sets sysctl parameters for Kubernetes networking. Disables swap. On worker nodes it formats /dev/vdc as XFS and mounts it at /var/lib/longhorn — the raw /dev/vdb is left unformatted for DirectPV.

k3s_server — installs k3sup on your local machine if not present, generates a random cluster token, writes the k3s server config to the controller, and uses k3sup to install the k3s server and write a local kubeconfig. Key flags applied via config:

flannel-backend: "none"
disable-kube-proxy: true
disable-network-policy: true
disable:
  - traefik
  - servicelb
cluster-cidr: "10.42.0.0/16"
service-cidr: "10.43.0.0/16"

Flannel, kube-proxy, traefik, and servicelb are all disabled — Cilium handles everything.

cilium — installs Helm, renders the values template, and deploys Cilium via Helm. Waits for the DaemonSet to be ready before proceeding. Without this the workers can’t join because there is no CNI.

kubeProxyReplacement: true
routingMode: native
bgpControlPlane:
  enabled: true
l2announcements:
  enabled: true
hubble:
  relay:
    enabled: true
  ui:
    enabled: true

k3s_agent — uses k3sup join to register each worker against the controller. Waits until each node reports Ready.

argocd — installs ArgoCD via Helm, waits for the server deployment, saves the initial admin password to ~/.argocd-admin-password, then applies the App-of-Apps Application manifest that points ArgoCD at gitops/apps/argo-suite/ in the repo. ArgoCD syncs itself from that point.

3. Verify Cluster

#

Once the playbook completes the cluster should be up.

export KUBECONFIG=~/.kube/config

kubectl get nodes -o wide

NAME               STATUS   ROLES                  AGE   VERSION
k3s-controller-01  Ready    control-plane,master   8m    v1.35.1+k3s1
k3s-worker-01      Ready    <none>                 4m    v1.35.1+k3s1
k3s-worker-02      Ready    <none>                 3m    v1.35.1+k3s1
k3s-worker-03      Ready    <none>                 3m    v1.35.1+k3s1

Check Cilium status.

cilium status --wait

   /¯¯\
/¯¯\__/¯¯\    Cilium:             OK
\__/¯¯\__/    Operator:           OK
/¯¯\__/¯¯\    Envoy DaemonSet:    OK
\__/¯¯\__/    Hubble Relay:       OK
   \__/       ClusterMesh:        disabled

DaemonSet   cilium   Desired: 4, Ready: 4/4, Available: 4/4

4. GitOps — ArgoCD App-of-Apps

#

The ArgoCD role applies a single App-of-Apps Application pointing at gitops/apps/argo-suite/ in the repo. ArgoCD discovers and deploys every Application defined there.

kubectl get applications -n argocd

NAME                    SYNC STATUS   HEALTH STATUS
argocd                  Synced        Healthy
cert-manager            Synced        Healthy
cilium                  Synced        Healthy
envoy-gateway           Synced        Healthy
fluent-bit              Synced        Healthy
gateway-api             Synced        Healthy
k8s-monitoring          Synced        Healthy
keycloak                Synced        Healthy
keycloak-operator       Synced        Healthy
kube-prometheus-stack   Synced        Healthy
longhorn                Synced        Healthy
loki                    Synced        Healthy
postgres                Synced        Healthy
reflector               Synced        Healthy
sealed-secrets          Synced        Healthy

From here on, every change to the cluster is a git commit. Push a manifest change, ArgoCD detects it and reconciles.

5. BGP — Advertising LoadBalancer IPs

#

Once ArgoCD syncs the networking apps, Cilium’s BGP Control Plane takes over advertising LoadBalancer service IPs to the upstream router.

cilium bgp peers

Node               Local AS   Peer AS   Peer Address    Session State
k3s-controller-01  65000      65000     172.16.10.254   established
k3s-worker-01      65000      65000     172.16.10.254   established
k3s-worker-02      65000      65000     172.16.10.254   established
k3s-worker-03      65000      65000     172.16.10.254   established

The BGP peer config and LB IP pool are managed in gitops/cilium/config/. The pool is 172.16.1.1 – 172.16.1.254 — any LoadBalancer service gets an IP from this range automatically advertised to the router.

# gitops/cilium/config/lb-ip-pool.yaml
apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
  name: cluster-lb-pool
spec:
  blocks:
    - start: "172.16.1.1"
      stop: "172.16.1.254"

6. Teardown / Reset

#

To reset the cluster and start fresh without destroying the VMs.

ansible-playbook reset.yml

To destroy the VMs entirely.

cd infra/terraform
terraform destroy

Troubleshooting

#

Nodes stuck in NotReady after k3sup join — Cilium is not yet ready. Check cilium status on the controller. The Ansible play waits for the DaemonSet rollout but if it times out, run the cilium tag again.

ansible-playbook site.yml --tags cilium

BGP session stays Active and never reaches Established — the upstream router needs a BGP peer configured for each node IP (not just the gateway). Check your router config and confirm the local ASN matches.

ArgoCD shows Unknown or OutOfSync apps — the private repo credentials may not have been applied. Check gitops_repo_username and gitops_repo_token in inventory/group_vars/all.yml and re-run the argocd tag.

ansible-playbook site.yml --tags argocd

Check out Homelab Part 3 — Kubernetes for the manual setup this automates.