If you’ve already read the build logs — the k3s cluster, the storage classes, the failover drill — you know what I build. This post is about the part that doesn’t show up in a terraform apply output: how I actually decide something, in the moment a plan stops matching reality, and what I do about it afterward.
That’s a different question than “what technology did you pick,” and it’s the one I think actually separates senior from principal-level work. Anyone competent can execute a plan that goes to schedule. What I’m going to walk through is a plan that didn’t, twenty minutes into a homelab rebuild, and the small, unglamorous decision that followed it — because that decision is a better sample of how I’ll behave on your platform than any tool list is.
The plan breaks — the question is what you do in the next five minutes#
Every non-trivial build reaches a point where the thing in front of you disagrees with the thing you planned. A cloud image won’t boot. A vendor API returns something the docs didn’t mention. A regulatory requirement lands mid-sprint and the reference architecture you’d already signed off on doesn’t cover it. That moment isn’t rare, and it isn’t a failure of planning — it’s just what building things is actually like.
What I’ve learned to do in that moment, from a lot of years of it happening, is separate two questions that feel like one:
- What was I actually trying to protect with the original plan?
- Is the thing that broke the protection itself, or just one particular way of getting it?
Most bad decisions under pressure come from treating those as the same question — abandoning the actual goal because the specific vehicle for it stopped working, or worse, quietly pretending nothing was lost so the write-up looks cleaner than the decision was. I’ll walk through a recent, concrete case of getting that split right, and a smaller one that shows the same instinct at a different scale.
time it deserves?"} B -->|"yes"| C["Fix it, keep the original goal intact"] B -->|"no — not my bug,
not my problem to solve today"| D["What was I actually protecting?"] D --> E["Find another way to protect that"] E --> F["Name what gets lost, explicitly,
in the write-up — don't smooth it over"]
Rocky Linux 10, and a cloud image that wouldn’t boot#
I set out to move my homelab’s k3s automation onto Rocky Linux 10 on purpose. Not for familiarity — I’d already been running a lighter, non-SELinux distribution for the previous iteration, and it worked fine. I moved to Rocky specifically because it’s the same family of Linux I design platforms on at work: on OpenShift, SELinux enforcing and Security Context Constraints aren’t something a team opts into, the platform holds you to them by default. I wanted my own infrastructure carrying that same constraint instead of a friendlier one, because a homelab that never has to fight SELinux isn’t actually testing the thing I recommend to other engineers in a design review.
Rocky’s GenericCloud qcow2 wouldn’t boot on my Proxmox host. Not a terraform.tfvars mistake, not a bad cloud-init template — a broken image, on that particular hypervisor, and root-causing someone else’s cloud image wasn’t the problem I’d sat down to solve that evening.
Here’s the split from above, applied for real. What I was actually protecting was carrying a genuinely enforced OS-level constraint, not “running Rocky specifically.” What broke was one vehicle for getting there, not the goal itself. So I swapped the template to Ubuntu 24.04 LTS and moved on — and then said the cost of that swap plainly rather than let the post read as if nothing changed:
I lose the SELinux/SCC-style enforcement OpenShift gives me for free at work, and Ubuntu’s default AppArmor confinement isn’t a drop-in replacement for it.
What I refused to let slide with it was firewalld, configured with explicit rules rather than a flat, everything-trusts-everything home network — the cluster subnet trusted for etcd/kubelet/BGP/Cilium health traffic, the kube-apiserver port opened deliberately rather than the whole host. That posture never depended on which distro sat underneath it, so it didn’t need re-litigating just because the distro did.
The part of this I’d actually want a hiring panel to notice isn’t “he picked Ubuntu.” It’s that the decision took an afternoon, not a week, because I already knew which half of the original plan was load-bearing and which half was just the tool I happened to be using to get there. And the published post still says, in plain language, exactly what I gave up — not because a reviewer would catch it if I hid it, nobody’s auditing a homelab, but because a write-up that hides its own trade-offs is worthless as a record of how a decision actually went.
The same instinct, smaller stakes: what’s the default StorageClass#
I made a near-identical call, at a smaller scale, when I set the default StorageClass for that same cluster. I’ve already written up the mechanics in Building an HA k3s Control Plane on Ubuntu 24.04 LTS
— this isn’t a re-explanation, just the reasoning pulled out on its own.
The cluster has two storage options: NFS, safe and slow, backed by a plain export with no vendor API involved; and DirectPV, fast and node-local, backed by raw unformatted disks Terraform attaches to each worker. DirectPV is the more interesting technology and the more convenient default — right up until a worker’s disk dies and takes a non-replicated volume with it.
So NFS is the cluster default, with reclaimPolicy: Retain, and DirectPV is opt-in only, reserved for the narrow set of workloads that are already replicated at the application layer or fine losing cache-shaped data — MariaDB Galera, Prometheus, nothing else unless it asks for directpv-min-io by name. Same instinct as the Rocky decision: what was I actually protecting? Not “use the fastest storage available” — protecting against a helm uninstall I ran half-asleep quietly taking a database with it. Retain is itself a named trade — the cluster accumulates stale exports over time that someone has to go clean up — and I said so in the post rather than presenting the default as a free win.
Neither of these decisions was hard, technically. What made them worth writing up is that both took less than an hour to land on, because “what am I actually protecting, and what am I willing to spend to keep it” is a question I ask automatically now, not one I work through from scratch each time something breaks.
Where the same question shows up when the stakes are real#
Homelab decisions cost me an evening if I get them wrong. The version of this that actually matters is the day job — architecture decisions that other engineering teams build against, inside a regulated banking group, where getting the trade-off wrong shows up as an audit finding rather than a stale NFS export.
Translating POPIA and PCI-DSS into a reference architecture is the same split, at a scale where I don’t get to make the call alone. A regulatory requirement isn’t optional and it isn’t negotiable — but how a team meets it inside a given design usually has more than one honest answer, and my job is knowing which part of the constraint is the actual legal obligation and which part is just the first pattern that satisfied it. Handing engineering teams a reference architecture that pretends there’s no trade-off in it — no cost, no latency, no operational overhead attached to the guardrail — doesn’t survive contact with a real workload. Naming the trade explicitly, in the architecture doc itself, is what makes it something a team can actually adopt with their eyes open rather than something they route around the first time it’s inconvenient.
The FinOps governance work is the same instinct pointed at cost instead of security. Every cost-optimisation lever I put in front of an engineering team — reserved capacity, right-sizing, workload scheduling — has a matching thing it costs them: less headroom, more operational discipline required, a slower path to spinning something up. A governance programme that only ever presents the savings and never names what a team is giving up to get them doesn’t hold up once the first incident happens under-provisioned.
| Decision | What I was actually protecting | What got named as the cost |
|---|---|---|
| Rocky Linux 10 → Ubuntu 24.04 LTS | Carrying a genuinely enforced OS-level security constraint, not a specific distro | Loss of SELinux/SCC-style enforcement; AppArmor isn’t a drop-in replacement |
StorageClass default: NFS over DirectPV | Data not disappearing because of an accidental helm uninstall | Slower storage as the default; stale exports accumulate under Retain and need periodic cleanup |
| POPIA/PCI-DSS reference architectures | The actual regulatory obligation, not the first pattern that happened to satisfy it | Cost, latency, or operational overhead the guardrail adds — stated in the design doc, not discovered later |
| FinOps governance | Group-level cloud spend staying accountable | Headroom and operational slack a team gives up for the saving |
The rows differ enormously in stakes. The reasoning in each row is the same reasoning, run the same way, whether the outcome is “I lose SELinux on my own VMs for a weekend” or “a banking platform’s guardrail costs a team latency they have to plan around.”
I don’t optimise for the original plan being right. I optimise for the trade-off staying visible when the plan breaks — because the decisions I don’t have to defend later are the ones I wrote down honestly the first time.
Summary: the technology choice in either of these stories is genuinely the less interesting part — Rocky or Ubuntu, NFS or DirectPV, it wouldn’t much matter which way either one landed. What I’d want a hiring panel screening for Principal to actually take from this is the process underneath both calls: work out what the plan was really protecting, keep protecting that when the specific vehicle for it breaks, and put the cost in writing instead of letting the post read as if nothing was given up. That’s the habit that scales from a broken cloud image on my own Proxmox host to a reference architecture a dozen engineering teams build against inside a regulated bank — same question, asked the same way, regardless of what’s actually at stake.