Prometheus Behind Traefik and Authelia: Closing the Last Ingress Gap
Two files close the Prometheus ingress gap — then a latent cert-manager bug surfaces. The ESO key name mismatch that renewal tests don't catch.
Keeping the Bletchley Cluster Current: Talos, Kubernetes, and Every Helm Chart
Upgrading Talos, Kubernetes, and thirteen Helm charts with real workloads running — the safe order, the tools, and what went wrong. The how, not the what.
Backup Controller v0.11.0: Detecting What Wasn't Created by the Controller
The controller managed its own backups perfectly. It had no idea about the rest. v0.11.0 adds an audit script and two new alert conditions to close that gap.
Cluster Observability Part 6: Hardware Health Dashboards and Alerts
Grafana dashboards and 19 Prometheus alert rules for ZFS, SMART, disk temperature, fan speed, and Garage health — with manufacturer-sourced thresholds.
Cluster Observability Part 5: Hardware Health Collectors - ZFS, SMART, and Thermal
Adding ZFS, SMART, and thermal collectors to the Bletchley cluster — and three ARM64 image attempts before finding one that actually works.
stern: Real-Time Log Tailing Across Your Kubernetes Cluster
stern tails logs from multiple Kubernetes pods at once. I used it to find a pending Traefik upgrade and reconstruct a PVC resize across five pod types.
Closing the Backup Loop: A Custom Controller for Longhorn
New PVCs were silently unprotected until I noticed. I built a label-driven controller to close the loop — jobs created automatically, violations alerted, archives handled.
Secret Management Part 4: Worst Case Recovery — Rebuilding from First Principles
Losing OpenBao's Raft data is not inherently a data loss event — if every non-regeneratable secret is correctly externalised and maintained. The rebuild, mapped precisely.
Secret Management Part 3: Auto-Unseal with OpenBao on Proxmox
Auto-unseal via Transit on Proxmox LXC: what worked, what didn't, and the honest trade-off when the Shamir fallback assumption turned out to be wrong.
Secret Management Part 2: Securing the Remaining Cluster Secrets
Eight namespaces, eight ExternalSecrets, zero static tokens. Migrating every cluster credential to OpenBao — and learning which secrets don't need a vault at all.
Reflection: 25 Posts and a Cluster I Can Stand Behind
74 days, 25 posts, one Kubernetes cluster. A look back at what got built, what's still to come, and what I learned doing it.
Secret Management Part 1: Installing OpenBao and Securing the First Secret
Kubernetes Secrets aren't encrypted — just base64. I installed OpenBao and ESO to move the TransIP API key into a proper vault and verified cert-manager still works.