homelab-journey
Silent Failures: Monitoring Stateful Services After Restarts
Garage's node ID reverted, Longhorn backups failed silently, and OpenBao turned out to have the same unmonitored risk — three alerts, two reactive and one proactive.
Longhorn
Managing cloud-native distributed block storage for Kubernetes with Longhorn.
homelab-journey
Longhorn Disk Alerting: Getting the Signal Right
Replacing misleading volume-level Longhorn alerts with disk-level rules, recalibrating snapshot overhead to 50%, and how an unrelated incident accidentally produced the baseline data needed.
homelab-journey
Metrics Server on Talos: The Reboot That Broke Garage
Empty Longhorn panels led to metrics-server, a Talos cert quirk, a reboot that broke Garage, and 16 silent hours of failed backups.
homelab-journey
Longhorn Snapshot Overhead: Why the Alerts Were Right and Wrong
prometheus-server at 90% allocated, 5.6 GiB of real data. The alerts fired — but were they firing on the right thing? Snapshots, unreclaimed blocks, trim, and a PromQL join.
homelab-journey
Enabling fstrim on Longhorn Volumes in Talos Linux
Longhorn allocation never coming down? Here's how to enable automated fstrim via a RecurringJob on Talos Linux — and what it can't fix.
homelab-journey
Longhorn Storage Upgrade: Replacing 500GB NVMe with 1TB
Rolling NVMe upgrade on a live Talos cluster: phase=failed, the extraMounts unlock sequence, and why storageReserved needs an explicit value.
homelab-journey
Growing the Cluster: Adding rock5, rock6, and rock7 to Bletchley
Three new RK1 worker nodes, three different Longhorn disk problems. The right way to add a node — and what happened when I didn't follow it.
homelab-journey
Keeping the Bletchley Cluster Current: Talos, Kubernetes, and Every Helm Chart
Upgrading Talos, Kubernetes, and thirteen Helm charts with real workloads running — the safe order, the tools, and what went wrong. The how, not the what.
homelab-journey
Backup Controller v0.11.0: Detecting What Wasn't Created by the Controller
The controller managed its own backups perfectly. It had no idea about the rest. v0.11.0 adds an audit script and two new alert conditions to close that gap.
homelab-tools
stern: Real-Time Log Tailing Across Your Kubernetes Cluster
stern tails logs from multiple Kubernetes pods at once. I used it to find a pending Traefik upgrade and reconstruct a PVC resize across five pod types.
homelab-journey
Closing the Backup Loop: A Custom Controller for Longhorn
New PVCs were silently unprotected until I noticed. I built a label-driven controller to close the loop — jobs created automatically, violations alerted, archives handled.
homelab-tools
k9s: A Terminal UI for Navigating Your Kubernetes Cluster
k9s gives you an interactive terminal UI for your Kubernetes cluster. I installed it and immediately found a backup job that had been silently failing for eleven days.