homelab-journey
CloudNativePG Part 9: Authelia Gives Up Its PVC
Authelia's users were on CloudNativePG, its own state wasn't. Moving it took an inventory, a third tenant and a backup archive workflow.
Longhorn
Managing cloud-native distributed block storage for Kubernetes with Longhorn.
deep-dive
Wear Without Writes
One NVMe's reported wear sped up 4.5× the evening its writes almost stopped. Its spare blocks and error counters say the flash never noticed.
deep-dive
The Wear That Wasn't
An NVMe alert at 82% used life, on drives with full spare capacity and zero errors. The wear number turned out to be a clock, not an odometer.
homelab-journey
Silent Failures: Monitoring Stateful Services After Restarts
Garage's node ID reverted, Longhorn backups failed silently, and OpenBao turned out to have the same unmonitored risk — three alerts, two reactive and one proactive.
homelab-journey
Longhorn Disk Alerting: Getting the Signal Right
Replacing misleading volume-level Longhorn alerts with disk-level rules, recalibrating snapshot overhead to 50%, and how an unrelated incident accidentally produced the baseline data needed.
homelab-journey
Metrics Server on Talos: The Reboot That Broke Garage
Empty Longhorn panels led to metrics-server, a Talos cert quirk, a reboot that broke Garage, and 16 silent hours of failed backups.
homelab-journey
Longhorn Snapshot Overhead: Why the Alerts Were Right and Wrong
prometheus-server at 90% allocated, 5.6 GiB of real data. The alerts fired — but were they firing on the right thing? Snapshots, unreclaimed blocks, trim, and a PromQL join.
homelab-journey
Enabling fstrim on Longhorn Volumes in Talos Linux
Longhorn allocation never coming down? Here's how to enable automated fstrim via a RecurringJob on Talos Linux — and what it can't fix.
homelab-journey
Longhorn Storage Upgrade: Replacing 500GB NVMe with 1TB
Rolling NVMe upgrade on a live Talos cluster: phase=failed, the extraMounts unlock sequence, and why storageReserved needs an explicit value.
homelab-journey
Growing the Cluster: Adding rock5, rock6, and rock7 to Bletchley
Three new RK1 worker nodes, three different Longhorn disk problems. The right way to add a node — and what happened when I didn't follow it.
homelab-journey
Keeping the Bletchley Cluster Current: Talos, Kubernetes, and Every Helm Chart
Upgrading Talos, Kubernetes, and thirteen Helm charts with real workloads running — the safe order, the tools, and what went wrong. The how, not the what.
homelab-journey
Backup Controller v0.11.0: Detecting What Wasn't Created by the Controller
The controller managed its own backups perfectly. It had no idea about the rest. v0.11.0 adds an audit script and two new alert conditions to close that gap.