homelab-journey
CloudNativePG Part 1: Planning Highly Available PostgreSQL Across Two TuringPi Boards
Same partition, opposite outcome, depending only on which board the primary happens to be on.
homelab-journey
Documentation of my home lab evolution, from hardware builds to software experiments.
homelab-journey
Silent Failures: Monitoring Stateful Services After Restarts
Garage's node ID reverted, Longhorn backups failed silently, and OpenBao turned out to have the same unmonitored risk — three alerts, two reactive and one proactive.
homelab-journey
Fixing a Postgres Backup Failure: Region Mismatch and a Missing NAS Copy
A LogQL finding led to a one-line AWS CLI region fix for postgres housekeeping — and uncovered a second gap: backups never synced to the NAS.
homelab-journey
Longhorn Disk Alerting: Getting the Signal Right
Replacing misleading volume-level Longhorn alerts with disk-level rules, recalibrating snapshot overhead to 50%, and how an unrelated incident accidentally produced the baseline data needed.
homelab-journey
Metrics Server on Talos: The Reboot That Broke Garage
Empty Longhorn panels led to metrics-server, a Talos cert quirk, a reboot that broke Garage, and 16 silent hours of failed backups.
homelab-journey
Longhorn Snapshot Overhead: Why the Alerts Were Right and Wrong
prometheus-server at 90% allocated, 5.6 GiB of real data. The alerts fired — but were they firing on the right thing? Snapshots, unreclaimed blocks, trim, and a PromQL join.
homelab-journey
Enabling fstrim on Longhorn Volumes in Talos Linux
Longhorn allocation never coming down? Here's how to enable automated fstrim via a RecurringJob on Talos Linux — and what it can't fix.
homelab-journey
The Backup That Wasn't: Eight Days, Five Root Causes
A stale Grafana metric, eight days of silent failure, and five root causes that each made sense in isolation. How a backup stopped without anyone noticing.
homelab-journey
Longhorn Storage Upgrade: Replacing 500GB NVMe with 1TB
Rolling NVMe upgrade on a live Talos cluster: phase=failed, the extraMounts unlock sequence, and why storageReserved needs an explicit value.
homelab-journey
Growing the Cluster: Adding rock5, rock6, and rock7 to Bletchley
Three new RK1 worker nodes, three different Longhorn disk problems. The right way to add a node — and what happened when I didn't follow it.
homelab-journey
Cluster Observability Part 11: Log-Based Alerting with Loki Ruler
Loki ruler setup, four log-based alert rules from real Part 10 findings, a silent config gotcha, end-to-end test, and the complete dual alerting architecture.
homelab-journey
Cluster Observability Part 10: Hunting for Errors with LogQL
Six investigations across a Kubernetes cluster using LogQL: bootstrap artefacts, a silent 3-week backup failure, an Authelia crash sequence, and what high error volume actually means.