homelab-journey
Cluster Observability Part 7: Log Aggregation with Loki and Grafana Alloy
Loki + Grafana Alloy on ARM64: three deploy attempts, five Alloy River config gotchas, and logs finally flowing across all four cluster nodes.
homelab-journey
Documentation of my home lab evolution, from hardware builds to software experiments.
homelab-journey
PostgreSQL Backups on Kubernetes: Per-Database Dumps, Rotation, and Prometheus Monitoring
Per-database pg_dump to Garage S3, a touch file state machine, daily/weekly rotation, a Go Prometheus exporter, and Alertmanager rules for the PostgreSQL backup system on Bletchley.
homelab-journey
PostgreSQL on Bletchley: First Database on the Cluster
Setting up a shared PostgreSQL 17 StatefulSet on Kubernetes, Traefik TCP routing, an OpenBao v2.5.3 surprise, and migrating the Umami database.
homelab-journey
Prometheus Storage Overhead: Duplicate Scrapers and an Oversized Default Collector
Prometheus filling too fast — found a duplication and a 55% storage hog hiding in chart defaults. Fixed both, cut ingestion by 60%.
homelab-journey
Prometheus Behind Traefik and Authelia: Closing the Last Ingress Gap
Two files close the Prometheus ingress gap — then a latent cert-manager bug surfaces. The ESO key name mismatch that renewal tests don't catch.
homelab-journey
Keeping the Bletchley Cluster Current: Talos, Kubernetes, and Every Helm Chart
Upgrading Talos, Kubernetes, and thirteen Helm charts with real workloads running — the safe order, the tools, and what went wrong. The how, not the what.
homelab-journey
Backup Controller v0.11.0: Detecting What Wasn't Created by the Controller
The controller managed its own backups perfectly. It had no idea about the rest. v0.11.0 adds an audit script and two new alert conditions to close that gap.
homelab-journey
Cluster Observability Part 6: Hardware Health Dashboards and Alerts
Grafana dashboards and 19 Prometheus alert rules for ZFS, SMART, disk temperature, fan speed, and Garage health — with manufacturer-sourced thresholds.
homelab-journey
Cluster Observability Part 5: Hardware Health Collectors - ZFS, SMART, and Thermal
Adding ZFS, SMART, and thermal collectors to the Bletchley cluster — and three ARM64 image attempts before finding one that actually works.
homelab-journey
Closing the Backup Loop: A Custom Controller for Longhorn
New PVCs were silently unprotected until I noticed. I built a label-driven controller to close the loop — jobs created automatically, violations alerted, archives handled.
homelab-journey
Secret Management Part 4: Worst Case Recovery — Rebuilding from First Principles
Losing OpenBao's Raft data is not inherently a data loss event — if every non-regeneratable secret is correctly externalised and maintained. The rebuild, mapped precisely.
homelab-journey
Secret Management Part 3: Auto-Unseal with OpenBao on Proxmox
Auto-unseal via Transit on Proxmox LXC: what worked, what didn't, and the honest trade-off when the Shamir fallback assumption turned out to be wrong.