homelab-journey
Enabling fstrim on Longhorn Volumes in Talos Linux
Longhorn allocation never coming down? Here's how to enable automated fstrim via a RecurringJob on Talos Linux — and what it can't fix.
homelab-journey
Documentation of my home lab evolution, from hardware builds to software experiments.
homelab-journey
The Backup That Wasn't: Eight Days, Five Root Causes
A stale Grafana metric, eight days of silent failure, and five root causes that each made sense in isolation. How a backup stopped without anyone noticing.
homelab-journey
Longhorn Storage Upgrade: Replacing 500GB NVMe with 1TB
Rolling NVMe upgrade on a live Talos cluster: phase=failed, the extraMounts unlock sequence, and why storageReserved needs an explicit value.
homelab-journey
Growing the Cluster: Adding rock5, rock6, and rock7 to Bletchley
Three new RK1 worker nodes, three different Longhorn disk problems. The right way to add a node — and what happened when I didn't follow it.
homelab-journey
Cluster Observability Part 11: Log-Based Alerting with Loki Ruler
Loki ruler setup, four log-based alert rules from real Part 10 findings, a silent config gotcha, end-to-end test, and the complete dual alerting architecture.
homelab-journey
Cluster Observability Part 10: Hunting for Errors with LogQL
Six investigations across a Kubernetes cluster using LogQL: bootstrap artefacts, a silent 3-week backup failure, an Authelia crash sequence, and what high error volume actually means.
homelab-journey
Cluster Observability Part 9: Know Your Logs Before You Hunt Them
Before hunting errors with LogQL, you need to know your log formats. Format discovery, the Garage INFO trap, klog envelopes, cardinality constraints, and fixing missing kube-system static pod logs.
homelab-journey
Cluster Observability Part 8: Talos System Logs
Talos system logs via Vector: why loki.source.tcp doesn't exist, how Vector fills the gap, and fixing Alloy's node-local filter in the same pass.
homelab-journey
Cluster Observability Part 7: Log Aggregation with Loki and Grafana Alloy
Loki + Grafana Alloy on ARM64: three deploy attempts, five Alloy River config gotchas, and logs finally flowing across all four cluster nodes.
homelab-journey
PostgreSQL Backups on Kubernetes: Per-Database Dumps, Rotation, and Prometheus Monitoring
Per-database pg_dump to Garage S3, a touch file state machine, daily/weekly rotation, a Go Prometheus exporter, and Alertmanager rules for the PostgreSQL backup system on Bletchley.
homelab-journey
PostgreSQL on Bletchley: First Database on the Cluster
Setting up a shared PostgreSQL 17 StatefulSet on Kubernetes, Traefik TCP routing, an OpenBao v2.5.3 surprise, and migrating the Umami database.
homelab-journey
Prometheus Storage Overhead: Duplicate Scrapers and an Oversized Default Collector
Prometheus filling too fast — found a duplication and a 55% storage hog hiding in chart defaults. Fixed both, cut ingestion by 60%.