homelab-journey
Enabling fstrim on Longhorn Volumes in Talos Linux
Longhorn allocation never coming down? Here's how to enable automated fstrim via a RecurringJob on Talos Linux — and what it can't fix.
Kubernetes
Deep dives into K8s clusters, resource management, and cloud-native orchestration.
Prometheus
Fixing the Alerts: New Rules, Better Groups, Less Noise
Fan compound rule, seven new Prometheus rules, a six-group restructure, one Loki rule from the backup incident, and what's still on hold.
homelab-journey
The Backup That Wasn't: Eight Days, Five Root Causes
A stale Grafana metric, eight days of silent failure, and five root causes that each made sense in isolation. How a backup stopped without anyone noticing.
Observability
Auditing the Alerts: What's Noise, What's Missing, What's Broken
Four months of running alerting on a homelab cluster. One false positive, one correct-but-broken underlying issue, eight days of silent backup failure, and what a rule-by-rule review actually looks like.
homelab-journey
Longhorn Storage Upgrade: Replacing 500GB NVMe with 1TB
Rolling NVMe upgrade on a live Talos cluster: phase=failed, the extraMounts unlock sequence, and why storageReserved needs an explicit value.
homelab-journey
Growing the Cluster: Adding rock5, rock6, and rock7 to Bletchley
Three new RK1 worker nodes, three different Longhorn disk problems. The right way to add a node — and what happened when I didn't follow it.
homelab-journey
Cluster Observability Part 11: Log-Based Alerting with Loki Ruler
Loki ruler setup, four log-based alert rules from real Part 10 findings, a silent config gotcha, end-to-end test, and the complete dual alerting architecture.
homelab-journey
Cluster Observability Part 10: Hunting for Errors with LogQL
Six investigations across a Kubernetes cluster using LogQL: bootstrap artefacts, a silent 3-week backup failure, an Authelia crash sequence, and what high error volume actually means.
homelab-journey
Cluster Observability Part 9: Know Your Logs Before You Hunt Them
Before hunting errors with LogQL, you need to know your log formats. Format discovery, the Garage INFO trap, klog envelopes, cardinality constraints, and fixing missing kube-system static pod logs.
homelab-journey
Cluster Observability Part 7: Log Aggregation with Loki and Grafana Alloy
Loki + Grafana Alloy on ARM64: three deploy attempts, five Alloy River config gotchas, and logs finally flowing across all four cluster nodes.
homelab-journey
PostgreSQL Backups on Kubernetes: Per-Database Dumps, Rotation, and Prometheus Monitoring
Per-database pg_dump to Garage S3, a touch file state machine, daily/weekly rotation, a Go Prometheus exporter, and Alertmanager rules for the PostgreSQL backup system on Bletchley.
homelab-journey
PostgreSQL on Bletchley: First Database on the Cluster
Setting up a shared PostgreSQL 17 StatefulSet on Kubernetes, Traefik TCP routing, an OpenBao v2.5.3 surprise, and migrating the Umami database.