Enabling fstrim on Longhorn Volumes in Talos Linux
Longhorn allocation never coming down? Here's how to enable automated fstrim via a RecurringJob on Talos Linux — and what it can't fix.
Latest
Fixing the Alerts: New Rules, Better Groups, Less Noise
Fan compound rule, seven new Prometheus rules, a six-group restructure, one Loki rule from the backup incident, and what's still on hold.
The Backup That Wasn't: Eight Days, Five Root Causes
A stale Grafana metric, eight days of silent failure, and five root causes that each made sense in isolation. How a backup stopped without anyone noticing.
Auditing the Alerts: What's Noise, What's Missing, What's Broken
Four months of running alerting on a homelab cluster. One false positive, one correct-but-broken underlying issue, eight days of silent backup failure, and what a rule-by-rule review actually looks like.
Longhorn Storage Upgrade: Replacing 500GB NVMe with 1TB
Rolling NVMe upgrade on a live Talos cluster: phase=failed, the extraMounts unlock sequence, and why storageReserved needs an explicit value.
Growing the Cluster: Adding rock5, rock6, and rock7 to Bletchley
Three new RK1 worker nodes, three different Longhorn disk problems. The right way to add a node — and what happened when I didn't follow it.
Cluster Observability Part 11: Log-Based Alerting with Loki Ruler
Loki ruler setup, four log-based alert rules from real Part 10 findings, a silent config gotcha, end-to-end test, and the complete dual alerting architecture.
Cluster Observability Part 10: Hunting for Errors with LogQL
Six investigations across a Kubernetes cluster using LogQL: bootstrap artefacts, a silent 3-week backup failure, an Authelia crash sequence, and what high error volume actually means.
Cluster Observability Part 9: Know Your Logs Before You Hunt Them
Before hunting errors with LogQL, you need to know your log formats. Format discovery, the Garage INFO trap, klog envelopes, cardinality constraints, and fixing missing kube-system static pod logs.
Cluster Observability Part 8: Talos System Logs
Talos system logs via Vector: why loki.source.tcp doesn't exist, how Vector fills the gap, and fixing Alloy's node-local filter in the same pass.
Cluster Observability Part 7: Log Aggregation with Loki and Grafana Alloy
Loki + Grafana Alloy on ARM64: three deploy attempts, five Alloy River config gotchas, and logs finally flowing across all four cluster nodes.
PostgreSQL Backups on Kubernetes: Per-Database Dumps, Rotation, and Prometheus Monitoring
Per-database pg_dump to Garage S3, a touch file state machine, daily/weekly rotation, a Go Prometheus exporter, and Alertmanager rules for the PostgreSQL backup system on Bletchley.
homelab-journey
PostgreSQL on Bletchley: First Database on the Cluster
Setting up a shared PostgreSQL 17 StatefulSet on Kubernetes, Traefik TCP routing, an OpenBao v2.5.3 surprise, and migrating the Umami database.
homelab-journey
Prometheus Storage Overhead: Duplicate Scrapers and an Oversized Default Collector
Prometheus filling too fast — found a duplication and a 55% storage hog hiding in chart defaults. Fixed both, cut ingestion by 60%.
homelab-journey
Prometheus Behind Traefik and Authelia: Closing the Last Ingress Gap
Two files close the Prometheus ingress gap — then a latent cert-manager bug surfaces. The ESO key name mismatch that renewal tests don't catch.
homelab-journey
Keeping the Bletchley Cluster Current: Talos, Kubernetes, and Every Helm Chart
Upgrading Talos, Kubernetes, and thirteen Helm charts with real workloads running — the safe order, the tools, and what went wrong. The how, not the what.
homelab-journey
Backup Controller v0.11.0: Detecting What Wasn't Created by the Controller
The controller managed its own backups perfectly. It had no idea about the rest. v0.11.0 adds an audit script and two new alert conditions to close that gap.
homelab-journey
Cluster Observability Part 6: Hardware Health Dashboards and Alerts
Grafana dashboards and 19 Prometheus alert rules for ZFS, SMART, disk temperature, fan speed, and Garage health — with manufacturer-sourced thresholds.
homelab-journey
Cluster Observability Part 5: Hardware Health Collectors - ZFS, SMART, and Thermal
Adding ZFS, SMART, and thermal collectors to the Bletchley cluster — and three ARM64 image attempts before finding one that actually works.
homelab-tools
stern: Real-Time Log Tailing Across Your Kubernetes Cluster
stern tails logs from multiple Kubernetes pods at once. I used it to find a pending Traefik upgrade and reconstruct a PVC resize across five pod types.
homelab-journey
Closing the Backup Loop: A Custom Controller for Longhorn
New PVCs were silently unprotected until I noticed. I built a label-driven controller to close the loop — jobs created automatically, violations alerted, archives handled.
homelab-journey
Secret Management Part 4: Worst Case Recovery — Rebuilding from First Principles
Losing OpenBao's Raft data is not inherently a data loss event — if every non-regeneratable secret is correctly externalised and maintained. The rebuild, mapped precisely.
homelab-journey
Secret Management Part 3: Auto-Unseal with OpenBao on Proxmox
Auto-unseal via Transit on Proxmox LXC: what worked, what didn't, and the honest trade-off when the Shamir fallback assumption turned out to be wrong.
homelab-journey
Secret Management Part 2: Securing the Remaining Cluster Secrets
Eight namespaces, eight ExternalSecrets, zero static tokens. Migrating every cluster credential to OpenBao — and learning which secrets don't need a vault at all.