Enabling fstrim on Longhorn Volumes in Talos Linux
Longhorn allocation never coming down? Here's how to enable automated fstrim via a RecurringJob on Talos Linux — and what it can't fix.
Fixing the Alerts: New Rules, Better Groups, Less Noise
Fan compound rule, seven new Prometheus rules, a six-group restructure, one Loki rule from the backup incident, and what's still on hold.
The Backup That Wasn't: Eight Days, Five Root Causes
A stale Grafana metric, eight days of silent failure, and five root causes that each made sense in isolation. How a backup stopped without anyone noticing.
Auditing the Alerts: What's Noise, What's Missing, What's Broken
Four months of running alerting on a homelab cluster. One false positive, one correct-but-broken underlying issue, eight days of silent backup failure, and what a rule-by-rule review actually looks like.
Longhorn Storage Upgrade: Replacing 500GB NVMe with 1TB
Rolling NVMe upgrade on a live Talos cluster: phase=failed, the extraMounts unlock sequence, and why storageReserved needs an explicit value.
Growing the Cluster: Adding rock5, rock6, and rock7 to Bletchley
Three new RK1 worker nodes, three different Longhorn disk problems. The right way to add a node — and what happened when I didn't follow it.
Cluster Observability Part 11: Log-Based Alerting with Loki Ruler
Loki ruler setup, four log-based alert rules from real Part 10 findings, a silent config gotcha, end-to-end test, and the complete dual alerting architecture.
Cluster Observability Part 10: Hunting for Errors with LogQL
Six investigations across a Kubernetes cluster using LogQL: bootstrap artefacts, a silent 3-week backup failure, an Authelia crash sequence, and what high error volume actually means.
Cluster Observability Part 9: Know Your Logs Before You Hunt Them
Before hunting errors with LogQL, you need to know your log formats. Format discovery, the Garage INFO trap, klog envelopes, cardinality constraints, and fixing missing kube-system static pod logs.
Cluster Observability Part 8: Talos System Logs
Talos system logs via Vector: why loki.source.tcp doesn't exist, how Vector fills the gap, and fixing Alloy's node-local filter in the same pass.
Cluster Observability Part 7: Log Aggregation with Loki and Grafana Alloy
Loki + Grafana Alloy on ARM64: three deploy attempts, five Alloy River config gotchas, and logs finally flowing across all four cluster nodes.
PostgreSQL Backups on Kubernetes: Per-Database Dumps, Rotation, and Prometheus Monitoring
Per-database pg_dump to Garage S3, a touch file state machine, daily/weekly rotation, a Go Prometheus exporter, and Alertmanager rules for the PostgreSQL backup system on Bletchley.