Prometheus
Fixing the Alerts: New Rules, Better Groups, Less Noise
Fan compound rule, seven new Prometheus rules, a six-group restructure, one Loki rule from the backup incident, and what's still on hold.
Alertmanager
Observability
Auditing the Alerts: What's Noise, What's Missing, What's Broken
Four months of running alerting on a homelab cluster. One false positive, one correct-but-broken underlying issue, eight days of silent backup failure, and what a rule-by-rule review actually looks like.
homelab-journey
Cluster Observability Part 11: Log-Based Alerting with Loki Ruler
Loki ruler setup, four log-based alert rules from real Part 10 findings, a silent config gotcha, end-to-end test, and the complete dual alerting architecture.
homelab-journey
Cluster Observability Part 4: Alerting with Prometheus and Alertmanager
Enabling Alertmanager on the Bletchley cluster: alerting rules, SMTP delivery, end-to-end testing, and two Talos-specific surprises.