Prometheus

homelab-journey

Longhorn Disk Alerting: Getting the Signal Right

Replacing misleading volume-level Longhorn alerts with disk-level rules, recalibrating snapshot overhead to 50%, and how an unrelated incident accidentally produced the baseline data needed.

homelab-journey

Longhorn Snapshot Overhead: Why the Alerts Were Right and Wrong

prometheus-server at 90% allocated, 5.6 GiB of real data. The alerts fired — but were they firing on the right thing? Snapshots, unreclaimed blocks, trim, and a PromQL join.

Prometheus

Fixing the Alerts: New Rules, Better Groups, Less Noise

Fan compound rule, seven new Prometheus rules, a six-group restructure, one Loki rule from the backup incident, and what's still on hold.

Observability

Auditing the Alerts: What's Noise, What's Missing, What's Broken

Four months of running alerting on a homelab cluster. One false positive, one correct-but-broken underlying issue, eight days of silent backup failure, and what a rule-by-rule review actually looks like.

homelab-journey

PostgreSQL Backups on Kubernetes: Per-Database Dumps, Rotation, and Prometheus Monitoring

Per-database pg_dump to Garage S3, a touch file state machine, daily/weekly rotation, a Go Prometheus exporter, and Alertmanager rules for the PostgreSQL backup system on Bletchley.

homelab-journey

Prometheus Storage Overhead: Duplicate Scrapers and an Oversized Default Collector

Prometheus filling too fast — found a duplication and a 55% storage hog hiding in chart defaults. Fixed both, cut ingestion by 60%.

homelab-journey

Prometheus Behind Traefik and Authelia: Closing the Last Ingress Gap

Two files close the Prometheus ingress gap — then a latent cert-manager bug surfaces. The ESO key name mismatch that renewal tests don't catch.

homelab-journey

Cluster Observability Part 6: Hardware Health Dashboards and Alerts

Grafana dashboards and 19 Prometheus alert rules for ZFS, SMART, disk temperature, fan speed, and Garage health — with manufacturer-sourced thresholds.

homelab-journey

Cluster Observability Part 5: Hardware Health Collectors - ZFS, SMART, and Thermal

Adding ZFS, SMART, and thermal collectors to the Bletchley cluster — and three ARM64 image attempts before finding one that actually works.

homelab-journey

Cluster Observability Part 4: Alerting with Prometheus and Alertmanager

Enabling Alertmanager on the Bletchley cluster: alerting rules, SMTP delivery, end-to-end testing, and two Talos-specific surprises.

homelab-journey

Cluster Observability Part 3: Dashboards for Storage, Kubernetes, and Network

Importing Longhorn, Kubernetes, and resource dashboards onto the Bletchley cluster — and fixing the scrape config that was only collecting from one of four nodes.

homelab-journey

Cluster Observability Part 2: Grafana Dashboards

Adding Grafana to the Bletchley cluster: Longhorn-backed storage, pre-configured Prometheus datasource, and the Node Exporter Full dashboard showing live node metrics.