homelab-journey
Healthy, but Not Reading: Moving DSMR-reader to Bletchley
DSMR-reader was Ready but collecting nothing. A failed migration, three layers of monitoring, and tracking PostgreSQL inserts without a GRANT
Prometheus
Guides on metrics collection, alerting, and monitoring infrastructure using Prometheus.
cluster-observability
Getting etcd's Metrics Out of Talos
etcd on Talos is not a pod, so nothing scraped it. One patch, a restart Talos refuses, a rolling reboot, and alerts so the metrics can't go quiet.
cluster-observability
Teaching Prometheus to Read an eMMC
A flat line can mean a healthy eMMC or a dead collector. An arm64 exporter, eleven tested alert rules and a dashboard that tell the two apart.
deep-dive
The Disk With No S.M.A.R.T.
Six months of drive monitoring pointed at the wrong device. The one doing three times the writes has no S.M.A.R.T. at all.
cluster-observability
Cluster-Wide Certificate Expiry Monitoring
cert-manager covered 18 certificates. Thirteen more had no owner and five expire on one day. Closing the gap with x509-certificate-exporter.
deep-dive
The Alert That Named a Pod
A Prometheus alert named a pod IP, not a machine. The 'broken' label turned out to have logged every cluster-wide reboot for three months.
homelab-journey
CloudNativePG Part 4: Breaking a Throwaway Database on Purpose
Extending Prometheus and Alertmanager to cover CloudNativePG, then chaos-testing failover, node loss, and restores against a throwaway database.
homelab-journey
Longhorn Disk Alerting: Getting the Signal Right
Replacing misleading volume-level Longhorn alerts with disk-level rules, recalibrating snapshot overhead to 50%, and how an unrelated incident accidentally produced the baseline data needed.
homelab-journey
Longhorn Snapshot Overhead: Why the Alerts Were Right and Wrong
prometheus-server at 90% allocated, 5.6 GiB of real data. The alerts fired — but were they firing on the right thing? Snapshots, unreclaimed blocks, trim, and a PromQL join.
Prometheus
Fixing the Alerts: New Rules, Better Groups, Less Noise
Fan compound rule, seven new Prometheus rules, a six-group restructure, one Loki rule from the backup incident, and what's still on hold.
Observability
Auditing the Alerts: What's Noise, What's Missing, What's Broken
Four months of running alerting on a homelab cluster. One false positive, one correct-but-broken underlying issue, eight days of silent backup failure, and what a rule-by-rule review actually looks like.
homelab-journey
PostgreSQL Backups on Kubernetes: Per-Database Dumps, Rotation, and Prometheus Monitoring
Per-database pg_dump to Garage S3, a touch file state machine, daily/weekly rotation, a Go Prometheus exporter, and Alertmanager rules for the PostgreSQL backup system on Bletchley.