The Cost of Saying "I'm Still Here"
Control-plane eMMCs take 13–15 GB a day. Most of it is etcd recording twenty leases saying they're still alive, at about 22 KB a time.
Latest
Getting etcd's Metrics Out of Talos
etcd on Talos is not a pod, so nothing scraped it. One patch, a restart Talos refuses, a rolling reboot, and alerts so the metrics can't go quiet.
Wear Without Writes
One NVMe's reported wear sped up 4.5× the evening its writes almost stopped. Its spare blocks and error counters say the flash never noticed.
Teaching Prometheus to Read an eMMC
A flat line can mean a healthy eMMC or a dead collector. An arm64 exporter, eleven tested alert rules and a dashboard that tell the two apart.
The Disk With No S.M.A.R.T.
Six months of drive monitoring pointed at the wrong device. The one doing three times the writes has no S.M.A.R.T. at all.
Cluster-Wide Certificate Expiry Monitoring
cert-manager covered 18 certificates. Thirteen more had no owner and five expire on one day. Closing the gap with x509-certificate-exporter.
The Wear That Wasn't
An NVMe alert at 82% used life, on drives with full spare capacity and zero errors. The wear number turned out to be a clock, not an odometer.
The Traefik Dashboard That Routed to Itself
A dashboard broken since March turned out to be a memory amplifier. One visit took Traefik to 4 GB and hung a node.
The Alert That Named a Pod
A Prometheus alert named a pod IP, not a machine. The 'broken' label turned out to have logged every cluster-wide reboot for three months.
CloudNativePG Part 8: Two TLS Worlds and the Name Between Them
One PostgreSQL connection, two certificate authorities: a public CA proves the server while CloudNativePG keeps its internal CA for client certificates.
CloudNativePG Part 7: Turning Off the Thing You Built Four Months Ago
Deleting the legacy PostgreSQL instance took four commands. Proving it was safe to delete took a day and eleven read-only checks.
CloudNativePG Part 6: Migrating Umami Without Losing a Row
Moving a live 10 MB database onto CloudNativePG in under six minutes, and proving the verification harness could actually fail before trusting it.
homelab-journey
Secret Management Part 5: Replacing OpenBao's kv-admin Token with a Login
kv-admin had absorbed root's privileges and carried a deadline nobody tracked. A userpass login removed both — and the escalation made it possible.
homelab-journey
CloudNativePG Part 5: lldap's Database Goes First
LLDAP becomes the first real tenant on the shared CNPG cluster, and Authelia moves from a file to LDAP via one deadlocked rollout and a self-inflicted outage.
homelab-journey
Certificate Management Revisited: My “Internal CA” Wasn't a CA
A cert-manager selfSigned ClusterIssuer sat unused for 161 days before I discovered it couldn't provide the trust anchor my first real internal service needed. Here's the two-object bootstrap that turns it into a usable internal CA — and the openssl test that proves the difference.
homelab-journey
CloudNativePG Part 4: Breaking a Throwaway Database on Purpose
Extending Prometheus and Alertmanager to cover CloudNativePG, then chaos-testing failover, node loss, and restores against a throwaway database.
homelab-journey
CloudNativePG Part 3: Installation and Exposing It with MetalLB
Installing CloudNativePG on Bletchley: the Cluster, MetalLB exposure, automated backups, and every real bug hit getting there.
homelab-journey
CloudNativePG Part 2: Labeling Nodes and Auditing Existing Workloads
Labeling Bletchley's two boards for CloudNativePG, auditing existing workloads for the same partition risk, and confirming there's room for it.
homelab-journey
CloudNativePG Part 1: Planning Highly Available PostgreSQL Across Two TuringPi Boards
Same partition, opposite outcome, depending only on which board the primary happens to be on.
homelab-journey
Silent Failures: Monitoring Stateful Services After Restarts
Garage's node ID reverted, Longhorn backups failed silently, and OpenBao turned out to have the same unmonitored risk — three alerts, two reactive and one proactive.
homelab-journey
Fixing a Postgres Backup Failure: Region Mismatch and a Missing NAS Copy
A LogQL finding led to a one-line AWS CLI region fix for postgres housekeeping — and uncovered a second gap: backups never synced to the NAS.
homelab-journey
Longhorn Disk Alerting: Getting the Signal Right
Replacing misleading volume-level Longhorn alerts with disk-level rules, recalibrating snapshot overhead to 50%, and how an unrelated incident accidentally produced the baseline data needed.
homelab-journey
Metrics Server on Talos: The Reboot That Broke Garage
Empty Longhorn panels led to metrics-server, a Talos cert quirk, a reboot that broke Garage, and 16 silent hours of failed backups.
homelab-journey
Longhorn Snapshot Overhead: Why the Alerts Were Right and Wrong
prometheus-server at 90% allocated, 5.6 GiB of real data. The alerts fired — but were they firing on the right thing? Snapshots, unreclaimed blocks, trim, and a PromQL join.