The Cost of Saying "I'm Still Here"
Control-plane eMMCs take 13–15 GB a day. Most of it is etcd recording twenty leases saying they're still alive, at about 22 KB a time.
Control-plane eMMCs take 13–15 GB a day. Most of it is etcd recording twenty leases saying they're still alive, at about 22 KB a time.
etcd on Talos is not a pod, so nothing scraped it. One patch, a restart Talos refuses, a rolling reboot, and alerts so the metrics can't go quiet.
One NVMe's reported wear sped up 4.5× the evening its writes almost stopped. Its spare blocks and error counters say the flash never noticed.
A flat line can mean a healthy eMMC or a dead collector. An arm64 exporter, eleven tested alert rules and a dashboard that tell the two apart.
Six months of drive monitoring pointed at the wrong device. The one doing three times the writes has no S.M.A.R.T. at all.
cert-manager covered 18 certificates. Thirteen more had no owner and five expire on one day. Closing the gap with x509-certificate-exporter.
An NVMe alert at 82% used life, on drives with full spare capacity and zero errors. The wear number turned out to be a clock, not an odometer.
A dashboard broken since March turned out to be a memory amplifier. One visit took Traefik to 4 GB and hung a node.
A Prometheus alert named a pod IP, not a machine. The 'broken' label turned out to have logged every cluster-wide reboot for three months.
One PostgreSQL connection, two certificate authorities: a public CA proves the server while CloudNativePG keeps its internal CA for client certificates.
Deleting the legacy PostgreSQL instance took four commands. Proving it was safe to delete took a day and eleven read-only checks.
Moving a live 10 MB database onto CloudNativePG in under six minutes, and proving the verification harness could actually fail before trusting it.