CloudNativePG Part 7: Turning Off the Thing You Built Four Months Ago

Deleting the legacy PostgreSQL instance took four commands. Proving it was safe to delete took a day and eleven read-only checks.

CloudNativePG Part 6: Migrating Umami Without Losing a Row

Moving a live 10 MB database onto CloudNativePG in under six minutes, and proving the verification harness could actually fail before trusting it.

Secret Management Part 5: Replacing OpenBao's kv-admin Token with a Login

kv-admin had absorbed root's privileges and carried a deadline nobody tracked. A userpass login removed both — and the escalation made it possible.

CloudNativePG Part 5: lldap's Database Goes First

LLDAP becomes the first real tenant on the shared CNPG cluster, and Authelia moves from a file to LDAP via one deadlocked rollout and a self-inflicted outage.

Certificate Management Revisited: My “Internal CA” Wasn't a CA

A cert-manager selfSigned ClusterIssuer sat unused for 161 days before I discovered it couldn't provide the trust anchor my first real internal service needed. Here's the two-object bootstrap that turns it into a usable internal CA — and the openssl test that proves the difference.

CloudNativePG Part 4: Breaking a Throwaway Database on Purpose

Extending Prometheus and Alertmanager to cover CloudNativePG, then chaos-testing failover, node loss, and restores against a throwaway database.

CloudNativePG Part 3: Installation and Exposing It with MetalLB

Installing CloudNativePG on Bletchley: the Cluster, MetalLB exposure, automated backups, and every real bug hit getting there.

CloudNativePG Part 2: Labeling Nodes and Auditing Existing Workloads

Labeling Bletchley's two boards for CloudNativePG, auditing existing workloads for the same partition risk, and confirming there's room for it.

CloudNativePG Part 1: Planning Highly Available PostgreSQL Across Two TuringPi Boards

Same partition, opposite outcome, depending only on which board the primary happens to be on.

Silent Failures: Monitoring Stateful Services After Restarts

Garage's node ID reverted, Longhorn backups failed silently, and OpenBao turned out to have the same unmonitored risk — three alerts, two reactive and one proactive.

Fixing a Postgres Backup Failure: Region Mismatch and a Missing NAS Copy

A LogQL finding led to a one-line AWS CLI region fix for postgres housekeeping — and uncovered a second gap: backups never synced to the NAS.

Longhorn Disk Alerting: Getting the Signal Right

Replacing misleading volume-level Longhorn alerts with disk-level rules, recalibrating snapshot overhead to 50%, and how an unrelated incident accidentally produced the baseline data needed.

homelab-journey

Metrics Server on Talos: The Reboot That Broke Garage

Empty Longhorn panels led to metrics-server, a Talos cert quirk, a reboot that broke Garage, and 16 silent hours of failed backups.

homelab-journey

Longhorn Snapshot Overhead: Why the Alerts Were Right and Wrong

prometheus-server at 90% allocated, 5.6 GiB of real data. The alerts fired — but were they firing on the right thing? Snapshots, unreclaimed blocks, trim, and a PromQL join.

homelab-journey

Enabling fstrim on Longhorn Volumes in Talos Linux

Longhorn allocation never coming down? Here's how to enable automated fstrim via a RecurringJob on Talos Linux — and what it can't fix.

Prometheus

Fixing the Alerts: New Rules, Better Groups, Less Noise

Fan compound rule, seven new Prometheus rules, a six-group restructure, one Loki rule from the backup incident, and what's still on hold.

homelab-journey

The Backup That Wasn't: Eight Days, Five Root Causes

A stale Grafana metric, eight days of silent failure, and five root causes that each made sense in isolation. How a backup stopped without anyone noticing.

Observability

Auditing the Alerts: What's Noise, What's Missing, What's Broken

Four months of running alerting on a homelab cluster. One false positive, one correct-but-broken underlying issue, eight days of silent backup failure, and what a rule-by-rule review actually looks like.

homelab-journey

Longhorn Storage Upgrade: Replacing 500GB NVMe with 1TB

Rolling NVMe upgrade on a live Talos cluster: phase=failed, the extraMounts unlock sequence, and why storageReserved needs an explicit value.

homelab-journey

Growing the Cluster: Adding rock5, rock6, and rock7 to Bletchley

Three new RK1 worker nodes, three different Longhorn disk problems. The right way to add a node — and what happened when I didn't follow it.

homelab-journey

Cluster Observability Part 11: Log-Based Alerting with Loki Ruler

Loki ruler setup, four log-based alert rules from real Part 10 findings, a silent config gotcha, end-to-end test, and the complete dual alerting architecture.

homelab-journey

Cluster Observability Part 10: Hunting for Errors with LogQL

Six investigations across a Kubernetes cluster using LogQL: bootstrap artefacts, a silent 3-week backup failure, an Authelia crash sequence, and what high error volume actually means.

homelab-journey

Cluster Observability Part 9: Know Your Logs Before You Hunt Them

Before hunting errors with LogQL, you need to know your log formats. Format discovery, the Garage INFO trap, klog envelopes, cardinality constraints, and fixing missing kube-system static pod logs.

homelab-journey

Cluster Observability Part 8: Talos System Logs

Talos system logs via Vector: why loki.source.tcp doesn't exist, how Vector fills the gap, and fixing Alloy's node-local filter in the same pass.