Cluster Observability Part 3: Dashboards for Storage, Kubernetes, and Network
Importing Longhorn, Kubernetes, and resource dashboards onto the Bletchley cluster — and fixing the scrape config that was only collecting from one of four nodes.
Version Control for the Cluster: Installing Forgejo on Bletchley
Installing Forgejo on Bletchley: self-hosted git, infra/apps repo structure, accidentally committing secrets, SQLite WAL, and a two-layer backup to Garage S3.
Backup Validation: Restoring from Longhorn Backups
Testing the backup chain: etcd encryption with age, and a full Grafana PVC delete-and-restore to confirm Longhorn backups actually work.
Offloading Bletchley Backups: Traefik, Wildcard Certs, and rclone
Exposing Garage S3 via Traefik, why Synology Cloud Sync failed, and how rclone solved the offsite backup problem instead.
Backup Infrastructure: Garage S3, ZFS, and Longhorn Backup Targets
Building the local backup layer for the Bletchley cluster: ZFS mirror on rock3's SATA SSDs, Garage S3, NFS, and Longhorn recurring backups.
Certificate Management: cert-manager on the Bletchley Cluster
Adding TLS to the Bletchley cluster with cert-manager, TransIP DNS-01 challenges, Let's Encrypt staging and production issuers, and automatic HTTP→HTTPS redirects.
Cluster Networking: MetalLB and Traefik
Adding MetalLB and Traefik to Bletchley: real IPs for LoadBalancer services, hostname-based routing, and a reader-suggested improvement that preserves source IPs from day one.
Cluster Observability Part 2: Grafana Dashboards
Adding Grafana to the Bletchley cluster: Longhorn-backed storage, pre-configured Prometheus datasource, and the Node Exporter Full dashboard showing live node metrics.
Cluster Observability Part 1: Prometheus and Node Exporter on Talos
Installing Prometheus and node exporter on Talos Linux: namespace labelling, values files, the duplicate pod gotcha, and confirming all four nodes are being scraped.
Longhorn Deep Dive: Replicas, Versions, and Upgrades
The decisions behind the Longhorn installation: why 2 replicas on a 1Gb cluster, how version pinning protects reproducibility, and when to upgrade.
Setting Up Longhorn Storage on the Bletchley Cluster
Installing Longhorn distributed storage on a Talos Linux cluster: NVMe preparation, Helm install, and why the namespace label matters before anything else.
Upgrading Talos Linux Nodes: Adding a Missing Extension
How I discovered a missing system extension before it caused problems, and upgraded all four Bletchley cluster nodes in 15 minutes. Plus: what changes when you have running workloads.