Igor

homelab-journey

Cluster Observability Part 9: Know Your Logs Before You Hunt Them

Before hunting errors with LogQL, you need to know your log formats. Format discovery, the Garage INFO trap, klog envelopes, cardinality constraints, and fixing missing kube-system static pod logs.

homelab-journey

Cluster Observability Part 8: Talos System Logs

Talos system logs via Vector: why loki.source.tcp doesn't exist, how Vector fills the gap, and fixing Alloy's node-local filter in the same pass.

homelab-journey

Cluster Observability Part 7: Log Aggregation with Loki and Grafana Alloy

Loki + Grafana Alloy on ARM64: three deploy attempts, five Alloy River config gotchas, and logs finally flowing across all four cluster nodes.

homelab-journey

PostgreSQL Backups on Kubernetes: Per-Database Dumps, Rotation, and Prometheus Monitoring

Per-database pg_dump to Garage S3, a touch file state machine, daily/weekly rotation, a Go Prometheus exporter, and Alertmanager rules for the PostgreSQL backup system on Bletchley.

homelab-journey

PostgreSQL on Bletchley: First Database on the Cluster

Setting up a shared PostgreSQL 17 StatefulSet on Kubernetes, Traefik TCP routing, an OpenBao v2.5.3 surprise, and migrating the Umami database.

homelab-journey

Prometheus Storage Overhead: Duplicate Scrapers and an Oversized Default Collector

Prometheus filling too fast — found a duplication and a 55% storage hog hiding in chart defaults. Fixed both, cut ingestion by 60%.

homelab-journey

Prometheus Behind Traefik and Authelia: Closing the Last Ingress Gap

Two files close the Prometheus ingress gap — then a latent cert-manager bug surfaces. The ESO key name mismatch that renewal tests don't catch.

homelab-journey

Keeping the Bletchley Cluster Current: Talos, Kubernetes, and Every Helm Chart

Upgrading Talos, Kubernetes, and thirteen Helm charts with real workloads running — the safe order, the tools, and what went wrong. The how, not the what.

homelab-journey

Backup Controller v0.11.0: Detecting What Wasn't Created by the Controller

The controller managed its own backups perfectly. It had no idea about the rest. v0.11.0 adds an audit script and two new alert conditions to close that gap.

homelab-journey

Cluster Observability Part 6: Hardware Health Dashboards and Alerts

Grafana dashboards and 19 Prometheus alert rules for ZFS, SMART, disk temperature, fan speed, and Garage health — with manufacturer-sourced thresholds.

homelab-journey

Cluster Observability Part 5: Hardware Health Collectors - ZFS, SMART, and Thermal

Adding ZFS, SMART, and thermal collectors to the Bletchley cluster — and three ARM64 image attempts before finding one that actually works.

homelab-tools

stern: Real-Time Log Tailing Across Your Kubernetes Cluster

stern tails logs from multiple Kubernetes pods at once. I used it to find a pending Traefik upgrade and reconstruct a PVC resize across five pod types.