CloudNativePG Part 2: Labeling Nodes and Auditing Existing Workloads

Introduction

Part 1 settled the interesting questions — where the primary lives, why async beat sync, why the label isn't called zone. None of it is deployable yet. The topology.bletchley/board label the whole anti-affinity design leans on doesn't exist on a single node. Part 3 can't install anything against a label that isn't there.

So before touching the operator, three things needed doing: apply the label to all seven nodes, check whether anything already running on the cluster has the same partition-visibility exposure Part 1 designed CNPG's placement around, and confirm there's actually room for three more Postgres instances alongside everything else. None of these are hard decisions on their own — the hard thinking happened in Part 1 — but skipping straight to installing the operator without doing them is exactly the kind of shortcut that turns into a Part 4 incident.

This post is part of the CloudNativePG sub-series.

• Part 1: Planning Across Two Boards
• Part 2: Labeling Nodes and Auditing Existing Workloads

🏠 This is part of the Homelab Journey series - building a production Kubernetes cluster from scratch.

• CloudNativePG Part 1: Planning Highly Available PostgreSQL Across Two TuringPi Boards
• CloudNativePG Part 2: Labeling Nodes and Auditing Existing Workloads (you are here)

This post picks up directly from CloudNativePG Part 1 — the topology and primary-placement reasoning here was all decided there. This post is the mechanical follow-through.

Applying the Board Label

Two small patch files, one value each:

# board-patch-bletchley1.yaml
machine:
  nodeLabels:
    topology.bletchley/board: bletchley1

# board-patch-bletchley2.yaml
machine:
  nodeLabels:
    topology.bletchley/board: bletchley2

The only real question was whether this needed a reboot. Talos's own docs say nodeLabels changes apply live, and a previous patch on this cluster had already proven it in practice. Ran it with --mode no-reboot anyway rather than trust that blind — that mode fails loudly instead of silently rebooting if the assumption turns out wrong.

Applied bletchley2 first, then bletchley1 — control plane last, on purpose:

for node in rock5 rock6 rock7; do
  talosctl patch mc -n $node --patch @board-patch-bletchley2.yaml
done
for node in rock1 rock2 rock3 rock4; do
  talosctl patch mc -n $node --patch @board-patch-bletchley1.yaml
done

Seven for seven: Applied configuration without a reboot. The control plane never blinked — same node age before and after, no unexpected quorum event. kubectl get nodes --show-labels confirms the split: rock1–4 carry topology.bletchley/board=bletchley1, rock5–7 carry bletchley2. Both files are committed now, with a short entry added to the repo's README alongside the rest.

Auditing the Rest of the Cluster

Nothing else on Bletchley runs HA today — that's the whole reason this series exists. What I hadn't actually checked was whether that held everywhere, or whether something already runs more than one replica without carrying the specific risk shape Part 1 designed CNPG's placement around: an operator-promoted role, a primary that can keep accepting writes while isolated from the board that manages it.

kubectl get deploy,sts,daemonset -A -o wide across every namespace confirmed the baseline and settled the exceptions. Garage, Loki, Alertmanager, the NFS provisioner, OpenBao, and the Postgres StatefulSet this series is working toward replacing are all singletons, as expected — one replica each, nothing to diverge from. The only workloads actually running more than one replica are CoreDNS and Longhorn's CSI sidecars and UI, and neither carries Part 1's risk: CoreDNS replicas are stateless and interchangeable, no primary involved, while the Longhorn controllers use leader election through the API server — the same mechanism the CNPG operator relies on — but they don't hold data of their own, so an isolated leader just loses its lease and a clean new one gets elected.

CNPG's 3-instance, operator-promoted pattern really is new to this cluster. Nothing else needs board-aware anti-affinity of its own.

Garage was worth a second look on the way past, though. It's already pinned to rock3 — bletchley1 — via a plain nodeSelector, set up long before this series existed, for reasons that line up with Part 1's reasoning exactly without ever using the word "board." The instinct wasn't new. It just hadn't been generalized or named yet.

One more thing turned up while checking pod placement: postgres-0, the current single-instance Postgres backing Umami, is running on rock5 — bletchley2, not bletchley1. Not replicated, so no divergence risk, but it's sitting on the board without the control plane. Small, accidental example of exactly what Part 1 was designed around.

Resource Headroom

The obvious way to estimate CNPG's footprint: take what Postgres costs today (postgres-0 requests 100m CPU / 256Mi, uses 10m / 50Mi in practice) and multiply by three. Two problems with that. postgres-0 only holds Umami's data — the eventual CNPG cluster consolidates Umami, lldap, and Authelia, so it undersells future volume. More fundamentally, three replicated instances carry fixed overhead regardless of data size: WAL streaming to two replicas, continuous archiving, a Barman Cloud Plugin sidecar per instance. That floor exists even against an empty database.

Used a conservative generic estimate instead: roughly 1 CPU / 1Gi request per instance, plus real headroom for the Barman sidecar. Real-world reports put that sidecar spiking to around 0.9 CPU and 350MiB depending on backup and WAL activity — not the negligible afterthought "sidecar" usually implies.

Checked against actual per-node numbers, both requested and in use:

                 CPU free (req)   Mem free (req)   Mem free (live)
rock1–4          ~6.3–6.9 cores   ~5.6–7.5Gi       ~4.1–5.9Gi
rock5            ~6.8 cores       ~16Gi            ~14Gi
rock6–7          ~6.7–6.9 cores   ~7.4–7.5Gi       ~5.4–5.9Gi

Comfortable either way. The live-usage column matters more than it looks, though: kubectl top nodes showed several nodes running well above their requests (rock2 requests 18% of its memory, uses 44%). Postgres tends to use whatever memory it's given for caching rather than sitting near its request, so live usage is the number to size against here, not the flattering request-based percentage.

One risk worth naming, not solving yet: pinning two of three CNPG instances to bletchley1 won't push existing pods off those nodes automatically — nothing in this stack rebalances running pods. But if memory gets genuinely tight, kubelet's own pressure-based eviction can kick in, and the live-usage numbers above make that a real possibility, not a hypothetical. Bletchley2 has spare capacity if that ever needs solving, rock6 and rock7 especially. Watching this after Part 3 and Part 4 put real load on the cluster, not pre-solving it on guesswork.

What's Working Now

• ✅ topology.bletchley/board applied to all seven nodes — rock1–4 as bletchley1, rock5–7 as bletchley2
• ✅ Applied live, --mode no-reboot, zero disruption to the control plane
• ✅ Workload audit complete — nothing else on the cluster carries CNPG's specific risk shape
• ✅ Resource headroom confirmed comfortable, including under live-usage accounting
• ⚠️ Known limitation: the "the cluster already balances itself sensibly across both boards" property is current state, not an enforced invariant — worth revisiting once CNPG is actually running and putting real load on rock4

What's Next

Part 3 installs the operator for real: the CNPG Cluster resource, the board-anti-affinity rules this label exists to support, and the dedicated MetalLB LoadBalancer Service Part 1 decided on. It's also where the assumptions from Parts 1 and 2 get tested against actual scheduling behavior instead of reasoned about in advance.

← Previous: CloudNativePG Part 1: Planning Highly Available PostgreSQL Across Two TuringPi Boards

Questions or suggestions? Leave a comment below or reach out at igor@vluwte.nl.