CloudNativePG Part 1: Planning Highly Available PostgreSQL Across Two TuringPi Boards

Introduction

Postgres on Bletchley currently runs as a single, non-HA StatefulSet backing Umami — the first database I moved onto the cluster — with a custom pg_dump pipeline shipping backups to Garage S3, built and later patched over two other posts. It works, but it's a single point of failure, and I already know it needs to become something better: lldap and Authelia are next in line for the auth-replacement project, and both need a Postgres backend that can survive a node dying without taking authentication down with it.

The operator choice for that replacement — CloudNativePG (CNPG) — got decided in an earlier planning session, back when Bletchley was still one physical board. That session settled which operator and why; it never got as far as where the instances actually live. Since then, a second TuringPi 2 board joined the cluster, turning Bletchley from four nodes on one board into seven nodes spread across two. That changes the topology question in a way that's worth working through properly before deploying anything.

This post is that working-through. No manifests yet, no operator installed — just the decisions and the failure-mode analysis that get made before touching either, written up because the reasoning turned out to be more interesting than the final answer.

This post is part of the CloudNativePG sub-series.

• Part 1: Planning Across Two Boards

🏠 This is part of the Homelab Journey series - building a production Kubernetes cluster from scratch.

• Silent Failures
• CloudNativePG Part 1: Planning Highly Available PostgreSQL Across Two TuringPi Boards (you are here)

The Board Situation

This single fact drives almost every decision below, so it's worth stating plainly rather than assuming it's obvious.

Bletchley is built from two physical TuringPi 2 carrier boards, each holding up to four RK1 compute modules. They're two separate pieces of hardware, connected to each other only through the network switch — no shared power, no shared backplane, nothing beyond an ordinary network link between them.

• bletchley1 — the original board, four nodes: rock1, rock2, rock3, rock4. rock1–3 run the Kubernetes control plane (API server, etcd, scheduler). rock4 is a worker.
• bletchley2 — added in the last post, three nodes: rock5, rock6, rock7, all workers. rock5 has 16GB of RAM; the rest of the fleet has 8GB.

The detail that matters most: the Kubernetes control plane exists only on bletchley1. There's no API server, no etcd member, nothing control-plane-related running on bletchley2 at all. Every decision below about where the CNPG primary lives comes back to this one fact — whichever board hosts the control plane is the board the CNPG operator can always see and act on, no matter what happens to the network path to the other board.

The operator choice itself I'm not re-litigating here — CNPG over Zalando's operator, Crunchy PGO, StackGres, or hand-rolled Patroni was already decided, mainly on the strength of active development and backup support that fits Garage directly. What's new is the two-board topology question that decision left open.

The Question That Actually Mattered

Going in, I assumed the interesting question would be about quorum: three instances, tolerate one loss, standard stuff. It turned out the actual hinge of the whole design was something else — where does the primary live, given that the control plane only exists on one of the two boards? I got there by asking a handful of questions in order, and one of them was a wrong turn worth keeping in, because it's the kind of wrong turn that only shows up once you actually interrogate an assumption instead of running with it.

"Can I just pin the primary to rock5 — it's got the extra RAM?"

My first instinct was straightforward: rock5 has 16GB against 8GB everywhere else, so put the primary there and let the replicas run on the smaller nodes. Reasonable instinct, wrong mental model. CNPG has no durable concept of pinning "the primary" to a specific node, because the primary isn't a pod with a fixed home — it's a role that moves. Any of the three instances can hold it, and which one does changes over the cluster's lifetime as failovers happen. You can bias which pod starts as instance-1, but you can't nail the primary role to a piece of hardware and expect it to stay there.

That distinction is what makes the RAM idea worth mentioning at all — it's exactly the kind of decision that looks sound when you're thinking about steady-state performance and looks wrong the moment you start thinking about what happens after a failover. I'm keeping it in because it's where the RAM-driven placement idea started, and it's the same idea I end up reversing a few questions later for a completely different reason.

"Will MetalLB actually know which instance is primary?"

A reasonable thing to check before relying on it, and no — MetalLB doesn't know anything about Postgres roles. It's a dumb IP-advertisement layer: it announces a LoadBalancer IP on the network and forwards packets to whatever Kubernetes Service currently backs it. The actual failover-aware part is Kubernetes' own Service/Endpoints mechanism reacting to labels CNPG maintains on the pods (role=primary, role=replica). When CNPG promotes a new primary, it relabels the pod, Kubernetes updates the Endpoints for the -rw Service, and MetalLB just keeps forwarding to whatever that Service currently points at. MetalLB is the delivery truck, not the dispatcher.

"For backups — does it hit a standby, or the live primary?"

CNPG's default backup target is prefer-standby, meaning it prefers to run Barman Cloud backups against a replica rather than the primary. The reasoning is obvious once stated: backup I/O — reading through the whole dataset to ship it to S3 — competes for disk and CPU with whatever's serving live traffic. Push it onto a replica and the instance actually taking writes doesn't have to share resources with its own backup job. With three instances in play, there's always a replica available to take that job, so this one was a quick confirmation rather than a real decision point.

"I assumed 3 instances was about quorum — what about a network partition between the two boards?"

This is the question the whole post turns on, and it's the one that unwound the RAM-based placement decision.

The instinct behind "three instances gives you quorum" comes from etcd, Raft, and similar systems, where the nodes themselves vote and a majority is required to agree on truth. CNPG doesn't work that way. The three Postgres instances don't vote among themselves at all — the CNPG operator, running as a controller against the Kubernetes API, unilaterally decides when and what to promote, based on what it can observe. "Quorum" isn't really the operating model here; "operator-observed state" is.

That distinction matters enormously given the board situation. The API server and etcd live only on bletchley1. If bletchley2 loses network contact with bletchley1 — a switch misbehaving, a cable fault, anything that severs the link between the two boards without actually killing either one — then any CNPG instance running on bletchley2 also loses its connection to the API server the operator uses to observe and manage it.

Here's where the RAM-driven placement idea from earlier actually breaks: if the primary had been placed on rock5 (bletchley2) and that board loses contact with bletchley1, the operator can neither see nor manage that primary anymore. Meanwhile the primary itself is not aware anything is wrong — from its own point of view, it's a healthy Postgres instance, still accepting connections and writes on whatever network path is still reachable to it. Any client with an already-open connection — a pooled connection, a live session — can keep writing to it as long as that specific path holds, because an open connection doesn't need to re-resolve the Service to know where to send its next query.

That's not classic two-master split-brain, where the entire client population actively splits across two writable instances — new connection attempts would still resolve through the Service to whichever instance currently carries the primary label. But it is a real risk: an isolated primary can keep accepting writes that later diverge from whatever gets promoted in its place, and those writes need reconciling once the partition heals. This lines up with how CNPG's own maintainers describe the trade-off — they deliberately don't auto-fence an isolated primary (CNPG's fencing feature is a distinct, manually-triggered mechanism, not an automatic partition response), and instead expect pg_rewind to reconcile the old primary against the new one once connectivity returns, discarding whatever it accepted while cut off.

Flip the placement, though, and the same partition becomes a non-event. With the primary on bletchley1 — same board as the control plane — a board-to-board network partition just means the two replicas on bletchley2 fall behind and catch up once the link is restored. The operator never loses sight of the primary, so it never has a reason to attempt a promotion at all. No stalled writes, no isolated-primary risk in this direction — just ordinary replication lag. This is the reversal: rock5's extra RAM stopped mattering the moment "can the operator see and manage this instance during a partition" became the actual constraint, and that constraint points squarely at bletchley1.

Worth being precise about the limits of this: placing the primary on bletchley1 reduces this risk, it doesn't eliminate every path to it. If an earlier, unrelated failover has already promoted a bletchley2 replica to primary before a partition occurs, the "primary lives on bletchley1" assumption no longer holds until someone notices and switches it back manually.

"So the primary's on rock4 — does that mean it just stops taking writes if it can't see the other two?"

Close, but not quite — and the gap between the guess and the answer is exactly where sync vs. async replication gets pinned down. My assumption was that Postgres would behave the way I'd half-remembered synchronous systems behaving: no acknowledgment from a standby, no accepted write. That's true — but only under synchronous replication, and CNPG's default is asynchronous.

Under async replication (the default, and what I'm choosing here), the primary keeps accepting writes with no wait on any standby acknowledgment at all. If rock4 loses its network path to the bletchley2 replicas, it doesn't stall — it just keeps running, and the replicas fall behind, catching up once the path returns. Under synchronous replication, the primary would instead refuse to acknowledge writes it couldn't get confirmed on at least one standby — trading availability for the guarantee that anything acknowledged is safely replicated somewhere else. There's also failoverQuorum (CNPG 1.28+), a related but distinct safeguard that only matters on top of synchronous replication: it stops the operator from promoting a candidate unless it can prove that candidate holds every synchronously-committed write. Without synchronous replication in play, failoverQuorum has nothing to check.

I'm going with async. Synchronous + failoverQuorum together would give zero data loss on failover, but at the cost of writes stalling any time the standbys are unreachable — and the specific scenario that combination protects against (an isolated primary diverging from what gets promoted) is already made far less likely by putting the primary on bletchley1 in the first place. What's left over — rock4 itself failing outright, not just losing network to the other board — is a rarer failure mode than a network hiccup between two boards connected only by a switch, so I'm accepting a small, rare data-loss window in that specific case rather than trading away write latency for a benefit that mostly duplicates what placement already buys.

"Wait — is what I just described actually split-brain?"

A late self-check, and worth including honestly rather than editing it out of the final version. My first draft of the paragraph above reached for "split-brain," because it's the familiar, dramatic term for two things disagreeing about who's in charge. On rereading it, that's not quite what's happening here. Split-brain implies two live primaries that clients are actively divided across. What CNPG's failure mode actually describes is narrower and more specific: an isolated primary may continue accepting writes that later need reconciliation once the partition heals and pg_rewind runs. New connections still resolve to whichever instance holds the primary label — there's no client population split evenly across two "true" primaries.

The more precise phrasing is also the more defensible one if anyone who actually knows CNPG internals reads this closely. It's a small correction, but it's the same kind of check worth running on your own explanation before publishing it that you'd run on a config before applying it.

The Decision

Working through those six questions in order landed on the topology below — which is really the point of walking through them: by the time you get here, you've reconstructed the reasoning yourself rather than taking it on faith.

Topology: 3 instances — 2 on bletchley1, 1 on bletchley2 — enforced with pod anti-affinity keyed on a new topology.bletchley/board label (bletchley1 / bletchley2). I'm deliberately not calling this label zone. topology.kubernetes.io/zone implies a level of independent failure-domain infrastructure — separate power, separate uplink, separate physical location — that two TuringPi boards sharing a switch and a power circuit in the same rack simply don't have. board describes exactly what it is and nothing more.

I considered two alternatives: four instances split 2-and-2 for true symmetric redundancy (rejected — more resource footprint than CNPG's HA minimum actually requires), and three instances with plain node anti-affinity and no board awareness at all (rejected — it wouldn't reliably guarantee the board spread that's the entire point here).

Primary placement: instance-1 soft-preferred onto rock4, not rock5. Bigger RAM lost out to "the operator can only manage what it can reach."

Replication: async, CNPG's default. Sync + failoverQuorum deferred — see the reasoning above. Installing CNPG ≥1.28 regardless, so switching to sync + failoverQuorum later stays a config change rather than an upgrade project.

External access: a dedicated MetalLB LoadBalancer Service, via CNPG's managed.services.additional, selector-matched to the primary role rather than routed through Traefik. Postgres traffic doesn't gain anything from going through Traefik — it was only ever a candidate because Traefik happens to be the existing entrypoint pattern for everything else. A CNPG Pooler (PgBouncer) in front is deferred until actual external connection volume justifies the extra layer.

Backups: the native Barman Cloud Plugin, targeting a new, dedicated Garage bucket — not the existing postgres-backups bucket the old pg_dump pipeline uses. Keeping them separate means the old and new backup pipelines can run side by side during the migration instead of racing each other into the same bucket.

Storage: Longhorn PVCs, but on a dedicated single-replica StorageClass rather than the cluster's default of 2 replicas. This one's worth explaining rather than just stating: Longhorn's replication protects against storage-layer failures, while CNPG already protects against database-instance failures — and with three Postgres replicas, streaming replication, continuous WAL archiving, and Barman backups all doing that job, doubling storage on top is redundancy the design doesn't need. Dropping it buys back lower write latency (no second or third network hop per write), less network traffic, and less disk consumption on top of the ×3 footprint CNPG's instances already carry.

Risks and Open Items Going In

A few things I'm carrying into the next posts, flagged honestly rather than glossed over:

• Board labels don't exist yet. First implementation step, low risk but blocking — nothing else here works without it.
• Resource headroom for 3 CNPG instances isn't verified yet against everything else already scheduled across the seven nodes.
• managed.services.additional is a less-travelled path than CNPG's built-in -rw/-ro/-r Services. It needs hands-on verification that failover actually updates it correctly — I'm not assuming it just works.
• The "primary stays on bletchley1" property is a starting condition, not an enforced invariant. Nothing automatically snaps the primary back to bletchley1 after an unrelated failover promotes a bletchley2 replica. Operational commitment going forward: after any failover, check where the primary landed, and if it's not on bletchley1, manually promote it back (kubectl cnpg promote) once the promoted replica has caught up. Manual for now — worth revisiting whether it should be alerted on once the cluster is actually live.
• Migrating Umami off the working non-HA instance carries the usual dump/restore risk, to be handled the same way the original Docker-to-Bletchley migration was: row-count verification before cutting over.
• Accepting async replication means accepting a small, rare data-loss window if rock4 fails outright — not partitions, an actual node failure. Consciously accepted, documented here so it isn't forgotten later.

What's Next

Decisions are made; nothing's deployed yet. The rest of this series follows the shape the risks above suggest — each step isolating one new risk instead of stacking several at once. Parts 2 and 3 are fairly firm; Parts 4 through 7 depend on those two actually holding up on real hardware, so treat this as a plan rather than a promise:

• Part 2 — labeling rock1–4 as bletchley1 and rock5–7 as bletchley2, and a pass over existing workloads to check whether any of them want board-awareness too.
• Part 3 — installing the CNPG operator, deploying the Cluster resource, and wiring up the dedicated MetalLB LoadBalancer Service. This is also where Parts 1–2's assumptions get checked against what actually happens on real hardware.
• Part 4 — a throwaway database, killed on purpose: primary pod termination, a deliberate board-partition test, and a backup/restore round-trip through the Barman Cloud Plugin, all with zero blast radius on anything real.
• Part 5 — lldap's database, bootstrapped fresh rather than migrated, unblocking the parked auth-replacement work a step early.
• Part 6 — the actual Umami migration, only attempted once the pattern's proven twice over on lower-stakes data.
• Part 7 — decommissioning the old StatefulSet and its custom backup pipeline, after a soak period.

← Previous: Silent Failures

Questions or suggestions? Leave a comment below or reach out at igor@vluwte.nl.