CloudNativePG Part 3: Installation and Exposing It with MetalLB

Introduction

Parts 1 and 2 of this series were entirely about getting ready: designing the topology across Bletchley's two TuringPi boards, working out how CloudNativePG's operator-driven failover interacts with a physical partition, and confirming the cluster actually had room — and no competing constraints — for what was about to land. None of that touched a running cluster. Part 3 is where the plan meets real hardware.

By the end of this post there's a working bletchley-pg cluster: three PostgreSQL 17 instances split across the two boards, a dedicated MetalLB LoadBalancer reachable from outside the cluster, and backups running automatically to Garage — all without a single application database created yet. That last part is deliberate, not an oversight, and I'll get to why.

Most of what follows isn't the manifests themselves — those were already sketched out in Parts 1 and 2 — it's what happened once I applied them. A backup mechanism that looked fine on paper but was already deprecated. And a CLI default that quietly assumed a backup path this cluster doesn't use.

🏠 This is part of the Homelab Journey series - building a production Kubernetes cluster from scratch.

• CloudNativePG Part 2
• CloudNativePG Part 3: Installation and Exposing It with MetalLB (you are here)

This post assumes you've already done CloudNativePG Part 1 (the topology and design decisions) and CloudNativePG Part 2 (board labels live, workload audit clean). This post assumes both are done and doesn't re-derive them.

Closing Out the Open Questions

A handful of decisions were deliberately left open across Parts 1 and 2 because they didn't affect the topology or failure-mode design — but they block writing an actual manifest. Quick recap before getting into the install itself:

DNS. I'd originally hoped to reuse postgres.vluwte.nl directly, but that name is live right now, routing to the old single-instance Postgres StatefulSet that still backs Umami. Cutting over immediately would take Umami's database offline before it's actually migrated. So CNPG gets its own interim name, pg.vluwte.nl, and postgres.vluwte.nl moves over in one clean cutover once the old StatefulSet is retired (Part 7).

PostgreSQL version. 17, matching Umami's current database exactly. PG 18 is the current stable release and would be the cheaper point to absorb a version bump, but that would bundle a major-version upgrade into Part 6's migration instead of keeping it isolated. One new variable per part is the whole point of this series' structure — the version bump becomes its own project, after this series wraps.

CNPG version vs. Kubernetes compatibility. This one needed real research, not just a preference. Bletchley runs Kubernetes v1.36.2 (confirmed in Part 2), and CNPG's own supported releases table only lists 1.30.x as officially supporting 1.36 — 1.29.x has it under "tested, not supported" only, and its support window ends sooner besides. So: CNPG 1.30.x.

Install method. Helm chart over the raw manifest. Automation is explicitly part of what this cluster is for, and Helm is the better-integrated path for whichever GitOps tool eventually gets adopted (Flux or ArgoCD, still undecided). The real catch — Helm doesn't refresh CRDs shipped in a chart's crds/ folder on helm upgrade — stays a manual step for now, deferred until the CD tool choice makes it worth automating properly.

With those settled, the actual sequence was: storage and backup target first, then the operator, then the cluster itself, then external access, then a failover test, then automating the backup schedule.

The Backup Mechanism Wasn't What I Thought

Part 1 called for "native Barman Cloud Plugin backups," but my first draft of the manifest used .spec.backup.barmanObjectStore — the in-tree field, configured directly on the cluster resource. That's the legacy mechanism. It's been deprecated since CNPG 1.26, and it's slated for removal in 1.30 — the exact version this cluster targets. Writing it into the manifest wouldn't have just been outdated style, it likely wouldn't have worked at all.

The current approach splits backup configuration into its own component entirely: the Barman Cloud Plugin, installed separately from the operator (same namespace, cnpg-system, but its own Deployment), using the CNPG-I plugin architecture. Instead of one inline block, there are two resources — an ObjectStore describing the backup target, and a .spec.plugins stanza on the cluster referencing it by name:

# apps/databases/objectstore-cnpg-backups.yaml
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: cnpg-backups-store
  namespace: databases
spec:
  retentionPolicy: "30d"
  configuration:
    destinationPath: "s3://cnpg-backups/"
    endpointURL: http://garage-s3.garage.svc.cluster.local:3900
    s3Credentials:
      accessKeyId:
        name: cnpg-backup-credentials
        key: ACCESS_KEY_ID
      secretAccessKey:
        name: cnpg-backup-credentials
        key: ACCESS_SECRET_KEY
    wal:
      compression: bzip2
  instanceSidecarConfiguration:
    env:
      - name: AWS_DEFAULT_REGION
        value: garage

Two things in there deserve their own explanation, because both cost me actual debugging time.

The Region Field Is Broken, Not Just Unreliable

My first version of this manifest had region: garage sitting under s3Credentials, as a plain string — mirroring how you'd set the region for a real AWS bucket. Applying it failed outright:

The ObjectStore "cnpg-backups-store" is invalid:
* spec.configuration.s3Credentials.region: Invalid value: "string": spec.configuration.s3Credentials.region in body must be of type object: "string"

There's an open GitHub issue (#9724) describing this field is not mapped to the environment variable by the plugin or operator. As a result, Barman (and the underlying Boto3 library) defaults to us-east-1 unless AWS_DEFAULT_REGION is explicitly set in instanceSidecarConfiguration.env. Since Garage isn't real AWS S3 and always needs an explicit region for request signing set to 'garage'.

Installing the Plugin Itself

The plugin isn't a config flag on the operator — it's installed separately, with cert-manager as a prerequisite (already running on this cluster from an earlier post):

kubectl apply -f \
  https://github.com/cloudnative-pg/plugin-barman-cloud/releases/download/v0.14.0/manifest.yaml
kubectl rollout status deployment -n cnpg-system barman-cloud

Worth checking the releases page before running that — I started on v0.13.0, and v0.14.0 had shipped three days before I actually got to this step. Upgrading was painless (a plain re-apply, no CRD migration needed outside the one breaking release back at 0.8.0), but it's a good habit to check first rather than after.

Deploying the Cluster

With storage, the operator, and the backup target all in place, the cluster resource itself:

# apps/databases/cluster.yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: bletchley-pg
  namespace: databases
spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:17

  inheritedMetadata:
    labels:
      backup.vluwte.nl/enabled: "false"

  storage:
    storageClass: longhorn-single-replica
    size: 20Gi

  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          preference:
            matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                values: ["rock4"]

  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.bletchley/board
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          cnpg.io/cluster: bletchley-pg

  plugins:
    - name: barman-cloud.cloudnative-pg.io
      isWALArchiver: true
      parameters:
        barmanObjectName: cnpg-backups-store

  managed:
    services:
      additional:
        - selectorType: rw
          serviceTemplate:
            metadata:
              name: bletchley-pg-external
              annotations:
                metallb.io/loadBalancerIPs: "10.0.140.101"
            spec:
              type: LoadBalancer

Two pieces here are worth calling out specifically, because both are easy to get subtly wrong.

The board-spread guarantee isn't affinity, it's topologySpreadConstraints. My first instinct was podAntiAffinityType: required with the board as the topology key — "no two instances on the same board." With 3 instances and only 2 boards, that's mathematically impossible to satisfy and would leave the cluster permanently unschedulable. topologySpreadConstraints with maxSkew: 1 is the actual right tool: the pod-count difference between the two boards can never exceed 1, so 2-1 is valid and 3-0 isn't, and whenUnsatisfiable: DoNotSchedule hard-rejects any placement that would violate it rather than silently landing somewhere the design didn't want.

The primary landing on rock4 is a preference, not a guarantee. nodeAffinity here is soft — preferredDuringSchedulingIgnoredDuringExecution — because CNPG applies affinity uniformly to all instances; there's no way to target just the primary. It worked on this first deploy, and namespace/topology decisions elsewhere in this file exist specifically because "primary stays on bletchley1" is an operational commitment I have to check and enforce manually after any failover, not something the cluster guarantees on its own. More on that below.

Applying it, and watching it actually come up healthy, took about four minutes:

igor@granite bletchley % kubectl get cluster bletchley-pg -n databases
NAME           AGE     INSTANCES   READY   STATUS                     PRIMARY
bletchley-pg   4m21s   3           3       Cluster in healthy state   bletchley-pg-1

igor@granite bletchley % kubectl -n databases get pods -o wide
NAME             READY   STATUS    RESTARTS   AGE     IP             NODE    NOMINATED NODE   READINESS GATES
bletchley-pg-1   2/2     Running   0          6m9s    10.244.3.197   rock4   <none>           <none>
bletchley-pg-2   2/2     Running   0          4m31s   10.244.4.125   rock5   <none>           <none>
bletchley-pg-3   2/2     Running   0          3m30s   10.244.3.199   rock4   <none>           <none>

Two on rock4, one on rock5 — the intended 2/1 board split, and the primary landed exactly where the soft preference asked it to.

One deliberate scope note: this is where Part 3 ends. Not a single application database gets created here — the cluster comes up healthy, empty, and externally reachable, and that's it. This series deliberately isolates one new risk per part: Part 4 is a throwaway database specifically so failover, partition, and backup/restore get chaos-tested with nothing real at stake; Part 5 is lldap, the first real workload but not a migration; Part 6 is Umami, only once the whole stack has been proven twice over. Bootstrapping any database here — even a disposable one — would jump ahead of that structure.

Two Gotchas That Nearly Cost Me a Working Backup

Everything above got the cluster running and WAL archiving working. But kubectl cnpg status kept showing this:

Continuous Backup status (Barman Cloud Plugin)
No recovery window information found in ObjectStore 'cnpg-backups-store' for server 'bletchley-pg'
Working WAL archiving:        OK

Which makes sense in hindsight — WAL archiving alone doesn't make a cluster restorable. A recovery window needs an actual base backup as its anchor, and nothing in the plan up to this point had ever triggered one.

kubectl cnpg backup Has a Trap Built Into Its Defaults

Triggering one seemed straightforward:

igor@granite bletchley % kubectl cnpg backup bletchley-pg -n databases
backup/bletchley-pg-20260802191133 created

The command appeared to succeed, but the backup actually failed:

NAME                          AGE   CLUSTER        METHOD              PHASE       ERROR
bletchley-pg-20260802191133   21m   bletchley-pg   barmanObjectStore   failed      cannot proceed with the backup as the cluster has no backup section

The CLI's --method flag defaults to barmanObjectStore — the exact legacy mechanism this whole post was about not using. Since this cluster never configures it, every unflagged kubectl cnpg backup invocation fails the same way. The fix is just being explicit every time:

kubectl cnpg backup -n databases bletchley-pg \
  --method=plugin --plugin-name=barman-cloud.cloudnative-pg.io

A CRD Name Collision Hid a Working Backup

With the right flags, the backup actually completed — but I didn't know that at first, because checking it looked like this:

igor@granite bletchley % kubectl get backup -n databases
No resources found in databases namespace.

That's not an error, which is what made it convincing. It turns out at least one other CRD on this cluster — almost certainly Longhorn's own backups.longhorn.io, since Longhorn is the storage provisioner here — also registers a backups plural. kubectl get backup silently resolved to the wrong one and returned a perfectly valid, perfectly empty answer instead of erroring on the ambiguity. The fix is the fully-qualified resource name:

igor@granite bletchley % kubectl get backups.postgresql.cnpg.io -n databases
NAME                          AGE     CLUSTER        METHOD    PHASE       ERROR
bletchley-pg-20260802192513   7m42s   bletchley-pg   plugin    completed

There it was the whole time. kubectl cnpg status confirmed it properly a moment later:

First Point of Recoverability:  2026-08-02 19:25:17 CEST
Last Successful Backup:         2026-08-02 19:25:17 CEST

Good to know for anything else CNPG-related on this cluster: assume short resource names are contested here, and reach for backups.postgresql.cnpg.io (and probably clusters.postgresql.cnpg.io) explicitly.

Automating Backups

A single manual backup proves the pipeline works, but nothing was actually keeping backups happening going forward. The declarative, cron-driven equivalent is ScheduledBackup — same method/pluginConfiguration shape as the manual backup, plus a schedule. One catch: CNPG's cron format has a leading seconds field, six parts instead of the usual five.

# apps/databases/scheduledbackup-cnpg-backups.yaml
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: bletchley-pg-daily
  namespace: databases
spec:
  schedule: "0 15 5 * * *"       # daily at 05:15
  backupOwnerReference: cluster
  cluster:
    name: bletchley-pg
  method: plugin
  pluginConfiguration:
    name: barman-cloud.cloudnative-pg.io

The next morning, it fired entirely on its own:

NAME                                AGE   CLUSTER        METHOD    PHASE       ERROR
bletchley-pg-daily-20260803051500   12h   bletchley-pg   plugin    completed

The object's own labels (cnpg.io/scheduled-backup: bletchley-pg-daily, cnpg.io/immediateBackup: "false") confirm it was a genuine cron-triggered run, not the "backup immediately on creation" behavior. It also picked up the correct WAL timeline automatically — timeline 3, the one the failover testing below had already left the cluster on — which is a small but reassuring sign the scheduler tracks live cluster state rather than anything stale.

Testing Failover

The actual point of this whole design — verifying the external LoadBalancer Service tracks the primary through a real failover, not just in theory.

igor@granite bletchley % kubectl cnpg promote bletchley-pg bletchley-pg-2 -n databases
Node bletchley-pg-2 in cluster bletchley-pg will be promoted

bletchley-pg-2 runs on rock5 — bletchley2. The old primary (bletchley-pg-1, rock4) visibly restarted during demotion, dropping from 2/2 Running to Completed and back to Running with one restart recorded — expected behavior (the postgres process restarts into standby/recovery mode to follow the new timeline), not a crash, though it's a little alarming to watch live if you don't already know that's coming.

Within about 45 seconds:

Primary instance:        bletchley-pg-2
Status:                  Cluster in healthy state

And the external Service followed automatically, no manual intervention:

igor@granite bletchley % kubectl get svc bletchley-pg-external -n databases -o wide
NAME                    TYPE           CLUSTER-IP     EXTERNAL-IP    SELECTOR
bletchley-pg-external   LoadBalancer   10.108.183.2   10.0.140.101   cnpg.io/cluster=bletchley-pg,cnpg.io/instanceRole=primary

That selector is the entire mechanism — a plain label match CNPG keeps pointed at whichever pod currently holds the primary role. Nothing failover-plugin-specific about it, which is reassuring; it means the behavior isn't tied to some fragile custom logic.

Since the new primary landed on bletchley2, per the Part 1 operational commitment I promoted it back manually:

igor@granite bletchley % kubectl cnpg promote bletchley-pg bletchley-pg-1 -n databases
Node bletchley-pg-1 in cluster bletchley-pg will be promoted

Which also worked cleanly, landing the primary back on rock4. One thing I hadn't planned for: each promotion produced exactly one failed WAL-archive entry for that timeline's .history file (00000002.history, then 00000003.history), self-resolving each time with no backlog. It showed up on both failovers, not just once, which makes it look like a genuine, repeatable characteristic of how the plugin handles a timeline switch — not a fluke worth chasing down, but worth knowing about if you ever see it.

Lessons Learned

1. Read the CRD source, not just the issue tracker, when a field looks broken. The s3Credentials.region bug report made it sound like a propagation problem. It's actually a type mismatch — the field is a Secret reference, not a string — which changes how you should think about "fixing" it (don't; use the sidecar env var instead).
2. CLI defaults can silently assume the wrong thing. kubectl cnpg backup's default --method pointed at a mechanism this cluster deliberately doesn't use, and failed with a clear error — but only because I happened to check. Always pass --method/--plugin-name explicitly on this cluster from now on.
3. A clean "No resources found" isn't proof of absence. Short resource names can resolve ambiguously when multiple CRDs share a plural. The fully-qualified name is the only way to be sure you're looking at the right thing.
4. topologySpreadConstraints and pod anti-affinity solve different problems. A "required" anti-affinity that sounds like the right board-spread guarantee can actually make a Cluster unschedulable. Skew-based spread constraints are the correct primitive when the pod count doesn't evenly divide across domains.

What's Working Now

• ✅ CNPG operator 1.30.0 (Helm) and Barman Cloud Plugin 0.14.0 running in cnpg-system
• ✅ bletchley-pg Cluster healthy — 3 instances, PostgreSQL 17, confirmed 2/1 board spread
• ✅ Backups running end-to-end: Garage-backed ObjectStore, bzip2 WAL compression, 30d retention, daily ScheduledBackup confirmed firing on its own
• ✅ External access live at pg.vluwte.nl / 10.0.140.101, postgres.vluwte.nl untouched
• ✅ Failover tested in both directions, external Service tracked the primary automatically each time
• ✅ Everything committed to git, including README documentation for both new namespaces
• ⬜ No application database yet — deliberately, per this series' own risk-isolation design
• ⬜ Operator and plugin installation itself still isn't GitOps-tracked — deferred until a CD tool is chosen

Part 3 ended up validating more than just the manifests. It confirmed that the architectural decisions from Parts 1 and 2 actually survive contact with reality: workloads land where expected, failover updates external access automatically, and backups are happening without manual intervention. The remaining parts can now focus on databases themselves rather than the platform underneath them.

What's Next

Part 4 puts a throwaway database on this Cluster and tries to break it on purpose: chaos-testing failover under real load, a board-level partition test, and — for the first time — an actual restore from these backups rather than just confirming they complete.

← Previous: CloudNativePG Part 2

Questions or suggestions? Leave a comment below or reach out at igor@vluwte.nl.