CloudNativePG Part 10: The Certificate That Rejects Its Own Fix

CloudNativePG's webhook certificate renewed on schedule. Then I broke it on purpose, and the only write refused all evening was refused by the fix.

Share

Introduction

When CloudNativePG builds a cluster, it also builds a small private PKI for itself: a CA for the operator, a CA for the cluster, and the certificates those two sign. On Bletchley that happened on 2 August, which meant five certificates with the same 90-day clock, all expiring on 31 October. CloudNativePG is documented to renew them on its own. This cluster had never lived through that.

One of the five is different from the others. It belongs to the operator's admission webhook, and if it goes wrong, the Kubernetes API server refuses every write to every CloudNativePG resource. That includes the kubectl apply you would reach for to fix it. That is where the title comes from.

In September I wrote down what I expected to happen, including one prediction that could be wrong. On 24 October the renewal came. A week later I broke the webhook on purpose, twice, to see what the failure actually looks like and how it recovers. The only write refused that evening wasn't caused by either of the failures I deliberately introduced. It was caused by the recovery.


This post is part of the CloudNativePG sub-series.

🏠 This is part of the Homelab Journey series - building a production Kubernetes cluster from scratch.


The One With Teeth

An admission webhook is a check the API server runs before it stores an object. The CloudNativePG operator runs one: whenever anyone creates or updates a Cluster, Backup, ScheduledBackup, Database or Pooler, the API server sends the object to the operator over HTTPS and waits for an answer. The mutating half fills in defaults; the validating half rejects specs that are invalid or unsafe. There are nine entries in total, spread over two configurations:

Configuration Entries
cnpg-mutating-webhook-configuration mbackup, mcluster, mdatabase, mscheduledbackup
cnpg-validating-webhook-configuration vbackup, vcluster, vscheduledbackup, vdatabase, vpooler

All nine have failurePolicy: Fail. If the API server cannot complete the TLS handshake with the webhook, it does not let the write through unchecked; it refuses it.

That sounds worse than it is, and it is worth being exact about. Nothing that is running stops. PostgreSQL keeps serving, replication keeps going, existing connections are untouched, and the operator can still write status (status updates are not covered by the webhook). What stops is the creation or modification of the resource types covered by these webhook configurations, by anyone. That includes the operator itself: every night at 05:15 it creates a Backup object for the scheduled backup, and that write goes through the same webhook.

There is one more detail that makes this certificate unusual, and it turned out to matter. The caBundle in each entry, the field that tells the API server which certificates to trust, does not contain a CA certificate. It contains the webhook's serving certificate itself.

That means the API server is not trusting a certificate authority that can vouch for successive serving certificates. It is trusting one specific certificate. When that certificate changes, the bundle and the certificate being served must change together. The Secret, the nine bundle entries and the certificate the operator actually serves must all agree.

Why that is unusual, and what it means for monitoring, is the subject of a companion deep dive, out on Saturday.

The loop in the title is real, but smaller than it looks. The webhook guards the CloudNativePG API group and nothing else. The webhook configurations, the operator's Deployment and its Secret all live outside it, so every way out stays open:

  1. Delete or patch the webhook configurations (both of them).
  2. Set failurePolicy: Ignore on the affected entries.
  3. Delete the webhook certificate's Secret and restart the operator, which generates a new one.

The diagram puts the two side by side. Read it from the top left: a write goes to the API server, which asks the webhook; if the certificate check fails, the write is refused. The red dashed arrow is the problem in the title: the fix is a write too, so it goes round the same loop. Everything on the right sits outside that loop, so none of it can be refused. The warning on the third comes back later.

Two panels. On the left, inside the CloudNativePG API group: a kubectl apply of a Cluster, Backup, ScheduledBackup, Database or Pooler goes to the API server, which calls the CloudNativePG webhook (nine entries) over HTTPS. The certificate check fails because the trusted certificate and the served one differ, and the write is refused (failurePolicy Fail). A dashed red arrow from the refusal back to kubectl apply reads: the fix is a write too. On the right, outside the loop and not guarded by the webhook, three escape hatches: patch or delete the webhook configurations; set failurePolicy to Ignore on the affected entries; delete the Secret and restart the operator, which has a red border and a warning sign. Below them: each of these works, because the webhook cannot block it. A line at the bottom reads: unaffected throughout, running PostgreSQL, replication, client connections and status updates.
The loop is real, but the way out never goes through it. The marked escape hatch comes back later.

Five Certificates and a Prediction

These are the five, all with the same 31 October expiry:

Secret What it is Born on 2 August
cnpg-system/cnpg-ca-secret The operator's CA ~13:40
cnpg-system/cnpg-webhook-cert The webhook's serving certificate ~13:40
databases/bletchley-pg-ca The cluster's client CA 15:32
databases/bletchley-pg-replication streaming_replica's client certificate 15:32
databases/bletchley-pg-server The cluster's original server certificate 15:32

One day, two times: I installed the operator first and created the cluster almost two hours later, and the certificates kept that gap.

The fifth one was the prediction. In Part 8 I replaced the operator's server certificate with a Let's Encrypt certificate for postgres.vluwte.nl, by setting serverTLSSecret and serverCASecret on the Cluster. The documentation says that in that case the operator "still creates and manages the two secrets related to client certificates". Nothing about the server side. So in September I wrote it down: bletchley-pg-server will not be renewed. If that was right, it would quietly walk towards expiry while everything else renewed around it.


24 October

The renewal came on time, a little after the seven-day mark: the operator's pair at 13:53 UTC, the cluster's pair at 16:00. Everything I had planned to check came back clean:

  • Nothing restarted. All three instances reloaded their certificates and PostgreSQL received a SIGHUP, within the same second. The postmasters' start times were unchanged.
  • The webhook certificate changed in all ten places at once: the Secret and all nine bundle entries carried the new serial.
  • No alert fired. With renewal at seven days and the expiry alert at three, silence was the right answer.
  • The cluster CA kept its key. The old server certificate still verified against the renewed CA, so existing client certificates survive a CA renewal.

And the prediction held. bletchley-pg-server was untouched: same serial, still expiring on 31 October. The operator had not just stopped renewing it, it no longer listed it at all among the certificates in the cluster's status:

kubectl -n databases get cluster bletchley-pg -o jsonpath='{.status.certificates.expirations}' | jq
{
  "bletchley-pg-ca": "2027-01-22 15:55:58 +0000 UTC",
  "bletchley-pg-replication": "2027-01-22 15:55:58 +0000 UTC",
  "postgres-vluwte-ca": "2045-09-02 23:59:59 +0000 UTC",
  "postgres-vluwte-tls": "2026-12-06 16:52:44 +0000 UTC"
}

All three instances were serving the Let's Encrypt certificate, so I deleted the Secret on 27 October, a day before the expiry alert would have fired. The operator did not recreate it, and clients connecting with sslmode=verify-full noticed nothing. A certificate nobody served had been a certificate nobody would miss, except the alert.

How the renewal looked from inside, and why the timestamps nearly fooled me, is in the deep dive. For this post the point is simpler: the mechanism worked, and the only certificate that needed attention was the one the mechanism had stopped caring about.


Breaking It on Purpose

A clean renewal proves the happy path. It does not show what the failure looks like, whether the way out really works, or whether the new alerts see anything. So on 30 October I broke the webhook, deliberately, on a healthy cluster.

The obvious plan was to delete the webhook certificate's Secret. The checks I ran first said that would most likely show nothing: the Secret is mounted into the operator as an optional volume, so deleting it empties the files, but the operator keeps the certificate it already loaded in memory, and that certificate still matches the bundle. Writes would keep working. So I split the test into two parts.

Part A: a wrong certificate in one entry

The cleanest way to produce a rejection is to make the bundle disagree with what is served. I picked vpooler, the entry for Pooler objects: it has no mutating partner, and I have no Poolers. Breaking it is a fire drill in an empty room.

First, a probe that reaches the webhook without changing anything:

kubectl apply --dry-run=server -f pooler-probe.yaml
# pooler.postgresql.cnpg.io/probe-dryrun created (server dry run)

Then I put a real but wrong certificate into vpooler's bundle, the cluster's own CA, and ran the probe again:

Error from server (InternalError): Internal error occurred: failed calling webhook "vpooler.cnpg.io": failed to call webhook: Post "https://cnpg-webhook-service.cnpg-system.svc:443/validate-postgresql-cnpg-io-v1-pooler?timeout=10s": tls: failed to verify certificate: x509: certificate signed by unknown authority

This is the line a reader will paste into a search engine, so it is worth reading slowly. InternalError is the API server's label, and it reads like a bug; it is not. The quoted name tells you which entry failed. The end of the line says why: the bundle does not match the certificate the webhook serves. An expired certificate would end in certificate has expired instead.

The kubectl patch that broke the bundle had gone through without complaint while Pooler writes were being refused, which is the escape hatch in action: the webhook configuration is outside the loop.

Then I waited, polling the bundle every five seconds. The operator put the right certificate back by itself, 5 minutes and 3 seconds after the break, without a restart. The new mismatch alert went pending 40 seconds after the break and cleared when the operator fixed it. It never fired, and that is on purpose: its for: is 15 minutes, three of the operator's repair cycles. A break that heals itself should not wake anyone up.

Part B: delete the Secret

Then the original plan: delete cnpg-webhook-cert and see what happens.

For 18 minutes, nothing was refused. A dry-run change to the Cluster, every ten seconds, went through every time. Within about a minute the kubelet had emptied the mounted volume, and the operator's certificate reads started failing, six per minute. The new read-errors alert fired almost immediately, and the email arrived at 19:29 UTC:

/image
file: screenshot-cnpg-part10-certwatcher-errors-email.png
alt: Alertmanager email with the subject FIRING 1 CNPGOperatorCertWatcherErrors cnpg-operator. The labels are alertname CNPGOperatorCertWatcherErrors, instance cnpg-operator, job cnpg-operator, namespace cnpg-system, node rock7, pod cnpg-cloudnative-pg-59659f766d-plqpl and severity warning. The description reads: 6 failed certificate reads in 15 minutes. The operator may be serving a stale certificate.
caption: The warning arrived about a minute and a half after the Secret was deleted. Nothing had been refused, and nothing would be.

What did not happen: the operator never recreated the Secret. It repairs a wrong bundle within five minutes, but a missing certificate is only noticed at startup. After 18 minutes I stopped waiting and used the documented way out.


The Fix Is the Outage

kubectl -n cnpg-system rollout restart deploy/cnpg-cloudnative-pg

The loop that had been probing every ten seconds printed this:

19:45:48 secret:MISSING bundle:7035E4F41E34A35D6B826BDA62F22416 probe:ok
19:45:58 secret:5DA4F5B9C37F7DFB2DA03560B952F242 bundle:5DA4F5B9C37F7DFB2DA03560B952F242 probe:REFUSED: Error from server (InternalError): Internal error occurred: failed calling webhook "mcluster.cnpg.io": failed to call webhook: Post "https://cnpg-webhook-service.cnpg-system.svc:443/mutate-postgresql-cnpg-io-v1-cluster?timeout=10s": tls: failed to verify certificate: x509: certificate signed by unknown authority
19:46:08 secret:5DA4F5B9C37F7DFB2DA03560B952F242 bundle:5DA4F5B9C37F7DFB2DA03560B952F242 probe:ok

The restart was at 19:45:47. Eleven seconds later, the new operator pod had created a new certificate and written it into all nine bundle entries. And a real Cluster write was refused: the only refusal of the whole evening, through mcluster, with the same "unknown authority" as Part A.

The operator's log explains the gap. The new pod loaded its configuration at 19:45:52, but its webhook server only started serving at 19:46:00.9. In between, the bundle already pinned the new certificate, and the only pod that could answer was the old one, still shutting down, serving the old certificate. The API server compared the two and refused. Ten seconds later the new pod was serving, and the probe went through again. Recovery took 21 seconds from the restart.

So the failure I spent two parts trying to cause was caused by the step that fixes it. Deleting the Secret refused nothing; restarting the operator, the documented remedy, refused a write. The reason is the pinned leaf from the first section. If the bundle held a CA, the old and the new certificate would both have been accepted during the handover, and there would have been no window at all. Because the bundle holds one exact certificate, every change of that certificate is a switch with two sides, and for a few seconds they disagree.

Twenty seconds is not much, and it only matters if a write happens to land in it. But it changes the advice. A restart is not free, and it should not happen at 05:15, when the operator creates its own nightly backup. None of the three new alerts saw this moment, either; why not is in the deep dive.


What Is Written Down Now

The recovery is now written down in the README I keep for the databases, based on what happened rather than on what I expected. Its core is a decision table:

Situation What to do Tested
A bundle entry disagrees with the Secret, the Secret is valid Wait 5 to 10 minutes; the operator repairs it Yes, Part A
The Secret is missing Restart the operator; don't wait. Expect up to ~20 seconds of refused writes. Not near 05:15 UTC Yes, Part B
The certificate has expired Delete the Secret, then restart (turning it into the case above) No
The operator cannot start, and a write must happen now failurePolicy: Ignore on the entries, as briefly as possible, then revert No

The last two rows say "No" because I have not done them. They follow from the tested rows, but a runbook should say which of its steps have actually been run.

One more consequence of the test: Part B gave the webhook a new certificate, so it no longer renews together with the operator's CA. In January the CA renews around the 15th and the webhook certificate around the 21st. That turns out to be useful: the first time the CA renews on its own, I can check whether it keeps its key, the same way the cluster CA did in October.


Lessons Learned

  1. Test the remedy, not just the failure. The failure behaved as predicted. The fix was the only part that surprised me.
  2. A restart is not free when a bundle pins a leaf. Every change of that certificate has a moment where the two sides disagree. Know when your own writes happen, and don't restart then.
  3. Deleting a certificate can be less dramatic than it sounds. A process that has already loaded its certificate keeps using it. The danger is in what replaces it.
  4. Write the prediction down before the event. "bletchley-pg-server will not renew" was easy to check because it was written in September, not reconstructed in November.

What's Next

The monitoring behind this post gets its own post on Saturday: why two metrics with "certificate" in their name could not see the renewal, why the expiry clock cannot warn in time, and the one place none of the new alerts looks. After that, January: the operator's CA renews on its own for the first time. And while checking client connections for this post I found that one tenant has been falling back to an unencrypted connection; that is a story of its own.


← Previous: And Then There Were Three


Questions or suggestions? Leave a comment below or reach out at igor@vluwte.nl.