One Certificate in Ten Places
Monitoring a certificate you don't renew: counters that count reads, a clock that can't warn, and a check blind to the one refusal that happened
Introduction
On Wednesday, CloudNativePG Part 10 told the story: CloudNativePG renewed its own certificates on 24 October without a hitch, I broke its admission webhook on purpose a week later, and the only write refused all evening was caused by the recovery. That post was about what happened. This one is about what I could see while it happened.
Because that is the uncomfortable part. Before the renewal I had metrics with "certificate" in their name, an expiry exporter covering every certificate in the cluster, and a threshold written down in my notes. Looked at closely, almost every one of them answered a different question than the one I was asking. The counters counted file reads. The clock either fired every cycle or not in time. A timestamp on the certificate was not the time it was issued. And the sharpest check I built afterwards stayed perfectly calm through the one refusal that actually happened.
The thread through all of it: a certificate like this lives in three places, the Secret, the bundle that says what to trust, and the copy a process actually serves. Every signal looks at some of them. None looks at all three.
š¬ This is part of the Technical Deep Dives series - the details behind the homelab journey.
- Handling a Password You Never See
- One Certificate in Ten Places (you are here)
The Bundle Holds the Leaf
CloudNativePG does something unusual with its admission webhook: it tells the API server to trust one specific certificate instead of a certificate authority. To see why that matters, start with what is normal.
An admission webhook has a caBundle: the certificates the API server should trust when it connects to the webhook over HTTPS. Normally that is a CA. The component that runs the webhook keeps the CA's key, issues itself a serving certificate, and can replace that certificate whenever it likes: any new certificate from the same CA is trusted automatically. The bundle only has to change when the CA does, which is rare.
When I enabled the x509 exporter's caBundle sources in September (Cluster-Wide Certificate Expiry Monitoring), it showed seven components in this cluster with a bundle: MetalLB, frr-k8s, Longhorn, External Secrets, cert-manager, the TransIP DNS webhook and CloudNativePG. Six of them put a CA in it. CloudNativePG puts the serving certificate itself, the leaf, in all nine of its webhook entries.
That removes the slack. With a CA in the bundle, renewing the serving certificate is a one-sided change: the operator swaps its certificate and the API server keeps trusting it. With the leaf in the bundle, every renewal is a two-sided change. The Secret, the nine bundle entries and the served certificate must switch together, and anything that switches early or late is refused. That is the root of the twist in Part 10.
It also gives you something to check. If it is the same certificate in ten places, the ten places must agree, exactly:
count(
count by (serial_number) (
x509_cert_not_after{secret_name="cnpg-webhook-cert"}
or x509_cert_not_after{cabundle_resource_name=~"cnpg-.*"}
)
)
One distinct serial across the Secret and the nine entries is healthy. Two means something did not switch. It was 1 in September (B81366DCā¦), and still 1 after the 24 October renewal, with every series on the new serial (7035E4F4ā¦).

Two Metrics That Count Reads
The first thing I did in September was the obvious one: grep the operator's metrics for cert. Two came back:
certwatcher_read_certificate_total 333339
certwatcher_read_certificate_errors_total 0
A rising counter and a healthy zero. It is easy to read that as "the certificate is being watched, and all is well". Neither metric knows what a date is. They come from controller-runtime's certificate watcher, the code that loads the webhook's serving certificate from disk, and they count how often it read the file and how often that failed.
To see what that means in practice, I looked at the read rate across the renewal on 24 October:
rate(certwatcher_read_certificate_total{job="cnpg-operator"}[1h]) * 60
It was flat at 6 reads per minute, one every 10 seconds, all afternoon. The renewal landed at 13:53:50 and the line did not move. The operator later confirmed the interval itself, when it logged "Starting certificate poll+watcher" ⦠"interval":10 on startup. (In September I had estimated 10.9 seconds by dividing the counter by the pod's uptime. Division is not a measurement; rate() is.)
So the counter cannot see a renewal. What it can see is whether the operator is reading its certificate at all, and whether those reads fail, and that is what it is now used for. When I deleted the Secret in Part 10, the error counter climbed six per minute and an alert fired within a minute and a half. A metric with the noun in its name is not necessarily about the property you care about. Sometimes it is about something else that is still worth knowing.

Why the Clock Cannot Warn You
The other obvious signal is the expiry date itself. The x509 exporter publishes it for every certificate, so an alert on "expires in less than N days" is one line. The question is N, and the answer turned out to be smaller than I had written down.
CloudNativePG renews its certificates seven days before they expire. In normal operation, then, the lowest remaining lifetime you will ever see is seven days: the certificate counts down to seven and jumps back to ninety. My notes said to alert at 21 days. That alert would have fired on 10 October, stayed on for two weeks until the renewal cleared it, and done the same again every 90 days. An alert that fires every cycle as normal operation is noise by construction, and noise is how real alerts get ignored.
Any threshold of seven days or more has that problem. The one in place sits at three days, inside the renewal lead time, so in normal operation it never fires. That also means it is the last line: when it fires, the renewal has already failed, and there are three days left to find out why. Early warning cannot come from the clock. It has to come from watching the mechanism that renews.

The clock did find one certificate in October, and it is a fitting one. bletchley-pg-server was the certificate the operator had stopped caring about after Part 8, and it was the only one that walked towards the three-day line. The clock caught exactly what the mechanism no longer covered.
What 24 October Looked Like From Inside
The renewal itself went cleanly, but reading the evidence afterwards nearly led me to two wrong conclusions.
notBefore is not when the certificate was made. The renewed webhook certificate has notBefore 13:48:50. The operator's log, and the moment the exporter saw the new serial, both put the change about five minutes later. My first reading was a five-minute delay between the operator writing the certificate and anyone noticing. That delay did not exist. The Secret's managedFields gives the time it was actually written:
kubectl -n cnpg-system get secret cnpg-webhook-cert \
-o jsonpath='{.metadata.managedFields[*].time}{"\n"}'
# 2026-10-24T13:53:50Z
Exactly five minutes after notBefore. The cluster's certificates showed the same offset (notBefore 15:55:58, written 16:00:59), and so did the original server certificate from August. CloudNativePG backdates notBefore by five minutes, a common allowance for clocks that are slightly out of step. Harmless, as long as you do not use it as the issue time.
The renewal is not exactly at seven days. With the real write times, the operator's pair renewed 13 minutes 32 seconds after its seven-day mark, and the cluster's pair 28 minutes 9 seconds after its own. Two different offsets for one rule point to a check that runs periodically inside the operator, not a timer set for the exact moment.
A certificate travels two different ways. From the same afternoon:
| Path | Written | Picked up | Delay |
|---|---|---|---|
| Webhook certificate, Secret mounted as a volume | 13:53:50 | certificate watcher, 13:54:18 | 28 s |
| Cluster certificates, read through the API | 16:00:59 | all three instances, 16:00:59 | same second |
| The exporter, on its own poll | 13:53:50 | new serial at 13:54:30 | ~40 s |
The volume path waits for the kubelet to sync the mounted Secret. The instance manager in each PostgreSQL pod watches the Secret through the API, writes the files itself and reloads PostgreSQL, so there is nothing to wait for.
One log line for the whole event. The renewal of four certificates produced exactly one line in the operator's log, and it is controller-runtime's, not CloudNativePG's: "Updated current TLS certificate". The CA renewals and the cluster's certificates log nothing at the operator. The sentence a future reader would search for barely exists.
The cluster's CA kept its key. The old server certificate, signed by the August CA, still verified against the October CA. The CA was re-signed, not replaced, so client certificates signed by it stay valid across a CA renewal. Whether the operator's own CA does the same is still open; it renews for the first time in January.
Three Rules, and the Place None of Them Looks
Watching the mechanism instead of the clock came down to three rules, now in the cluster's certificate alert group:
| Alert | Fires when | for |
Seen in action |
|---|---|---|---|
CNPGWebhookCertMismatch |
More than one serial across the Secret and the nine bundle entries | 15m | Pending 40 s after the Part A break; never fired |
CNPGOperatorCertWatcherStalled |
The operator reads its certificate zero times in 5 minutes, or the metric is absent | 10m | Not triggered (needs the operator to be down) |
CNPGOperatorCertWatcherErrors |
Any failed certificate read in 15 minutes | 0m | Fired and emailed in Part B |
The for: 15m on the mismatch rule is deliberate. In Part A the operator repaired a wrong bundle entry by itself in five minutes. A mismatch that heals itself should stay pending; one that survives three repair cycles needs a person. The stalled rule's absent() clause also makes it the only check that notices the operator being gone altogether.
Then came the moment in Part 10 that none of them saw. The operator restarted, the new pod created a new certificate and wrote it into all nine bundle entries, and for about ten seconds the old pod, still shutting down, kept serving the old certificate. A real Cluster write was refused. Throughout that window, the serial count stayed at 1.
That is not a bug in the rule. The Secret and the bundle agreed, both on the new certificate. The disagreement was with the third place a certificate lives: the copy a process is actually serving. The exporter reads Secrets and bundles, objects at rest in the cluster. It cannot see what is on the wire, and neither can anything built on it.
| Where the certificate lives | State | Seen by |
|---|---|---|
| The Secret | at rest | The x509 exporter, the expiry alerts, the mismatch rule |
| The nine bundle entries | at rest | The x509 exporter, the mismatch rule |
| What the pod serves | on the wire | Nothing, today |
Seeing the third would take a probe that connects to the webhook and compares the certificate it is presented with the Secret at rest. I have not built it. A ten-second, self-healing window during a deliberate restart does not justify it on its own, but the external PostgreSQL certificate has exactly the same gap (cert-manager and the exporter both read it at rest), and cert-manager renews it every 60 days, the first time this week. One probe design can cover both.
The Relabel That Did Half Its Job
The stalled rule depends on rate() over the operator's read counter, and that ran into an old problem from The Alert That Named a Pod: the scrape jobs for CloudNativePG had no instance label of their own, so Prometheus filled it in with the pod's IP address. Over 90 days the operator had appeared under 8 different identities, and the three PostgreSQL instances under 28, partly because Flannel had handed the same pod network to different nodes over time. An alert email would have named an IP address, and every operator restart started a new series.
So I gave both jobs a stable instance: the pod name for the PostgreSQL instances, which CloudNativePG keeps stable, and a fixed cnpg-operator for the operator. I also added pod and node labels to the operator job, to correlate it with anything else that happens on the same pod or node. The changeover looked as expected: the old series fading out, the new one ramping up.

My note on the change said a fixed instance would keep one series across restarts, so rate() would carry straight through. Two days later the restart in Part 10 showed otherwise:

Both series have instance="cnpg-operator". They are still two series, because Prometheus identifies a series by all of its labels, and pod and node changed with the restart. The fix made alerts and emails readable and selectors stable. It did not make the series continuous, and it could not while those two labels were there.
I kept them anyway: knowing which pod and node an alert came from is worth more here than an unbroken graph, and none of the rules needs continuity across a restart. What I did fix was the comment in the scrape configuration that claimed otherwise. The same thing had happened once before, unnoticed: when I added a namespace label to this job months ago, one series split into two (40 samples without it, 762 with it). Every label you add is part of the identity, whether you meant it to be or not.
Lessons Learned
- An expiry threshold must sit inside the renewal lead time. Above it, the alert fires every cycle by design. Inside it, it is a last line, not an early warning.
- A metric with the noun in its name is not necessarily about the property. The certificate counters count file reads. Useful, but not for expiry.
- Don't assume a
caBundleholds a CA. Look. When it pins a leaf, every renewal is a two-sided switch with no overlap. notBeforeis not the issue time. Use the object'smanagedFieldsfor when something was actually written.- A check at rest cannot see what is served. Know which of the places a certificate lives your monitoring actually reads.
- Every label is part of a series' identity. A stable
instancedoes not keep a series together if another label changes.
What's Next
In January the operator's CA renews on its own for the first time, a week before the webhook certificate, and I can check whether it keeps its key the way the cluster CA did. Before that, the external PostgreSQL certificate, renewed by cert-manager every 60 days, is the natural place to start on a probe that looks at the third place: what is actually served.
ā Previous: CloudNativePG Part 10: The Certificate That Rejects Its Own Fix
Questions or suggestions? Leave a comment below or reach out at igor@vluwte.nl.