Cluster-Wide Certificate Expiry Monitoring

Introduction

Certificates on Bletchley felt handled. cert-manager issues them and renews them. Prometheus scrapes its metrics. Since Fixing the Alerts there are rules as well, at 14 days warning and 7 days critical.

That was true for 18 certificates. It is also why I never looked any further.

CloudNativePG made me look. Its certificates come from the operator, not from cert-manager. I went to check when they expire and found a date: 31 October. Nothing in the cluster would tell me whether that date passed without a renewal. And it was not only CloudNativePG. Thirteen Secrets hold certificates with no Certificate object behind them, so there is no cert-manager renewal date to inspect and no expiry metric at all.

This post is about that second half. I wanted three things:

• every certificate stored in the Kubernetes API known
• every expiry date tracked continuously
• renewals visible as a date that moves, instead of configuration I trust

Renewing anything early, forcing a rotation, testing reload paths: all out of scope. This is about visibility.

🔭 This is part of the Cluster Observability series - making the Bletchley cluster observable, from metrics and logs to the alerts built on them.

• Fixing the Alerts
• Cluster-Wide Certificate Expiry Monitoring (you are here)

What was already covered

cert-manager's half needs nothing. Let's go over what is already there.

There currently are 18 Certificate objects. Each one publishes a status.renewalTime I can read. Prometheus scrapes certmanager_certificate_expiration_timestamp_seconds for all of them. Two rules fire at 14 and 7 days. That half is monitored end to end.

The more useful number is how early each certificate renews. I assumed 60 days for the Let's Encrypt ones as this is what I documented above my alert rules. So I measured all 18, by subtracting notAfter from status.renewalTime:

Eleven of the 18 come from Let's Encrypt, one per (public) hostname. The other seven are internal: three from my own CA, two for the TransIP webhook, and the Barman pair. The addendum lists all 18 with their issuers.

Five lead times, and they come from two causes.

Thirteen certificates take cert-manager's default, which renews once a third of the lifetime is left. That is 30 days on a 90-day Let's Encrypt certificate, 121 days on a one-year certificate, and ~608 days on the five-year TransIP CA. One rule, three lifetimes.

Five set renewBefore themselves. The Barman pair use 360h, which is the 15 days. authelia-ca-trust and lldap-ldap-tls use 720h, pulling two one-year certificates down to 30 days. internal-ca-root uses 8760h.

My assumed 60 days is neither of those. So the comment was wrong twice over: wrong lead time, and a certificate count that had long since gone stale.

The shortest lead time is the Barman pair, at 15 days, and it is a number I chose myself. My 14-day warning sits one day inside it. That is correct behaviour, because the warning should only fire when a renewal is actually late. The margin is still one day, in a value I can change.

The census

First question: how many certificates are there at all? In Kubernetes they live in Secrets, usually under tls.crt or ca.crt. So the census is a loop over every Secret and every such key:

kubectl get secret -A -o json | jq -r '
  .items[] | .metadata.namespace as $ns | .metadata.name as $n |
  (.data // {}) | to_entries[] | select(.key | test("\\.crt$")) |
  "\($ns) \($n) \(.key) \(.value)"' \
| while read -r ns name key b64; do
    cert=$(echo "$b64" | base64 -d 2>/dev/null) || continue
    end=$(echo "$cert" | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
    [ -z "$end" ] && continue
    flag="OK"
    echo "$cert" | openssl x509 -noout -checkend 7776000 >/dev/null 2>&1 || flag="<90d"
    echo "$cert" | openssl x509 -noout -checkend 2592000 >/dev/null 2>&1 || flag="<30d"
    subj=$(echo "$cert" | openssl x509 -noout -subject 2>/dev/null | sed 's/^subject=//')
    printf '%-5s %-14s %-34s %-9s %-30s %s\n' "$flag" "$ns" "$name" "$key" "$end" "$subj"
  done

It prints one line per key, and flags anything inside 90 or 30 days. The full output is in the addendum at the end of this post.

42 certificate-bearing keys across 31 Secrets. They come in four combinations of Secret type and key:

That table matters later. Hold on to it.

The 13 outside cert-manager

Cross-reference those 31 Secrets against the 18 Certificate objects. Thirteen are left over, not backed by a Certificate object:

The expiry column is read from the certificates. The renewer column is a guess. I inferred it from whichever component created each Secret, and verified none of it.

"No Certificate object" does not mean "not renewed". Most of this list is probably rotated by the component that made it. What it lacks is anything I can inspect.

The five CloudNativePG certificates share a birthday. All five were minted on 2 August, when CloudNativePG went onto the cluster, with the operator's default 90-day lifetime. The documentation says the operator renews 7 days before expiry. The operator Deployment sets neither environment variable that changes that, so I expect a renewal around 24 October.

Expecting it is the problem. Nothing would have told me if it did not happen.

Choosing a tool

Five options were on the table.

The deciding question: which signal is still true if the renewal mechanism lies? cert-manager's metric reports what cert-manager believes about its own certificates. The exporter reports what is actually in the Secret.

The CronJob lost twice. I did not know the second reason yet when I chose, and it comes up in the reconciliation.

What it costs

One pod that can read every Secret in the cluster.

The exporter needs get, list and watch on Secrets. The Kubernetes API has no field-level read, so that ServiceAccount can read every private key and token here. resourceNames cannot narrow it, because the exporter uses LIST and WATCH. The only lever is includeNamespaces, and that silently misses any namespace added later.

I took the trade deliberately, and wrote it into the monitoring README so it is not something I rediscover.

The default that would have missed every CA

Enix publishes the exporter as an OCI chart. I installed 4.2.0. The values file is short, and its first block is the most consequential setting in this post:

# apps/monitoring/x509-certificate-exporter/values.yaml
secretsExporter:
  secretTypes:
    - type: kubernetes.io/tls
      key: tls.crt
    - type: kubernetes.io/tls
      key: ca.crt
    - type: Opaque
      key: tls.crt
    - type: Opaque
      key: ca.crt

exposeNotBeforeMetric: true

prometheusServiceMonitor:
  create: false                  # no Prometheus Operator
prometheusRules:
  create: false                  # make own rules

The chart's default secretTypes is one entry: type kubernetes.io/tls, key tls.crt.

Every CA this census found is an Opaque Secret with a ca.crt. Install with defaults, and the tool built for exactly this job reports a healthy-looking list of leaf certificates with every CA missing.

That is why the values file has four entries, one per combination in the census table. The chart README offers a shorter way to write it, a single keyPatterns regex. It does not exist in 4.2.0's values. The README is ahead of the release.

exposeNotBeforeMetric is off by default. I turned it on, so both ends of each certificate's validity are in Prometheus.

The two create: false lines are for a cluster without the Prometheus Operator. Bletchley runs the prometheus-community prometheus chart, so there are no ServiceMonitor or PrometheusRule CRDs to create. The bundled rules would have been wrong anyway. The alerting section says why.

Install

helm -n monitoring install x509-certificate-exporter \
  oci://quay.io/enix/charts/x509-certificate-exporter --version 4.2.0 \
  -f apps/monitoring/x509-certificate-exporter/values.yaml

Pulled: quay.io/enix/charts/x509-certificate-exporter:4.2.0
Digest: sha256:02522d40d3f9b5a96e171a65d28137348e382ea5d39af8afb6ae34d1340d4216
NAME: x509-certificate-exporter
LAST DEPLOYED: Sun Sep 13 14:10:21 2026
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1

The startup log answered the questions I had:

level=INFO msg=starting version=4.2.0
level=INFO msg="maxprocs: Updating GOMAXPROCS=1: determined from CPU quota"
level=INFO msg="GOMEMLIMIT is updated" GOMEMLIMIT=141557760 previous=9223372036854775807
level=INFO msg="kube-apiserver reachable" version=v1.36.2
level=INFO msg="Listening on" address=[::]:9793
level=INFO msg="TLS is disabled." http2=false address=[::]:9793
level=INFO msg="initial sync complete" source_name=cluster-secrets namespace="" elapsed=919.044314ms

The pod came up on rock6, so the arm64 image works. namespace="" confirms the scope is cluster-wide. The initial sync of every Secret took 919 ms. The listener is plain HTTP, which is fine for a scrape that never leaves the cluster.

I also checked the security context. PodSecurity restricted has been a recurring theme here:

kubectl -n monitoring get deploy x509-certificate-exporter \
  -o jsonpath='{.spec.template.spec.securityContext}{"\n"}{.spec.template.spec.containers[0].securityContext}{"\n"}'

{"runAsNonRoot":true,"seccompProfile":{"type":"RuntimeDefault"}}
{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"readOnlyRootFilesystem":true,"runAsGroup":65534,"runAsUser":65534}

Compliant with restricted, with nothing configured.

That is worth saying out loud. A few minutes later, a Prometheus upgrade printed restricted warnings for prometheus-server, its config reloader and alertmanager. A maintained chart getting this right out of the box is the norm. The violators are the exception.

Scrape job

The scrape job follows the existing garage pattern in the Prometheus values:

  - job_name: x509-certificate-exporter
    static_configs:
      - targets: ['x509-certificate-exporter.monitoring.svc.cluster.local:9793']

A Service name rather than pod discovery, on purpose. The chart's Service is headless, so the name resolves to the pod IP. But with static_configs, the instance label is the target string exactly as written. After The Alert That Named a Pod I wanted a label that survives pod replacement by construction, not by relabelling.

up{job="x509-certificate-exporter"}   1
count(x509_cert_not_after)            65
x509_source_up                        1
x509_source_errors_total              17 series, all 0

All 17 error reasons are zero, permission_denied included. The RBAC reaches every namespace.

65 = 42 + 22 + 1

The census found 42 keys. The exporter reported 65 series. Before trusting either number, I wanted the difference to close exactly.

One thing to fix in the vocabulary first. A series is one certificate seen in one place, so the counts in this section are observations, not certificates. The same certificate can appear under two Secret keys, and again in a webhook's caBundle, and each of those is its own series. That is not double counting. Each is a separate place the certificate has to be correct.

It does:

One Let's Encrypt Secret, labels trimmed, produces three series:

x509_cert_not_after{issuer_CN="ISRG Root X1", subject_CN="Root YR",
  secret_key="tls.crt", secret_name="authelia-tls", secret_namespace="authelia"}   1.977782399e+09
x509_cert_not_after{issuer_CN="Root YR", subject_CN="YR1",
  secret_key="tls.crt", secret_name="authelia-tls", secret_namespace="authelia"}   1.851551999e+09
x509_cert_not_after{issuer_CN="YR1", subject_CN="auth.bletchley.vluwte.nl",
  secret_key="tls.crt", secret_name="authelia-tls", secret_namespace="authelia"}   1.792296288e+09

What openssl x509 doesn't tell you

The 23 extra certificates were no surprise to the exporter. They were a surprise to me. My census could not see them.

openssl x509 reads the first certificate in a PEM blob and stops. Every Let's Encrypt intermediate, every cross-signed root, and the Longhorn CA in front of its leaf were invisible to the loop. My census under-reported the cluster by more than a third, and reported no error while doing it. That is the second reason the CronJob lost.

The reconciliation turned up three more things worth keeping:

• Two Root YR certificates, and they are not the same certificate. The one inside every Let's Encrypt chain is cross-signed by ISRG Root X1 and expires in 2032. The one in databases/postgres-vluwte-ca is self-issued and expires in 2045. That is the pin-versus-chain distinction from CloudNativePG Part 8, now visible as two series.
• Let's Encrypt alternates intermediates. Six of my hosts sit on YR1, five on YR2. This matters later, for renewal detection.
• Duplicates are kept, not dropped. One Secret can hold the same certificate under ca.crt and tls.crt. That becomes two series with one serial, told apart by secret_key. The Barman pair is that case, because a selfSigned issuer copies the leaf into ca.crt.

What the API server actually trusts

A second place to look

The exporter reads one source that no Secret-based approach can: the caBundle field on admission webhooks, API services and CRD conversion webhooks. It is off by default. I enabled all four resource kinds, including the two the chart leaves off:

cabundlesExporter:
  enabled: true
  resources:
    mutating: true
    validating: true
    apiservice: true
    crdConversion: true

This source reads objects containing public certificates, so it does not add the private-key exposure created by Secret access. Both extra kinds earned their place: v1alpha1.acme.transip.nl and bgppeers.metallb.io. An aggregated API or a conversion webhook with an expired bundle fails just as quietly as anything else.

Twelve resources carry a caBundle, with 25 entries between them:

APIService                      v1alpha1.acme.transip.nl                   1 entry
CustomResourceDefinition        bgppeers.metallb.io                        1 entry
MutatingWebhookConfiguration    cert-manager-webhook                       1 entry
MutatingWebhookConfiguration    cnpg-mutating-webhook-configuration        4 entries
MutatingWebhookConfiguration    longhorn-webhook-mutator                   1 entry
ValidatingWebhookConfiguration  cert-manager-webhook                       1 entry
ValidatingWebhookConfiguration  cnpg-validating-webhook-configuration      5 entries
ValidatingWebhookConfiguration  externalsecret-validate                    1 entry
ValidatingWebhookConfiguration  frr-k8s-validating-webhook-configuration   1 entry
ValidatingWebhookConfiguration  longhorn-webhook-validator                 1 entry
ValidatingWebhookConfiguration  metallb-webhook-configuration              6 entries
ValidatingWebhookConfiguration  secretstore-validate                       2 entries

Those 25 series hold only seven distinct certificates. All seven were already visible as Secret series, so this source added no new certificates.

What it added is a second observation point. One side is what the Secret holds. The other is what the API server has been told to trust. The value is the comparison.

With both sources enabled, the exporter reports 90 series.

A caBundle is not necessarily a CA

The name promises a bundle of CAs. Six of the seven components here deliver that. MetalLB, frr-k8s, Longhorn, ESO, cert-manager and the TransIP webhook all embed their CA.

CloudNativePG embeds the serving leaf. All nine entries, four mutating and five validating, carry cnpg-webhook-service.cnpg-system.svc. The serial matches the certificate in the cnpg-webhook-cert Secret. One certificate in ten places.

That removes the slack from the October renewal. A bundle holding a CA still verifies a renewed leaf, so a missed repatch is survivable. A bundle holding the leaf pins that exact certificate. Every renewal has to update all nine entries in lockstep, and a miss breaks admission immediately.

I first counted eight entries. I found the ninth, mbackup.cnpg.io, only when I printed every series instead of counting them. Enumerate, don't count.

It is an exact equality, so it gives a check that is not about expiry at all:

count(
  count by (serial_number) (
      x509_cert_not_after{secret_name="cnpg-webhook-cert"}
   or x509_cert_not_after{cabundle_resource_name=~"cnpg-.*"}
  )
) > 1

Today that returns nothing: one distinct serial. More than one means something was not updated, and unlike an expiry comparison it catches a partial repatch.

What happens to it in October belongs to a later CloudNativePG post. That is where this finding gets its ending.

The dashboard

I imported Grafana dashboard 13922. It needed two edits.

First, it predates exporter v4. x509_read_errors is now x509_source_errors_total, so the health and error panels stayed empty until I replaced the name in each of them. x509_cert_not_after is unchanged, so the expiry panels worked straight away.

Second, it covers every source kind the exporter supports: host-path files, kubeconfigs, ConfigMaps, CRLs, PKCS#12 and JKS keystores. None of those are enabled here. Their panels were permanently empty, which is correct rather than broken, and I removed them. A permanently empty panel is the visual version of an alert that always fires, which was the whole argument of Auditing the Alerts. The CA bundle panels went with them, and can come back now that the source is enabled.

One source I left off deliberately. ConfigMap scanning for ca.crt would pick up kube-root-ca.crt, and Kubernetes publishes that into every namespace: the same cluster CA, twenty-odd times.

Three panels in the overview row are worth reading closely.

The total reads 90, and it is count(x509_cert_not_after). That is exactly what the arithmetic predicted: 65 from Secrets, plus 25 caBundle entries. Expired, within 7 days and within 28 days are all zero. The two soonest rows, authelia-tls and traefik-dashboard-tls, sit at about 4.5 weeks. Both are Let's Encrypt certificates with a 30 day lead time, so neither is due. The table also lists barman-cloud-client-tls twice: the duplicate case again, one certificate under two keys.

The Media panel disagrees with the total. It reports one source kind, Kubernetes Secret, holding 65 certificates, at 100%. Those are the 65 from revision 1. The 25 caBundle entries are in the total but not in the breakdown. One panel says 90; its neighbour says 65 is everything. Same class of problem as the renamed metric: it was written before the source existed. I have not read its query yet, so for now I distrust the breakdown rather than the total.

The panel labelled Exporters reads 34. There is one exporter. 34 is the number of x509_source_errors_total series: 17 failure reasons, two sources, cluster-secrets and cluster-cabundles. It would have read 17 before I enabled caBundles. It will move again if a third source appears. Whatever it counts, it is not exporters.

So two of those panels measure something other than their label, and neither number looks wrong at a glance. 65 is a plausible certificate count, and 34 is a plausible anything. An imported dashboard is a set of claims someone else wrote. I trust the ones I have traced back to a query, which here means the total and the two expiry tiles.

The split behind that 34 is useful in itself. Read errors are counted per source now. So the read-error rule I left for later, increase(x509_source_errors_total[1h]) > 0, can name the source that went unreadable. That matters, because a namespace the exporter cannot read produces an absence rather than an alert. The certificate count drops quietly, and no expiry rule fires: the series it would evaluate are gone.

One last thing before the alerting section. The dashboard's own warning and critical variables default to 28 and 7 days, the same numbers the chart wanted to alert on. Here they are harmless: a variable colours a cell, a rule sends mail. It is still worth knowing which of the two you are looking at.

Alerting: why 3 days, not 28

The chart ships rules at 28 days warning and 14 days critical. Both sit above the Barman lead time of 15 days, and far above CloudNativePG's 7. Enabled, they would fire during normal operation, before the renewal they are supposed to watch had even been attempted.

There are two different alerts hiding in that number. A lead-time alert says a certificate is approaching its renewal window, so it has to know the window. A safety-net alert says a certificate is nearly expired and no replacement has appeared, so it only has to sit below every window. The chart ships the first kind. I want the second.

Initially I picked a threshold per certificate, from lead times I mostly guessed. The table at the top of this post is what measuring produced instead, and the two had little in common. So the rules I kept need no lead time at all.

The two cert-manager rules I already had moved into a new certificates group, unchanged. The x509 rules sit behind them. The group reads as two halves: what cert-manager covers, and what it cannot.

          # -- X509 Certificate Exporter ---------------------------------------
          - alert: CertificateExpiringImminent
            expr: x509_cert_not_after - time() < 3 * 86400
            for: 15m
            labels:
              severity: critical
            annotations:
              summary: >-
                Certificate expires in {{ $value | humanizeDuration }}:
                {{ $labels.secret_namespace }}/{{ $labels.secret_name }}
              description: >-
                {{ $labels.subject_CN }} (issuer {{ $labels.issuer_CN }}) in secret
                {{ $labels.secret_namespace }}/{{ $labels.secret_name }} key {{ $labels.secret_key }}
                expires in {{ $value | humanizeDuration }} and no replacement has appeared.
                Where the renewal mechanism is known it runs at least 7 days ahead, so this is a
                renewal that did not happen, or a certificate with nobody to renew it.
          - alert: CertificateManualRenewalDue
            expr: >-
              x509_cert_not_after{secret_name=~"postgres-vluwte-ca|openbao-transit-ca|longhorn-webhook-ca|external-secrets-webhook"}
              - time() < 90 * 86400
            for: 1h
            labels:
              severity: warning
            annotations:
              summary: >-
                Manually-managed certificate expires in {{ $value | humanizeDuration }}:
                {{ $labels.secret_namespace }}/{{ $labels.secret_name }}
              description: >-
                Nothing is believed to renew this certificate automatically. Renewal is a human act,
                so this fires 90 days ahead.
          - alert: X509ExporterDown
            expr: up{job="x509-certificate-exporter"} == 0 or x509_source_up == 0
            for: 10m
            labels:
              severity: warning
            annotations:
              summary: "Certificate exporter is down — no certificate expiry is being monitored"
              description: >-
                While this is firing, none of the certificate expiry alerts can fire either,
                because the series they evaluate are absent rather than stale.

Three days as a safety net

CertificateExpiringImminent sits below every lead time in the cluster. cert-manager's shortest is 15 days. CloudNativePG's is 7, so if this rule fires for one of its certificates, the renewal was already four days overdue.

That is the whole design. It cannot fire during normal operation, for any certificate, including ones added next year that nobody has classified. It needs no exception for cert-manager's 18, and no knowledge of who renews what. It fires when a renewal is late.

It evaluates per series, with no aggregation. Every certificate in a bundle is judged on its own date, so a chain cannot hide an expiring leaf behind a root with years left.

Three days is not much warning. That is deliberate. For a certificate that renews itself, a human is only needed once the automation has failed. The question the alert answers is "has it failed", not "is it coming up".

Why not one day, then? One day would be just as safe from false positives, and would leave no room to do anything about it. Three days covers a weekend.

Ninety days where the remedy is me

Three days is useless when renewal is a human act. CertificateManualRenewalDue gives four Secrets 90 days' notice.

Two are there on a fact. postgres-vluwte-ca and openbao-transit-ca were built by hand, and no software anywhere has the job of renewing them.

The other two are there on an inference, and probably a wrong one. longhorn-webhook-ca and external-secrets-webhook were each created by a controller that very likely rotates them. If the remedy is "restart the controller and it mints a new one", then 90 days of notice buys nothing, and the 3-day rule already covers the failure. I left them in because their dates are in 2036. The way to settle it is to read how dynamiclistener and ESO rotate, not to wait a decade and watch.

Watching the watcher

X509ExporterDown exists because a dead exporter is silent. No certificate alert can fire, and nothing says why. Same shape as the rules in Silent Failures, and for the same reason.

A label that isn't there

The annotations above are still wrong for the source I added last. I am leaving them in the post rather than quietly fixing them, because the shape of the mistake is the interesting part.

Both x509 rules build their summary from {{ $labels.secret_namespace }}/{{ $labels.secret_name }}. A caBundle series has neither label.

Say CertificateExpiringImminent fires for one of the nine CloudNativePG webhook entries. The alert itself is correct. The mail arrives titled "Certificate expires in 2d 3h: /", and the description reads "in secret / key ". The only label every series carries is subject_CN, and in that mail it would have been the one thing identifying the certificate.

It is not urgent. All seven caBundle certificates are also in Secrets, so a Secret series fires alongside, with a full subject line. It is still the alert that reads worst in exactly the case that justified adding the source.

The lesson is that an annotation is only as portable as the least-labelled series its expression can match. Lead with subject_CN. Treat secret_* as the optional extra.

Validation, and what it doesn't cover

All five rules showed inactive on /alerts after the reload. I also ran each expression against live data in Grafana. Each returned nothing, which is correct: nothing is due.

I did not force any of them to fire. The Prometheus to Alertmanager to email path is already proven by the rules in the cluster and cnpg groups. That leaves three things untested. I would rather name them than let them pass as coverage:

1. The for: durations. Nothing confirms that a condition which persists becomes a firing alert rather than flapping.
2. Annotation templating. A label a series does not carry renders as empty text. Nothing here proves otherwise for a rule that has never fired. The caBundle gap above is that failure exactly, found by reading label sets rather than by receiving a bad mail.
3. The negative case beyond today's data. An empty result proves the expression does not match today's values. Only a unit test proves it cannot match the values a certificate passes through mid-cycle. That is the mistake the chart's 28/14 defaults would have made.

promtool test rules closes the third one cheaply: synthetic series, nothing sent. It is on the list.

Expiry is not renewal

The third goal was to see a renewal happen. The obvious query is changes(x509_cert_not_after[90d]). It does not work.

Every series carries serial_number, and a renewed certificate has a new serial. So a renewal never changes the value of a series. It ends one series and starts another, and changes() on either returns zero.

Aggregating the serial away with without (serial_number) is also wrong. It keeps issuer_CN, and Let's Encrypt alternates between YR1 and YR2. A certificate renewed onto the other intermediate would again look like one series ending and another starting.

The fix is to group on the labels that identify a certificate's slot, and that should survive a renewal:

changes(
  max by (secret_namespace, secret_name, secret_key, subject_CN) (x509_cert_not_after)
  [90d:1h]
)

With 120 days of retention, one 90-day CloudNativePG cycle fits.

There is one known hole. The query assumes subject_CN survives a rotation. For Longhorn it does not. Its CA is CN=dynamiclistener-ca@1772051961, and that number is a Unix timestamp: late February 2026, when the cluster was built. Mint a new CA and the subject changes with it. The panel would then show a new certificate rather than a date that moved.

The panel does not exist yet. It needs a few weeks of data to show anything. The first real test is the Barman pair on 16 October, then the CloudNativePG set around 24 October.

What this still cannot see

The exporter reads certificates at rest, inside the cluster. Three things stay outside it.

Files outside the cluster. The talosconfig client certificate (O=os:admin) and the kubeconfig client certificate (O=system:masters) live on granite, my workstation. Both expire on 22 February 2027. That is separate work, planned for November. The PKI that issues them is in 1Password, so expiry there is an inconvenience rather than a lockout. It is still a date nothing watches.

Talos machine PKI. Not Kubernetes objects.

What is actually served. A Secret that was renewed but never reloaded looks perfectly healthy to the exporter. The barman-cloud-*-tls Secrets carry no cnpg.io/reload label, and the Barman plugin sits on the WAL archiving path. That one is noted, not yet acted on. For admission webhooks, the caBundle comparison covers part of this. It catches configuration that was not updated, though not what a webhook server actually presents.

What's Working Now

• ✅ x509-certificate-exporter 4.2.0, cluster-wide, reading Secrets and caBundle fields. Compliant with PodSecurity restricted, no read errors
• ✅ All 31 certificate-bearing Secrets and all 12 caBundle-bearing resources visible, reconciled exactly against a hand census
• ✅ Scraped by Prometheus under a stable instance label, with 120 days of history
• ✅ Dashboard 13922 adapted for v4 metric names, with the empty source-kind panels removed, and its total matching the reconciled 90 observations
• ✅ Five rules in a certificates group: cert-manager's two, a 3-day safety net, 90 days for hand-made certificates, and a rule for the exporter itself
• ✅ The CloudNativePG Secret versus caBundle serial check has a baseline of 1

What I still don't know

Goals 1 and 2 are met within the Kubernetes API boundary. Every certificate stored in the Secrets and caBundle fields I am monitoring is visible, and every expiry date is in Prometheus with history. Outside that boundary nothing changed.

1. Whether the presumed renewers actually renew. The renewer column for the 13 is inferred from whichever component created each Secret. October settles it for the short-lived ones. For the 2029, 2036 and 2045 dates, waiting is not a strategy; reading upstream behaviour is.
2. Whether CloudNativePG renews on the schedule its documentation states. Seven days before expiry is documented, not observed here.
3. Whether dynamiclistener and ESO rotate their webhook CAs. If they do, CertificateManualRenewalDue should cover two Secrets rather than four.
4. How the rules behave when they fire. The for: durations and the annotation templating are untested by choice, and the caBundle annotation gap is one failure I already know is in there.
5. What two dashboard panels are counting. The Media breakdown calls 65 of 90 certificates 100%, and the Exporters panel reads 34 for one exporter. Neither number is explained yet.
6. Whether a renewed certificate is actually served. Everything here is read at rest. A Secret that renews and is never reloaded still looks healthy.

Lessons Learned

1. "Handled" is worth auditing precisely when it feels true. cert-manager was doing its job so well that it hid everything it was never responsible for.
2. An expiry threshold has to sit inside the renewal lead time. Read the lead time from status.renewalTime. This cluster has five different ones, and my guess of 60 days matched none of them.
3. Check a tool's defaults against your own data. The default secretTypes would have produced a clean dashboard with every CA missing.
4. openssl x509 reads the first certificate and stops. Any inventory built on it silently drops intermediates and roots.
5. Don't assume a caBundle holds a CA. When it pins a leaf, every rotation becomes a change on two sides with no grace period.
6. Enumerate, don't count. A partial listing hid the ninth webhook entry.
7. Never put a count in a comment. It is correct once.
8. A Certificate, its Secret and the subject are three different names. barman-cloud-client writes into barman-cloud-client-tls, and lldap-ldap-tls writes into lldap-tls. Cross-reference through spec.secretName, not by guessing.
9. A lead-time alert and a safety-net alert are not the same alert. One predicts normal operation and has to know the renewal window. The other detects failure and only has to sit below it.

Conclusion

At the start of this I thought I had certificate monitoring. I had cert-manager metrics, expiry alerts and a Grafana dashboard.

What I actually had was monitoring for the 18 certificates cert-manager knows about. I had mistaken one certificate-management system for certificate management itself.

The census found 13 more Secrets. The exporter found 23 certificates my openssl loop could not see. The caBundle scan found a second place where the API server depends on certificate state, with one certificate sitting in ten of them. None of that was broken. It was unobserved, which is a different thing and a worse one, because it cannot be fixed until it is counted.

So the useful line is not cert-manager against everything else. It is what certificate state I can observe, and what I can prove about it. Inside the Kubernetes API that list is now long and reconciled. Outside it, the answer has not changed at all.

What's Next

The five certificates expiring on 31 October are the real test. Does the operator renew them on time? Do all nine webhook entries follow? What does the serial check do while it happens? That is a later CloudNativePG post, written after the event rather than before it.

Before then: a promtool unit test for the rules, the renewal-observed panel on the dashboard, and an answer on how dynamiclistener and ESO rotate. Then the manual-renewal list will hold only certificates that really need a human.

Addendum: the raw census, and where the numbers meet

The 42 lines the loop printed, with the <90d and <30d flags it computes from openssl x509 -checkend:

OK    authelia       authelia-ca-trust                  ca.crt    Aug 12 19:47:32 2036 GMT       CN=vluwte internal CA
OK    authelia       authelia-ca-trust                  tls.crt   Aug 19 19:48:03 2027 GMT       CN=authelia-internal-ca-trust
<90d  authelia       authelia-tls                       tls.crt   Oct 18 04:04:48 2026 GMT       CN=auth.bletchley.vluwte.nl
OK    cert-manager   cert-manager-webhook-ca            ca.crt    Mar  6 19:44:38 2027 GMT       CN=cert-manager-webhook-ca
OK    cert-manager   cert-manager-webhook-ca            tls.crt   Mar  6 19:44:38 2027 GMT       CN=cert-manager-webhook-ca
OK    cert-manager   cert-manager-webhook-transip-ca    ca.crt    Mar  5 20:00:08 2031 GMT       CN=ca.cert-manager-webhook-transip.cert-manager
OK    cert-manager   cert-manager-webhook-transip-ca    tls.crt   Mar  5 20:00:08 2031 GMT       CN=ca.cert-manager-webhook-transip.cert-manager
OK    cert-manager   cert-manager-webhook-transip-webhook-tls ca.crt    Mar  5 20:00:08 2031 GMT       CN=ca.cert-manager-webhook-transip.cert-manager
OK    cert-manager   cert-manager-webhook-transip-webhook-tls tls.crt   Mar  6 20:00:08 2027 GMT       
OK    cert-manager   internal-ca-root                   ca.crt    Aug 12 19:47:32 2036 GMT       CN=vluwte internal CA
OK    cert-manager   internal-ca-root                   tls.crt   Aug 12 19:47:32 2036 GMT       CN=vluwte internal CA
<90d  cnpg-system    barman-cloud-client-tls            ca.crt    Oct 31 13:58:40 2026 GMT       CN=barman-cloud-client
<90d  cnpg-system    barman-cloud-client-tls            tls.crt   Oct 31 13:58:40 2026 GMT       CN=barman-cloud-client
<90d  cnpg-system    barman-cloud-server-tls            ca.crt    Oct 31 13:58:40 2026 GMT       CN=barman-cloud
<90d  cnpg-system    barman-cloud-server-tls            tls.crt   Oct 31 13:58:40 2026 GMT       CN=barman-cloud
<90d  cnpg-system    cnpg-ca-secret                     ca.crt    Oct 31 13:40:18 2026 GMT       OU=cnpg-system, CN=cnpg-ca-secret
<90d  cnpg-system    cnpg-webhook-cert                  tls.crt   Oct 31 13:40:18 2026 GMT       CN=cnpg-webhook-service.cnpg-system.svc
<90d  databases      bletchley-pg-ca                    ca.crt    Oct 31 15:32:50 2026 GMT       OU=databases, CN=bletchley-pg
<90d  databases      bletchley-pg-replication           tls.crt   Oct 31 15:32:50 2026 GMT       CN=streaming_replica
<90d  databases      bletchley-pg-server                tls.crt   Oct 31 15:32:50 2026 GMT       CN=bletchley-pg-rw
OK    databases      postgres-vluwte-ca                 ca.crt    Sep  2 23:59:59 2045 GMT       C=US, O=ISRG, CN=Root YR
<90d  databases      postgres-vluwte-tls                tls.crt   Dec  6 16:52:44 2026 GMT       CN=postgres.vluwte.nl
OK    external-secrets external-secrets-webhook           ca.crt    Mar 29 19:55:02 2036 GMT       O=external-secrets, CN=external-secrets
OK    external-secrets external-secrets-webhook           tls.crt   Mar 29 19:55:02 2036 GMT       CN=external-secrets-webhook.external-secrets.svc
<90d  forgejo        forgejo-tls                        tls.crt   Dec 10 06:26:34 2026 GMT       CN=git.vluwte.nl
<90d  garage         garage-s3-tls                      tls.crt   Dec  4 18:26:17 2026 GMT       CN=s3.bletchley.vluwte.nl
<90d  it-tools       it-tools-tls                       tls.crt   Oct 28 17:34:43 2026 GMT       CN=it-tools.vluwte.nl
OK    lldap          lldap-tls                          ca.crt    Aug 12 19:47:32 2036 GMT       CN=vluwte internal CA
OK    lldap          lldap-tls                          tls.crt   Aug 18 19:08:53 2027 GMT       CN=lldap.lldap.svc.cluster.local
<90d  lldap          lldap-web-tls                      tls.crt   Nov 16 18:14:36 2026 GMT       CN=accounts.vluwte.nl
<90d  longhorn-system longhorn-tls                       tls.crt   Dec  1 17:17:20 2026 GMT       CN=longhorn.bletchley.vluwte.nl
OK    longhorn-system longhorn-webhook-ca                tls.crt   Feb 23 19:39:21 2036 GMT       O=dynamiclistener-org, CN=dynamiclistener-ca@1772051961
OK    longhorn-system longhorn-webhook-tls               tls.crt   Sep 11 19:40:34 2027 GMT       O=dynamic, CN=dynamic
OK    metallb-system frr-k8s-webhook-server-cert        ca.crt    May 22 13:15:07 2036 GMT       O=frrk8s, CN=cert
OK    metallb-system frr-k8s-webhook-server-cert        tls.crt   May 25 13:15:07 2027 GMT       CN=frr-k8s-webhook-service.metallb-system.svc
OK    metallb-system metallb-webhook-cert               ca.crt    Mar  1 19:18:38 2036 GMT       O=metallb, CN=cert
OK    metallb-system metallb-webhook-cert               tls.crt   Mar  4 19:18:38 2027 GMT       CN=metallb-webhook-service.metallb-system.svc
<90d  monitoring     alertmanager-tls                   tls.crt   Oct 24 11:04:37 2026 GMT       CN=alertmanager.bletchley.vluwte.nl
<90d  monitoring     grafana-tls                        tls.crt   Dec  1 17:15:41 2026 GMT       CN=grafana.bletchley.vluwte.nl
<90d  monitoring     prometheus-tls                     tls.crt   Nov 21 06:09:03 2026 GMT       CN=prometheus.bletchley.vluwte.nl
OK    openbao        openbao-transit-ca                 ca.crt    Apr  6 10:44:54 2029 GMT       CN=openbao-transit.luwte.net
<90d  traefik        traefik-dashboard-tls              tls.crt   Oct 18 07:17:31 2026 GMT       CN=traefik.bletchley.vluwte.nl

Four checks fall out of this. All four close.

The split is 13 and 18. 42 keys across 31 Secrets. Thirteen of those Secrets have no Certificate object. The other 18 are exactly the cert-manager set, one Secret per Certificate. Every expiry date in the table of 13 above is a line in this output.

The key counts explain the four-row secretTypes block. 27 tls.crt keys, 15 ca.crt keys, and 11 Secrets carrying both. Four Secrets have a ca.crt and nothing else: cnpg-ca-secret, bletchley-pg-ca, postgres-vluwte-ca and openbao-transit-ca. Those four alone are what the chart's default would have dropped.

The 22 missing chain certificates sit in 11 lines. The eleven Let's Encrypt Secrets each appear once, with only a tls.crt. openssl x509 read the leaf in each and stopped. The intermediate and the cross-signed root behind those eleven lines are the 22 the exporter found and this output cannot show.

The dashboard's Top Issuers panel checks the whole thing from the other direction. It counts by issuer rather than by Secret, across both sources. Nothing in it depends on my arithmetic:

Read the cnpg-ca-secret row again. 11 is 1 plus 1 plus 9: the one certificate in ten places, arrived at without counting webhook entries by hand.

The cert row is the other lesson on that panel. Two unrelated CAs, MetalLB's and frr-k8s's, both with CN=cert, add up into one row. A common name is not an identity. That is exactly why the renewal-detection query groups on a Secret's own coordinates and not on a name.

The 18 Certificates, with their issuers

Read from the cluster, not inferred:

NS                NAME                                       SECRET                                     ISSUER                                  KIND            RENEWBEFORE
authelia          authelia-ca-trust                          authelia-ca-trust                          internal-ca-issuer                      ClusterIssuer   720h
authelia          authelia-tls                               authelia-tls                               letsencrypt-production                  ClusterIssuer   <none>
cert-manager      cert-manager-webhook-transip-ca            cert-manager-webhook-transip-ca            cert-manager-webhook-transip-selfsign   <none>          <none>
cert-manager      cert-manager-webhook-transip-webhook-tls   cert-manager-webhook-transip-webhook-tls   cert-manager-webhook-transip-ca         <none>          <none>
cert-manager      internal-ca-root                           internal-ca-root                           internal-ca                             ClusterIssuer   8760h
cnpg-system       barman-cloud-client                        barman-cloud-client-tls                    selfsigned-issuer                       Issuer          360h
cnpg-system       barman-cloud-server                        barman-cloud-server-tls                    selfsigned-issuer                       Issuer          360h
databases         postgres-vluwte-tls                        postgres-vluwte-tls                        letsencrypt-production                  ClusterIssuer   <none>
forgejo           forgejo-tls                                forgejo-tls                                letsencrypt-production                  ClusterIssuer   <none>
garage            garage-s3-tls                              garage-s3-tls                              letsencrypt-production                  ClusterIssuer   <none>
it-tools          it-tools-tls                               it-tools-tls                               letsencrypt-production                  ClusterIssuer   <none>
lldap             lldap-ldap-tls                             lldap-tls                                  internal-ca-issuer                      ClusterIssuer   720h
lldap             lldap-web-tls                              lldap-web-tls                              letsencrypt-production                  ClusterIssuer   <none>
longhorn-system   longhorn-tls                               longhorn-tls                               letsencrypt-production                  ClusterIssuer   <none>
monitoring        alertmanager-tls                           alertmanager-tls                           letsencrypt-production                  ClusterIssuer   <none>
monitoring        grafana-tls                                grafana-tls                                letsencrypt-production                  ClusterIssuer   <none>
monitoring        prometheus-tls                             prometheus-tls                             letsencrypt-production                  ClusterIssuer   <none>
traefik           traefik-dashboard-tls                      traefik-dashboard-tls                      letsencrypt-production                  ClusterIssuer   <none>

Three things are visible here and nowhere else in this post.

Six issuers, three of them selfSigned bootstraps. letsencrypt-production signs the 11 public hostnames. internal-ca-issuer signs two internal certificates, and it is backed by internal-ca-root, which the selfSigned internal-ca ClusterIssuer minted. The TransIP webhook runs the same two-step bootstrap, with cert-manager-webhook-transip-selfsign minting the CA that then signs the webhook's own certificate. And selfsigned-issuer in cnpg-system signs the Barman pair.

That last one explains a duplicate. A selfSigned issuer produces a certificate that signs itself, and cert-manager writes the same certificate into ca.crt as well as tls.crt. So both Barman Secrets hold one certificate under two keys, which is the collision the exporter keeps as two series. Certificate Management Revisited is where that distinction first bit me.

The renewBefore column is the entire lead-time table. Thirteen blanks, so thirteen defaults at a third of the lifetime. Five explicit values: 360h twice, 720h twice, 8760h once.

And the names do not line up. barman-cloud-client writes into barman-cloud-client-tls. lldap-ldap-tls writes into lldap-tls. Any mapping from a Certificate to a Secret has to go through spec.secretName.

Two small corrections while I was in here. The leaf in cert-manager-webhook-transip-webhook-tls has no CN at all, which is why the lead-time table names the Secret instead of a subject. And my inventory note had the MetalLB CA expiring 4 March 2036; the certificate says 1 March. The leaf date it alerts on, 4 March 2027, was right.

← Previous: The Wear That Wasn't

Questions or suggestions? Leave a comment below or reach out at igor@vluwte.nl.