Certificate Management Revisited: My “Internal CA” Wasn't a CA

Introduction

Back in March I set up cert-manager on the Bletchley cluster and created three ClusterIssuers: letsencrypt-staging, letsencrypt-production, and one called internal-ca. The first two have been issuing certificates for public services ever since. The third has sat there, untouched, for 161 days.

What I wrote about it at the time was this:

The internal-ca issuer uses cert-manager's built-in self-signed CA capability. It's prepared here but not yet used for any services — it exists for future internal services that don't need publicly-trusted certificates. When that time comes, the only change needed is an annotation on an Ingress.

That last sentence is wrong, and I only found out because I finally had a use for it. CloudNativePG Part 5 deploys LLDAP as the first real workload on the new database cluster, and Authelia will authenticate against it over LDAPS — which means one program verifying another program's certificate, with no human around to wave a failed check through. internal-ca cannot do that. It was never able to do that.

This post is the correction: what a selfSigned issuer actually is, why it works fine for a browser and not at all for a service, and the two-object bootstrap that turns it into a real internal CA. It took about fifteen minutes to fix. Understanding why it was broken is the part worth writing down.

🏠 This is part of the Homelab Journey series - building a production Kubernetes cluster from scratch.

• CloudNativePG Part 4
• Certificate Management Revisited: Building a Real Internal CA (you are here)

This post assumes you already have cert-manager running. If you're starting from scratch, Certificate Management: cert-manager on the Bletchley Cluster covers the installation, the TransIP DNS-01 webhook, and the Let's Encrypt issuers.

What I Actually Needed

Authelia has been running since April against a file-based user database with Redis for sessions, both on a filesystem mount. CloudNativePG Part 5 replaces that with LLDAP — a proper directory, with its database on the new cluster. It's the first real workload to land there, and it's deliberately first: LLDAP starts empty, so there is no migration to get wrong.

That makes Authelia an LDAP client, and the hop between the two runs across the pod network. Encrypting it is the obvious thing to do while both are being built.

LDAPS is where the distinction bites. Authelia is a programmatic TLS client doing verified TLS: it opens a connection to LLDAP, is handed a certificate, and decides whether to trust it — before any traffic flows, with no user interaction. To make that decision it needs a trust anchor ahead of time. Something durable, mounted into the pod, that covers certificates it has never seen.

A browser hitting an Ingress works differently — not because it has a trust anchor, but because it doesn't need one. Verification fails there too. What the browser can do is stop, show the failure, and offer a human the chance to override it and continue anyway. Nothing durable is learned; the check is simply waived by someone who chose to waive it. That is the scenario my March note had in mind, and for that scenario the note is correct — an annotation genuinely is all it takes.

The assumption just doesn't survive contact with anything that verifies rather than merely presents.

What internal-ca Actually Was

The manifest is four lines:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca
spec:
  selfSigned: {}

A selfSigned issuer produces self-signed certificates: each one is signed with its own private key rather than by anything above it. That does not make them CA certificates. Unless a certificate carries CA:TRUE in its Basic Constraints and the usages that go with it, it is simply a self-signed leaf — an end-entity certificate that happens to have signed itself, and that cannot sign anything else.

Which is why "CA" in the name was doing no work. There was no CA anywhere in the picture: no certificate authorised to sign others, and therefore no shared anchor to hand a client — nothing that would cover a certificate the client hasn't already met. Every certificate that issuer produced stood alone, related to nothing else it had ever produced.

The obvious workaround is to mount ca.crt from a certificate issued in the client's own namespace. cert-manager populates that key on every certificate Secret, so it looks like it should work. For a selfSigned certificate, ca.crt is a copy of the leaf itself. You'd be handing Authelia a copy of a certificate that isn't the one LLDAP will present. I'll prove that later in the post rather than just asserting it.

None of which the cluster ever complained about. kubectl get clusterissuer reported READY=True on it for 161 days — I'll come back to that at the end, once there's something to compare it against.

The Two-Step Bootstrap

The fix is the standard cert-manager pattern, and it takes two objects:

1. Use the existing selfSigned issuer exactly once, to mint a real CA certificate (isCA: true). In this setup, that's the useful role for selfSigned: bootstrapping the CA certificate, and nothing after it.
2. Create a ca-type ClusterIssuer backed by that certificate's Secret. Everything internal issues from there, sharing one root.

The mildly embarrassing part: this cluster has been doing it correctly all along, in a corner I installed and never looked at again. The TransIP DNS-01 webhook ships with exactly this pair — cert-manager-webhook-transip-selfsign (selfSigned) bootstrapping cert-manager-webhook-transip-ca (ca). The pattern I needed was already running, three lines further down in kubectl get issuer.

Their Ready conditions are worth putting side by side, because the difference is the whole post in miniature:

# cert-manager-webhook-transip-selfsign
status:
  conditions:
  - reason: IsReady
    status: "True"
    type: Ready

# cert-manager-webhook-transip-ca
status:
  conditions:
  - message: Signing CA verified
    reason: KeyPairVerified
    status: "True"
    type: Ready

KeyPairVerified versus IsReady. cert-manager can meaningfully verify the second one — there's a signing keypair behind it to check. For the first, Ready means little more than "the object parses".

One pre-flight check first

A ca-type ClusterIssuer reads its Secret from cert-manager's cluster resource namespace, not from wherever you feel like putting it. Worth confirming before writing the manifest:

kubectl get deploy -n cert-manager cert-manager \
  -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n'

["--v=2"
"--cluster-resource-namespace=$(POD_NAMESPACE)"
...

Set explicitly, but to $(POD_NAMESPACE) — so it resolves to whichever namespace cert-manager runs in, which is cert-manager. Same outcome as the default, with the useful property that it would follow cert-manager if the deployment ever moved.

I also checked that nothing was currently issuing from internal-ca, which returned nothing, as expected after 161 days.

The manifest

Both objects in one file, deliberately:

# infra/certificates/internal-ca-bootstrap.yaml
---
# Step 1 of the bootstrap: use the existing selfSigned ClusterIssuer once, to
# mint a real CA certificate.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: internal-ca-root
  namespace: cert-manager      # must be cert-manager's cluster resource namespace
spec:
  isCA: true
  commonName: 'vluwte internal CA'
  secretName: internal-ca-root
  duration: 87600h             # 10 years
  renewBefore: 8760h           # 1 year
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: internal-ca          # the existing selfSigned ClusterIssuer
    kind: ClusterIssuer
    group: cert-manager.io
---
# Step 2: the issuer everything internal actually uses from here on. Leaves
# issued from this share one root, which is what makes ca.crt meaningful.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca-issuer
spec:
  ca:
    secretName: internal-ca-root

Ten years on the root because rotating it means redistributing trust to everywhere it has been mounted. renewBefore: 8760h gives a year of warning. ECDSA P-256 because it's smaller and universally supported by the TLS stacks involved.

Note that this is a single-tier PKI: the root signs leaves directly. Public CAs keep the root offline and issue through an intermediate, so a compromised signing key can be replaced without redistributing the anchor. At this scale the extra tier buys complexity rather than safety.

I left internal-ca alone because it is now intentionally the bootstrap issuer, but the name remains misleading. internal-ca-issuer is the one consumers should use. If I rename the bootstrap later, that will be a separate cleanup rather than part of this fix.

Applying It

kubectl apply -f infra/certificates/internal-ca-bootstrap.yaml

certificate.cert-manager.io/internal-ca-root created
clusterissuer.cert-manager.io/internal-ca-issuer created

kubectl wait --for=condition=Ready certificate/internal-ca-root -n cert-manager --timeout=60s
kubectl get clusterissuer internal-ca-issuer

certificate.cert-manager.io/internal-ca-root condition met

NAME                 READY   AGE
internal-ca-issuer   True    12s

Clean on the first apply — though if you're tailing the controller logs, don't be alarmed by what goes past. Applying both objects together briefly produces Secret not found errors, because the ClusterIssuer can reconcile before the Certificate has created its Secret. cert-manager requeues the issuer and it becomes Ready once the Secret exists. Ordering inside the file doesn't matter.

Verifying the Root Is a Real CA

A certificate is only a CA if its Basic Constraints say so. Check, don't assume:

kubectl get secret internal-ca-root -n cert-manager \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/internal-ca-root.crt

openssl x509 -in /tmp/internal-ca-root.crt -noout -subject -issuer -dates
openssl x509 -in /tmp/internal-ca-root.crt -noout -text | grep -A2 'Basic Constraints'
openssl x509 -in /tmp/internal-ca-root.crt -noout -text | grep -A2 'Key Usage'

subject=CN=vluwte internal CA
issuer=CN=vluwte internal CA
notBefore=Aug 15 19:47:32 2026 GMT
notAfter=Aug 12 19:47:32 2036 GMT

            X509v3 Basic Constraints: critical
                CA:TRUE

            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign

CA:TRUE is the whole thing in two words. That is the field separating this root from everything internal-ca was producing.

Subject and Issuer are identical, and that is correct here. A root is always self-signed — there is nothing above it to sign it. This is where my March mistake actually lived: I'd read "self-signed" as a synonym for "not a real CA". It isn't. It describes who signed the certificate, not what the certificate is permitted to do. Basic Constraints answers the second question, and only that one determines whether leaves can chain to it.

The Secret itself has three keys — ca.crt, tls.crt, tls.key. For the root, ca.crt and tls.crt are the same certificate, which is unremarkable. What matters is what those same three keys mean on a leaf.

The Test That Actually Matters

Nothing so far proves the issuer is useful — only that the root is shaped like a CA. The test that matters is issuing an actual leaf from it and checking what lands in the Secret. A scratch namespace keeps it out of the way of anything real:

kubectl create namespace ca-test

# applied with: cat <<'EOF' | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ca-test-leaf
  namespace: ca-test
spec:
  secretName: ca-test-leaf-tls
  commonName: ca-test.ca-test.svc.cluster.local
  dnsNames:
    - ca-test.ca-test.svc.cluster.local
    - ca-test.ca-test.svc
  duration: 2160h
  renewBefore: 360h
  usages:
    - server auth
    - digital signature
    - key encipherment
  issuerRef:
    name: internal-ca-issuer
    kind: ClusterIssuer
    group: cert-manager.io

Deliberately shaped like the LDAPS certificate LLDAP will get: server auth usage, internal service DNS names, a 90-day lifetime renewed 15 days out. If it works here it works there.

kubectl wait --for=condition=Ready certificate/ca-test-leaf -n ca-test --timeout=60s

certificate.cert-manager.io/ca-test-leaf condition met

Then pull both halves of the Secret out and interrogate them. /tmp/internal-ca-root.crt is the root I extracted in the previous section:

kubectl get secret ca-test-leaf-tls -n ca-test -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/leaf.crt
kubectl get secret ca-test-leaf-tls -n ca-test -o jsonpath='{.data.ca\.crt}'  | base64 -d > /tmp/leaf-ca.crt

openssl x509 -in /tmp/leaf.crt -noout -subject -issuer
openssl verify -CAfile /tmp/internal-ca-root.crt /tmp/leaf.crt
diff /tmp/internal-ca-root.crt /tmp/leaf-ca.crt && echo "ca.crt matches root"
openssl x509 -in /tmp/leaf.crt -noout -text | grep -A1 'Subject Alternative Name'

subject=CN=ca-test.ca-test.svc.cluster.local
issuer=CN=vluwte internal CA

/tmp/leaf.crt: OK

ca.crt matches root

            X509v3 Subject Alternative Name:
                DNS:ca-test.ca-test.svc.cluster.local, DNS:ca-test.ca-test.svc

Signed by the root, verifies against the root, and its ca.crt is the root byte for byte. That last line is the property every future consumer depends on: any workload can mount ca.crt from its own namespace's certificate Secret and get a trust anchor that covers certificates it has never seen.

Which answers a question worth asking out loud, because it looks strange in the Part 5 manifests: why does Authelia need a certificate at all, when it's a client and never presents one? It doesn't, really — it needs the root, and ca.crt is the only route cert-manager offers for delivering it. cert-manager writes a Certificate's Secret into that Certificate's own namespace and nowhere else, so a certificate issued in authelia is simply the supported way to get the root into the authelia namespace. Its tls.crt and tls.key go unused; the certificate exists as a delivery mechanism for the third key in the Secret.

The alternatives are worse: copy the Secret across namespaces by hand and it goes stale at the next renewal, or install trust-manager and run another controller for what one Certificate object already does. Issuing a leaf nobody presents feels wasteful for about a minute, and then it stops mattering — it renews itself, and the root it carries stays correct.

The same test against the old issuer

Worth doing once, to see the failure rather than take my word for it. Same namespace, same manifest, one field changed:

 metadata:
-  name: ca-test-leaf
+  name: ca-test-selfsigned-leaf
   namespace: ca-test
 spec:
-  secretName: ca-test-leaf-tls
+  secretName: ca-test-selfsigned-leaf-tls
   commonName: ca-test.ca-test.svc.cluster.local
   ...
   issuerRef:
-    name: internal-ca-issuer
+    name: internal-ca          # the OLD selfSigned ClusterIssuer
     kind: ClusterIssuer
     group: cert-manager.io

Names aside, that's seven characters of real difference between a certificate that works and one that can't. It goes Ready just as fast:

kubectl wait --for=condition=Ready certificate/ca-test-selfsigned-leaf -n ca-test --timeout=60s

kubectl get secret ca-test-selfsigned-leaf-tls -n ca-test -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/ss-leaf.crt
kubectl get secret ca-test-selfsigned-leaf-tls -n ca-test -o jsonpath='{.data.ca\.crt}'  | base64 -d > /tmp/ss-leaf-ca.crt

openssl x509 -in /tmp/ss-leaf.crt -noout -subject -issuer
openssl x509 -in /tmp/ss-leaf.crt -noout -text | grep -A2 'Basic Constraints'

subject=CN=ca-test.ca-test.svc.cluster.local
issuer=CN=ca-test.ca-test.svc.cluster.local

            X509v3 Basic Constraints: critical
                CA:FALSE

Three lines, and the distinction from the top of the post is right there. Subject equals issuer, so it signed itself. CA:FALSE, so it is not a CA. Self-signed leaf, nothing more — and compare that to the root two sections up, where subject also equalled issuer but Basic Constraints said CA:TRUE. Same self-signing, opposite authority.

diff /tmp/ss-leaf.crt /tmp/ss-leaf-ca.crt && echo "ca.crt IS the leaf — there is no CA here"

ca.crt IS the leaf — there is no CA here

There it is, demonstrated rather than asserted: ca.crt on a selfSigned certificate is a copy of the certificate itself. Mounting it into Authelia would have handed it a duplicate of a certificate LLDAP isn't presenting.

And the pair of verify commands — identical invocation, same -CAfile, one leaf from each issuer:

openssl verify -CAfile /tmp/internal-ca-root.crt /tmp/leaf.crt
openssl verify -CAfile /tmp/internal-ca-root.crt /tmp/ss-leaf.crt

/tmp/leaf.crt: OK

CN=ca-test.ca-test.svc.cluster.local
error 18 at 0 depth lookup: self-signed certificate
error /tmp/ss-leaf.crt: verification failed

I'd expected error 20, unable to get local issuer certificate. Error 18 is DEPTH_ZERO_SELF_SIGNED_CERT, and it's the better message: OpenSSL isn't saying it looked for an issuer and came up empty. It's saying this certificate signed itself, there is nowhere to look.

One last one, which is the punchline for the whole Authelia framing:

openssl verify -CAfile /tmp/ss-leaf.crt /tmp/ss-leaf.crt

/tmp/ss-leaf.crt: OK

The certificate validates perfectly — provided you already trust the exact certificate you're validating. That's the circular step a human can perform manually when a browser presents a certificate warning: inspect the certificate, decide to override verification, and continue. A programmatic TLS client has no equivalent human interaction. It has to decide before first contact, from what it was given in advance — and a selfSigned issuer has nothing to give it but the certificate it hasn't met yet.

Then the scratch namespace goes, taking both Certificates and both Secrets with it:

kubectl delete namespace ca-test
rm -f /tmp/leaf.crt /tmp/leaf-ca.crt /tmp/ss-leaf.crt /tmp/ss-leaf-ca.crt /tmp/internal-ca-root.crt

All five /tmp files were public certificates, no key material, so that last line is tidiness rather than hygiene. The one file worth being careful with never left the cluster: tls.key in the internal-ca-root Secret is the private key for the root, and nothing in this post ever needed to read it.

The Part Nothing Warned Me About

Back to the promised comparison. Same command before the bootstrap and after it:

kubectl get clusterissuer -o wide

# before
NAME                     READY   STATUS                                                 AGE
internal-ca              True                                                           161d
letsencrypt-production   True    The ACME account was registered with the ACME server   161d
letsencrypt-staging      True    The ACME account was registered with the ACME server   161d

# after
NAME                     READY   STATUS                                                 AGE
internal-ca              True                                                           161d
internal-ca-issuer       True    Signing CA verified                                    16m
letsencrypt-production   True    The ACME account was registered with the ACME server   161d
letsencrypt-staging      True    The ACME account was registered with the ACME server   161d

READY is True on every row, before and after. The column you'd instinctively check never once distinguished a working issuer from one that couldn't do the job. Only STATUS moved — it renders the Ready condition's message, and a selfSigned issuer sets none, because there is nothing to verify. The new ca issuer has a signing keypair behind it, so cert-manager has something real to report.

Worth being precise about what that blank means: it is not cert-manager flagging a problem, it's cert-manager having nothing to say. The information was never hidden from me — it just wasn't information until I had a reason to want it.

The same holds a layer deeper. Both throwaway Certificates went Ready in well under a second. Both controller logs are structurally identical — Certificate must be re-issued → CertificateRequest Approved → Ready → oldStatus="False" status="True" — and neither contains a single warning. cert-manager was entirely content to issue a certificate that nothing in the cluster could verify.

Nothing in Kubernetes was ever going to tell me. The distinction only becomes visible one layer down at openssl, or at 3am when Authelia refuses to connect.

The AGE column does some unplanned narrative work too: 161d beside 16m is the five-month gap between writing a confident note and finding out it was wrong.

Cleaning Up the Documentation

infra/certificates/README.md described clusterissuer-internal-ca.yaml as "ClusterIssuer for internal CA certificates". That's now actively wrong: the object is the bootstrap, and using it for internal certificates is precisely the mistake being corrected. It got a new row for the bootstrap file, a corrected description, and a line saying to issue from internal-ca-issuer.

Fixing the objects without fixing the description would leave the trap armed for the next person to walk into it, who is me in five months.

What's Working Now

• ✅ internal-ca-root — a real CA certificate in the cert-manager namespace, CA:TRUE, ECDSA P-256, valid to August 2036
• ✅ internal-ca-issuer — a ca-type ClusterIssuer reporting Signing CA verified, for every internal service from here on
• ✅ Leaves issued from it verify against the root, and carry the root as ca.crt — mountable in any namespace, no trust-manager, no cross-namespace secret copying
• ✅ internal-ca retained as the one-time selfSigned bootstrap, and now documented as such
• ⚠️ Known limitation: the root is only trusted where its ca.crt is explicitly mounted. Nothing on my laptop or in a browser trusts it, by design — public-facing services stay on Let's Encrypt.

Lessons Learned

1. Test with the consumer's constraints, not your own. I validated the March setup the way a human validates things — by looking at it. Authelia doesn't get to look at it. The test that mattered took two openssl verify calls and would have caught this in March.
2. The pattern was already in the cluster. The TransIP webhook had been doing this correctly since day one. Reading the objects I install and never touch again is apparently a habit worth having.
3. Fix the documentation in the same pass. Correcting the manifests without correcting the prose that describes them leaves the misunderstanding intact for whoever reads it next.

What's Next

Immediate: CloudNativePG Part 5 was the thing that surfaced this, and it's now unblocked. LLDAP gets deployed against a fresh database on the cluster and an LDAPS certificate from internal-ca-issuer, and Authelia gets a second certificate issued in its own namespace purely so the root can be mounted from ca.crt into its trust directory — replacing the file-and-Redis backend it has been using since April.

Future: Every internal service that verifies rather than merely presents can now use the same anchor — service-to-service TLS, mail, backup endpoints. That was the stated purpose of internal-ca in March. It's finally true.

← Previous: CloudNativePG Part 4

Questions or suggestions? Leave a comment below or reach out at igor@vluwte.nl.