Certificate Management: cert-manager on the Bletchley Cluster

Introduction

The previous post ended with Grafana and Longhorn reachable via real hostnames on VLAN 140 — but still showing a “Not Secure” badge in the browser.
For a quick experiment HTTP is fine, but for a persistent cluster it isn’t. Credentials and metrics travel in plain text, and modern browsers treat HTTP as a warning rather than a default.

This post adds TLS across the cluster using cert-manager, the Kubernetes-native certificate management controller.

The setup here goes beyond the minimum. Two certificate authorities are configured: Let's Encrypt production for publicly-trusted certificates, and an internal CA for cluster-internal services that don't need public trust. A Let's Encrypt staging issuer is also configured — not as a curiosity, but as the correct way to validate the full setup before touching production rate limits. All certificate issuance happens via DNS-01 challenge against the TransIP API, which means the API key never leaves the cluster and there's no need to expose any service to the internet for validation.

By the end of this post, Grafana and Longhorn are both serving HTTPS with valid Let's Encrypt certificates, HTTP redirects automatically to HTTPS, and the cluster is ready to issue certificates for any new service with a single annotation.

🏠 This is part of the Homelab Journey series - building a production Kubernetes cluster from scratch.

• Cluster Networking: MetalLB and Traefik
• Certificate Management: cert-manager on the Bletchley Cluster (you are here)
• Backup Infrastructure

This post assumes MetalLB and Traefik are already running on the cluster, with Grafana and Longhorn accessible via hostname over HTTP. If you're starting from scratch, the earlier posts in this series cover each of those components.

How the Pieces Fit Together

Before installing anything, it helps to understand what goes where — especially when credentials are involved.

The key security principle: credentials stay in Kubernetes Secrets. Config files only reference secret names, never the values themselves.

The full stack is four components installed in sequence:

1. cert-manager — the core controller that manages certificates as Kubernetes resources
2. cert-manager-webhook-transip — a webhook that cert-manager calls to create DNS TXT records via the TransIP API
3. A Kubernetes Secret — holding the TransIP RSA private key
4. Three ClusterIssuers — letsencrypt-staging, letsencrypt-production, and internal-ca

The internal-ca issuer is prepared here but not yet used for any services. It's ready for future internal-only services that don't need publicly-trusted certificates — cluster dashboards, internal tools, anything that stays on VLAN 140.

Why DNS-01 and Not HTTP-01

Let's Encrypt offers two ways to prove you control a domain. HTTP-01 works by placing a file at a known URL on the domain — which requires the service to be reachable from the internet. DNS-01 works by creating a TXT record in the domain's DNS — which requires API access to the DNS provider but no inbound internet access at all.

The Bletchley cluster lives entirely on VLAN 140. None of its services are reachable from the internet. HTTP-01 is therefore not an option.

DNS-01 is also better for wildcard certificates. A single *.bletchley.vluwte.nl certificate covers every service on the cluster without a separate challenge per hostname. That's not used here — individual per-service certificates are issued instead — but the option exists.

The DNS provider for vluwte.nl is TransIP, which has an API that cert-manager can use via a webhook. The webhook is what bridges cert-manager's standard DNS-01 interface to TransIP's specific API format.

A Note on Let's Encrypt Rate Limits

Let's Encrypt production has rate limits — most relevantly, 5 duplicate certificates per week per domain. During initial setup, debugging, and configuration changes, it's easy to burn through those quickly without realising it.

The staging environment is the solution. It runs the same validation as production, issues certificates from an untrusted CA, and has much more generous rate limits. The correct workflow is:

1. Configure everything against staging first
2. Confirm the full DNS-01 challenge flow works end to end
3. Switch to production only once staging is confirmed

This is not just a precaution for initial setup — it's the right approach any time you're changing issuer configuration or adding new domains. Using staging until you're confident, then flipping to production, is the pattern used in production Kubernetes environments.

Installation

Working Directory

All cert-manager config files live in their own subdirectory, consistent with the pattern used for other components:

mkdir -p ~/talos-cluster/bletchley/cert-manager
cd ~/talos-cluster/bletchley/cert-manager

cert-manager

Add the Helm repo and check available versions:

helm repo add cert-manager https://charts.jetstack.io
helm repo update
helm search repo cert-manager/cert-manager --versions | head -5

The latest stable at the time of writing is v1.19.4. Pin to that.

cert-manager installs CRDs (Custom Resource Definitions) that the Kubernetes API needs before any Certificate or ClusterIssuer resources can be created. The crds.enabled: true setting in the values file handles this as part of the Helm install — no separate CRD installation step needed. There's also a change in cert-manager >= v1.18.0 worth noting: the default private key rotation policy changed from Never to Always, meaning cert-manager now generates a new private key on every certificate renewal. This is slightly more secure and is the correct default.

cat > cert-manager-values.yaml << 'EOF'
crds:
  enabled: true
extraArgs:
  - --dns01-recursive-nameservers=8.8.8.8:53,1.1.1.1:53
  - --dns01-recursive-nameservers-only
EOF

The two extraArgs entries deserve explanation — they're not obvious and they matter. cert-manager's DNS-01 propagation check runs inside the cluster, which means it queries internal DNS by default. On a split-DNS setup where internal DNS doesn't mirror the public zone, the TXT record that TransIP just created is invisible internally. cert-manager sees the propagation check fail and never submits the challenge to Let's Encrypt.

The fix is to tell cert-manager to use public resolvers (8.8.8.8 and 1.1.1.1) for its propagation check — the --dns01-recursive-nameservers flag sets which resolvers to use, and --dns01-recursive-nameservers-only tells cert-manager to skip internal DNS entirely. This requires, as I restrict outgoing DNS requests, a firewall rule allowing the cluster nodes (10.0.140.0/24) to reach 8.8.8.8:53 and 1.1.1.1:53 on UDP/TCP port 53. These flags affect only cert-manager's DNS propagation check — all other cluster DNS (CoreDNS, service discovery, pod lookups) continues using internal DNS as before.

kubectl create namespace cert-manager

helm install cert-manager cert-manager/cert-manager \
  --namespace cert-manager \
  --version v1.19.4 \
  --values cert-manager-values.yaml

Unlike Longhorn and MetalLB, cert-manager does not need the privileged pod security label — it doesn't touch host disks or devices. A standard namespace is sufficient.

Verify all three pods are running:

kubectl get pods -n cert-manager

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-75dcc595b6-pml4h              1/1     Running   0          29s
cert-manager-cainjector-8674b7cd5c-bw4n7   1/1     Running   0          29s
cert-manager-webhook-6fd77ccbb-mfzfr       1/1     Running   0          29s

Three pods: the controller, the CA injector, and the webhook. All three need to be Running before proceeding.

cert-manager-webhook-transip

The TransIP webhook extends cert-manager with the ability to create TXT records via the TransIP API. It's deployed into the same cert-manager namespace and communicates with cert-manager over the Kubernetes webhook mechanism.

helm repo add cert-manager-webhook-transip \
  https://demeesterdev.github.io/cert-manager-webhook-transip/
helm repo update
helm search repo cert-manager-webhook-transip --versions | head -5

Latest stable at time of writing: 0.1.4.

cat > cert-manager-webhook-transip-values.yaml << 'EOF'
groupName: acme.transip.nl

certManager:
  namespace: cert-manager
  serviceAccountName: cert-manager

secretName:
  - transip-secret
EOF

The serviceAccountName: cert-manager here refers to the Kubernetes service account used internally by cert-manager — not the TransIP account name. The TransIP account name goes in the ClusterIssuer config later.

helm install cert-manager-webhook-transip \
  cert-manager-webhook-transip/cert-manager-webhook-transip \
  --namespace cert-manager \
  --version 0.1.4 \
  --values cert-manager-webhook-transip-values.yaml

The TransIP Credentials Secret

The TransIP API uses key-pair authentication. The RSA private key is stored as a Kubernetes Secret — it goes into the cluster once and never needs to be placed on any node or copied to any host. The TransIP account name is not sensitive and does not go in the Secret; it appears as plain text in the ClusterIssuer config.

kubectl create secret generic transip-secret \
  --namespace cert-manager \
  --from-literal=privateKey="$(cat <<'EOF'
-----BEGIN RSA PRIVATE KEY-----
<your TransIP RSA private key here>
-----END RSA PRIVATE KEY-----
EOF
)"

Verify the Secret exists with one data key (the output shows the key name and byte count, not the value itself):

kubectl describe secret transip-secret -n cert-manager

A note on Kubernetes Secrets: they are base64 encoded, not encrypted. Anyone with kubectl access to the cluster can decode a Secret value trivially. The security model relies on Kubernetes RBAC controlling who can read Secrets, and on etcd access controls. For this homelab setup on an isolated VLAN with controlled access, this is an acceptable interim position — but it's a known gap. Encrypting Secrets properly, using a tool like Sealed Secrets, is a future improvement worth making before storing anything more sensitive than an API key.

ClusterIssuers

Three ClusterIssuers are created. The structure is identical across the two Let's Encrypt issuers — only the server URL and the Secret name for the ACME account key differ.

clusterissuer-letsencrypt-staging.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: your-email@example.com
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
      - dns01:
          webhook:
            groupName: acme.transip.nl
            solverName: transip
            config:
              accountName: your-transip-username
              ttl: 300
              privateKeySecretRef:
                name: transip-secret
                key: privateKey

clusterissuer-letsencrypt-production.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: your-email@example.com
    privateKeySecretRef:
      name: letsencrypt-production
    solvers:
      - dns01:
          webhook:
            groupName: acme.transip.nl
            solverName: transip
            config:
              accountName: your-transip-username
              ttl: 300
              privateKeySecretRef:
                name: transip-secret
                key: privateKey

clusterissuer-internal-ca.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca
spec:
  selfSigned: {}

The internal-ca issuer uses cert-manager's built-in self-signed CA capability. It's prepared here but not yet used for any services — it exists for future internal services that don't need publicly-trusted certificates. When that time comes, the only change needed is an annotation on an Ingress.

Apply all three:

kubectl apply -f clusterissuer-letsencrypt-staging.yaml
kubectl apply -f clusterissuer-letsencrypt-production.yaml
kubectl apply -f clusterissuer-internal-ca.yaml

Verify all three are ready:

kubectl get clusterissuer

NAME                     READY   AGE
internal-ca              True    5s
letsencrypt-production   True    10s
letsencrypt-staging      True    17s

All three READY: True. The ACME account registration with Let's Encrypt happened automatically when the issuers were created — cert-manager contacted the staging and production APIs and registered using the email address in the spec.

Testing Before Touching Production

With the infrastructure in place, the correct next step is to issue a test certificate against staging before touching any running services. This validates the full chain: cert-manager → TransIP webhook → TransIP API → DNS TXT record → Let's Encrypt validation → certificate issued.

cat > test-certificate.yaml << 'EOF'
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test-staging
  namespace: cert-manager
spec:
  secretName: test-staging-tls
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
  dnsNames:
    - test.bletchley.vluwte.nl
EOF

kubectl apply -f test-certificate.yaml
kubectl get certificate -n cert-manager -w

Within about 90 seconds, the certificate transitions to READY: True. The TransIP webhook created the _acme-challenge.test.bletchley.vluwte.nl TXT record, the public DNS resolvers confirmed it was visible, and Let's Encrypt staging validated and issued the certificate.

Once confirmed, clean up the test:

kubectl delete certificate test-staging -n cert-manager
kubectl delete secret test-staging-tls -n cert-manager

Adding TLS to Grafana and Longhorn

With staging proven, the Ingress resources for Grafana and Longhorn are updated to add TLS. Two additions are needed: a cert-manager.io/cluster-issuer annotation telling cert-manager which issuer to use, and a tls block telling Traefik which Secret to serve as the certificate.

The correct (and slightly unintuitive) procedure when switching a certificate from staging to production is: delete the existing Secret first, then apply the updated Ingress. Deleting the Secret forces cert-manager to issue a fresh certificate immediately; applying the Ingress first may leave cert-manager confused about whether it needs to act.

Staging first — ingress-grafana.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana
  namespace: monitoring
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-staging
spec:
  ingressClassName: traefik
  rules:
  - host: grafana.bletchley.vluwte.nl
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: grafana
            port:
              number: 80
  tls:
  - hosts:
    - grafana.bletchley.vluwte.nl
    secretName: grafana-tls

kubectl apply -f ingress-grafana.yaml
kubectl get certificate -n monitoring -w

READY: True in about 90 seconds. Staging confirmed. Switch to production:

# Delete staging secret first, then apply updated Ingress
kubectl delete secret grafana-tls -n monitoring
# Edit ingress-grafana.yaml: change letsencrypt-staging → letsencrypt-production
kubectl apply -f ingress-grafana.yaml

The same sequence for Longhorn:

kubectl delete secret longhorn-tls -n longhorn-system
# Edit ingress-longhorn.yaml: change letsencrypt-staging → letsencrypt-production
kubectl apply -f ingress-longhorn.yaml

Watch both certificates reach READY: True:

kubectl get certificate -A -w

Verify the production certificates are correctly issued:

kubectl describe certificate grafana-tls -n monitoring

Key fields to confirm:

• Issuer Ref: letsencrypt-production — correct issuer
• Status: True / Ready — certificate is valid
• Not After — 90-day validity from Let's Encrypt
• Renewal Time — cert-manager will auto-renew 30 days before expiry

HTTP to HTTPS Redirect

With TLS in place, plain HTTP should redirect to HTTPS rather than serving content. Traefik handles this via a Middleware resource.

cat > middleware-redirect-https.yaml << 'EOF'
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: redirect-to-https
  namespace: traefik
spec:
  redirectScheme:
    scheme: https
    permanent: true
EOF

kubectl apply -f middleware-redirect-https.yaml

The middleware is referenced in Ingress annotations using the format <namespace>-<middlewarename>@kubernetescrd. Add it to both Ingress files alongside the existing cert-manager annotation:

annotations:
  cert-manager.io/cluster-issuer: letsencrypt-production
  traefik.ingress.kubernetes.io/router.middlewares: traefik-redirect-to-https@kubernetescrd

Apply both updated Ingresses:

kubectl apply -f ingress-grafana.yaml
kubectl apply -f ingress-longhorn.yaml

Verify the redirect is working with curl, which shows the redirect without following it:

curl -v http://grafana.bletchley.vluwte.nl 2>&1 | grep -E "Location:|< HTTP"

< HTTP/1.1 301 Moved Permanently
< Location: https://grafana.bletchley.vluwte.nl/

301 Moved Permanently to the HTTPS URL. The redirect is permanent — browsers will cache it and go directly to HTTPS on subsequent visits.

What's Working Now

✅ cert-manager v1.19.4 installed
✅ cert-manager-webhook-transip v0.1.4 installed
✅ TransIP RSA private key stored as Kubernetes Secret
✅ letsencrypt-staging ClusterIssuer — for testing new certificate configs
✅ letsencrypt-production ClusterIssuer — for live certificates
✅ internal-ca ClusterIssuer — prepared for future internal-only services
✅ DNS-01 challenge flow validated end to end via staging
✅ Grafana — Let's Encrypt production certificate, auto-renewing
✅ Longhorn — Let's Encrypt production certificate, auto-renewing
✅ HTTP → HTTPS redirect on both services (301 Permanent)

Adding TLS to any new service on the cluster now takes two annotations and a tls block in the Ingress — cert-manager handles everything else.

Lessons Learned

1. The nameservers flag lives on the controller, not the ClusterIssuer. The --dns01-recursive-nameservers setting is a cert-manager controller flag set via extraArgs in the Helm values, not a field in the ClusterIssuer YAML. Attempting to add it to the ClusterIssuer spec results in a strict decoding error — the field simply doesn't exist there.
2. Split-DNS breaks the propagation check silently. cert-manager confirmed the DNS TXT record was created (the TransIP webhook worked), and dig @8.8.8.8 confirmed it was publicly visible — but cert-manager was querying internal DNS, where the record didn't exist. The challenge stayed pending indefinitely with no obvious error message pointing to the real cause. The fix is --dns01-recursive-nameservers-only combined with public resolver addresses.
3. Delete the Secret before switching issuers. When switching an Ingress annotation from letsencrypt-staging to letsencrypt-production, delete the existing TLS Secret first. cert-manager detects the missing Secret and issues a fresh certificate from the new issuer immediately. Applying the updated Ingress before deleting the Secret can result in cert-manager not acting, leaving the old staging certificate in place.
4. Always use staging first. The staging environment runs the same DNS-01 validation as production and exposes the same failure modes. Using it until the full chain is confirmed costs nothing and protects production rate limits. The five-duplicate-certificates-per-week limit is easy to exhaust during debugging without realising it.

What's Next

cert-manager is running and the two main cluster services have valid certificates. A few things are intentionally deferred:

The Traefik dashboard currently has no Ingress — it's accessible via IP only. Adding an Ingress with a cert-manager annotation follows exactly the same pattern as Grafana and Longhorn.

The internal-ca issuer is configured but unused. It's ready for future services on VLAN 140 that don't need publicly-trusted certificates — anything internal-only that should still be served over TLS without relying on Let's Encrypt.

Certificate extraction for external hosts is also possible via cert-manager. The TransIP API key stays in the cluster; cert-manager issues the certificate and stores it in a Kubernetes Secret; a script extracts it and copies it to the external host. This replaces the current manual process for vluwte.nl on sulu.luwte.net while keeping credentials off the host.

← Previous: Cluster Networking: MetalLB and Traefik
→ Next: Backup Infrastructure

Questions or suggestions? Leave a comment below or reach out at igor@vluwte.nl.