Bringing the Registry Home: Retiring the Insecure Registry Patch

Moving my container registry onto Bletchley behind a Let's Encrypt certificate, so no Talos node needs an insecure registry patch ever again.

Share

Introduction

Bletchley runs two images I build myself: the Longhorn backup controller and the eMMC wear exporter. Until last month both lived in a plain registry:2 container on docker.luwte.net, served over HTTP on port 5000. containerd will not talk plain HTTP to a registry unless it is told to, so every Talos node carried a small patch that said "this one is fine, skip the TLS".

That patch has bitten me twice. In The Backup That Wasn't, rock5, rock6 and rock7 had joined the cluster without it, the nightly PostgreSQL backup pod landed on rock5, and it sat in ImagePullBackOff for eight days. In October rock11 joined without it too. That time I caught it before anything needed to pull, but it was the same mistake made a second time, by the same person, with the same checklist in front of him.

A step that has to be remembered on every node will eventually be forgotten. So the goal this time was not a better checklist. It was to move the registry onto the cluster, put it behind Traefik with a Let's Encrypt certificate like every other service, and end up with nodes that need no registry configuration at all.

Along the way I found three traces of an earlier attempt to do exactly this, one of which had been quietly pointing at my PostgreSQL load balancer for two months.


🏠 This is part of the Homelab Journey series - building a production Kubernetes cluster from scratch.


Why the Registry Needed a Patch at All

containerd assumes every registry speaks HTTPS with a certificate it can verify. My registry on docker.luwte.net spoke HTTP. The fix lived in infra/talos/registry-patch.yaml and was applied to each node:

# infra/talos/registry-patch.yaml
machine:
  registries:
    mirrors:
      10.0.0.80:5000:
        endpoints:
          - http://10.0.0.80:5000
    config:
      10.0.0.80:5000:
        tls:
          insecureSkipVerify: true

There is nothing wrong with this file. The problem is where it lives. It is not part of anything Kubernetes knows about. No manifest depends on it, no controller reconciles it, and nothing fails until a pod that needs one of my images happens to be scheduled on a node that does not have it. That is a very quiet way to break.

It also has a cost I had not thought about until now: Talos deprecated .machine.registries in 1.12 in favour of separate RegistryMirrorConfig and RegistryTLSConfig documents. Keeping the patch would have meant rewriting it before the next upgrade round. Removing it means there is nothing to rewrite.


Choosing Where It Goes

I looked at more options than I needed to, mostly because Zot's pull-through cache is genuinely appealing. The table is where it landed.

Option What it does Why it did or didn't win
A: registry:2 in-cluster, HTTPS via Traefik The same registry:2 image as today, as a Deployment with a Longhorn PVC and an Ingress at registry.vluwte.nl with a cert-manager Let's Encrypt certificate. Chosen. Same software I already run, deployed like every other app. A publicly trusted certificate means containerd needs no Talos config at all. Costs a DNS record, an Ingress, and a cold-start dependency on Traefik.
B: registry:2 in-cluster, plain HTTP on its own MetalLB IP Same Deployment behind a LoadBalancer, with the Talos patch pointed at the new address. The most literal move, and it keeps the per-node patch: the thing that already failed twice. Kept as the fallback if HTTPS through Traefik gave trouble.
C: Forgejo's package registry The OCI registry built into Forgejo at git.vluwte.nl. Tried in May and had issues. A different piece of software from the one that works today.
D: Zot, with pull-through cache An OCI-native registry that can also cache Docker Hub, ghcr.io and others on demand. A cache only works if every node is told to use it: one mirror document per upstream registry, on every node. That brings back exactly the kind of per-node config I was trying to remove. Not a goal right now.
E: Harbor Projects, scanning, replication, proxy cache. Far too much for two images.
F: Stay on docker.luwte.net Status quo. Blocks retiring that host, which is the larger plan.

The deciding question was simple: can a node pull my images without any registry entry in its Talos config? Option A is the simplest answer that says yes. The software does not change, only where it runs and what certificate it presents.


Clearing Up the First Attempt

Before deploying anything I wanted to know exactly what each node had. I knew there had been an earlier, abandoned attempt to run a registry on Bletchley. I did not know how much of it was still around.

A stale mirror on one node

for n in rock1 rock2 rock3 rock5 rock6 rock7 rock9 rock11; do
  echo "== $n"; talosctl -n $n get mc -o jsonpath='{.spec}' | yq '.machine.registries'
done

rock4 is not in use, so it is not in the loop. Every node showed the 10.0.0.80 entry. rock1 showed something extra:

== rock1
mirrors:
  10.0.0.80:5000:
    endpoints:
      - http://10.0.0.80:5000
  10.0.140.101:5000:
    endpoints:
      - http://10.0.140.101:5000
    overridePath: true
config:
  10.0.0.80:5000:
    tls:
      insecureSkipVerify: true
  10.0.140.101:5000:
    tls:
      insecureSkipVerify: true

10.0.140.101 was the address of that earlier in-cluster registry. Since I exposed CloudNativePG through MetalLB, it has been bletchley-pg-external, the PostgreSQL load balancer. So rock1 had a registry mirror pointing at a database. Harmless in practice, because nothing pulled from that address any more:

kubectl get pods -A -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' \
  | grep 10.0.140.101

That returned nothing. But it was misleading, and it made rock1 different from every other node for no reason. It was also never in git: registry-patch.yaml only ever held the 10.0.0.80 entry. Someone (me) had edited rock1's live config by hand and never written it down.

The overridePath: true line is interesting in its own right. Talos describes it as "use the endpoint path as supplied, without adding /v2/". It exists for registries like Harbor or Zot, where the URL already carries a path such as /v2/<project>. A stock registry:2 serves everything under /v2/, so with overridePath: true containerd would request /<repo>/manifests/<tag> instead of /v2/<repo>/manifests/<tag> and get a 404. I never went back to confirm it, but I am fairly sure that one line is why the first attempt never worked. The working 10.0.0.80 entry does not set it.

Two config resources, not duplicated config

The same loop printed the registry block twice for rock2, rock6 and rock9, and once for everyone else. That looked like drift until I listed the config resources:

talosctl -n rock2 get mc
NODE    NAMESPACE   TYPE            ID           VERSION
rock2   config      MachineConfig   persistent   1
rock2   config      MachineConfig   v1alpha1     1
talosctl -n rock3 get mc
NODE    NAMESPACE   TYPE            ID         VERSION
rock3   config      MachineConfig   v1alpha1   1

Some nodes have two MachineConfig resources, persistent and v1alpha1, and get mc without an ID returns all of them. Two copies of the same config, not two configs. The fix for the check is to ask for the active one only:

talosctl -n $n get mc v1alpha1 -o jsonpath='{.spec}' | yq '.machine.registries'

Removing it

I removed the stale entry with talosctl edit mc, dry run first:

talosctl -n rock1.vluwte.nl edit mc --dry-run
@@ -45,15 +45,8 @@
             10.0.0.80:5000:
                 endpoints:
                     - http://10.0.0.80:5000
-            10.0.140.101:5000:
-                endpoints:
-                    - http://10.0.140.101:5000
-                overridePath: true
         config:
             10.0.0.80:5000:
-                tls:
-                    insecureSkipVerify: true
-            10.0.140.101:5000:
                 tls:
                     insecureSkipVerify: true

Read quickly, this diff looks like it removes the TLS setting for 10.0.0.80. It does not. The two tls blocks are identical, and the diff simply lined up the second one. What goes is the 10.0.140.101 key and one copy of the block beneath it. The real edit applied without a reboot, and both of rock1's config resources came back identical to every other node.

While I was in there I took stock of the old registry:

curl -s http://10.0.0.80:5000/v2/_catalog
{"repositories":["backup-controller","emmc-exporter","postgres-backup","postgres-backup-runner"]}

Four repositories, 204 MB on disk. Only two are in use: backup-controller:0.11.0 and emmc-exporter:0.1.1. The two postgres-backup images were orphaned when CloudNativePG took over database backups, and they were not coming along.


Deploying registry:2 on Bletchley

The registry gets its own directory, apps/registry/, with the same shape as every other app: namespace, PVC, Deployment, Service, Certificate and Ingress.

Running as non-root

The stock registry:2 image runs as root by default, but it does not need to. With a fixed UID and an fsGroup so the Longhorn volume is writable, the namespace can enforce the restricted Pod Security level:

# apps/registry/deployment.yaml (excerpt)
spec:
  replicas: 1
  strategy:
    type: Recreate            # RWO Longhorn PVC
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: registry
          image: registry:2.8.3
          env:
            - name: REGISTRY_STORAGE_DELETE_ENABLED
              value: "true"
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: data
              mountPath: /var/lib/registry
          readinessProbe:
            httpGet:
              path: /v2/
              port: http

REGISTRY_STORAGE_DELETE_ENABLED is there from day one. The old registry did not have it, which is why the orphaned images could never be cleaned up there without a restart.

Storage and backup

204 MB fits comfortably in 5 GiB, with room for whatever I build next. The PVC carries the labels my backup controller looks for:

# apps/registry/pvc-registry.yaml
metadata:
  name: registry-data
  namespace: registry
  labels:
    backup.vluwte.nl/enabled: "true"
    backup.vluwte.nl/name: "registry"
    backup.vluwte.nl/interval: "24"   # images change rarely
spec:
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi

The controller picked it up on its own:

NAME                       GROUPS   TASK     CRON            RETAIN   CONCURRENCY   AGE
backup-registry-registry   []       backup   30 0/24 * * ?   8        1             18m

Certificate and Ingress

The certificate follows the same pattern as the rest of the cluster: an explicit Certificate from the letsencrypt-production ClusterIssuer, validated over DNS-01, as set up in the cert-manager post. registry.vluwte.nl exists only in internal DNS, pointing at Traefik on 10.0.140.100. It does not need to be publicly reachable to get a publicly trusted certificate.

The Ingress is deliberately not behind Authelia. containerd and Docker cannot follow a ForwardAuth redirect to a login page, so it would simply break every pull. That leaves the registry exactly as open as the old one: anyone on the internal network can push. Acceptable for now, and noted.

First start

registry-598cbcc64b-7gt7n   0/1     ContainerCreating   0          16s   <none>          rock11
registry-598cbcc64b-7gt7n   1/1     Running             0          22s   10.244.11.157   rock11

Ready in 22 seconds, with no warnings from the restricted namespace. That is a nice contrast with Calibre-Web, whose init system needs privileged. The log had one warning worth reading:

level=warning msg="No HTTP secret provided - generated random secret. This may cause problems
with uploads if multiple registries are behind a load-balancer."

The registry signs upload state with that secret. With several replicas behind a load balancer, an upload could hit a replica with a different secret and fail. With one replica, the only effect is that an upload in progress during a pod restart has to be retried. If this ever runs more than one replica, the secret goes into OpenBao. For now it stays as it is.

Then, from my workstation, with no -k:

curl -s https://registry.vluwte.nl/v2/
{}

An empty JSON object is what a healthy registry returns, and getting it without -k means the certificate verified against my Mac's ordinary trust store.


The Moment That Mattered

skopeo copies an image straight from one registry to another without a local Docker in between:

for img in backup-controller:0.11.0 emmc-exporter:0.1.1; do
  skopeo copy --src-tls-verify=false \
    docker://10.0.0.80:5000/$img docker://registry.vluwte.nl/$img
done

My first run printed zsh: command not found: # twice, because I had pasted a comment from my notes onto the for line. Interactive zsh does not treat # as a comment unless interactivecomments is set. It ran # as a command, failed, and carried on. Both copies completed: fifteen layers for the backup controller, one for the exporter. That also proved large uploads through Traefik work, which was one of the risks I had written down beforehand.

Then the test the whole plan was built around. A throwaway pod, pinned to rock11, pulling from the new registry:

kubectl run regtest --image=registry.vluwte.nl/emmc-exporter:0.1.1 --restart=Never \
  --overrides='{"spec":{"nodeName":"rock11"}}'
kubectl describe pod regtest | grep -A5 Events
Normal  Pulling  19s   kubelet  Pulling image "registry.vluwte.nl/emmc-exporter:0.1.1"
Normal  Pulled   19s   kubelet  Successfully pulled image "registry.vluwte.nl/emmc-exporter:0.1.1" in 62ms
Normal  Created  19s   kubelet  Container created

rock11 has never been told that registry.vluwte.nl exists. There is no entry for it in its Talos config, and there never will be. It pulled the image in 62 milliseconds because, as far as containerd is concerned, this is just another registry on the internet with a valid certificate.


Repointing the Consumers

With the old registry still running, each consumer could be switched one at a time and switched back by reverting a single line in git.

The eMMC exporter: every node at once

The exporter is a DaemonSet, so changing its image is a cluster-wide test for free. One pod per node, each one pulling from the new registry:

kubectl apply -f apps/monitoring/emmc-monitoring/emmc-exporter.yaml
kubectl -n monitoring get pods -o wide | grep emmc-exporter
emmc-exporter-9lfqm   1/1   Running   0   112s   10.244.1.5      rock1
emmc-exporter-gsfrp   1/1   Running   0   101s   10.244.11.159   rock11
emmc-exporter-k4fc5   1/1   Running   0   43s    10.244.2.9      rock5
emmc-exporter-kx65q   1/1   Running   0   66s    10.244.8.25     rock6
emmc-exporter-mxc4c   1/1   Running   0   78s    10.244.10.11    rock9
emmc-exporter-s8ppb   1/1   Running   0   55s    10.244.9.91     rock3
emmc-exporter-vpml5   1/1   Running   0   90s    10.244.3.140    rock2
emmc-exporter-xc4k6   1/1   Running   0   31s    10.244.6.49     rock7

Eight nodes, eight replacements, zero restarts. Including rock5, rock6 and rock7, the three nodes that started the whole story in May.

The backup controller: a third trace

The backup controller is the one I was most careful with, because it is the thing that guards everything else. kubectl diff showed the expected image change, and something I did not expect:

     kubectl.kubernetes.io/last-applied-configuration: |
       {..."image":"10.0.140.101:5000/backup-controller:0.1.0",...}
@@ -34,7 +34,7 @@
-        image: 10.0.0.80:5000/backup-controller:0.11.0
+        image: registry.vluwte.nl/backup-controller:0.11.0

The last-applied-configuration annotation still pointed at 10.0.140.101:5000 and version 0.1.0. The last time I had run kubectl apply on this Deployment was during that first registry attempt, most likely when I first deployed the controller in April. Every upgrade since, all the way to 0.11.0, had been done some other way. The live image and git agreed; the annotation kubectl uses to calculate changes was months out of date. This apply brought all three back in line, and it was the third trace of the first attempt: after the rock1 mirror and the reused IP address.

The new pod pulled from registry.vluwte.nl and was running in 9 seconds. Its first full reconcile was clean, including the registry's own new backup job:

[unchanged     ] registry/registry-data: job='backup-registry-registry' cron='30 0/24 * * ?' retain=8
[reconcile] complete — 8 valid, 0 skipped, 0 conflicts

One small thing showed up on the way out. The old pod ended in Error instead of Completed. The controller does not handle SIGTERM cleanly, so when Kubernetes stops it, it exits non-zero, most likely while its watch stream is mid-read. Harmless here, because the new pod was already serving, but it is a bug, and it is now on the list.

The real proof came that night at 00:30, when every backup job ran from the new image, including the registry's first backup of itself.


Retiring the Old Registry

Once both consumers had run for a while from the new registry, the old path could go:

  • The registry container on docker.luwte.net was stopped. Its /registry directory stayed on disk for a few weeks in case I needed to roll something back.
  • The 10.0.0.80:5000 entry was removed from every node. No node on Bletchley has any registry configuration now.
  • infra/talos/registry-patch.yaml was deleted from git, and its line was removed from the node provisioning checklist. The next node I add will not need it, and cannot forget it.
  • Docker Desktop on my workstation lost both insecure-registries entries. registry.vluwte.nl needs none.
  • The orphaned postgres-backup images went with the old registry, without ever being copied.

There is one loose end that comes with putting the registry inside the cluster: the backup controller's image now lives in the cluster it backs up. After a total loss, the registry would be gone before I could restore anything with the controller. The answer is that the image is reproducible, and the build is a single docker build. The source lives in Forgejo, which also runs on Bletchley, so on its own that would just move the problem one step back. What breaks the loop is that Forgejo's backups leave the cluster: to the NAS, and from the NAS to the cloud. After a total loss, the order is to restore the source from one of those copies, build the image, and push it to a fresh registry.


Lessons Learned

  1. Remove the step instead of remembering it. The insecure registry patch failed twice through the same human error. A better checklist would have reduced the odds. Removing the need for the patch reduced them to zero.
  2. A trusted certificate is configuration you never have to distribute. Let's Encrypt plus internal DNS gave every node trust in the registry without touching a single node.
  3. overridePath is for registries with a path in the URL. Harbor and Zot need it. A stock registry:2 breaks with it.
  4. talosctl get mc without an ID returns every config resource. Nodes with a persistent copy show everything twice. Ask for v1alpha1 when comparing nodes.
  5. Hand edits outlive the reason for them. The .101 mirror was never in git, so nothing ever reminded me to remove it. The stale last-applied annotation was the same problem in a different place.
  6. A DaemonSet is a free cluster-wide test. Changing the exporter's image proved every node could pull, in under two minutes.

What's Working Now

  • ✅ registry:2 running on Bletchley at registry.vluwte.nl, non-root under the restricted Pod Security level
  • ✅ Let's Encrypt certificate via cert-manager, trusted by containerd and Docker with no extra configuration
  • ✅ Registry storage on Longhorn, backed up daily by the backup controller
  • ✅ Backup controller and eMMC exporter pulling from the new registry on every node
  • ✅ No registry configuration on any Talos node, and no registry patch in git
  • ✅ The registry container on docker.luwte.net retired
  • ⚠️ Known limitation: no authentication on push, same as before

What's Next

Immediate:

  • Make the backup controller exit cleanly on SIGTERM.

Future:

  • Add htpasswd authentication for pushes, so only my workstation can publish images.
  • Revisit Zot if Docker Hub rate limits ever start to hurt. The per-node mirror config would then buy something real.
  • Keep moving the remaining services off docker.luwte.net.

← Previous: One Certificate in Ten Places


Questions or suggestions? Leave a comment below or reach out at igor@vluwte.nl.