Offloading Bletchley Backups: Traefik, Wildcard Certs, and rclone

Introduction

The previous post set up the local backup layer: Garage S3 on rock3's ZFS pool, Longhorn backing up every four hours, etcd snapshots landing in a second bucket. One copy, on the cluster itself. That's better than nothing, but it doesn't complete the 3-2-1 backup chain. If the cluster and rock3 both fail, the backups go with them.

The third copy needs to leave the cluster. The Synology NAS already handles encrypted offsite backups for other services — the goal was to extend that to the cluster backups with minimal new infrastructure. All the backup data is already in Garage S3. The Synology just needs a way to reach it.

This post covers that path: exposing Garage externally via a Traefik HTTPS ingress, issuing a wildcard Let's Encrypt certificate via the TransIP DNS-01 webhook, and running rclone on the Synology to pull both backup buckets on a daily schedule. There are a few detours along the way — Synology Cloud Sync turned out to be incompatible with Garage, and the wildcard cert became necessary for reasons that weren't obvious upfront.

🏠 This is part of the Homelab Journey series - building a production Kubernetes cluster from scratch.

• Backup Infrastructure
• Offloading Bletchley Backups: Traefik, Wildcard Certs, and rclone (you are here)
• Backup Validation

This post assumes Garage is running on the cluster and Longhorn is configured with a backup target pointing at it. It also assumes cert-manager is installed with the TransIP DNS-01 webhook configured. The previous two posts in this series cover both.

Why the Synology Can't Reach Garage Directly

Garage's S3 API is exposed inside the cluster as a ClusterIP service — garage-s3.garage.svc.cluster.local:3900. ClusterIP services are only reachable from inside the cluster. The Synology sits on the home network, not inside the Kubernetes pod network.

The options are: expose Garage via a LoadBalancer service (gets a MetalLB IP, reachable from the network), or expose it via Traefik with an Ingress (reachable via hostname, TLS included). The Traefik path was chosen because it follows the same pattern as every other externally reachable service on the cluster, gives a proper HTTPS endpoint, and the TLS certificate comes from Let's Encrypt automatically via cert-manager.

Exposing Garage via Traefik

DNS Record

First, add a DNS A record on the internal DNS server:

s3.bletchley.vluwte.nl → 10.0.140.100

This points at the MetalLB VIP — the same IP that Grafana, Longhorn, and every other ingress-routed service uses. Traefik routes by the Host header, so all services can share the same IP.

Ingress Manifest

The Ingress follows exactly the same pattern as the Grafana and Longhorn ingresses from the networking post:

# ingress-garage-s3.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: garage-s3
  namespace: garage
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
    traefik.ingress.kubernetes.io/router.middlewares: traefik-redirect-to-https@kubernetescrd
spec:
  ingressClassName: traefik
  rules:
    - host: s3.bletchley.vluwte.nl
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: garage-s3
                port:
                  number: 3900
  tls:
    - hosts:
        - s3.bletchley.vluwte.nl
      secretName: garage-s3-tls

The cert-manager.io/cluster-issuer annotation tells cert-manager to automatically issue a certificate for s3.bletchley.vluwte.nl using the letsencrypt-production ClusterIssuer. The certificate request, DNS-01 challenge, and secret creation all happen automatically.

This file lives in ~/talos-cluster/bletchley/garage/ — alongside the other Garage manifests, because the Ingress is part of the Garage service configuration.

Apply and watch the certificate issue:

kubectl apply -f ~/talos-cluster/bletchley/garage/ingress-garage-s3.yaml
kubectl get certificate -n garage -w

The certificate goes from False to True within about 90 seconds once the DNS TXT record propagates. If the challenge stalls at pending, check:

kubectl get challenge -n garage -w

Once the certificate is READY: True, verify the endpoint:

curl -v https://s3.bletchley.vluwte.nl 2>&1 | grep -E "SSL|issuer|subject|HTTP"

Expected output confirms TLS 1.3, a valid Let's Encrypt certificate, and a 403 response from Garage — which is the correct response for an unauthenticated request to the root endpoint.

The Synology Cloud Sync Problem

With the HTTPS endpoint working, the next step was configuring Synology Cloud Sync to pull both buckets. Cloud Sync supports S3-compatible storage and seemed like the natural choice — the Synology already uses it for other cloud providers.

The first gotcha: Cloud Sync expects a bare hostname without the https:// prefix. s3.bletchley.vluwte.nl, not https://s3.bletchley.vluwte.nl. The field label is "Server address" and it prepends the scheme itself.

With that corrected, Cloud Sync successfully listed both buckets — longhorn-backups and etcd-backups both appeared in the dropdown. Progress.

But Cloud Sync wouldn't advance past bucket selection. Watching the Traefik access logs revealed the problem:

"GET /?location= HTTP/1.1" 400

Cloud Sync was sending GetBucketLocation requests to the root endpoint — /?location= without a bucket name in the path. This is a malformed S3 request. Garage returns 400, Cloud Sync gets confused, and the setup wizard stalls.

The deeper issue is that Cloud Sync expects virtual-hosted style S3 addressing for some operations — it expects to reach longhorn-backups.s3.bletchley.vluwte.nl rather than s3.bletchley.vluwte.nl/longhorn-backups. Garage supports path-style addressing, but Cloud Sync's GetBucketLocation call was going to the wrong host with the wrong request format.

There's no way to configure path-style addressing in Cloud Sync's interface. The only fix is to make virtual-hosted style work — which means a wildcard DNS record and a wildcard certificate.

Adding Wildcard Support

Wildcard DNS Record

Add a second DNS record on the internal DNS server:

*.s3.bletchley.vluwte.nl → 10.0.140.100

This means longhorn-backups.s3.bletchley.vluwte.nl and etcd-backups.s3.bletchley.vluwte.nl both resolve to the MetalLB VIP.

Wildcard Certificate

The existing certificate only covers s3.bletchley.vluwte.nl. Wildcard certs can't be issued from an Ingress annotation — cert-manager can only issue wildcard certs from a standalone Certificate resource with DNS-01 challenge, because HTTP-01 validation can't prove control of a wildcard domain.

# garage-s3-wildcard-cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: garage-s3-wildcard-tls
  namespace: garage
spec:
  secretName: garage-s3-wildcard-tls
  issuerRef:
    name: letsencrypt-production
    kind: ClusterIssuer
  dnsNames:
    - "*.s3.bletchley.vluwte.nl"

Updated Ingress

The Ingress gets a second host rule and a second TLS entry:

spec:
  rules:
    - host: s3.bletchley.vluwte.nl
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: garage-s3
                port:
                  number: 3900
    - host: "*.s3.bletchley.vluwte.nl"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: garage-s3
                port:
                  number: 3900
  tls:
    - hosts:
        - s3.bletchley.vluwte.nl
      secretName: garage-s3-tls
    - hosts:
        - "*.s3.bletchley.vluwte.nl"
      secretName: garage-s3-wildcard-tls

Apply both:

kubectl apply -f ~/talos-cluster/bletchley/garage/garage-s3-wildcard-cert.yaml
kubectl apply -f ~/talos-cluster/bletchley/garage/ingress-garage-s3.yaml

Both certificates issue successfully and Traefik immediately picks them up. Even with the wildcard infrastructure in place, Cloud Sync still couldn't proceed — the GetBucketLocation 400 error persisted. Cloud Sync appears to have a fundamental incompatibility with Garage's handling of that API call, regardless of addressing style.

Rather than spending more time debugging Cloud Sync, the pragmatic decision was to switch tools.

rclone on the Synology

rclone is a command-line sync tool with first-class S3 support and explicit path-style addressing. It's available as a Docker container, which runs cleanly in Synology Container Manager without any native installation.

Credentials

Before configuring rclone, create a dedicated Synology key in Garage with access to both buckets:

kubectl exec -n garage garage-0 -- /garage key create synology-key
kubectl exec -n garage garage-0 -- /garage bucket allow \
  --read --write --owner longhorn-backups \
  --key synology-key
kubectl exec -n garage garage-0 -- /garage bucket allow \
  --read --write --owner etcd-backups \
  --key synology-key

Save the Key ID and Secret Key to 1Password.

Separate credentials per consumer is good practice — if the Synology key ever needs to be rotated or revoked, it doesn't affect Longhorn's access.

rclone Configuration

The rclone config file lives at /volume1/docker/rclone/config/rclone.conf on the Synology:

[garage]
type = s3
provider = Other
access_key_id = YOUR_KEY_ID
secret_access_key = YOUR_SECRET_KEY
region = garage
endpoint = https://s3.bletchley.vluwte.nl

The region = garage value must match the s3_region set in garage.toml. Without this, rclone sends us-east-1 in the auth header and Garage rejects the request with an AuthorizationHeaderMalformed error.

Test the connection:

rclone ls garage:longhorn-backups

Both buckets list their contents correctly.

Syncing

Test a full sync to confirm data flows:

rclone sync garage:longhorn-backups /bletchley/longhorn --progress
# Transferred: 39.553 MiB / 39.553 MiB, 100%, 17.747 MiB/s
# Transferred: 423 / 423, 100%

rclone sync garage:etcd-backups /bletchley/etcd --progress
# Transferred: 31.801 MiB / 31.801 MiB, 100%, 23.989 MiB/s
# Transferred: 2 / 2, 100%

423 objects from Longhorn, 2 encrypted etcd snapshots. Both synced in under 3 seconds.

Scheduled Task via Container Manager

The rclone Docker image has no scheduler — it's a sync tool, not a daemon. Scheduling is handled by Synology Task Scheduler running docker run commands.

The task script (in /volume1/docker/rclone/config/ for reference, run via Task Scheduler):

# Clean previous logfiles
cat /dev/null > /volume1/dump/Bletchley/rclone-longhorn.log
cat /dev/null > /volume1/dump/Bletchley/rclone-etcd.log

docker run --rm \
  -v /volume1/docker/rclone/config:/config/rclone \
  -v /volume1/dump/Bletchley:/bletchley \
  rclone/rclone \
  sync garage:longhorn-backups /bletchley/longhorn \
  --log-file /bletchley/rclone-longhorn.log \
  --log-level INFO

docker run --rm \
  -v /volume1/docker/rclone/config:/config/rclone \
  -v /volume1/dump/Bletchley:/bletchley \
  rclone/rclone \
  sync garage:etcd-backups /bletchley/etcd \
  --log-file /bletchley/rclone-etcd.log \
  --log-level INFO

# Mail the results
(echo "To: recipient@example.com
From: sender@example.com
Subject: Bletchley rclone results
Content-Type: text/plain

=== longhorn ===" && grep -v " INFO " /volume1/dump/Bletchley/rclone-longhorn.log \
  && echo "=== etcd ===" \
  && grep -v " INFO " /volume1/dump/Bletchley/rclone-etcd.log) | ssmtp recipient@example.com

The task runs daily at 05:00 — after the Longhorn recurring backup at 04:00 and the etcd backup CronJob at 02:00. After each run, the results of the jobs are mailed so there's visibility into whether the sync succeeded without having to SSH into the Synology. This will only be required if no data gets copied.

One operational note: Container Manager sends an "unexpected container stop" notification for every docker run --rm container that exits. This is because the container terminates after completing the sync, and Container Manager interprets any unplanned exit as unexpected. Since the only Docker containers on this Synology are these rclone sync tasks, the notification rule for container stop was disabled in Control Panel → Notification → Rules. The email from the task script provides the visibility instead.

Volume Mount Layout

The destination folder structure on the Synology:

/volume1/dump/Bletchley/
  longhorn/     ← longhorn-backups bucket contents
  etcd/         ← etcd-backups bucket contents
  rclone-longhorn.log
  rclone-etcd.log

The Complete Backup Chain

With rclone running, the full 3-2-1 backup chain is operational:

1. Snapshot — Longhorn creates a point-in-time snapshot of each PVC
2. Local backup — Longhorn backs up the snapshot to Garage S3 on rock3's SATA ZFS pool (every 4 hours, retain 4)
3. Offsite — rclone pulls both Garage buckets to the Synology at 05:00 daily → Synology HyperBackup pipelines to cloud S3 (encrypted)

For etcd/Talos state, talos-backup runs as a CronJob at 02:00, encrypts the snapshot with age, and writes it to the etcd-backups Garage bucket. rclone picks this up in the 05:00 daily sync.

Cleaning Up

rclone uses path-style addressing, so it reaches s3.bletchley.vluwte.nl/longhorn-backups directly — it never needed virtual-hosted style. The wildcard infrastructure added for Cloud Sync is now dead weight: unused DNS record, unused certificate, unused host rule in the Ingress. Left in place it would consume a Let's Encrypt rate limit slot, add noise to kubectl get certificate, and leave a *.s3.bletchley.vluwte.nl entry in Traefik routing that serves nothing.

Remove the wildcard certificate first:

kubectl delete certificate garage-s3-wildcard-tls -n garage
kubectl delete secret garage-s3-wildcard-tls -n garage

cert-manager stops managing it and the Let's Encrypt certificate is no longer renewed. The garage-s3-wildcard-cert.yaml file can be deleted from the repo.

Revert the Ingress to its original single-host form — remove the *.s3.bletchley.vluwte.nl host rule and the second TLS entry:

# ingress-garage-s3.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: garage-s3
  namespace: garage
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
    traefik.ingress.kubernetes.io/router.middlewares: traefik-redirect-to-https@kubernetescrd
spec:
  ingressClassName: traefik
  rules:
    - host: s3.bletchley.vluwte.nl
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: garage-s3
                port:
                  number: 3900
  tls:
    - hosts:
        - s3.bletchley.vluwte.nl
      secretName: garage-s3-tls

kubectl apply -f ~/talos-cluster/bletchley/garage/ingress-garage-s3.yaml

Finally, remove the wildcard DNS record from the internal DNS server. .s3.bletchley.vluwte.nl no longer needs to resolve anywhere.

Verify the cleanup:

kubectl get certificate -n garage
kubectl get ingress -n garage

Expected: one certificate (garage-s3-tls, READY: True), one Ingress with a single host rule. Clean.

What's Working Now

• ✅ s3.bletchley.vluwte.nl HTTPS endpoint with valid Let's Encrypt certificate
• ✅ Traefik Ingress routing s3.bletchley.vluwte.nl to Garage
• ✅ Dedicated synology-key Garage credentials for the Synology
• ✅ rclone pulling longhorn-backups and etcd-backups to Synology daily at 05:00
• ✅ Email notification after each sync run with full log output
• ✅ Synology HyperBackup already in place to pipeline to cloud S3
• ⚠️ Synology Cloud Sync is incompatible with Garage — abandoned. Wildcard infrastructure built for it has been removed.

What's Next

Both backup copies exist — one on Garage, one on the Synology, and from there to cloud S3. But backups are only worth as much as the last successful restore. The next post runs the actual test: stop Grafana, delete its PVC, restore from backup, and confirm the data is intact.

← Previous: Backup Infrastructure
→ Next: Backup Validation

Questions or suggestions? Leave a comment below or reach out at igor@vluwte.nl.