CloudNativePG Part 7: Turning Off the Thing You Built Four Months Ago
Deleting the legacy PostgreSQL instance took four commands. Proving it was safe to delete took a day and eleven read-only checks.
Introduction
Part 6 moved Umami's database onto the CloudNativePG cluster without losing a row. What it left behind is the state every migration passes through and few of them leave: two PostgreSQL services running, one of them serving nothing. The old shared StatefulSet in namespace postgres was still up, still being backed up nightly, still exposed on its own Traefik entrypoint, still holding a frozen copy of a database that had moved on without it. It went onto the cluster at the end of April and came off at the end of August, which is a short life for something with its own storage, its own backup system, its own exporter and its own entrypoint.
Deleting the remaining Kubernetes resources took four commands and a few seconds. Getting to the point where those four commands were defensible took a day, and that gap is what this post is about.
A teardown is not really a deletion task. It is a proof. The claim is that nothing depends on this any more, and until that claim has evidence behind it, deleting is guessing in a confident tone of voice. Building gives you feedback for free, because the thing works or it doesn't. A cluster is exactly as quiet after a correct deletion as after a catastrophic one, so the evidence has to be gathered in advance or not at all.
What follows is that proof, and mostly what made it harder than it looks: dependencies that no kubectl query reports, monitoring that outlives the thing it monitors, an ordering constraint hidden inside the teardown itself, and one shared component that had to be changed in passing. The commands and manifests are all in the addendum at the end, so the body sticks to the evidence rather than the keystrokes.
This post is part of the CloudNativePG sub-series.
- Part 5: lldap's Database Goes First
- Part 6: Migrating Umami
- Part 7: Turning Off the Thing You Built Four Months Ago
🏠 This is part of the Homelab Journey series - building a production Kubernetes cluster from scratch.
- Migrating Umami
- CloudNativePG Part 7: Turning Off the Thing You Built Four Months Ago (you are here)
This post assumes Umami's database has already been migrated onto CloudNativePG. If you're joining here, Part 6 is the migration this one cleans up after, and its diagram shows the layout everything below is removing from.
What the proof has to establish
Before anything destructive, I wrote down what "safe to delete" would have to mean, in terms specific enough to check:
- The instance holds no database anyone still uses, and no role that owns anything elsewhere.
- Nothing outside the instance still reads from it, writes to it, or holds a credential that only it can satisfy.
- Nothing that monitors it will be left pointing at an address that no longer answers.
- Umami, the one service that ever lived there, keeps serving throughout, and I can show that rather than assume it.
Eleven read-only checks answered those, run as one block with the output saved. It took about an hour, and it is the reason the rest of the day was boring.
The first answer was reassuring. The instance held one user database, umami, at 10 MB, one non-superuser role, also umami, and one client connection, which was the psql session asking the question. The shared instance built to host many databases ended its life having hosted exactly one.
The second answer was a small correction rather than a discovery, and it sets the tone for everything after. Both backup CronJobs already showed SUSPEND=True. I had suspended them during Part 6, before the split between these two posts was settled, so my plan for Part 7 still listed suspending them as step one. A plan describes the cluster you intended. The checks describe the cluster you have. Where they differ, the checks win, and the only way to know they differ is to look.
Dependencies that no kubectl query reports
The hard part of the proof is the second bullet, because the honest version of it is "nothing anywhere depends on this", and Kubernetes only knows about Kubernetes. Two of the four things this instance touched were not visible from the cluster at all.
A reader outside the cluster
The postgres-backups bucket in Garage held 14 objects and 1.4 MB. Who was reading them is a property of the object store, not of the cluster, so the object store is what you ask:
==== KEYS FOR THIS BUCKET ====
Permissions Access key Local aliases
R GK2b0a066525ce37bfec0e8bee synology-key
RW GK7787bd30cd867d4d0aeccdd1 postgres-backup
Two keys, not one. The read-only key belongs to the Synology, which pulls a copy off Garage on a schedule, set up back in the offloading-backups post. That job is deliberate and it was doing its job. The point is that nothing on the Kubernetes side references it, so a dependency check that stopped at kubectl would have returned one consumer and been wrong.
It also changes the order of what follows, because the Synology job is an rclone sync rather than an rclone copy. Sync mirrors deletions. My plan said to retire it first "or it starts erroring", which is true and completely misses the point: had it still been scheduled when I emptied the bucket, the next run would have faithfully propagated the deletion and wiped the copy that is the entire safety net for the retention month. The direction of a mirror is invisible until the source disappears.
A role shared by two namespaces
Umami's credential plumbing was spread across three namespaces, and one object in it was live. This is where my check phase went from ten checks to eleven, because the tenth came back ambiguous and I only noticed while reading it.
Listing every ExternalSecret and SecretStore looked conclusive. It wasn't, because every namespace names its store openbao, and the store name is not the thing that breaks. The OpenBao role behind it is, and answering that takes a second listing with one more column:
NS NAME ROLE
...
databases umami umami
postgres openbao postgres
umami openbao umami
Role postgres is used by exactly one store, so removing it is safe. But umami/openbao, which I was about to delete along with its namespace, shares role umami with databases/umami, which is the live tenant. Two stores, different names, different namespaces, one role, and nothing in the first listing hints at it. Two checks, one real answer, and the second one only got written because the first was read carefully rather than ticked off.
The same neighbourhood held two name traps, both of which come down to a suffix. secret/umami/db is the disposable rollback credential and secret/umami/db-credential is the live one, so the destructive name is the shorter one. ExternalSecret umami-db in namespace postgres is disposable, while umami-db-owner in namespace databases is live, so every command needs its -n.
With that established the deletions were unremarkable, and the evidence afterwards is the part that matters: bao read auth/kubernetes/role/umami still returns the role with policies [umami], and umami-db-owner in namespace databases reports SecretSynced with a fresh sync. Deleting a Kubernetes namespace cannot reach into OpenBao and remove a role, so that pair of commands was never the real danger. The danger is the tidy-up pass afterwards, when role/umami sits in a list of things that look like they belong to a service that no longer exists.
One more piece of evidence, from dropping the database itself:
pg_terminate_backend
----------------------
(0 rows)
Nothing was connected. The step I had written as the risky one had nothing to do, which is the cleanest possible confirmation that Part 6's cutover really did move the traffic.
Monitoring outlives the thing it monitors
The third bullet turned out to have two halves, and both of them are about the backup system I built for this instance rather than the instance itself.
The first half is ordering, and it is about my own code rather than anything upstream. The backup CronJob here is a script I wrote for this instance, and it enumerates databases at runtime: with zero user databases it dumps nothing, uploads nothing and exits 0, which my exporter then reports as a fresh successful backup. That is a bug in the script's success semantics, not a harmless edge case. It treats "nothing to do" and "did the job" as the same outcome, and publishes a healthy last-success timestamp for a backup that contains nothing at all.
I have known about it since I wrote it and never fixed it, so dropping the database with the job still scheduled would have reproduced it. Suspending first avoids it, and that is the whole reason the first teardown step comes before the second.
The second half is that suspending starts a clock. PostgresBackupMissing fires after 25 hours, and the last successful backup was 2026-08-30 03:00:07, so the alert I built for exactly this condition was due at roughly 04:05 the next morning. It would have been right to fire, since backups genuinely had stopped, but it is noise about a decision already made. That puts a real deadline on a teardown that otherwise has none. The rule came out with about eleven hours to spare.
Removing that system is three edits, not one. Deleting the exporter's Service kills the scrape target, but it does not remove the scrape job pointing at it, and it does not remove the two alert rules that read its metrics. So I went looking for what would notice, expecting an uncomfortable answer: is there a generic scrape-target-down rule that will start complaining the moment the exporter disappears?
There isn't. Every up == 0 rule in the stack is pinned to a named job, and none of them matches postgres-backup. Deleting the exporter traded one alert for none, silently. That is a real gap in my alerting, found while proving something unrelated, and it is on the todo list now rather than in this post.
Three blocks came out of prometheus-values.yaml, the scrape job and both alert rules, with GarageS3AuthErrors sitting immediately after them and staying. Verification was three greps with numbers I could state in advance: zero, one, zero.
The order the proof imposes
Two of the day's steps failed on the first attempt, and both failures were confirmation flags rather than mistakes with consequences. The first was Garage refusing key delete without --yes. Nothing happened, exit code 1, key intact, and while adding the flag I re-read my own ordering and found it backwards.
postgres-backup is the only RW key on that bucket, since the Synology's is read-only. It is therefore the only credential that can empty it. Garage refuses to delete a non-empty bucket with BucketNotEmpty (409), and its CLI has no object-delete verb at all, so emptying is an S3 operation that needs a pod carrying that key. Delete the key first and there is no way left to empty the bucket, which leaves an undeletable bucket and a key that exists only to serve it. Bucket first, key last.
That credential lives in secret/postgres-backup-s3, in namespace postgres, which the teardown deletes two steps later. So the bucket work has to finish before the namespace goes, and the constraint comes from the teardown itself rather than from anything I planned. A refusal that costs nothing is worth more than a command that works.
Changing shared infrastructure to remove one port
The riskiest step was not deleting anything of mine. It was removing the postgres entrypoint from Traefik, because that upgrade rewrites a Service that ten Ingress objects sit behind.
It failed the first time, for a reason nowhere on my risk list: helm upgrade traefik traefik/traefik with no --version resolves to the newest chart in the repo cache rather than the chart the release is running, and the newer schema rejects logs, a key the deployed values use. Helm refused before touching anything. Pinned to the deployed 40.2.0 it went through, and the Service came out with web and websecure only, three ports before and two after.
Then I probed all ten hosts, treating any HTTP status as a pass. Nine answered. The tenth, traefik.bletchley.vluwte.nl, returned 000, meaning no HTTP response at all.
The instinct is to assume you caused it. The cheapest way to find out is to diff the release you changed against the last one that worked:
10,16d9
< ports:
< postgres:
< expose:
< default: true
< exposedPort: 5432
< port: 5432
< protocol: TCP
The only difference between the last known-good revision and the current one is the block I meant to remove. That turns "did I break something?" into a one-line answer, and it is the single most useful command in the whole teardown.
The access log then named the actual fault in one line:
10.244.5.116 - - [30/Aug/2026:15:45:32 +0000] "GET / HTTP/1.1" 499 21 "-" "-" 11918 "traefik-traefik-dashboard-traefik-bletchley-vluwte-nl@kubernetes" "http://10.244.5.116:8000" 30ms
Client and backend addresses are both Traefik's own pod, and the router name ends @kubernetes rather than @kubernetescrd, so a duplicate plain Ingress is proxying the dashboard back into Traefik's web entrypoint to be routed again. Broken since that Ingress was created in March, already in my cluster todo with a fix worked out, and nothing to do with this teardown.
The deletion itself
With the proof complete, the destructive part is four commands: delete the StatefulSet, the PVC, the Service, then the namespace. Longhorn released and removed the PV on its own, leaving no orphan. The namespace took the remaining ExternalSecrets, the openbao SecretStore and the old failed-job history with it. It was silent and it took seconds, and there is no rollback past it.
The OpenBao footprint went next, two secret paths, the role and the policy. That output is worth reading carefully, because it flatters you. Success! Data deleted (if it existed) is the same message for a path that was never there. The checks are what established these paths were real and the role unshared. The output on its own proves nothing.
What's Working Now
The after-picture, next to the thing it replaced:
igor@granite bletchley % kubectl get ns postgres
Error from server (NotFound): namespaces "postgres" not found
igor@granite bletchley % kubectl -n databases get cluster
NAME AGE INSTANCES READY STATUS PRIMARY
bletchley-pg 28d 3 3 Cluster in healthy state bletchley-pg-1
One PostgreSQL where there were two, and it is the three-instance one.
- ✅ Namespace
postgresis gone, along with the StatefulSet, its 10Gi Longhorn volume, the Service and the IngressRouteTCP. - ✅ The backup system built alongside it is gone in all three of its parts, workloads, scrape job and alert rules, with
GarageS3AuthErrorsuntouched beside them. - ✅ The
postgres-backupsbucket and its RW key are deleted, four Garage keys remain, and the Synology holds its copy until October. - ✅ Traefik exposes
webandwebsecureonly, and nine of ten Ingress hosts answer. The tenth is a pre-existing dashboard loop unrelated to this work. - ✅ The shared
umamiOpenBao role and policy survived, andumami-db-ownerstill reportsSecretSyncedon its own hourly refresh. - ✅ Umami never stopped writing. The event counter reads 742 at migration, 746 before the drop, 747 after it, and 748 once the teardown was finished.
That counter is the through-line of Parts 6 and 7, and it is the fourth bullet of the proof. Four readings, one of them taken across a DROP DATABASE, and no downtime anywhere in between.
What I'd Do Differently Next Time
The proof held, and Umami never noticed. Six things about how I got there would change.
- Run the verification loop before the change, not only after. I wrote the ten-host
curlprobe as an after-picture, so whentraefik.bletchley.vluwte.nlcame back000I had nothing to compare it against and spent the next stretch establishing that a fault from March was not mine. A baseline taken five minutes earlier would have answered that instantly. Any verification worth running afterwards is worth running first. - Not run the fix for a problem that hasn't happened. My plan carried a
kubectl patch --field-manager=helmas a remedy for the field-manager conflict I hit in PostgreSQL on Bletchley: Moving the First Database to the Cluster, and during the failed upgrade I ran it preemptively. The conflict then never recurred, and I can no longer say whether that is because the earlier ownership transfer held or because I had just transferred it again. A remedy executed early is evidence destroyed. Next time it stays in the plan as a response, with the trigger written next to it. - Put the version pin in the plan, not in the muscle memory. Both Helm upgrades were written into the plan as commands. One had
--versionand worked; the other didn't and turned a one-line values change into an attempted chart upgrade, stopped by a schema check. The fix is not "remember to pin", it is that a plan step containinghelm upgradeis not finished until it carries the version the release is actually running. - Ask the non-Kubernetes systems first, and as a named step. My check phase found the Synology's read key on the backup bucket, but only because I ran
garage bucket infofor other reasons. The teardown template now gets an explicit step for dependents that live outside the cluster: object-store keys, DNS records, NAS jobs, anything holding a credential. It is the question most likely to be answered wrongly by confident reasoning. - Retire a monitor and its alert rules in one change. Suspending the backup CronJobs during Part 6 and removing
PostgresBackupMissinga day later left a 25-hour fuse burning across the gap for no benefit. Either both go together, or the plan carries the deadline in writing. It cleared by eleven hours, which is closer than I would like for something that was entirely predictable. - File the defect the moment the check phase finds it. The absence of any generic scrape-target-down alert came out of a grep I ran for a different purpose, and it sat in my head for the rest of the day before reaching the todo file that evening. A check phase generates findings that have nothing to do with the task; they need somewhere to go at the moment they appear, or they leave with the terminal session.
The pattern behind most of these is the same. My risk list predicted nothing that happened: the step billed as most dangerous went cleanly, and the two things that actually bit were a missing --yes and a missing --version. What saved the day was not foresight about which step was risky, it was cheap evidence gathered before and after each one. So next time I would spend less effort ranking the dangers and more on making every step answerable.
What's Next
Immediate: postgres.vluwte.nl currently resolves to an address that no longer answers on 5432. Part 8 covers the hostname move from pg.vluwte.nl, a publicly-trusted certificate for the CNPG endpoint, and Umami's sslmode=require.
Dated follow-ups: the retained Synology copy of the old dumps gets deleted in early October. CNPG's internal CA expires on October 31 and nothing currently alerts on it.
Carried forward: the missing generic scrape-target-down alert, and the traefik.bletchley.vluwte.nl routing loop, which predates all of this and now has a diagnosis to go with the existing todo entry. Two orphaned images remain in the local registry.
← Previous: Migrating Umami
Questions or suggestions? Leave a comment below or reach out at igor@vluwte.nl.
Addendum: Commands and Manifests
Everything that was run or removed, in execution order. The body of the post shows the evidence; this is the keystrokes, verbatim.
The check phase, C1 to C11
All read-only. Run as one block, output saved, before anything destructive.
C1. What the legacy instance actually holds:
kubectl -n postgres exec -i postgres-0 -- psql -U postgres -c '\l+'
kubectl -n postgres exec -i postgres-0 -- psql -U postgres -c '\du'
kubectl -n postgres exec -i postgres-0 -- psql -U postgres -c \
"SELECT datname, usename, client_addr, state FROM pg_stat_activity WHERE backend_type = 'client backend';"
C2. Everything in namespace postgres:
kubectl -n postgres get all,pvc,secret,externalsecret,secretstore,cronjob,job,ingressroutetcp
C3. Everything left in namespace umami:
kubectl -n umami get all,pvc,secret,externalsecret,secretstore,ingress,ingressroute
C4. Who else uses the Traefik postgres entrypoint:
kubectl get ingressroutetcp -A -o custom-columns=\
NS:.metadata.namespace,NAME:.metadata.name,ENTRYPOINTS:.spec.entryPoints
kubectl -n traefik get svc traefik -o jsonpath='{.spec.ports}' | jq
C5. The event count on CNPG, the reading repeated throughout:
kubectl -n databases exec -i pg-migrate -- env PGPASSWORD="$UMAMI_PG_PW" \
psql -h bletchley-pg-rw.databases.svc.cluster.local -U umami -d umami -At -c \
"SELECT count(*) || ' | ' || max(created_at) FROM website_event;"
C6. What is in the backup bucket:
kubectl run -it --rm debug \
--image=10.0.0.80:5000/postgres-backup-runner:1.0.1 \
--restart=Never -n postgres \
--env="AWS_ACCESS_KEY_ID=$(kubectl get secret -n postgres postgres-backup-s3 -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d)" \
--env="AWS_SECRET_ACCESS_KEY=$(kubectl get secret -n postgres postgres-backup-s3 -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 -d)" \
-- sh -c '
aws s3 ls s3://postgres-backups/ --recursive --summarize \
--endpoint-url http://garage-s3.garage.svc.cluster.local:3900
'
C7. The Garage bucket and its keys:
kubectl exec -n garage statefulset/garage -- /garage bucket info postgres-backups
kubectl exec -n garage statefulset/garage -- /garage key list
kubectl exec -n garage statefulset/garage -- /garage key info postgres-backup
C8. Alert rules that reference the exporter, and whether any generic up rule covers it:
kubectl -n monitoring get cm prometheus-server -o jsonpath='{.data.alerting_rules\.yml}' \
| grep -n -B3 -A8 'postgres_backup'
kubectl -n monitoring get cm prometheus-server -o jsonpath='{.data.alerting_rules\.yml}' \
| grep -n -B5 -A5 'up ==\|up{'
C9. Who reads the OpenBao postgres paths. The last two listings here look like they answer the question and don't, which is what C11 is for:
bao kv list secret/postgres
bao read auth/kubernetes/role/postgres
bao policy read postgres
kubectl get externalsecret -A -o custom-columns=\
NS:.metadata.namespace,NAME:.metadata.name,STORE:.spec.secretStoreRef.name
kubectl get secretstore -A
C10. The deployed Prometheus chart version, needed to pin the upgrade:
helm list -n monitoring
C11. Which OpenBao role each SecretStore actually uses. Added mid-check-phase, once C9's output showed that every namespace spells its store name openbao. This is the listing that answers the question C9 was asked:
kubectl get secretstore -A -o custom-columns=\
NS:.metadata.namespace,NAME:.metadata.name,ROLE:.spec.provider.vault.auth.kubernetes.role
The teardown, in order
R1. Suspend the backup jobs, before dropping anything:
kubectl -n postgres patch cronjob postgres-backup -p '{"spec":{"suspend":true}}'
kubectl -n postgres patch cronjob postgres-housekeeping -p '{"spec":{"suspend":true}}'
kubectl -n postgres get cronjob
R2. Drop the legacy database and role. Re-run C5 immediately before this:
kubectl -n postgres exec -i postgres-0 -- psql -U postgres -c \
"SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'umami';"
kubectl -n postgres exec -i postgres-0 -- psql -U postgres -c 'DROP DATABASE umami;'
kubectl -n postgres exec -i postgres-0 -- psql -U postgres -c 'DROP ROLE umami;'
R3. Retire the rollback credential, and confirm the live one survives. Note which of the two paths is which:
bao kv metadata delete secret/umami/db
bao kv get -format=json secret/umami/db-credential | jq '.data.data | keys'
R4. Remove Umami's legacy residue:
kubectl delete namespace umami
kubectl -n postgres delete externalsecret umami-db
kubectl -n postgres delete job init-umami-db
R5. Prove the shared OpenBao role survived. Run immediately after R4:
bao read auth/kubernetes/role/umami
kubectl -n databases annotate externalsecret umami-db-owner force-sync="$(date +%s)" --overwrite
kubectl -n databases get externalsecret umami-db-owner
Recreation command, if it had not survived:
bao write auth/kubernetes/role/umami \
bound_service_account_names=external-secrets \
bound_service_account_namespaces=external-secrets \
policies=umami ttl=1h
R6. Delete the backup workloads:
kubectl -n postgres delete cronjob postgres-backup postgres-housekeeping
kubectl -n postgres delete deployment postgres-backup
kubectl -n postgres delete service postgres-backup-metrics
R7. Remove the Prometheus config. Three blocks out of apps/monitoring/prometheus/prometheus-values.yaml.
The scrape job:
- job_name: postgres-backup
scrape_interval: 1h
static_configs:
- targets:
- postgres-backup-metrics.postgres.svc.cluster.local:8080
Both alert rules, with their comment block. GarageS3AuthErrors follows immediately after these in the file and stays:
# -- PostgreSQL Backups --------------------------------------------
# Monitors the postgres-backup exporter which reads backup state
# from Garage S3 hourly. PostgresBackupMissing fires when no
# successful backup has been recorded in 25 hours (90000s).
# PostgresBackupSizeDeviation fires when the dump size deviates
# significantly from the historical average — catches silent
# failures where a dump completes but produces an anomalous result.
- alert: PostgresBackupMissing
expr: time() - postgres_backup_last_success_timestamp_seconds > 90000
for: 5m
labels:
severity: warning
annotations:
summary: "PostgreSQL backup missing for {{ $labels.database }}"
description: "No successful backup in the last 25 hours for database {{ $labels.database }}"
# Postgres dump more than 20% smaller than previous day
- alert: PostgresBackupSizeDeviation
expr: |
(postgres_backup_dump_size_bytes offset 24h - postgres_backup_dump_size_bytes)
/ postgres_backup_dump_size_bytes offset 24h > 0.20
for: 5m
labels:
severity: warning
annotations:
summary: "PostgreSQL backup size deviation for {{ $labels.database }}"
description: "Dump for {{ $labels.database }} is more than 20% smaller than yesterday"
Apply and confirm:
helm upgrade prometheus prometheus-community/prometheus -n monitoring \
-f apps/monitoring/prometheus/prometheus-values.yaml --version 29.21.0
kubectl -n monitoring get cm prometheus-server -o jsonpath='{.data.alerting_rules\.yml}' | grep -c 'postgres_backup' # want 0
kubectl -n monitoring get cm prometheus-server -o jsonpath='{.data.alerting_rules\.yml}' | grep -c 'GarageS3AuthErrors' # want 1
kubectl -n monitoring get cm prometheus-server -o jsonpath='{.data.prometheus\.yml}' | grep -c 'postgres-backup' # want 0
R8. The bucket and the key, in four steps: retire the Synology sync, empty the bucket, delete the bucket, delete the key.
R8a, on the Synology, is the only step that happens outside the cluster. This is the postgres block removed from the rclone job, and the reason it has to go first is in its fifth line:
docker run --rm \
-v /volume1/docker/rclone/config:/config/rclone \
-v /volume1/dump/Bletchley:/bletchley \
rclone/rclone \
sync garage:postgres-backups /bletchley/postgres \
--log-file /bletchley/rclone-postgres.log \
--log-level INFO
The other buckets in that job stay, and so does synology-key itself; only its access to this one bucket disappears with the bucket. What the job already pulled stays behind at /volume1/dump/Bletchley/postgres, with rclone-postgres.log beside it, and that copy is the retention month. Deleting both is R8e, dated early October.
The rest runs against the cluster:
# R8b: empty the bucket, using the only RW key
kubectl run -it --rm debug \
--image=10.0.0.80:5000/postgres-backup-runner:1.0.1 \
--restart=Never -n postgres \
--env="AWS_ACCESS_KEY_ID=$(kubectl get secret -n postgres postgres-backup-s3 -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d)" \
--env="AWS_SECRET_ACCESS_KEY=$(kubectl get secret -n postgres postgres-backup-s3 -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 -d)" \
-- sh -c '
aws s3 rm s3://postgres-backups/ --recursive \
--endpoint-url http://garage-s3.garage.svc.cluster.local:3900
echo "--- remaining: ---"
aws s3 ls s3://postgres-backups/ --recursive --summarize \
--endpoint-url http://garage-s3.garage.svc.cluster.local:3900
'
# R8c: delete the bucket
kubectl exec -n garage statefulset/garage -- /garage bucket delete postgres-backups --yes
# R8d: delete the RW key, last
kubectl exec -n garage statefulset/garage -- /garage key delete GK7787bd30cd867d4d0aeccdd1 --yes
kubectl exec -n garage statefulset/garage -- /garage key list
R9. Confirm the instance is empty. Nothing returned means safe to proceed:
kubectl -n postgres exec -i postgres-0 -- psql -U postgres -At -c \
"SELECT datname FROM pg_database WHERE NOT datistemplate AND datname <> 'postgres';"
R10. Delete the TCP route:
kubectl -n postgres delete ingressroutetcp postgres
R11. Remove the Traefik postgres entrypoint. The block deleted from infra/networking/traefik/values-patch.yaml:
ports:
postgres:
expose:
default: true
exposedPort: 5432
port: 5432
protocol: TCP
Capture the before-state first, including plain Ingress objects, which the IngressRoute listing does not cover:
kubectl get ingressroute,ingressroutetcp -A -o custom-columns=\
NS:.metadata.namespace,NAME:.metadata.name,ENTRYPOINTS:.spec.entryPoints
kubectl get ingress -A
kubectl -n traefik get pods
helm list -n traefik
Then upgrade, pinned to the version helm list reported:
helm upgrade traefik traefik/traefik -n traefik \
-f infra/networking/traefik/traefik-values.yaml \
-f infra/networking/traefik/values-patch.yaml \
--version 40.2.0
Only if a field-manager conflict on .spec.externalTrafficPolicy actually appears:
kubectl patch service traefik -n traefik \
--type=merge --field-manager=helm \
-p='{"spec":{"externalTrafficPolicy":"Local"}}'
Verify. Any HTTP status is a pass; a connection failure or timeout is not:
kubectl -n traefik get svc traefik -o jsonpath='{.spec.ports}' | jq
kubectl -n traefik get pods
for h in auth.bletchley.vluwte.nl git.vluwte.nl s3.bletchley.vluwte.nl \
it-tools.vluwte.nl accounts.vluwte.nl longhorn.bletchley.vluwte.nl \
alertmanager.bletchley.vluwte.nl grafana.bletchley.vluwte.nl \
prometheus.bletchley.vluwte.nl traefik.bletchley.vluwte.nl; do
printf '%-40s ' "$h"
curl -s -o /dev/null -w '%{http_code}\n' -m 5 "https://$h"
done
And, when one of them fails, the was-it-me check:
diff <(helm get values traefik -n traefik --revision 13) \
<(helm get values traefik -n traefik --revision 14)
R12. Delete the StatefulSet and its storage. No rollback after this, and R8 must be complete first:
kubectl -n postgres delete statefulset postgres
kubectl -n postgres delete pvc data-postgres-0
kubectl -n postgres delete service postgres
kubectl delete namespace postgres
kubectl get pv pvc-07bc4f25-d702-4cd4-be3c-403a0fe8a24c
R13. Remove the OpenBao footprint. Role and policy umami are deliberately not touched:
bao kv metadata delete secret/postgres/superuser
bao kv metadata delete secret/postgres/backup-s3
bao delete auth/kubernetes/role/postgres
bao policy delete postgres
R14. What came out of the repo
Deleted, the whole apps/postgres/ tree:
apps/postgres/README.md
apps/postgres/namespace.yaml
apps/postgres/statefulset.yaml
apps/postgres/rbac/openbao-role.yaml
apps/postgres/_template/init-job-template.yaml
apps/postgres/_template/ADDING-A-DATABASE.md
apps/postgres/externalsecret/externalsecret-umami.yaml
apps/postgres/externalsecret/externalsecret-superuser.yaml
apps/postgres/externalsecret/externalsecret-backup-s3.yaml
apps/postgres/ingresses/ingressroutetcp.yaml
apps/postgres/jobs/init-umami.yaml
apps/postgres/service/service.yaml
apps/postgres/backup/cronjob-backup.yaml
apps/postgres/backup/cronjob-housekeeping.yaml
apps/postgres/backup/deployment-exporter.yaml
apps/postgres/backup/service-exporter.yaml
Also deleted: apps/umami/externalsecret.yaml, and both container sources under build/ (postgres-backup, the Go exporter, and postgres-backup-runner, the image behind the CronJobs and the debug-pod recipe above). Both stay in git history, which is not a reason to keep them in the tree; it does mean the debug-pod recipe no longer has an image, so it had to come out of my technical reference too.
Modified: apps/monitoring/prometheus/prometheus-values.yaml and infra/networking/traefik/values-patch.yaml. READMEs updated: apps/umami/README.md, infra/networking/README.md.
Then the sweep for anything that still points at a service that no longer exists:
grep -rn --exclude-dir=.git \
-e 'apps/postgres' -e 'postgres-backup' -e 'postgres\.postgres\.svc' \
-e 'postgres\.vluwte\.nl' -e 'entryPoints.*postgres' .
Final verification
# 1. The legacy instance is gone
kubectl get ns postgres
kubectl get all -n postgres
# 2. The shared umami role and policy survived R13
bao read auth/kubernetes/role/umami
bao policy read umami
bao kv get -format=json secret/umami/db-credential | jq '.data.data | keys'
# 3. The live tenant still syncs and still writes
kubectl -n databases get externalsecret umami-db-owner
kubectl -n databases exec -i pg-migrate -- env PGPASSWORD="$UMAMI_PG_PW" \
psql -h bletchley-pg-rw.databases.svc.cluster.local -U umami -d umami -At -c \
"SELECT count(*) || ' | ' || max(created_at) FROM website_event;"
# 4. One PostgreSQL on the cluster, not two
kubectl get pods -A | grep -i postgres
kubectl -n databases get cluster
One caveat on that last pair. kubectl get pods -A | grep -i postgres returning nothing is not the proof it looks like, because CNPG's pods are named bletchley-pg-* and would not match either way. kubectl -n databases get cluster is the line that actually carries the evidence.
← Previous: Migrating Umami
Questions or suggestions? Leave a comment below or reach out at igor@vluwte.nl.