PostgreSQL on Bletchley: First Database on the Cluster
Setting up a shared PostgreSQL 17 StatefulSet on Kubernetes, Traefik TCP routing, an OpenBao v2.5.3 surprise, and migrating the Umami database.
Introduction
Every cluster eventually needs a database. For Bletchley, that moment arrived as part of the Umami migration — the analytics database had been running in a Docker container on docker.luwte.net, outside the cluster, since the beginning. Moving it onto the cluster meant solving the more general problem first: how do you run PostgreSQL on Kubernetes in a way that's operationally manageable long-term?
The answer I settled on is a single shared PostgreSQL 17 StatefulSet — one instance, multiple databases, reachable from inside and outside the cluster. It mirrors the data.luwte.net model that already works well for the rest of the infrastructure. And it needed to be up and running before Umami could follow.
This post covers the design decision, how Traefik TCP routing works for exposing a PostgreSQL port, an unexpected detour through an OpenBao v2.5.3 breaking change, and the Umami migration itself. The backup system that came out of this work gets its own post.
🏠 This is part of the Homelab Journey series - building a production Kubernetes cluster from scratch.
- Prometheus Storage Overhead
- PostgreSQL on Bletchley: First Database on the Cluster (you are here)
This post assumes familiarity with the OpenBao secret management setup. If you're starting fresh, Secret Management Part 1 has the context.
Why a Shared Instance?
The first design question was whether to run one PostgreSQL instance per application or a single shared one. The per-app approach has clear isolation benefits — a misconfiguration in one database can't affect another, and you can tune each instance independently. But in a homelab running on four ARM64 nodes, that isolation comes at a real cost: every PostgreSQL StatefulSet needs its own Longhorn PVC, its own ExternalSecret, its own backup job. Two or three databases in and you're managing a small fleet of database servers.
The shared instance model is simpler and it already works in this infrastructure. data.luwte.net runs MariaDB with one database per application, and adding a new one is straightforward. The same pattern scales just as cleanly for PostgreSQL: one StatefulSet, one namespace, one set of backup jobs. Adding Umami was a bao kv put and a Kubernetes init Job. Adding the next application will be the same.
The tradeoff I accepted is a single point of failure at the database tier. For a homelab, that's fine. Longhorn replicates the underlying storage volume across nodes, which protects against disk or node loss, but this is still a single PostgreSQL instance without automatic database failover.
Exposing PostgreSQL Through Traefik TCP
Getting external access working was the first interesting problem. PostgreSQL runs on port 5432, which is a raw TCP connection — not HTTP, not something Traefik handles by default. The plan was to add a dedicated LoadBalancer Service with a new MetalLB IP, but it turned out Traefik already has what's needed: TCP entrypoints.
Adding a postgres entrypoint to Traefik's Helm values exposes port 5432 on the existing MetalLB IP (10.0.140.100), alongside HTTP/HTTPS. No second IP required.
The updated values-patch.yaml:
service:
spec:
externalTrafficPolicy: Local
loadBalancerIP: 10.0.140.100
type: LoadBalancer
ports:
postgres:
port: 5432
expose:
default: true
exposedPort: 5432
protocol: TCP
Apply with:
helm upgrade traefik traefik/traefik \
-n traefik \
-f traefik-values.yaml \
-f values-patch.yaml
After helm upgrade, the Traefik pod picks up the new entrypoint:
kubectl describe pod -n traefik | grep -A20 "Args"
# Args:
# --entryPoints.postgres.address=:5432/tcp
# --entryPoints.web.address=:8000/tcp
# --entryPoints.websecure.address=:8443/tcp
An IngressRouteTCP then routes connections on that entrypoint to the PostgreSQL Service in the postgres namespace. Connections to postgres.vluwte.nl:5432 are only reachable from the internal network — postgres.vluwte.nl resolves exclusively on the internal DNS, and the MetalLB IP is not exposed to the public internet. Do not expose a database port publicly without additional controls such as firewall restrictions, VPN access, or mTLS — a plain TCP port with password authentication is not sufficient on its own.
The IngressRouteTCP manifest:
#apps/postgres/ingresses/ingressroutetcp.yaml
---
# IngressRouteTCP — routes TCP traffic on the postgres entrypoint to the
# postgres ClusterIP Service.
#
# Traefik matches all SNI traffic on port 5432 and forwards to
# postgres.postgres.svc.cluster.local:5432.
#
# HostSNI(`*`) is required for plain TCP (no TLS at the Traefik level).
# TLS is handled at the PostgreSQL level if needed — configure ssl_mode
# in the client connection string.
#
# Depends on:
# - postgres entrypoint defined in Traefik values-patch.yaml
# - postgres Service (apps/postgres/service/service.yaml)
apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata:
name: postgres
namespace: postgres
spec:
entryPoints:
- postgres
routes:
- match: HostSNI(`*`)
services:
- name: postgres
port: 5432
kubectl apply -f apps/postgres/ingresses/ingressroutetcp.yaml
The Helm Field Manager Conflict
The upgrade didn't go smoothly on the first attempt. Running helm upgrade failed with:
conflict occurred while applying object traefik/traefik /v1, Kind=Service:
Apply failed with 1 conflict: conflict with "kubectl-patch" using v1: .spec.externalTrafficPolicy
This is a Kubernetes field manager problem. At some earlier point, kubectl patch had set externalTrafficPolicy directly on the Service — which means kubectl owns that field. When Helm tries to set the same field via server-side apply, the two field managers conflict and Helm refuses to proceed.
The fix is to transfer field ownership back to Helm:
kubectl patch service traefik -n traefik \
--type=merge \
--field-manager=helm \
-p='{"spec":{"externalTrafficPolicy":"Local"}}'
After that, helm upgrade succeeded. The lesson is worth internalising: if you ever use kubectl patch or kubectl edit on a resource that Helm also manages, you can end up with this split ownership situation. The symptom is always that specific conflict with "kubectl-patch" error on upgrade.
Secrets and the OpenBao v2.5.3 Detour
Credentials follow the same pattern as every other namespace: secrets stored in OpenBao, synced into Kubernetes via External Secrets Operator. Two secrets were needed before anything else could start:
# Inside the OpenBao pod
bao kv put secret/postgres/superuser \
POSTGRES_PASSWORD=<generated-password>
bao kv put secret/umami/db \
POSTGRES_USER=umami \
POSTGRES_PASSWORD=<generated-password>
Both recorded in 1Password under bletchley — postgres superuser and bletchley — umami postgres user.
Setting up the OpenBao policy and Kubernetes auth role for the postgres namespace requires root — the kv-admin token I originally created doesn't have sys/policies or auth/kubernetes/role capabilities. Generating a new root token on Bletchley has been straightforward using recovery key shares. Except this time it wasn't.
OpenBao v2.5.3 introduced a security fix (CVE-2026-5807) that disables the unauthenticated root token generation endpoints by default via a new config flag: disable_unauthed_generate_root_endpoints = true. Every attempt at bao operator generate-root returned either a 405 or a 403, with no obvious indication why.
The fix was to temporarily add disable_unauthed_generate_root_endpoints = false to the listener stanza in openbao-values.yaml, run helm upgrade, restart the pod, and then complete the standard generate-root flow using recovery key shares from 1Password. Once the policy and role were created, the flag was removed and the pod restarted again to close the window.
The policy itself:
bao policy write postgres - <<EOF
path "secret/data/postgres/*" {
capabilities = ["read"]
}
EOF
bao write auth/kubernetes/role/postgres \
bound_service_account_names=external-secrets \
bound_service_account_namespaces=external-secrets \
policies=postgres \
ttl=1h
When kv-admin was first created, it only needed KV access — setting policies was a root task, and root was still easy to get at that point. With the OpenBao v2.5.3 change making root generation more involved, it made sense to extend kv-admin to cover policy and auth role management so routine namespace onboarding wouldn't require root in the future.
bao policy write kv-admin - <<EOF
path "secret/data/*" {
capabilities = ["create", "update", "read", "delete"]
}
path "secret/metadata/*" {
capabilities = ["list", "read", "delete"]
}
path "sys/policies/acl/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
path "auth/kubernetes/role/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
EOF
Future namespace additions can now be done entirely with the kv-admin token. Root is reserved for break-glass scenarios.
Standing Up the StatefulSet
With secrets in place, the deployment follows the standard order: namespace → ExternalSecrets → wait for sync → Service → StatefulSet.
#apps/postgres/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: postgres
#apps/postgres/externalsecret/externalsecret-superuser.yaml
---
# SecretStore — connects ESO to OpenBao using Kubernetes auth.
# Scoped to the postgres namespace (least-privilege, consistent with cluster pattern).
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: openbao
namespace: postgres
spec:
provider:
vault:
server: "http://openbao.openbao.svc:8200"
path: "secret"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "postgres"
---
# ExternalSecret — pulls the postgres superuser password from OpenBao.
#
# OpenBao path: secret/postgres/superuser
# Required key: POSTGRES_PASSWORD
#
# Store the superuser password in OpenBao before applying:
# bao kv put secret/postgres/superuser POSTGRES_PASSWORD=<generated-password>
# Also record in 1Password as "bletchley — postgres superuser".
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: postgres-superuser
namespace: postgres
spec:
refreshInterval: 1h
secretStoreRef:
name: openbao
kind: SecretStore
target:
name: postgres-superuser
creationPolicy: Owner
data:
- secretKey: POSTGRES_PASSWORD
remoteRef:
key: secret/postgres/superuser
property: POSTGRES_PASSWORD
The SecretStore uses Kubernetes auth to authenticate against OpenBao. This assumes ESO already has the ClusterRole binding that allows it to perform TokenReview requests against the Kubernetes API — that RBAC was set up when ESO was installed on Bletchley. If you're adapting this to a fresh cluster, check the Secret Management Part 1 post for the required permissions.
Apply in order — the StatefulSet needs the Secret to exist before the pod starts:
kubectl apply -f apps/postgres/namespace.yaml
kubectl apply -f apps/postgres/externalsecret/externalsecret-superuser.yaml
# Wait for ESO to sync before continuing
kubectl get externalsecret -n postgres postgres-superuser
NAME STORETYPE STORE REFRESH INTERVAL STATUS READY LAST SYNC
postgres-superuser SecretStore openbao 1h SecretSynced True 11m
The first sync attempt failed — the ExternalSecret was showing SecretSyncedError. Comparing the failing SecretStore spec with a working one (the Authelia namespace) revealed two differences: the server URL used the full cluster DNS name (openbao.openbao.svc.cluster.local:8200) instead of the short form (openbao.openbao.svc:8200), and the spec included an unnecessary serviceAccountRef. Both corrected, and the ExternalSecret synced immediately.
#apps/postgres/service/service.yaml
---
# ClusterIP Service — makes PostgreSQL reachable from other namespaces at:
# postgres.postgres.svc.cluster.local:5432
apiVersion: v1
kind: Service
metadata:
name: postgres
namespace: postgres
labels:
app: postgres
spec:
type: ClusterIP
selector:
app: postgres
ports:
- name: postgres
port: 5432
targetPort: 5432
protocol: TCP
#apps/postgres/statefulset.yaml
---
# PostgreSQL 17 StatefulSet — shared instance for all cluster databases.
#
# Storage: Longhorn RWO PVC, 10Gi initial allocation.
# Superuser password pulled from OpenBao via ESO (postgres-superuser Secret).
# One database + user per application — do not use the superuser in applications.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres
namespace: postgres
labels:
app: postgres
spec:
serviceName: postgres
replicas: 1
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres:17.9-alpine
ports:
- containerPort: 5432
name: postgres
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-superuser
key: POSTGRES_PASSWORD
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 1000m
memory: 1Gi
readinessProbe:
exec:
command:
- pg_isready
- -U
- postgres
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
livenessProbe:
exec:
command:
- pg_isready
- -U
- postgres
initialDelaySeconds: 30
periodSeconds: 30
timeoutSeconds: 5
volumeClaimTemplates:
- metadata:
name: data
# Labels patched directly on PVC - cannot be set here after initial creation
# backup.vluwte.nl/enabled: "false"
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 10Gi
kubectl apply -f apps/postgres/service/service.yaml
kubectl apply -f apps/postgres/statefulset.yaml
# Verify PostgreSQL is accepting connections
kubectl exec -n postgres pod/postgres-0 -- pg_isready -U postgres
# /var/run/postgresql:5432 - accepting connections
Longhorn provisioned the 10Gi PVC automatically. One thing to note: the Longhorn backup controller on Bletchley uses a label on the PVC to decide whether to back up a volume. For PostgreSQL, Longhorn volume snapshots are the wrong tool — they capture raw storage state without any database consistency guarantees. The right approach is pg_dump to S3, which produces a consistent, application-level backup. The label needs to be patched directly on the PVC after creation to opt it out of Longhorn snapshots (volumeClaimTemplates can't set labels after the PVC exists):
kubectl patch pvc data-postgres-0 -n postgres \
--type=merge \
-p='{"metadata":{"labels":{"backup.vluwte.nl/enabled":"false","backup.vluwte.nl/name":"postgres"}}}'
Migrating the Umami Database
Setting Up the Umami Database
With the StatefulSet running, the next step is creating the umami user and database. This needs a few things in place first: an OpenBao policy and Kubernetes auth role for the umami namespace, and the umami credentials available as a Secret in the postgres namespace so the init Job can read them.
The OpenBao policy and role follow the same pattern as postgres:
bao policy write umami - <<EOF
path "secret/data/umami/*" {
capabilities = ["read"]
}
EOF
bao write auth/kubernetes/role/umami \
bound_service_account_names=external-secrets \
bound_service_account_namespaces=external-secrets \
policies=umami \
ttl=1h
The init Job runs in the postgres namespace and needs the umami password available there to create the user. That means a second ExternalSecret — one that syncs the same OpenBao secret (secret/umami/db) into the postgres namespace. This is intentional: namespaces are an isolation boundary, and Secrets don't cross them. The Umami application will have its own ExternalSecret in the umami namespace pulling the same credentials — that will be covered in the application migration post.
#apps/postgres/externalsecret/externalsecret-umami.yaml
---
# ExternalSecret — pulls umami DB credentials into the postgres namespace.
#
# The init Job (jobs/init-umami.yaml) runs in the postgres namespace and needs
# the umami password available as a Secret there to create the umami user.
# This is separate from apps/umami/externalsecret.yaml which serves the
# Umami application in the umami namespace.
#
# OpenBao path: secret/umami/db
# Resulting Secret name: umami-db (referenced by jobs/init-umami.yaml)
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: umami-db
namespace: postgres
spec:
refreshInterval: 1h
secretStoreRef:
name: openbao
kind: SecretStore
target:
name: umami-db
creationPolicy: Owner
data:
- secretKey: POSTGRES_PASSWORD
remoteRef:
key: secret/umami/db
property: POSTGRES_PASSWORD
Apply the ExternalSecret and wait for it to sync before running the init Job:
kubectl apply -f apps/postgres/externalsecret/externalsecret-umami.yaml
kubectl get externalsecret -n postgres umami-db
With the ExternalSecret synced, it's time to create the user and database. A Kubernetes Job is the right primitive here rather than an initContainer on the Umami Deployment. An initContainer would work, but it would require the Umami pod to have access to PostgreSQL superuser credentials — a broader permission than an application pod should have. The Job runs once, with elevated privileges scoped only to the provisioning step, and then exits. The Umami application itself only ever receives its own database credentials, which keeps the runtime permission model aligned with the principle of least privilege.
#apps/postgres/jobs/init-umami.yaml
---
# Init Job — creates the umami database and user in PostgreSQL.
#
# Prerequisites before applying:
# 1. postgres StatefulSet is running and ready
# 2. postgres-superuser Secret exists (ESO has synced it)
# 3. umami-db Secret exists in the postgres namespace (ESO has synced it from
# secret/umami/db in OpenBao — see apps/postgres/externalsecret/externalsecret-umami.yaml)
#
# This job is safe to re-run: CREATE USER IF NOT EXISTS and CREATE DATABASE are
# safe to re-run. Re-apply if the job fails or is deleted.
apiVersion: batch/v1
kind: Job
metadata:
name: init-umami-db
namespace: postgres
spec:
backoffLimit: 4
template:
spec:
restartPolicy: OnFailure
containers:
- name: init
image: postgres:17.9-alpine
command:
- sh
- -c
- |
set -e
echo "Creating umami user..."
psql -h postgres.postgres.svc.cluster.local -U postgres <<EOF
DO \$\$
BEGIN
IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'umami') THEN
CREATE USER umami WITH PASSWORD '$UMAMI_PASSWORD';
END IF;
END
\$\$;
EOF
echo "Creating umami database..."
psql -h postgres.postgres.svc.cluster.local -U postgres <<'EOF'
SELECT 'CREATE DATABASE umami OWNER umami'
WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'umami')\gexec
EOF
echo "Done."
resources:
requests:
cpu: 50m
memory: 64Mi
env:
- name: PGPASSWORD
valueFrom:
secretKeyRef:
name: postgres-superuser
key: POSTGRES_PASSWORD
- name: UMAMI_PASSWORD
valueFrom:
secretKeyRef:
name: umami-db
key: POSTGRES_PASSWORD
kubectl apply -f apps/postgres/jobs/init-umami.yaml
kubectl logs -n postgres job/init-umami-db -f
Two things worth noting in the Job's shell script. First, the database creation uses \gexec, a psql meta-command that only works via a heredoc — not with the -c flag, hence the <<'EOF' block. Second, \$\$ in the user creation block escapes the PL/pgSQL dollar-quoting at the shell level so psql receives the literal $$ it needs — the unquoted <<EOF allows this shell expansion while also expanding $UMAMI_PASSWORD. Known limitation: a password containing a single quote will break the SQL syntax. Since password generation is controlled in this setup, restricting generated passwords to alphanumeric characters when storing them in OpenBao is an acceptable mitigation.
With all this prepared, the data migration becomes straightforward.
# Dump from the source
pg_dump -h docker.luwte.net -U umami -d umami -Fc -f /tmp/umami.dump
# Restore into the new instance
pg_restore -h postgres.vluwte.nl -U umami -d umami /tmp/umami.dump
Row counts verified before cutting over:
psql -h docker.luwte.net -U umami -d umami \
-c "SELECT schemaname, relname, n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC;"
schemaname | relname | n_live_tup
------------+--------------------+------------
public | website_event | 251
public | session | 105
public | _prisma_migrations | 0
public | website | 0
public | user | 0
Compared with the counts after:
psql -h postgres.vluwte.nl -U umami -d umami \
-c "SELECT schemaname, relname, n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC;"
schemaname | relname | n_live_tup
------------+--------------------+------------
public | website_event | 251
public | session | 105
public | _prisma_migrations | 14
public | website | 1
public | user | 1
Identical on both sides.
Cutover
Updating Umami's DATABASE_URL in /opt/umami/.env on sulu.luwte.net:
DATABASE_URL=postgresql://umami:<password>@postgres.vluwte.nl:5432/umami
Restart Umami. Dashboard loads, analytics data intact, new events being recorded. The internal DNS hostname resolves correctly from sulu.luwte.net — which was the whole point of the Traefik TCP route.
Decommissioning the Old Container
With Umami confirmed working against the new database, the Docker container on docker.luwte.net was stopped and removed:
docker compose stop postgres
docker compose rm postgres
The /postgresql LVM volume was left in place for a week before deletion — a good habit when decommissioning anything that held production data. If something unexpected surfaced, the data was still there.
What's Working Now
- ✅ PostgreSQL 17.9 StatefulSet running in
postgresnamespace on Bletchley - ✅ External access via
postgres.vluwte.nl:5432(Traefik TCP on10.0.140.100) - ✅
umamidatabase migrated fromdocker.luwte.net, Umami pointing at the new instance - ✅
postgres-umamiDocker container ondocker.luwte.netdecommissioned - ✅
kv-adminpolicy extended — routine namespace onboarding no longer requires root
Adding a new database for the next application is now: store credentials in OpenBao, apply an ExternalSecret, run an init Job. The infrastructure for it is already there.
What's Next
The backup system that came out of this work deserves its own post — it grew from a simple CronJob into something considerably more complete, with per-database dumps, rotation, a Go-based Prometheus exporter, and Alertmanager rules. That's the next post.
The Umami application itself (still running on sulu.luwte.net) will move onto the cluster in a later post, once the database foundation is stable.
← Previous: Prometheus Storage Overhead
Questions or suggestions? Leave a comment below or reach out at igor@vluwte.nl.