Handling a Password You Never See

Introduction

In CloudNativePG Part 9, Authelia became the third tenant on bletchley-pg, and that meant one new database password. It has two readers: CloudNativePG, which sets it on the Postgres role, and Authelia, which logs in with it. Both get it from OpenBao through External Secrets Operator (ESO).

So there is one credential, but not one reader. OpenBao holds the value. ESO copies it into two Kubernetes Secrets in two namespaces: one in databases, which CloudNativePG uses to set the Postgres role's password, and one in authelia, which Authelia reads as a mounted file. The challenge is to manage the credential's permissions, lifecycle, and delivery without ever exposing its value.

The rule on this cluster is that every secret lives in OpenBao and no password is ever typed into a command or printed on screen. That rule is easy to state and turned out to have more edges than I expected. Generating the password stored a byte I did not mean to store, turning a 64-character password into a 65-character one. Removing the bad version ran into a permission my all-powerful operator policy did not have. Checking that both readers got the same value meant comparing two things I was not allowed to look at. And while tidying up, I found an old secret that had outlived its reader.

None of these is a big story on its own. Together they are most of what "handle a secret properly" means in practice.

🔬 This is part of the Technical Deep Dives series - technical concepts explained in depth, independent of the homelab journey.

• I'm Still Here
• Handling a Password You Never See (you are here)

This post builds on CloudNativePG Part 9, where the manifests for both sides are in the addendum. The OpenBao and ESO setup itself goes back to Secret Management Part 1.

Where a Secret Lives Is a Permissions Decision

Authelia already had an entry in OpenBao, secret/authelia/chart-secrets, holding the three keys its Helm chart needs. The obvious move was to add the database password there as a fourth key. I did not: the databases namespace needs this password too, and anything allowed to read chart-secrets can also read storage.encryption.key, the key that encrypts Authelia's 2FA data at rest. So the password got its own entry, secret/authelia/postgres, and the databases side got a policy for that one path:

# Read-only access to the Authelia database credential, for the SecretStore in
# the databases namespace (password for the CNPG role `authelia`).
# Deliberately one path, not secret/data/authelia/* — that prefix also holds
# chart-secrets, including storage.encryption.key.
path "secret/data/authelia/postgres" {
  capabilities = ["read"]
}

This is narrower than the pattern I was copying. Umami's policy is a wildcard:

bao policy read umami; bao policy read authelia-eso

path "secret/data/umami/*" {
  capabilities = ["read"]
}
path "secret/data/authelia/*" {
  capabilities = ["read"]
}

For Umami, umami/* is fine, because that prefix holds nothing but the database credential. For Authelia, authelia/* would have handed the database side the encryption key. Copying a pattern means copying its intent (each reader gets only what it needs), not its literal wildcard. Authelia's own role, authelia-eso, keeps its wildcard: it already reads the encryption key and needs nothing new.

What the Roles Actually Isolate

To decide whether the databases side needed its own role at all, I read how the existing ones are bound:

bao read auth/kubernetes/role/umami

bound_service_account_names                 [external-secrets]
bound_service_account_namespaces            [external-secrets]
token_policies                              [umami]

authelia-eso looked the same. Three different things are easy to blur here: a Kubernetes namespace, a Kubernetes ServiceAccount, and an OpenBao Kubernetes-auth role. An OpenBao role is bound to a ServiceAccount, and both of these roles are bound to ESO's own ServiceAccount, in ESO's own namespace.

The consequence: a SecretStore in databases does not authenticate as a databases identity. None of my SecretStores sets a serviceAccountRef, so ESO always logs in to OpenBao with its own ServiceAccount. Moving a SecretStore to another namespace does not change who OpenBao thinks is asking.

That corrected an assumption I had written down a few steps earlier: that authelia-eso was tied to the authelia namespace and therefore unusable from databases. It is not tied to the authelia namespace; it is tied to ESO's own ServiceAccount. A SecretStore in any namespace that names role authelia-eso gets Authelia's secrets. On this cluster, the roles separate policies (what can be read), not namespaces (who may ask).

On a single-operator cluster that is an acceptable trade-off, and it was the path of least resistance when I set up ESO. But it is not what names like authelia-eso suggest. Real per-namespace isolation would need a serviceAccountRef on every SecretStore, and roles bound to those namespaces' ServiceAccounts. For this change, the dedicated authelia-db role still matters: it keeps the policy minimal, even though the binding is shared.

The 65th Byte

The password was generated straight into OpenBao, so it never touched the terminal, the shell history or the process list:

openssl rand -hex 32 | bao kv put -cas=0 secret/authelia/postgres username=authelia password=-

password=- tells bao to read the value from stdin. Then I checked what was stored, without displaying it:

bao kv get -format=json secret/authelia/postgres | jq '.data.data | {keys: keys, username: .username, password_length: (.password | length)}'

{
  "keys": [
    "password",
    "username"
  ],
  "username": "authelia",
  "password_length": 65
}

65, not 64. openssl rand -hex 32 generates 32 random bytes and formats them as 64 hexadecimal characters. Its text output ends with a newline. Because password=- reads stdin as-is, that newline became part of the stored value.

That byte would have reached both readers. CloudNativePG sets the role password from the Secret, and Authelia reads it from a mounted file. Whether each of them keeps or trims a trailing newline, I do not know, and I did not want to find out during a cutover. If one trims and the other does not, you get a login failure with a password that looks right everywhere you inspect it. My own probe would not have helped either: it read the file with $(cat …), which strips trailing newlines, so it reads the file differently from a program that does not.

The fix is one tr:

openssl rand -hex 32 | tr -d '\n' | bao kv put secret/authelia/postgres username=authelia password=-

The same pitfall applies to every key=- write in this cluster's history. Whether any older secret carries the same newline is now on my todo list.

The Version You Cannot Destroy

Writing a clean version 2 was easy. Getting rid of the 65-byte version 1 was not:

bao kv destroy -versions=1 secret/authelia/postgres

Error writing data to secret/destroy/authelia/postgres: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/secret/destroy/authelia/postgres
Code: 403. Errors:

* 1 error occurred:
* permission denied

The token carried kv-admin, the operator policy I log in with since Secret Management Part 5. Its own comments call it a path to full privilege: it can write any policy and bind any role. Yet it cannot destroy one version of one secret.

The reason is how KV version 2 splits its API into separate paths, each needing its own capability. Removing something comes in three strengths:

• Soft delete marks a version as deleted. Its data stays, and it can be undeleted. A DELETE on secret/data/<path> does this for the latest version only; choosing a version needs secret/delete/.
• Destroy permanently removes the data of chosen versions, through secret/destroy/. The metadata, and the record that those versions existed, stay.
• Metadata delete removes the entire KV entry, including all its secret data, version history, and metadata, through secret/metadata/. Writing to the path afterwards starts again at version 1.

kv-admin covers secret/data/* and secret/metadata/*. So it can soft-delete the latest version, which here was the clean version 2, and it can remove the whole entry. It cannot touch one chosen older version, which is exactly what I asked for. That left me choosing between widening the policy and deleting the entire entry. Since the entry had not yet been consumed, deleting and recreating it was the narrower operational change.

I had three ways out:

bao kv metadata delete secret/authelia/postgres
openssl rand -hex 32 | tr -d '\n' | bao kv put -cas=0 secret/authelia/postgres username=authelia password=-
bao kv get -format=json secret/authelia/postgres | jq '{version: .data.metadata.version, password_length: (.data.data.password | length)}'

{
  "version": 1,
  "password_length": 64
}

metadata delete only reports "Data deleted (if it existed)", which proves nothing. The proof is the next line: -cas=0 only writes when no entry exists at that path, and it succeeded as a fresh version 1.

Verifying Without Looking

With a clean value in OpenBao, ESO delivered it to both namespaces. Now I needed to know that both Secrets held exactly the same bytes, without displaying either. Three checks did that, each answering a different question.

Is it the right shape? A length check, here on the Kubernetes side:

kubectl -n databases get secret authelia-db-owner -o json \
  | jq '{type, keys: (.data | keys), password_length: (.data.password | @base64d | length)}'

{
  "type": "kubernetes.io/basic-auth",
  "keys": [
    "password",
    "username"
  ],
  "password_length": 64
}

Is it the same value in both places? A hash of each side, compared by the shell:

[ "$(kubectl -n authelia get secret authelia -o jsonpath='{.data.storage\.postgres\.password\.txt}' | shasum -a 256)" = \
  "$(kubectl -n databases get secret authelia-db-owner -o jsonpath='{.data.password}' | shasum -a 256)" ] \
  && echo "identical in both namespaces" || echo "DIFFERENT"

identical in both namespaces

Neither value reaches the screen: only the comparison result does. A length check alone would not have been enough here, since two different random strings of 64 characters have the same length.

Does it actually work? A throwaway pod that mounts the Authelia-side Secret key as a file, logs in with it, and reports who it is and how it is connected. It returned authelia|authelia and t|TLSv1.3. The pod is in the addendum of Part 9. It proves the password CloudNativePG set on the role is the one Authelia will read, which neither of the first two checks can.

Retiring a Secret Takes Two Deletions

The last finding came while rewriting Authelia's README, where I found a reference to a Secret called authelia-users, said to come from secret/data/authelia/users-database. My recollection was that no such OpenBao entry had ever existed. Git disagreed:

git log --oneline -- apps/authelia/externalsecret-authelia.yaml

ddaa154 authelia: retire file backend and transition overlay
4f5b691 secret management: migrate remaining cluster secrets to OpenBao via ESO

The ExternalSecret had existed, and was removed in CloudNativePG Part 5 when Authelia switched from its file backend to lldap. The Kubernetes side was clean. I expected the OpenBao entry to be gone too:

bao kv metadata get secret/authelia/users-database

current_version             3
updated_time                2026-04-06T07:16:52.874825965Z

====== Version 1 ======
created_time     2026-04-05T19:36:38.541315938Z
destroyed        false
...

Three versions, untouched since April 6, holding the pre-lldap user list with its password hashes. It had not been the live authentication backend since August, but hashes are still credential material, and still worth treating as a secret. Deleting the ExternalSecret could never have removed it: ESO only reads from OpenBao. With creationPolicy: Owner, deleting an ExternalSecret removes the Kubernetes Secret it created, and nothing upstream. Retiring a secret's consumer is two deletions, and I had only done the first.

Before deleting it, I checked that nothing still read the path, in Kubernetes or in any policy:

kubectl get externalsecrets -A -o json | jq -r '.items[] | select(tostring | test("users-database")) | "\(.metadata.namespace)/\(.metadata.name)"'
# (no output)
bao kv metadata delete secret/authelia/users-database
for p in $(bao policy list); do bao policy read "$p" | grep -q 'users-database' && echo "$p"; done

No ExternalSecret referenced it, and no policy named it. (The policy loop prints No policy named: root along the way: root is listed but has no readable body.) The old user list now exists nowhere in the cluster.

Lessons Learned

1. Where a secret lives decides who can read it. A separate path is what let one reader have a single-path policy.
2. Know what your roles bind to. Mine bind to ESO's own ServiceAccount, so they limit what can be read, not who can ask.
3. key=- stores exactly what you pipe in, newline included. A length check without displaying the value catches it.
4. A powerful policy is not every capability. KV version 2 splits destroy, delete and undelete into their own paths.
5. Verify identity, not just shape. Two random strings of the same length are not the same string. A hash comparison shows whether they are, without showing either.
6. Deleting a consumer is not deleting the secret. ESO reads from OpenBao and never writes back, so the source has to be cleaned up separately.
7. Git history beats memory. "That never existed" was wrong, and git show settled it in one command.

What's Next

• Check the older secrets for a trailing newline. Every entry written with key=- is a candidate; a length check per key will tell.
• Export the hand-written ESO policies into git. Only kv-admin and now authelia-db are tracked. The others exist only inside OpenBao.
• Decide whether per-namespace isolation is worth it. That means serviceAccountRef on every SecretStore and a role per namespace ServiceAccount. Not urgent on a single-operator cluster, but it should be a decision rather than a default.

← Previous: CloudNativePG Part 9

Questions or suggestions? Leave a comment below or reach out at igor@vluwte.nl.