Secret Management Part 5: Replacing OpenBao's kv-admin Token with a Login

Introduction

Recently the kv-admin token expired in the middle of a policy change. Recovering it meant the break-glass root-generation ceremony, which since OpenBao v2.5.3 is no longer a single command. Part 4 had already written the runbook for exactly this, and having it did not stop the afternoon going sideways. A procedure you never have to run beats one you have written down.

The obvious response is to alert on the token's remaining TTL. That is the wrong fix, and working out why is most of this post. The right one removes the thing that expires.

There is a second thread underneath, and it is the more uncomfortable one. kv-admin was created in Part 1 specifically to avoid using root for routine work. By the time I came to replace it, it had become root-equivalent.

This post is part of the Secret Management sub-series.

• Part 3: Auto-Unseal on Proxmox
• Part 4: Worst Case Recovery
• Part 5: Replacing the kv-admin Token with a Login

🏠 This is part of the Homelab Journey series - building a production Kubernetes cluster from scratch.

• CloudNativePG Part 5
• Secret Management Part 5: Replacing the kv-admin Token with a Login (you are here)

This post assumes you have OpenBao running with a policy-scoped operator token, roughly as Part 1 set it up. The auto-unseal arrangement that makes the pod restart here survivable is Part 3.

The Chore, and What It Actually Costs

Routine OpenBao work is done with a static kv-admin token, pasted into BAO_TOKEN at the start of a session. Its TTL is capped at 768 hours (32 days) by OpenBao's default max_lease_ttl, so it has to be recreated roughly monthly.

I track that by hand, with a calendar entry set a few days early on purpose, so a busy week does not turn into a missed deadline. That buffer is the entire mitigation, and it works right up until it doesn't.

$ bao token lookup
creation_ttl        768h
display_name        token-kv-admin
entity_id           n/a
expire_time         2026-09-03T07:03:31Z
issue_time          2026-08-02T07:03:31Z
orphan              false
path                auth/token/create
policies            [default kv-admin]
renewable           true
ttl                 257h36m11s

Two fields in there are worth pausing on, because both come back later.

renewable true: so why not simply renew it? Because a non-periodic renewable token can only be renewed up to the mount's max_lease_ttl, measured from creation. August 2 plus 768h is September 3, and no amount of renewing moves that ceiling. Renewable is not extendable.

orphan false. This token has a parent. Revoking the parent takes it with it, silently, at whatever arbitrary moment that happens. It was never quite the dependable thing it felt like.

And entity_id n/a, display_name token-kv-admin: every action this token has ever taken is attributable to exactly that string and to no person.

Why Alerting on the TTL Is the Wrong Fix

The first instinct was to alert on the remaining TTL. Seven days out, fire a warning, go and rotate.

OpenBao exposes no per-token expiry metric. There is no vault_token_ttl_seconds{accessor="..."} waiting to be scraped. Building that alert means writing an exporter that looks the token up by accessor on a timer and publishes seconds-to-expiry. In other words a bespoke component whose entire purpose is to remind me to perform a manual rotation I would still have to perform.

It is the same trap I wrote about in Auditing the Alerts: an alert that fires to tell you to go and do a chore is a chore with extra steps, plus a new moving part that can fail silently on its own.

So "done" here does not look like a working alert. It looks like this: no long-lived operator credential exists, so there is no expiry date to track, alert on, or miss.

The Options, and Why the Simplest One Won

userpass won because it is the simplest thing on the list that works. The auth method ships with OpenBao, enabling it is one command, and it adds nothing to the cluster. Every other option either builds something or leans on something.

It also keeps the credential separate, which I now see as a feature rather than a cost. Every other account lives in LLDAP; this one lives only in 1Password. That means no shared lifecycle, no MFA, and a second operator must be added by hand. But OpenBao holds the secrets the cluster runs on, so I would rather its front door not depend on the directory, its database, and the SSO in front of both.

Tokens Are Receipts, Not Credentials

The industry split is between human and machine authentication, and the rule is identical on both sides: a token is never the credential, it is the receipt. Something you have (a password, an OIDC session, a ServiceAccount JWT) is exchanged for a short-lived token. When the token dies you exchange again.

Bletchley already gets the machine half right. ESO uses Kubernetes auth, chosen in Part 2 precisely because the static-token approach ran into this same 768h cap. kv-admin was the last place where a token was the credential, and bao auth list makes the point in four lines:

Path           Type          Accessor                    Description
----           ----          --------                    -----------
kubernetes/    kubernetes    auth_kubernetes_134e09b7    n/a
token/         token         auth_token_532bdf92         token based credentials

Machines have a proper auth method. Humans have a token.

What kv-admin Actually Granted

Before changing anything, I read the policy I had been using for four months.

path "secret/data/*" {
  capabilities = ["create", "update", "read", "delete"]
}
path "secret/metadata/*" {
  capabilities = ["list", "read", "delete"]
}
path "sys/policies/acl/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
path "auth/kubernetes/role/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

The bottom two rules are the interesting ones. Write on sys/policies/acl/* allows authoring a policy that grants anything. Write on auth/kubernetes/role/* allows binding that policy to a ServiceAccount. Together they are a path to full privilege. kv-admin is root-equivalent by escalation: it can rewrite the policy that governed itself and bind arbitrary policies to Kubernetes identities.

This was not carelessness, which is what makes it worth writing down. When the v2.5.3 default closed the unauthenticated root path, policy management had to come from somewhere, and kv-admin was the only credential left. The escalation is the visible scar of losing the root path — and because it worked, nobody looked at it again.

There is nothing to fix, incidentally. The same person holds the same access either way, and narrowing the policy would be theatre. But it changes what the token is: not a scoped operator credential that avoids root, but root wearing a different name and carrying an expiry date.

The Migration

The whole cutover is runtime API calls. No Helm, no pod restart, and reversible right up until the final command. That property is worth designing for deliberately. It confines the risky part of the work to one line at the end.

Extending the policy

bao auth enable userpass is a write to sys/auth/userpass. Creating the user is a write to auth/userpass/users/a-igor. The kv-admin policy granted neither, and the default policy attached to every token does not cover them. So step one is extending the policy and here the escalation stops being an observation and becomes the mechanism.

kv-admin can write sys/policies/acl/*, which includes its own policy. Policy contents are evaluated per request rather than baked into the token at creation, so the moment the updated policy is written the existing token gains the new capabilities. No re-issue, no login, no root.

Two commands minutes apart, same token. Before:

$ bao auth list
Error listing enabled authentications: Error making API request.

URL: GET http://127.0.0.1:8200/v1/sys/auth
Code: 403. Errors:

* 1 error occurred:
	* permission denied

After writing the extended policy, a full listing. That is the central irony paying off early: the escalation that made kv-admin root-equivalent is exactly what let me migrate away from it without root.

A 403 that looked like a broken tunnel

Getting there was not clean. The first attempt to write the policy from my laptop over a port-forward failed:

$ read -rs BAO_TOKEN
$ bao policy write kv-admin apps/openbao/policies/kv-admin.hcl
Handling connection for 8200
Code: 403. Errors:

* permission denied

I diagnosed that as "the port-forward doesn't work from here", went in via kubectl exec instead, and lost time. Wrong diagnosis, and the evidence against it is printed right there. Handling connection for 8200 is kubectl reporting that it accepted the connection and forwarded it. The 403 came back from OpenBao, so the request arrived and was answered. The tunnel was working the whole time.

The actual bug is one word. read -rs BAO_TOKEN sets a shell variable, not an environment variable, and bao is a separate process that inherits only exported ones. It made the request unauthenticated, which OpenBao correctly answered with a 403. BAO_ADDR worked because the line above happened to use export — which is precisely what made the failure look like connectivity rather than credentials.

read -rs BAO_TOKEN
export BAO_TOKEN

The sudo capability, and a guardrail that evaporates

With the policy extended, enabling the auth method should have been one command:

$ bao auth enable userpass
URL: POST http://127.0.0.1:8200/v1/sys/auth/userpass
Code: 403. Errors:

* 1 error occurred:
	* permission denied

create, read and update on sys/auth/userpass are not enough. Enabling an auth method also requires the sudo capability, because auth-method management is one of OpenBao's root-protected paths. The asymmetry is what made it confusing: bao auth list — a read of sys/auth — works without it. Only the mutation is gated.

path "sys/auth/userpass" {
  capabilities = ["create", "read", "update", "sudo"]
}

Re-apply, retry, Success! Enabled userpass auth method at: userpass/.

This is the sharpest version of the escalation argument. OpenBao does guard auth-method management, and root-protected paths exist precisely so a well-scoped policy cannot quietly enable a new way in. The guardrail is real and correctly designed. It is also one line in a file kv-admin was already permitted to edit. The policy granted itself sudo and the guardrail evaporated.

The user must be created with its password

The plan said: create the user, then set the password separately, so it never appears on a command line. OpenBao disagrees.

$ bao write auth/userpass/users/a-igor \
    token_policies=kv-admin token_ttl=8h token_max_ttl=24h
Code: 400. Errors:

* missing password

The password is required at creation; there is no create-then-set sequence. Everything goes in one write, with the password on stdin so it reaches neither the terminal nor shell history:

read -rs BAO_NEW_PASSWORD
printf '%s' "$BAO_NEW_PASSWORD" | bao write auth/userpass/users/a-igor \
  token_policies=kv-admin \
  token_ttl=8h \
  token_max_ttl=24h \
  password=-
unset BAO_NEW_PASSWORD

The safety goal survives intact — the password reaches the process on stdin, never as an argument. Only the sequencing was wrong.

printf, not echo — and tr -d '\n' if you pipe it in from a password manager. A trailing newline becomes part of the stored password, and the resulting failure is maximally confusing: the password looks right, bao login rejects it, and nothing in the error hints at an invisible character.

The username is a-igor, house convention for admin accounts, matching the accounts created in LLDAP two posts ago.

Verifying before breaking anything

unset BAO_TOKEN
bao login -method=userpass username=a-igor

The token helper caches the session, so every command after this needed no environment variable at all — including the one that shows what the login produced:

bao token lookup

Against the same command run on the old token at the start:

Four fields that changed meaning. The actor is now named by how they authenticated and as whom; logging in through an auth method creates an identity entity, so actions are attributable to a person rather than to a credential. And orphan true arrives for free — the property the transit seal token had to be created with -orphan to get, and the one the static token never had.

A KV read and bao policy list both worked under the new token, which was the functional check.

Self-service password change works as well, and it uses the dedicated sub-path rather than a write to the user itself:

read -rs BAO_NEW_PASSWORD
printf '%s' "$BAO_NEW_PASSWORD" | bao write auth/userpass/users/a-igor/password password=-
unset BAO_NEW_PASSWORD

Writing to auth/userpass/users/a-igor instead would restate every field on the user, and any field left out is silently reset — token_ttl back to the mount default, taking the 8-hour session with it.

Revoking the Old Token

Only once the login was proven. Revocation is by self, not by accessor: kv-admin has no auth/token/revoke-accessor capability, and the default policy grants only the -self variants, so accessor-based revocation returns a 403. A token revoking itself needs no extra privilege and leaves no doubt about which token died.

$ bao token revoke -self
Success! Revoked token (if it existed)

That message is not proof. OpenBao returns the same string whether or not there was a token to revoke — it cannot tell "revoked yours" from "that one was already gone", so it declines to claim either. The proof is the next command:

$ bao token lookup
URL: GET http://127.0.0.1:8200/v1/auth/token/lookup-self
Code: 403. Errors:

* permission denied

The token still had 254 hours left, and revoking it early rather than letting it lapse is the whole point: keeping it as a fallback would mean keeping a credential with a date on it.

Then a fresh bao login, which worked with no other credential in existence.

Telemetry, While the Pod Was Down Anyway

Everything above was runtime-only. Enabling telemetry is the one step that touches the values file and needs a pod delete, so it went in the same change, closing a standing TODO — OpenBao has never been scraped.

The rule for this restart is simple: confirm the Proxmox transit instance is up and unsealed first. If it is sealed or unreachable when the pod comes back, Bletchley comes back sealed and everything ESO-dependent fails behind it.

The config needs one setting that reads alarming:

telemetry {
  unauthenticated_metrics_access = true
}

/v1/sys/metrics otherwise requires a token, and giving Prometheus a rotating one means a bearer-token file plus something to refresh it. That awkwardness is why the flag exists. The exposure is narrow and worth naming rather than hand-waving: request counts, latencies, per-mount route counters, token and lease counts, seal state, Go runtime stats. No secret values, no policy contents, no token material. What leaks is structure. Any pod that can reach port 8200 can also reach the OpenBao API listener, so the telemetry endpoint does not introduce a new network destination; it changes what that listener exposes without authentication.

One detail in the scrape config is worth stealing:

extraScrapeConfigs: |
  - job_name: openbao
    metrics_path: /v1/sys/metrics
    params:
      format: ['prometheus']
    static_configs:
      - targets: ['openbao-internal.openbao.svc.cluster.local:8200']

The target is openbao-internal, not openbao. The chart creates two Services. openbao is the ordinary one that applications connect to, and like any Service it lists only the pods currently passing their readiness probe. openbao-internal is the headless companion that gives the StatefulSet's pod a stable DNS name, and the chart sets publishNotReadyAddresses: true on it — so it lists the pod whether or not it is ready.

That difference determines whether Prometheus retains a target during a sealed state, which is necessary for this alert to work—but I haven't yet observed the sealed state in practice. A sealed OpenBao is not ready, so scraping openbao would mean the target vanishes at precisely the moment the seal metric becomes interesting, leaving absent() as the only available signal and no way to tell "sealed" from "deleted".

Two rules went into the existing cluster alert group. OpenBaoSealed fires on vault_core_unsealed == 0 after three minutes — long enough that an ordinary restart auto-unseals without paging. OpenBaoUnreachable fires on up{job="openbao"} == 0 after five, and deliberately conflates "sealed and not serving" with "simply down", because both need attention now and the distinction would not change what I do next.

They join OpenBaoUnhealthy, which was already in that group from a blackbox probe against /v1/sys/health, so the alerts page now shows three OpenBao rules, all inactive. OpenBaoSealed is the one to treat with suspicion: it has never seen a sealed vault, so nothing has yet confirmed it fires.

What's Working Now

• ✅ userpass auth enabled, user a-igor bound to the kv-admin policy, token_ttl 8h / token_max_ttl 24h.
• ✅ Login over kubectl port-forward, with the token helper caching the session — no environment variable afterwards.
• ✅ Static kv-admin token revoked. No long-lived operator credential exists. Nothing to track, nothing to rotate on a schedule, no date to miss.
• ✅ Verified under the login — KV read, policy list, and a self-service password change.
• ✅ apps/openbao/policies/kv-admin.hcl in git, and provably the policy the vault is running.
• ✅ /v1/sys/metrics scraped against the headless service, with two alert rules loaded.
• ⚠️ A password, with no MFA and no lockout. Acceptable while the API is reachable only by port-forward on VLAN 140. It stops being acceptable the day an Ingress appears.
• ⚠️ A credential outside the directory. Every other account here lives in LLDAP; this one lives only in 1Password. Deliberate, and the point of the exercise — but it means no shared account lifecycle, and a second operator would be created by hand.
• ⚠️ Break-glass is still the expensive path. Losing the password means values edit → helm upgrade → pod delete → generate-root ceremony.
• ⬜ Housekeeping outstanding — mark the old token entry superseded, delete the calendar reminder, update the DR runbook.

Deferred deliberately: the audit device

Enabling a file audit device is the natural companion to this change. After the cutover the log would show a login event with a username and a distinct token per session, rather than "the kv-admin token did a thing" forever.

It is still the wrong change to bundle here, for one reason. If an audit device cannot write, OpenBao refuses requests — deliberate upstream behaviour, on the grounds that an unauditable vault must not serve traffic. The only persistent path available today is the 1Gi Raft PVC, sized for Raft state rather than for a log that grows with every request. A full volume would take the vault down quietly, looking like an outage rather than a disk problem. Doing it properly means a separate volume, rotation, and shipping to Loki — its own piece of work.

Lessons Learned

1. Automating the reminder is not automating the problem. The TTL exporter would have been a real component, correctly built, whose entire output was a notification that I still had a chore to do. Removing the chore was less work than monitoring it.
2. Renewable is not extendable. A non-periodic token cannot be renewed past max_lease_ttl measured from creation — which is the difference between it and the transit seal token, same vault and same 768h number, opposite fates.
3. A guardrail is only as strong as the weakest path to editing it. Root-protected paths and the sudo capability are well designed, and they are one line in a file kv-admin could already rewrite. Policy-write is the privilege that matters; everything else is downstream of it.
4. Scrape the endpoint that stays up when things break. The ready-only Service drops its endpoint at exactly the moment the seal metric becomes interesting.
5. The error is rarely where it appears to be. A 403 that looked like a broken tunnel was a missing export; a Success! that confirmed nothing needed a follow-up 403 to prove anything. In both cases the disambiguating evidence was the line above or the line below, never the error itself.

What's Next

The audit device, on its own volume, shipped to Loki, with an alert on audit write failure — that last part being the important one, since the failure mode is a hard stop on the whole vault rather than a gap in the logs. A NetworkPolicy on port 8200 admitting only external-secrets and monitoring. And MFA and lockout on the userpass method if OpenBao ever gets an Ingress, which is the condition that changes several answers in this post at once.

The dashboard side turned out to be a post of its own. Importing a community Grafana dashboard against a vault this quiet produces a screen full of empty panels for two entirely different reasons, neither of them OpenBao's fault.

← Previous: CloudNativePG Part 5

Questions or suggestions? Leave a comment below or reach out at igor@vluwte.nl.

Addendum: The Complete Configuration

apps/openbao/policies/kv-admin.hcl

path "secret/data/*" {
  capabilities = ["create", "update", "read", "delete"]
}
path "secret/metadata/*" {
  capabilities = ["list", "read", "delete"]
}

# The escalation described above: together these two are a path to full
# privilege — author any policy, bind it to any ServiceAccount. Present since
# the v2.5.3 default closed the unauthenticated root path and policy
# management had to live somewhere.
path "sys/policies/acl/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
path "auth/kubernetes/role/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# --- added for Part 5 --------------------------------------------------------

# Reading and listing sys/auth is NOT gated, which is why `bao auth list`
# works without the sudo capability below.
path "sys/auth" {
  capabilities = ["read", "list"]
}

# sudo is REQUIRED — sys/auth/* is a root-protected path. Without it,
# `bao auth enable userpass` returns 403 even with create/update.
path "sys/auth/userpass" {
  capabilities = ["create", "read", "update", "sudo"]
}

# Subsumes the /password sub-path, so no separate narrow rule is needed. A
# policy that can already rewrite every policy in the vault gains nothing
# from being narrow about its own user list.
path "auth/userpass/users/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

Applied from the git file, so the running policy provably originates from it:

bao policy write kv-admin apps/openbao/policies/kv-admin.hcl

apps/openbao/openbao-values.yaml — telemetry

Additions to apps/openbao/openbao-values.yaml:

telemetry {
  # Counters and summaries on /v1/sys/metrics are aggregated over this window,
  # not since-last-scrape. It must also EXCEED usage_gauge_period (default
  # 10m), or the identity and lease gauges exist for only part of each cycle.
  # NEVER 0 — that disables the endpoint entirely.
  prometheus_retention_time = "15m"
  disable_hostname          = true
}

listener "tcp" {
  tls_disable     = 1
  address         = "[::]:8200"
  cluster_address = "[::]:8201"

  # Operational telemetry only — no secret values, no policy contents, no
  # token material. Any pod that can reach it can already reach the API.
  # Revisit if OpenBao ever gets an Ingress.
  telemetry {
    unauthenticated_metrics_access = true
  }
}

updateStrategy: OnDelete means helm upgrade alone changes nothing — the pod must be deleted, and the Proxmox transit instance verified unsealed first.

apps/monitoring/prometheus/prometheus-values.yaml — the scrape job

additions to apps/monitoring/prometheus/prometheus-values.yaml:

# No ServiceMonitor: this is the community prometheus chart, not the Operator.
# Annotation-based discovery cannot express a query parameter, and this
# endpoint needs ?format=prometheus — a path annotation containing `?` gets
# URL-encoded and breaks. Hence an explicit job.
#
# TARGET IS THE HEADLESS SERVICE. `openbao` publishes only ready endpoints,
# and a sealed vault is not ready — the target would vanish at exactly the
# moment the seal metric matters. openbao-internal sets
# publishNotReadyAddresses: true.
extraScrapeConfigs: |
  - job_name: openbao
    metrics_path: /v1/sys/metrics
    params:
      format: ['prometheus']
    scheme: http
    static_configs:
      - targets: ['openbao-internal.openbao.svc.cluster.local:8200']

Alert rules — added to the existing cluster group

# -- OpenBao Seal Status: metric-based ------------------------------
# vault_core_unsealed is a gauge on the scraped metrics endpoint:
# 1 unsealed, 0 sealed. More specific than the probe above — it
# distinguishes "sealed" from "unreachable" — but it depends on
# OpenBao still serving /v1/sys/metrics while sealed, which has not
# yet been observed here (the only restart so far recovered in 11s,
# leaving no sealed window to test in). Treat as unproven until a
# real sealed period occurs; the probe alert above is the backstop
# that fires either way.
#
# for: 3m — shorter than the probe alert, so if metrics ARE served
# while sealed this fires first and names the condition precisely.
- alert: OpenBaoSealed
  expr: vault_core_unsealed == 0
  for: 3m
  labels: { severity: critical }
  annotations:
    summary: "OpenBao is sealed"
    description: "Transit auto-unseal has not resolved within 3m. Check that openbao-transit.luwte.net is up and unsealed. Bletchley cannot unseal the Proxmox instance; the dependency is one-directional."

# -- OpenBao Metrics Endpoint --------------------------------------
# Scrape health for the openbao job, which targets the headless
# service openbao-internal (publishNotReadyAddresses: true) rather
# than the load-balanced one; a sealed pod may drop out of the
# ready Service, which would make the target vanish exactly when it
# matters and turn a seal into an absent series.
#
# Deliberately conflates "sealed and not serving metrics" with
# "down": both mean ESO can no longer refresh secrets, and both need
# attention now. Expect this to fire alongside OpenBaoUnhealthy;
# Alertmanager groups them.
#
# NOTE: an expr matching no series never fires and reads as healthy.
# Verified 2026-08-24: up{job="openbao"} returns 1 every 15s.
- alert: OpenBaoUnreachable
  expr: up{job="openbao"} == 0
  for: 5m
  labels: { severity: critical }
  annotations:
    summary: "OpenBao metrics endpoint unreachable"
    description: "Sealed-and-not-serving, or down. Either way ESO-dependent workloads are failing silently."

← Previous: CloudNativePG Part 5

Questions or suggestions? Leave a comment below or reach out at igor@vluwte.nl.