homelab-journey

Growing the Cluster: Adding rock5, rock6, and rock7 to Bletchley

Three new RK1 worker nodes, three different Longhorn disk problems. The right way to add a node — and what happened when I didn't follow it.

Igor

— 9 min read
Share

Introduction

The TuringPi 2.5 supports four RK1 compute modules per board. The first Bletchley board has been fully populated since the beginning — three control plane nodes and one worker. The second board was always intended to add capacity, and with rock5, rock6, and rock7 now in place, the cluster has gone from four nodes to seven.

Joining new nodes to a Talos cluster is not complicated in itself. The tricky part, as I found out, is the interaction between Talos's volume manager and Longhorn's auto-discovery behaviour. Get the order of operations wrong and you end up with Longhorn happily claiming directories it has no business owning. This post covers the correct way to add a worker node to Bletchley — and then the honest account of what I actually did, which was not that.

🏠 This is part of the Homelab Journey series - building a production Kubernetes cluster from scratch.

This post builds on the Bletchley cluster setup. The Talos installation post covers the network design (VLAN 140, DHCP reservations, schematic IDs). The extensions post covers how Talos image upgrades and schematics work. The Longhorn setup post covers the extraMounts configuration and why the reboot sequence matters.

How to Add a Worker Node to Bletchley

The correct sequence. Read this before doing anything else.

Prerequisites

  • RK1 module flashed with the non-ZFS worker Talos image (Talos 1.12.6, extensions: nfsd, iscsi, util-linux-tools)
  • NVMe installed in the module and completely blank — no existing filesystem, no partition table
  • Access to the DHCP server

Step 1 — First boot and MAC capture

Boot the node. It comes up on the default VLAN and picks up a temporary DHCP address. At this point the node is not on VLAN 140 and not reachable on the cluster network — that's expected.

Before doing anything else, grab the MAC address from the DHCP server logs. Add it to the VLAN 140 DHCP pool with the correct fixed IP assignment. This must happen before the worker patch is applied, because when the node reboots onto VLAN 140 it needs to come up at the right address immediately.

Step 2 — Apply the worker patch

Apply the worker config using the temporary default-VLAN IP:

talosctl apply-config --insecure \
  --nodes <temporary-ip> \
  --config-patch @worker.patch.yaml \
  --file worker.yaml

Contents of infra/talos/worker.patch.yaml:

machine:
  install:
    disk: /dev/mmcblk0
  time:
    servers:
      - 192.168.0.138
  network:
    interfaces:
      - interface: end0
        vlans:
          - vlanId: 140
            dhcp: true

The node reboots and comes up on VLAN 140 at its fixed IP. All subsequent commands use the VLAN 140 address.

Step 3 — Verify the NVMe is blank

Before applying the Longhorn patch, confirm the NVMe has no filesystem:

talosctl get discoveredvolumes --nodes <nodename>.vluwte.nl

The NVMe should show as a bare disk with no filesystem. If it shows any filesystem (ext4, xfs, anything), wipe it:

talosctl wipe disk nvme0n1 --nodes <nodename>.vluwte.nl --drop-partition

The --drop-partition flag removes the partition table, which is required for Talos to create its own GPT layout cleanly on the next reboot.

Step 4 — Apply the Longhorn patch

With a confirmed blank NVMe, apply the Longhorn patch:

talosctl patch machineconfig \
  --nodes <nodename>.vluwte.nl \
  --patch @longhorn-patch.yaml

Contents of infra/talos/longhorn-patch.yaml:

machine:
  disks:
    - device: /dev/nvme0n1
      partitions:
        - mountpoint: /var/mnt/longhorn
  kubelet:
    extraMounts:
      - destination: /var/mnt/longhorn
        type: bind
        source: /var/mnt/longhorn
        options:
          - bind
          - rshared
          - rw

Reboot so Talos can partition the NVMe and mount it at /var/mnt/longhorn:

talosctl reboot --nodes <nodename>.vluwte.nl
kubectl get nodes <nodename> -w

After the node comes back Ready, verify the NVMe is correctly formatted:

talosctl get discoveredvolumes --nodes <nodename>.vluwte.nl
# Expected: nvme0n1 → gpt, nvme0n1p1 → xfs

Step 5 — Align the Kubernetes version

New nodes join at the version embedded in worker.yaml, which may lag behind the cluster. Check after joining:

kubectl get nodes

If the new node shows a different version, upgrade:

talosctl upgrade-k8s \
  --nodes rock1.vluwte.nl \
  --to <current-cluster-version>

This auto-discovers all nodes and rolls the update through control plane first, then workers.

Step 6 — Add logging and monitoring

Apply the per-node logging patch (endpoint is the node's own VLAN 140 IP):

talosctl patch machineconfig \
  --nodes <nodename>.vluwte.nl \
  --patch @logging-patch-<nodename>.yaml

Add the node to the Prometheus node-exporter scrape targets in apps/monitoring/prometheus/prometheus-values.yaml, then upgrade:

helm upgrade prometheus prometheus-community/prometheus \
  --namespace monitoring \
  --version <current-version> \
  --values apps/monitoring/prometheus/prometheus-values.yaml

Step 7 — Verify Longhorn

Longhorn runs a cluster-wide DaemonSet (longhorn-manager). The moment a new node becomes schedulable, Kubernetes automatically assigns a pod from that DaemonSet to it — no installation needed, but also no waiting. If the NVMe was blank and the Longhorn patch was applied before the node joined the cluster, Longhorn should have auto-registered the correct disk. Verify:

kubectl -n longhorn-system get nodes.longhorn.io <nodename> \
  -o jsonpath='{.spec.disks}' | jq .

The disk path should be /var/mnt/longhorn and storageReserved should be 80,530,636,800 bytes (~30% of 500 GB). If the entry looks correct, you are done.

If Longhorn has registered the wrong disk, or the entry has a UUID mismatch, see the repair steps below.

What I Actually Did

I did not follow that sequence. The longhorn-patch.yaml was applied before verifying that the NVMe disks were blank. Here is what that caused.

Why the wrong order matters

When longhorn-patch.yaml is applied to a node with a non-blank NVMe, Talos cannot partition the disk — it finds an existing filesystem and refuses to reformat it. Rather than failing visibly, it creates the /var/mnt/longhorn directory on the root filesystem as a fallback.

The longhorn-manager DaemonSet is already running cluster-wide. The moment a new node becomes schedulable, Kubernetes assigns a pod to it — and Longhorn immediately starts discovering disks, before any disk preparation has happened. Whatever it finds, it will try to use.

That is the state all three nodes were in when the problem surfaced.

How the problem was spotted

The Longhorn dashboard showed storage totals that were way off what three new 500 GB NVMe nodes should contribute. Instead of adding roughly 1 TB of schedulable storage, the numbers had barely moved. That was the trigger to start investigating.

rock5 — ext4 NVMe

Checking the disk state on rock5:

talosctl get discoveredvolumes --nodes rock5.vluwte.nl

rock5.vluwte.nl   runtime   DiscoveredVolume   nvme0n1   disk   500 GB   ext4

The NVMe had an existing ext4 filesystem on the whole disk — no partition table. The Talos boot log confirmed what was happening:

[talos] volume status "phase": "failed -> failed",
"error": "no disks matched for volume: 1 have wrong format"

Talos requires XFS and refused to touch the ext4 disk. The /var/mnt/longhorn mount was falling through to the root filesystem.

Fix: wipe, check and reboot.

#wipe partition table
talosctl wipe disk nvme0n1 --nodes rock5.vluwte.nl --drop-partition
#check wipe
talosctl get discoveredvolumes --nodes rock5.vluwte.nl
rock5.vluwte.nl   runtime     DiscoveredVolume   nvme0n1        2         disk        500 GB
#reboot node
talosctl reboot --nodes rock5.vluwte.nl
#check status of rock5
kubectl get nodes rock5 -w

After reboot the NVMe showed GPT with an XFS partition. Then the Longhorn cleanup.

The first attempt to remove the stale disk entry was rejected:

Delete Disk on node default-disk-b30600000000 error:
Please disable the disk and remove all replicas and backing images first

Trying to enable scheduling directly failed too, with a UUID mismatch:

record diskUUID doesn't match the one on the disk

The wipe had changed the NVMe's UUID. Longhorn still had the old UUID recorded and refused to accept the disk. The correct sequence — which took a couple of failed attempts to discover — is to disable scheduling and request eviction first, then remove:

kubectl -n longhorn-system patch nodes.longhorn.io rock5 \
  --type=merge \
  -p='{"spec":{"disks":{"default-disk-b30600000000":{"allowScheduling":false,"evictionRequested":true}}}}'

kubectl -n longhorn-system patch nodes.longhorn.io rock5 \
  --type=json \
  -p='[{"op": "remove", "path": "/spec/disks/default-disk-b30600000000"}]'

Then add the correct disk entry:

kubectl -n longhorn-system patch nodes.longhorn.io rock5 \
  --type=merge \
  -p='{"spec":{"disks":{"nvme-disk":{"path":"/var/mnt/longhorn","allowScheduling":true,"storageReserved":80530636800,"diskType":"filesystem"}}}}'

rock5 came up healthy and schedulable.

rock6 — same ext4 problem

rock6 had the same ext4 issue. Applied the same sequence: wipe → reboot → disable and evict stale entry → remove → add clean entry. Disk came up healthy with 490 GB available.

rock6 then showed ALLOWSCHEDULING: false at the node level. Attempting to enable it directly was rejected:

need to disable scheduling on node rock6 for node eviction, or cancel eviction to enable scheduling on this node

Eviction had been set at the node level during the cleanup and needed to be explicitly cancelled first:

kubectl -n longhorn-system patch nodes.longhorn.io rock6 \
  --type=merge \
  -p='{"spec":{"evictionRequested":false}}'

kubectl -n longhorn-system patch nodes.longhorn.io rock6 \
  --type=merge \
  -p='{"spec":{"allowScheduling":true}}'

rock7 — bare NVMe, wrong schematic, eMMC-backed Longhorn

rock7 had more problems than rock5 and rock6.

Wrong Talos schematic. rock7 had been flashed with the ZFS image as preparation for future S3 storage — the one used for rock3's SATA setup. This caused noticeable boot delays: the ZFS kernel module loaded on every boot and waited for a ZFS pool that didn't exist on rock7 yet.

The decision at this point was to standardise all three new nodes on the same schematic as rock1, rock2, and rock4. The schematic IDs were read directly from the running nodes rather than looking them up:

kubectl get node rock7 \
  -o jsonpath='{.metadata.annotations.extensions\.talos\.dev/schematic}'
# 777470099f5fed316bb4d2e60df2c754638d718ddcab319aad5f247f90a1517b

kubectl get node rock1 \
  -o jsonpath='{.metadata.annotations.extensions\.talos\.dev/schematic}'
# 9c8ed39c24b3518d9538cd50bdcb3d267da3d26623cbf97d6340406b0dab7f79

Upgrade rock7 to the correct schematic:

talosctl upgrade --nodes rock7.vluwte.nl \
  --image factory.talos.dev/installer/9c8ed39c24b3518d9538cd50bdcb3d267da3d26623cbf97d6340406b0dab7f79:v1.12.6

After reboot, the NVMe showed GPT with XFS — same as rock5 and rock6 after their wipes.

eMMC-backed Longhorn disk. Because rock7's NVMe had been blank when Longhorn first registered the node, Longhorn had fallen back to the eMMC boot disk as its backing. This needed to be removed before adding the NVMe entry.

Tried removing it via the Longhorn GUI — blocked. The GUI does not allow deleting a disk it considers the default. Used the same CLI sequence as rock5/rock6 to remove it, then added the correct NVMe entry:

kubectl -n longhorn-system patch nodes.longhorn.io rock7 \
  --type=merge \
  -p='{"spec":{"disks":{"default-disk-b30600000000":{"allowScheduling":false,"evictionRequested":true}}}}'

kubectl -n longhorn-system patch nodes.longhorn.io rock7 \
  --type=json \
  -p='[{"op": "remove", "path": "/spec/disks/default-disk-b30600000000"}]'

kubectl -n longhorn-system patch nodes.longhorn.io rock7 \
  --type=merge \
  -p='{"spec":{"disks":{"nvme-disk":{"path":"/var/mnt/longhorn","allowScheduling":true,"storageReserved":80530636800,"diskType":"filesystem"}}}}'

Same eviction cancellation needed as rock6:

kubectl -n longhorn-system patch nodes.longhorn.io rock7 \
  --type=merge \
  -p='{"spec":{"evictionRequested":false}}'

kubectl -n longhorn-system patch nodes.longhorn.io rock7 \
  --type=merge \
  -p='{"spec":{"allowScheduling":true}}'

What's Working Now

kubectl -n longhorn-system get nodes.longhorn.io

NAME    READY   ALLOWSCHEDULING   SCHEDULABLE   AGE
rock1   True    true              True          80d
rock2   True    true              True          80d
rock3   True    true              True          80d
rock4   True    true              True          80d
rock5   True    true              True          93m
rock6   True    true              True          91m
rock7   True    true              True          88m
  • ✅ rock5, rock6, rock7 joined to cluster, all Ready, all on Kubernetes 1.35.2
  • ✅ All three nodes visible in Longhorn, all schedulable
  • ✅ NVMe disks correctly formatted as XFS and mounted at /var/mnt/longhorn
  • ✅ All three nodes on the correct Talos schematic (matching rock1/rock2/rock4)
  • ✅ Longhorn dashboard: 7 nodes, 2.27 TiB total, 1.66 TiB schedulable, 8 volumes healthy
  • ✅ Logging and node-exporter monitoring extended to rock5, rock6, rock7

Lessons Learned

  1. NVMe must be blank before applying the Longhorn patch — if there is any existing filesystem, Talos cannot partition the disk. The /var/mnt/longhorn directory gets created on the root filesystem instead, Longhorn claims it immediately, and everything downstream is wrong.
  2. Grab the MAC before applying the worker patch — at first boot the node is on the default VLAN with a temporary address. That is the moment to get the MAC from the DHCP logs and add it to the VLAN 140 pool with a fixed IP, before the worker patch reboots the node onto VLAN 140.
  3. Two patches, applied in order — worker.patch.yaml first (VLAN, NTP, install disk), then longhorn-patch.yaml (NVMe partitioning and extraMounts) after confirming the NVMe is blank.
  4. Longhorn auto-discovers nodes immediately — the longhorn-manager DaemonSet is cluster-wide. The moment a new node becomes schedulable, a pod is assigned to it and disk discovery starts. Disk preparation must be complete before the node joins.
  5. Longhorn's diskUUID check is strict — wiping a disk changes its UUID. Longhorn refuses to use a disk where the registered UUID doesn't match. The fix is always to remove the stale entry and register a new one — you cannot just re-enable scheduling on the old entry.
  6. Disk removal requires the right sequence — disabling scheduling and requesting eviction must come before the remove patch. Trying to remove directly gets rejected. Trying to enable scheduling before cancelling eviction also gets rejected. The order is: disable + evict → remove → add clean entry → cancel eviction → enable scheduling.
  7. The Longhorn GUI cannot delete a disk it considers default — use the CLI patch sequence instead.
  8. Verify the schematic before flashing — new workers should use the non-ZFS schematic. Check against a known-good node:
    kubectl get node <name> -o jsonpath='{.metadata.annotations.extensions\.talos\.dev/schematic}'

← Previous: Log-Based Alerting

Questions or suggestions? Leave a comment below or reach out at igor@vluwte.nl.

Read more

This site uses essential cookies for functionality. No tracking cookies are used.