Fixing the Alerts: New Rules, Better Groups, Less Noise

Introduction

The audit post found the problems. This one fixes them.

Auditing the Alerts reviewed every rule in the cluster and came back with a clear list: one false positive to fix, seven rules to add, one timing adjustment, and a group structure that had grown messy enough to be worth reorganising. None of it was complicated in isolation, but the details matter — a rule with the wrong threshold is almost as bad as no rule at all.

This post covers everything that changed in prometheus-values.yaml: the fan rule redesign using a second metric as an intent signal, the new NTP, memory, certificate, and PVC rules with their threshold reasoning, the group restructure that took six alert categories and made them navigable, and one more rule that came out of The Backup That Wasn't — a straightforward Loki rule that would have caught eight days of silent failure on day one. The result is 51 Prometheus rules plus one new Loki rule, all passing validation, and a configuration that is easier to read and reason about.

One item is still on hold: the Longhorn fstrim problem. The prometheus-server PVC is still sitting at elevated allocation, retention extension is still deferred, and fixing that properly is the subject of the next post.

🔭 This is part of the Cluster Observability series — building observable infrastructure on a seven-node Talos Kubernetes cluster.

• The Backup That Wasn't
• Fixing the Alerts: New Rules, Better Groups, Less Noise (you are here)

Fixing NodeFanStopped

The fan alert was the clearest false positive in the audit. The original rule was:

- alert: NodeFanStopped
  expr: node_hwmon_fan_rpm{job="node-exporter", chip="platform_pwm_fan"} == 0
  for: 2m
  labels:
    severity: critical

This fires whenever fan RPM reads zero. The problem: the RK1 CPU fans on the TuringPi boards legitimately stop at low load. The PWM controller commands 0% duty cycle when the SoC is idle, and the fan spins down to 0 RPM. That is correct behaviour, not a failure. The rule had no way to tell the difference.

The fix is to add a second metric as an intent signal. node_cooling_device_cur_state{type="pwm-fan"} reflects what the controller is requesting: cur_state == 0 means "fan off, expected". cur_state > 0 means "fan should be spinning". The compound rule fires only when both conditions are true at the same time — 0 RPM while the controller is requesting non-zero duty.

# All nodes have a PWM fan (chip="platform_pwm_fan", sensor="fan1").
# Fans stop at low load — PWM controller commands 0% duty (cur_state=0)
# and RPM drops to 0. Alerting on 0 RPM alone caused false positives.
#
# node_cooling_device_cur_state{type="pwm-fan"} provides the intent:
# cur_state=0 → fan commanded off, 0 RPM expected.
# cur_state>0 → fan should spin; 0 RPM here is a failure.
- alert: NodeFanStopped
  expr: >
    node_hwmon_fan_rpm{job="node-exporter", chip="platform_pwm_fan"} == 0
    AND ON(instance)
    node_cooling_device_cur_state{job="node-exporter", type="pwm-fan"} > 0
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "Fan stopped on {{ $labels.instance }} while commanded to run"
    description: "Fan {{ $labels.sensor }} reading 0 RPM but cooling device controller is requesting non-zero duty cycle — possible fan failure."

The AND ON(instance) join matches both series on the instance label, so the alert fires per-node correctly. Severity is downgraded from critical to warning: a fan stopped at low load is not an immediate emergency, the SoC will throttle before thermal damage occurs. DiskTemperatureAlarm or NodeTemperatureCritical are the critical-severity signals for an actual thermal problem.

Validation: after applying the rule, a 45-second transient fan stop that morning produced no alert, correctly caught by the for: 2m holdoff. rock5 (where cur_state=0 and rpm=0 at idle) shows no result at all.

New Rules

NTP: NodeTimeDrift and NodeNTPNotSyncing

The 30-day baseline showed NTP health was good — offset sitting essentially at zero, occasional spikes to ~30ms that self-resolve within one scrape. But there were no rules covering it, so a genuine drift problem would have been invisible.

Two rules cover the two distinct failure modes:

# node_timex_offset_seconds measures actual NTP offset. Normal
# baseline is <0.03s — spikes to ~30ms are routine corrections.
# node_timex_estimated_error_seconds is not reliable on these nodes.
# NodeNTPNotSyncing fires immediately if the daemon stops syncing;
# NodeTimeDrift catches sustained drift if sync continues but drifts.
- alert: NodeTimeDrift
  expr: abs(node_timex_offset_seconds{job="node-exporter"}) > 1.0
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "Time drift on {{ $labels.instance }}"
    description: "NTP offset is {{ $value | printf \"%.3f\" }}s — clock may be drifting. Normal baseline is <0.03s."

- alert: NodeNTPNotSyncing
  expr: node_timex_sync_status{job="node-exporter"} == 0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "NTP not syncing on {{ $labels.instance }}"
    description: "node_timex_sync_status is 0 — NTP daemon may have stopped or lost connectivity."

The threshold choice for NodeTimeDrift is deliberate. Normal corrections peak around 30ms. One second is a 33x gap above that baseline — no legitimate NTP behaviour reaches it. The for: 15m holdoff ensures a brief NTP resync event with a high transient offset doesn't fire. NodeNTPNotSyncing has a shorter holdoff because a stopped NTP daemon is unambiguous and needs prompt attention.

node_timex_estimated_error_seconds was the first candidate metric. On these nodes it returns values in the 2-second range routinely, which made it useless for alerting without calibration. node_timex_offset_seconds is the actual measured offset and is reliable.

Memory Pressure: NodeMemoryPressure and NodeMemoryCritical

rock3 is the known hotspot: it hosts the control plane, Grafana, Garage, and the NFS server. A 7-day baseline showed it regularly sitting at 20–45% available. The thresholds are set below rock3's observed floor so the warning doesn't fire constantly on the busiest node:

# rock3 is the known hotspot (control plane + Grafana + Garage +
# NFS) and regularly sits at 20–45% available. Thresholds set below
# rock3's observed floor: warning <15% (~1.2 GiB), critical <5%
# (~400 MiB, near-OOM). Nodes have 8 GiB each.
- alert: NodeMemoryPressure
  expr: >
    node_memory_MemAvailable_bytes{job="node-exporter"}
    / node_memory_MemTotal_bytes{job="node-exporter"} < 0.15
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Low memory on {{ $labels.instance }}"
    description: "Less than 15% memory available on {{ $labels.instance }} — currently {{ $value | humanizePercentage }}."

- alert: NodeMemoryCritical
  expr: >
    node_memory_MemAvailable_bytes{job="node-exporter"}
    / node_memory_MemTotal_bytes{job="node-exporter"} < 0.05
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Critical memory pressure on {{ $labels.instance }}"
    description: "Less than 5% memory available on {{ $labels.instance }} — OOM risk. Currently {{ $value | humanizePercentage }}."

On 8 GiB nodes: warning fires below ~1.2 GiB available, critical below ~400 MiB. rock3 has a comfortable margin above 15%. The for: 5m holdoff filters brief spikes — memory pressure that persists for five minutes is genuinely actionable.

TLS Certificates: CertificateExpiringSoon and CertificateExpiryCritical

cert-manager is already being scraped through the kubernetes-pods job via pod annotation discovery. certmanager_certificate_expiration_timestamp_seconds tracks 11 certificates across the cluster. These rules are a safety net for the automation — if cert-manager's auto-renewal fails for any reason, the alert fires before a certificate expires and starts breaking services:

# cert-manager is scraped via kubernetes-pods job (annotation
# discovery). 11 certificates tracked across the cluster.
# Let's Encrypt certs renew at ~60 days; warning at 14d gives
# two renewal cycles of margin before critical at 7d.
- alert: CertificateExpiringSoon
  expr: >
    (certmanager_certificate_expiration_timestamp_seconds - time()) / 86400 < 14
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "Certificate {{ $labels.name }} expiring in less than 14 days"
    description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} (issuer: {{ $labels.issuer_name }}) expires in {{ $value | printf \"%.1f\" }} days."

- alert: CertificateExpiryCritical
  expr: >
    (certmanager_certificate_expiration_timestamp_seconds - time()) / 86400 < 7
  for: 1h
  labels:
    severity: critical
  annotations:
    summary: "Certificate {{ $labels.name }} expiring in less than 7 days"
    description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} (issuer: {{ $labels.issuer_name }}) expires in {{ $value | printf \"%.1f\" }} days — immediate action required."

Let's Encrypt renews at 60 days remaining. The warning threshold at 14 days gives two full renewal windows of margin before it fires, meaning it only fires if renewal has failed at least twice. Critical at 7 days is the "drop everything" signal. The for: 1h holdoff avoids noise during a renewal that momentarily lands below the threshold while being processed.

At the time of the audit, the shortest-lived certificate was it-tools-tls at ~31 days remaining — the first real test of these rules will be when that renewal cycle runs.

PVC Filesystem: PVCAlmostFull and PVCCriticallyFull

The audit surfaced an important distinction: LonghornVolumeAlmostFull measures Longhorn replica allocation, not filesystem usage inside the pod. These two numbers can diverge significantly — the prometheus-server volume demonstrates this clearly: 97% Longhorn allocation against 13% filesystem usage. The Longhorn alert fires correctly, but it can't tell you whether the filesystem is actually at risk.

kubelet_volume_stats measures the filesystem layer as seen from inside the pod, so it fills the gap:

# kubelet_volume_stats measures actual filesystem usage inside the pod,
# complementing LonghornVolumeAlmostFull which measures replica
# allocation. These two can diverge significantly.
# Warning >80%, critical >90%.
- alert: PVCAlmostFull
  expr: >
    kubelet_volume_stats_used_bytes{job="kubernetes-nodes"}
    / kubelet_volume_stats_capacity_bytes{job="kubernetes-nodes"} > 0.80
  for: 30m
  labels:
    severity: warning
  annotations:
    summary: "PVC {{ $labels.persistentvolumeclaim }} almost full"
    description: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} filesystem usage is above 80% — currently {{ $value | humanizePercentage }}."

- alert: PVCCriticallyFull
  expr: >
    kubelet_volume_stats_used_bytes{job="kubernetes-nodes"}
    / kubelet_volume_stats_capacity_bytes{job="kubernetes-nodes"} > 0.90
  for: 15m
  labels:
    severity: critical
  annotations:
    summary: "PVC {{ $labels.persistentvolumeclaim }} critically full"
    description: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} filesystem usage is above 90% — currently {{ $value | humanizePercentage }}."

The longer for on the warning (30 minutes) reflects that filesystem usage tends to grow gradually — a 30-minute window filters transient writes without missing a real fill event. Critical at 90% with a 15-minute holdoff is the point where action is genuinely urgent.

DeploymentReplicasUnavailable: Tuning the Holdoff

This rule existed. The change is simple: for: 5m to for: 15m. A rolling Helm upgrade that takes more than 5 minutes fires the original rule unnecessarily — Longhorn CSI components during the recent upgrade triggered it for roughly 4 days, which was a real condition, but shorter normal upgrades were also firing it. 15 minutes is long enough for normal rollouts to complete and short enough to catch a genuinely stuck deployment.

Group Restructure

The original alert configuration had grown organically. Rules were grouped loosely, with hardware rules mixed into the node group, PVC rules spread across multiple places, and no clear principle for what belonged together.

The new structure is based on one question for each rule: what subsystem does this tell you about?

The practical benefit is navigability. When a Longhorn alert fires, everything storage-related is in one place. When a certificate alert fires, the cluster group is the only place to look. Before the restructure, finding all the rules that touched storage required scanning across multiple groups.

One More Rule: ImagePullBackOff

The backup incident from The Backup That Wasn't produced one direct addition to the alert set: a Loki rule for ImagePullBackOff.

The backup pod was stuck in ImagePullBackOff on rock5 for eight days. Loki confirmed ~500 log lines per hour for the entire duration. A simple alert would have caught this on day one. The gap wasn't a missing Prometheus metric — it was a missing log-based rule.

This goes into apps/monitoring/loki/alerting-rules.yaml alongside the four rules added in Log-Based Alerting with Loki Ruler:

# -- Image Pull Failures -------------------------------------------
# Fires when kubelet logs sustained ImagePullBackOff entries. Sourced
# from talos-service/kubelet logs rather than Prometheus metrics —
# the kubelet log stream is the earliest signal for registry
# connectivity issues or missing images. for: 5m filters transient
# pull failures that self-resolve on retry. No namespace label
# available on this stream — use kubectl get pods -A | grep
# ImagePullBackOff to identify the affected pod.
- alert: PodImagePullBackOffFromLogs
  expr: |
    count_over_time(
      {job="talos-service", talos_service="kubelet"} |= "ImagePullBackOff" [10m]
    ) > 0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "ImagePullBackOff detected — check kubectl get pods -A"
    description: "Kubelet logs show repeated image pull failures. Run kubectl get pods -A | grep ImagePullBackOff to identify the affected pod."

The for: 5m holdoff filters brief pull failures that self-resolve on retry. A pod that is genuinely stuck will produce ImagePullBackOff log lines continuously — five minutes of sustained presence is unambiguous. The description points directly to the command to run rather than trying to surface pod context that isn't available as a Loki label on this stream. The five Loki log-based rules now mirror the same Prometheus alerting pipeline: Loki ruler → Alertmanager → email.

The ConfigMap is applied directly — no Helm upgrade needed. The loki-sc-rules sidecar watches for ConfigMaps labelled loki_rule: "true" and copies them into /var/loki/rules/fake/ where the ruler picks them up automatically:

kubectl apply -f apps/monitoring/loki/alerting-rules.yaml

The ruler API confirms all five rules in the configuration loaded:

kubectl port-forward -n monitoring svc/loki 3100:3100
curl localhost:3100/loki/api/v1/rules

Validation

promtool check rules validates all PromQL expressions — it catches syntax errors and invalid metric selectors before the rules reach Prometheus. Since the rules live inside prometheus-values.yaml as a Helm value, validation requires extracting them into a standalone file first. The extracted alerting_rules.yml is a test artifact only — created for the check, deleted after. The Loki rules in apps/monitoring/loki/alerting-rules.yaml are LogQL and validated separately when the ruler picks them up.

igor@granite prometheus % promtool check rules alerting_rules.yml
Checking alerting_rules.yml
  SUCCESS: 51 rules found

After applying the Helm upgrade, all new rules appeared as inactive in the Prometheus /alerts page — no unexpected pending states. NodeFanStopped has been silent since apply. The two LonghornVolumeAlmostFull and LonghornVolumeCriticallyFull alerts on the prometheus-server PVC are still active, as expected — that problem is not fixed by alerting changes.

What's Still on Hold

The Longhorn fstrim issue is unresolved and is the only remaining open item from the alerting review.

The prometheus-server Longhorn volume sits at elevated allocation because Longhorn holds onto every block address ever written, and trim doesn't reach the replica layer without StorageClass discard support. kubectl get sc longhorn -o yaml | grep -i discard returns nothing — discard is not configured. The util-linux-tools Talos extension is present on all nodes, so fstrim is available at the OS level, but manual kubectl annotate trim attempts have had no effect. The allocation floor was rising at approximately 1.4 GB per week at the time of the audit.

Retention extension from 20 days to 90 days is deferred until this is resolved. Extending retention while the allocation floor is rising would accelerate the problem — at the current write rate of ~257 MiB/day, 90-day retention would require ~23 GiB of actual filesystem space, but the Longhorn allocation floor would keep growing regardless.

The next post covers fixing the trim problem properly.

What's Working Now

• ✅ NodeFanStopped compound rule — no longer fires at idle, for: 2m holdoff validated
• ✅ NodeTimeDrift and NodeNTPNotSyncing — two NTP rules, 30-day baseline confirms no false positive risk
• ✅ NodeMemoryPressure and NodeMemoryCritical — thresholds set below rock3's observed floor
• ✅ CertificateExpiringSoon and CertificateExpiryCritical — 11 certificates tracked, first live test on it-tools-tls renewal
• ✅ PVCAlmostFull and PVCCriticallyFull — filesystem layer coverage, distinct from Longhorn replica allocation alerts
• ✅ DeploymentReplicasUnavailable — for: 15m filters rolling update noise
• ✅ Six-group rule structure — node / storage / hardware / kubernetes / backup / cluster
• ✅ PodImagePullBackOffFromLogs Loki rule — direct outcome of the backup incident
• ✅ 51 rules passing promtool check rules
• ⚠️ Known open item: Longhorn fstrim — prometheus-server PVC allocation floor still rising, retention extension still deferred

← Previous: The Backup That Wasn't

Questions or suggestions? Leave a comment below or reach out at igor@vluwte.nl.