Prometheus Storage Overhead: Duplicate Scrapers and an Oversized Default Collector

Introduction

Two weeks into running Prometheus on Bletchley, the PVC was already at 32GB. With a 40GB limit and a 30-day retention window, the numbers didn't add up — Prometheus was on track to fill the disk before the oldest data aged out. That's the kind of problem that looks boring from the outside but turns out to be genuinely interesting once you start digging.

The investigation revealed two separate problems stacked on top of each other: a duplication that I had inadvertently created myself, and a default collector that made perfect sense for a production Kubernetes cluster but had no business running in a homelab. Fixing both dropped the ingestion rate from roughly 3,000 samples per second to around 1,200 — a 60% reduction — without breaking a single Grafana dashboard.

This post documents what I found, what I changed, and what I learned about Prometheus scrape configuration along the way.

🏠 This is part of the Homelab Journey series - building a production Kubernetes cluster from scratch.

• Prometheus Behind Traefik and Authelia
• Prometheus Storage Overhead: Duplicate Scrapers and an Oversized Default Collector (you are here)

Finding the Problem

The first question was whether the scrape interval was the issue. A 1-minute interval is sensible homelab default and there's not much room to change it without sacrificing resolution on time-series data. Confirmed it was 1 minute across all jobs, and moved on.

The more productive angle was looking at what was being scraped and how much data each job was generating. Running a count of active targets revealed something immediately odd:

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=scrape_duration_seconds' \
  | jq '.data.result[] | "\(.metric.job) \(.metric.instance)"' \
  | sort | uniq -c

Both node-exporter and kubernetes-service-endpoints were listed — and both were hitting port 9100 on the same four physical machines. Port 9100 is the node exporter port. Two different jobs, same targets, same data.

Comparing series counts confirmed it:

# Series from kubernetes-service-endpoints hitting node-exporter
curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=count({job="kubernetes-service-endpoints", service="node-exporter-prometheus-node-exporter"})' \
  | jq '.data.result[0].value[1]'
# Result: 9583

# Series from the dedicated node-exporter job
curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=count({job="node-exporter"})' \
  | jq '.data.result[0].value[1]'
# Result: 9580

9,583 versus 9,580 — a difference of just 3 series. Nearly identical datasets being stored twice. The cause was straightforward once I saw it: kubernetes-service-endpoints uses Kubernetes service discovery and had automatically found the node-exporter Helm service (node-exporter-prometheus-node-exporter) and started scraping it via cluster IP — at the same time as the dedicated node-exporter job I had explicitly configured was scraping the same targets by hostname.

This was not intentional duplication; it was the Helm chart's default scraper doing its job a little too well.

Fix 1: Stopping the Duplicate

The cleanest fix was to keep the dedicated node-exporter job (which uses clean hostnames and explicit targets) and prevent kubernetes-service-endpoints from also scraping node-exporter endpoints.

The Prometheus Helm chart allows disabling built-in scrape jobs entirely, but that would have taken kube-state-metrics and kube-dns with it — both of which were being collected legitimately by the same job. The right approach was to override the job with a drop rule targeting the node-exporter service by name:

extraScrapeConfigs: |
  - job_name: kubernetes-service-endpoints
    # CUSTOM: overriding chart default to add drop rule for node-exporter
    # Chart version at time of override: prometheus-28.12.0
    # Check this block after chart upgrades
    honor_labels: true
    kubernetes_sd_configs:
    - role: endpointslice
    relabel_configs:
    - action: keep
      regex: true
      source_labels:
      - __meta_kubernetes_service_annotation_prometheus_io_scrape
    - action: drop
      regex: node-exporter-prometheus-node-exporter
      source_labels:
      - __meta_kubernetes_service_name
      # CUSTOM: drop node-exporter targets, already scraped by dedicated node-exporter job
    # ... remaining chart-default relabel rules unchanged

The custom job config opens with honor_labels: true, preserved from the chart default. Without it, if a scraped target already carries a label that Prometheus also attaches — such as job or instance — Prometheus would rename the target's version to exported_job or exported_instance rather than discard it. honor_labels: true prevents that renaming and lets the target's own label values take precedence.

The action: drop rule checks the Kubernetes service name and stops kubernetes-service-endpoints from scraping that target. Every other service it was previously collecting — kube-dns, kube-state-metrics — was unaffected.

Verification after applying the change:

curl -s 'http://localhost:9090/api/v1/targets' \
  | jq '.data.activeTargets[] | select(.labels.job=="kubernetes-service-endpoints") | .labels.service'

Expected output — node-exporter absent:

"kube-dns"
"kube-dns"
"prometheus-kube-state-metrics"
"prometheus-kube-state-metrics"

The label gap

Dropping kubernetes-service-endpoints from scraping node-exporter immediately broke the K8s Resource Monitoring dashboard. The reason: the kubernetes-service-endpoints job, because it uses Kubernetes service discovery, had been automatically attaching a node label to every series it collected. The dedicated node-exporter job, configured statically with hostnames, had no such label. Several dashboard queries group by (node), and those panels went blank.

The fix was to derive the node label from the target address at scrape time using a relabel rule:

- job_name: node-exporter
  static_configs:
    - targets:
      - rock1.vluwte.nl:9100
      - rock2.vluwte.nl:9100
      - rock3.vluwte.nl:9100
      - rock4.vluwte.nl:9100
  relabel_configs:
    - source_labels: [__address__]
      regex: '([^.]+)\..*'
      target_label: node
      replacement: '$1'

The regex extracts everything before the first . — so rock1.vluwte.nl:9100 becomes node="rock1". Dashboard panels that were grouping by (node) started working again, and as a side effect the instance labels in the surviving job are now clean hostnames rather than the pod-IP-based addresses that service discovery had been producing.

Fix 2: Disabling kubernetes-api-servers

After fixing the duplication, I ran a breakdown of series by job and found something striking: kubernetes-api-servers was responsible for 100,771 series — 55% of total storage. The Kubernetes API server exposes a very large number of histogram metrics, each of which multiplies into many series due to histogram bucket cardinality.

Before touching anything, I checked whether any dashboard or alert rule actually used these metrics:

grep -r 'apiserver_' /path/to/dashboard/files/
grep 'apiserver_' prometheus-values.yaml

Both returned nothing. In a production environment, API server metrics are genuinely useful for diagnosing latency issues, SLO breaches, and etcd health. In a homelab with four ARM nodes, I'm more likely to notice a hung kubectl command or a failing pod log than I am to be alerted by a p99 latency spike in API server histograms. The storage cost simply doesn't justify the diagnostic overhead for my use case.

Disabling the job was a one-line addition to prometheus-values.yaml:

scrapeConfigs:
  kubernetes-service-endpoints:
    enabled: false
  kubernetes-api-servers:
    enabled: false

Confirmed the job was gone from the Prometheus configmap:

kubectl get configmap prometheus-server -n monitoring -o yaml | grep 'job_name'

kubernetes-api-servers no longer appeared.

Result

Ingestion rate dropped from ~3,000 to ~1,200 samples/second. The screenshot at the top of this post shows both changes in sequence: the first partial drop at around 22:00 when the node-exporter duplication was removed, and then the sharp drop to ~1,200 the following morning when kubernetes-api-servers was disabled.

All dashboards verified and working. Disk usage won't drop immediately — Prometheus doesn't reclaim space when series stop being written, only when the blocks containing them fall outside the retention window. The high-ingestion data will be gone within 20 days.

Lessons Learned

1. Before dropping either of two duplicate jobs, compare the full label sets — not just the series counts. The 3-series difference between the two node-exporter jobs was tiny, but it turned out to be the node label, which was required by a dashboard. A small difference in series count can be a signal that one job carries labels the other doesn't.
2. Labels added by Kubernetes service discovery are not present on statically configured jobs. The node, namespace, and service labels all come from Kubernetes metadata and are only attached when a job uses kubernetes_sd_configs. If you switch from a service-discovery job to a static job and a dashboard depends on those labels, you'll need to reconstruct them via relabel_configs.
3. Helm chart defaults are tuned for production, not homelab. Both the duplication (node-exporter via kubernetes-service-endpoints) and the high-cardinality collector (kubernetes-api-servers) were chart defaults that made sense for a real cluster. For a homelab, the defaults warrant a review — not blindly, but with an understanding of what each job is actually collecting and whether you're using it.
4. Use rate(prometheus_tsdb_head_samples_appended_total[5m]) to see the effect of changes in near-real time. It shows the ingestion rate rather than cumulative storage, which makes it easy to see whether a change actually had the expected impact. I added Grafana community dashboard 19268 "Prometheus All Metrics" specifically for this kind of investigation.

← Previous: Prometheus Behind Traefik and Authelia

Questions or suggestions? Leave a comment below or reach out at igor@vluwte.nl.