undefined

Compare to recent history:
```promql

A growth rate > a few % per day on a stable application set is a red flag.

curl -s http://prometheus:9090/api/v1/status/tsdb | jq

undefined

---

"seriesCountByMetricName": [
  { "name": "http_request_duration_seconds_bucket", "value": 184320 },
  { "name": "go_gc_duration_seconds",               "value": 80 },
  ...
]

"labelValueCountByLabelName": [
  { "name": "url",       "value": 84210 },
  { "name": "trace_id",  "value": 41000 },
  { "name": "pod",       "value": 1820 }
]

undefined

Repeat per label, or use the helper:

```bash

undefined

undefined

If you see UUIDs, hashes, timestamps, or numeric IDs in the top values → that label has unbounded values from the source.

undefined

---

undefined

Diff externally. A new metric near the top of `seriesCountByMetricName` that wasn't there a week ago → that's your offender.

undefined

A vertical step in series count aligned with a deploy is conclusive.

---

undefined

A creation rate that materially exceeds the removal rate, sustained, means cardinality is on a one-way trip up. Common causes:

| Cause | Tell |
|---|---|
| Pod rollouts emitting `pod` label | Churn spike aligns with deploy timing; affects pod-discovered scrapes |
| `version` / `git_sha` / `image_tag` label on every metric | Churn spike on every deploy across many metrics |
| Ephemeral hostnames in `instance` | Cloud autoscaling event timing |
| Bug: dynamic label names | Churn climbs forever, never plateaus |
| Application bug emitting fresh UUIDs as labels | Linear unbounded growth, no deploy correlation |

undefined

Restarting Prometheus drops churned series but is not a fix. The fix is at the source.

---

undefined

undefined

undefined

Or normalize via relabel:
```yaml
metric_relabel_configs:
  - source_labels: [path]
    regex: /users/\d+
    target_label: path
    replacement: /users/:id

metric_relabel_configs:
  - source_labels: [__name__]
    regex: my_app_request_details
    action: drop

metric_relabel_configs:
  - regex: (instance|pod|node|host|job)
    action: labeldrop

- job_name: federate
  honor_labels: true
  metrics_path: /federate
  params:
    'match[]':
      - '{__name__=~".*:.*"}'  # Recording-rule naming convention only

Cardinality fire confirmed
│
├── Need to stop the bleeding NOW (production OOM, ingest 429s)
│   └── Apply emergency drop via metric_relabel_configs at scrape config
│       (also applies to Alloy/Agent — same syntax)
│       Then schedule the proper fix.
│
├── It's a Grafana Cloud Active Series bill issue, not a perf issue
│   ├── Cardinality is structural and you can't fix the app
│   │   └── Route to `adaptive-metrics` skill (post-ingest aggregation rules)
│   └── You want metric-by-metric DPM breakdown
│       └── Route to `dpm-finder` skill
│
├── It's a fixable application bug (unbounded label, debug metric in prod)
│   ├── Short-term: metric_relabel_configs drop at scrape
│   └── Long-term: fix in code; route to `prometheus-label-strategy` for design guidance
│
├── It's histogram cardinality
│   ├── Trim labels on the underlying histogram (14× win per label)
│   ├── Reduce bucket count if appropriate
│   └── Consider native histograms for high-resolution latency
│
└── It's churn (deploy-driven)
    ├── Remove `pod`, `version`, `git_sha` from application metrics
    ├── Use info-metric pattern for `version` (route to `prometheus-label-strategy`)
    └── Verify K8s SD relabel rules aren't carrying `uid` or other ephemeral fields

metric_relabel_configs:
  # Drop a specific bad metric
  - source_labels: [__name__]
    regex: bad_metric_name
    action: drop

  # Drop a high-cardinality label from all metrics
  - regex: bad_label_name
    action: labeldrop

  # Drop all labels matching a prefix (e.g., debug_*)
  - regex: debug_.*
    action: labeldrop

  # Drop a metric only when a specific label has unbounded values
  - source_labels: [__name__, path]
    regex: http_requests_total;/users/\d+
    action: drop

  # Aggregate path patterns to a template (replace the value)
  - source_labels: [path]
    regex: /users/\d+
    target_label: path
    replacement: /users/:id

  # Aggregate status codes to classes
  - source_labels: [status_code]
    regex: (\d)\d\d
    target_label: status_code
    replacement: ${1}xx

prometheus.relabel "drop_bad_metric" {
  forward_to = [prometheus.remote_write.default.receiver]

  rule {
    source_labels = ["__name__"]
    regex = "bad_metric_name"
    action = "drop"
  }

  rule {
    regex = "bad_label_name"
    action = "labeldrop"
  }
}