feat(tempo): add host.name resource dimension to span metrics (HOMELAB-1103) #227

Merged
aaron merged 2 commits from plane/HOMELAB-1103-tempo-host-filtering into live 2026-06-12 02:38:19 +00:00
Owner

What

Adds host.name as a Prometheus metric dimension and Grafana dashboard filter:

Phase 2: Tempo config

  • Overrides the chart's default config template to inject metrics_generator.processor.span_metrics.resource_dimensions: [host.name]
  • The override is needed because chart 1.14.0 doesn't expose processor config natively
  • Can be removed if upgrading to chart >= 1.24.2 (which supports metricsGenerator.processor natively)

Phase 3: Grafana dashboards

  • Adds host_name query variable to all 4 pi-telemetry dashboards
  • Updates all PromQL panel queries to include host_name=~"$host_name" filter
  • Default is "All" (backward-compatible via includeAll)

Dashboards updated

  • pi-sessions — 8 queries updated
  • pi-tools — 6 queries updated
  • pi-llm-performance — 6 queries updated
  • agent-telemetry — 6 queries updated

Verify

After merge and ArgoCD sync:

  • Tempo pods restart and pick up new config
  • traces_spanmetrics_calls_total{host_name="MacBook-Pro.local"} appears in Prometheus
  • traces_spanmetrics_calls_total{host_name="mgmt"} appears in Prometheus
  • Grafana dashboards load correctly with "Host" variable populated
  • Selecting "All" shows traces from both machines
  • Selecting a specific host filters panels to that machine

Closes HOMELAB-1103

## What Adds `host.name` as a Prometheus metric dimension and Grafana dashboard filter: ### Phase 2: Tempo config - Overrides the chart's default `config` template to inject `metrics_generator.processor.span_metrics.resource_dimensions: [host.name]` - The override is needed because chart 1.14.0 doesn't expose `processor` config natively - Can be removed if upgrading to chart >= 1.24.2 (which supports `metricsGenerator.processor` natively) ### Phase 3: Grafana dashboards - Adds `host_name` query variable to all 4 pi-telemetry dashboards - Updates all PromQL panel queries to include `host_name=~"$host_name"` filter - Default is "All" (backward-compatible via `includeAll`) ### Dashboards updated - `pi-sessions` — 8 queries updated - `pi-tools` — 6 queries updated - `pi-llm-performance` — 6 queries updated - `agent-telemetry` — 6 queries updated ## Verify After merge and ArgoCD sync: - [ ] Tempo pods restart and pick up new config - [ ] `traces_spanmetrics_calls_total{host_name="MacBook-Pro.local"}` appears in Prometheus - [ ] `traces_spanmetrics_calls_total{host_name="mgmt"}` appears in Prometheus - [ ] Grafana dashboards load correctly with "Host" variable populated - [ ] Selecting "All" shows traces from both machines - [ ] Selecting a specific host filters panels to that machine Closes HOMELAB-1103
feat(tempo): add host.name resource dimension to span metrics (HOMELAB-1103)
Some checks are pending
CI Review / pr-title (pull_request) Waiting to run
CI Review / helm-validate (pull_request) Waiting to run
CI Review / ai-review (pull_request) Waiting to run
Lint & Validate / terraform-validate (pull_request) Waiting to run
Lint & Validate / yaml-lint (pull_request) Waiting to run
Lint & Validate / shellcheck (pull_request) Waiting to run
0/0 projects applied successfully.
ac7251acb4
Phase 2: Override Tempo chart config template to add
metrics_generator.processor.span_metrics.resource_dimensions: [host.name]
so Prometheus metrics include host_name labels.

Phase 3: Add host_name dashboard variable and filter to all pi-telemetry
Grafana dashboards (pi-sessions, pi-tools, pi-llm-performance,
agent-telemetry). All PromQL panel queries now include
host_name=~"\" filter, backward-compatible with "All" default.
Merge branch 'live' into plane/HOMELAB-1103-tempo-host-filtering
Some checks failed
Lint & Validate / shellcheck (pull_request) Has been cancelled
CI Review / pr-title (pull_request) Has been cancelled
CI Review / helm-validate (pull_request) Has been cancelled
CI Review / ai-review (pull_request) Has been cancelled
Lint & Validate / terraform-validate (pull_request) Has been cancelled
Lint & Validate / yaml-lint (pull_request) Has been cancelled
c2407da0f2
aaron merged commit 97c083b5f4 into live 2026-06-12 02:38:19 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
aaron/infra-core!227
No description provided.