Homelab/infra-core
6
0
Fork 2
Code Issues Pull requests 35 Projects Releases 1 Packages Wiki Activity Actions

IR-20261112: the ld-runtime relay's failed alert sends become a counter + a warning rule (F-RELAY-ALERT-SHIPPING) #964

Open
pi-landing wants to merge 1 commit from plane/IR-20261112-relayalert into live
pull from: plane/IR-20261112-relayalert
merge into: Homelab:live
Conversation 0 Commits 1 Files changed 6 +905
pi-landing commented 2026-11-12 14:26:02 +00:00
Collaborator
Copy link

What this is

F-RELAY-ALERT-SHIPPING, part 1 of 2 (the other half is Homelab/prod). The #955 verdict relay inside ld-runtime posts warning alerts to Alertmanager and fails CLOSED: a failed send leaves a line in its container log and nothing retries it. Nothing watched that line, so the relay's failure mode was silence.

artefact file
Vector source → filter → counter core/manifests/monitoring/log-metrics/vector-ld-relay.yaml
PrometheusRule LDRelayAlertSendFailing (+ coverage rule) core/manifests/monitoring/rules/ld-relay-alerts.yaml
promtool cases core/manifests/monitoring/rule-tests/ld-relay-alerts.test.yaml (generated by ld-relay-alerts.gen.py)
end-to-end test of the shipped Vector config core/manifests/monitoring/rule-tests/ld-relay-vector-e2e.sh + ld-relay-log-fixtures.log

The third artefact — the dead-man that does not route through the Alertmanager it watches — is in Homelab/prod (hosts/mgmt/infra-watchers/ld-alertmanager-deadman.*), together with the Alertmanager egress rule the runtime needed. Any one of the three alone leaves a silent failure mode, which is the thing being fixed.

The filter matches the REAL log lines (the brief's string was wrong)

The first brief for this work named the line alert send failed. That string appears nowhere in the relay's format strings. Source read at infra-core 6de8ad695112a523afd11dabaf50bb938a6ab860:

  • cmd/ld-runtime/main.go:1279 — LD-RELAY ALERT metric=ld_relay_alert_send_total result=sent|failed total_sent=<n> total_failed=<n> <detail> — the alert was ACCEPTED by the alert manager | the alert was NOT delivered
  • relay/poller.go:1406 — LD-RELAY ALERT rail: the non-paging rail REFUSED the notification (<err>) — the alert was NOT delivered; the latch still stands and the next poll retries

So the config matches contains(msg, "result=failed") || contains(msg, "the alert was NOT delivered"). A filter on the brief's string would have shipped a counter that can never increment — a silent monitor on top of the silent monitor this change removes. The lane's rendered sample (sample-send-failure.log, copied to rule-tests/ld-relay-log-fixtures.log) proves it in both directions: old pattern 0 matches, these predicates 2, healthy lines 0.

Naming choice (f1 item 5)

The app's log line carries metric=ld_relay_alert_send_total with a result label. This exporter deliberately does not reuse that name; it emits ld_relay_alert_send_failed_total{node,pod}:

  • ld_relay_alert_send_total is the application's declared metric. The runtime exposes no /metrics and has no ServiceMonitor today, but the day it does, a log-derived series of the same name would merge with it and result would be a tag this exporter invented. Same collision-avoidance as the longhorn shipper's F7 note.
  • …_send_failed_total says what it counts, which is the only thing the rule selects on. Both rails (send path, non-paging-rail refusal) count into one series on purpose — sum by (node) would collapse a per-rail split anyway. Stated consequence: no denominator; sent-vs-failed lives in the log's total_sent/total_failed.

The rule

sum by (node) (increase(ld_relay_alert_send_failed_total[10m])) > 0
or
sum by (node) (ld_relay_alert_send_failed_total unless ld_relay_alert_send_failed_total offset 10m) > 0

severity: warning, for: 0m. The second leg is #953's B8 fix (f474c999) reproduced, not re-derived: Vector creates the counter at the first failure, so the realistic single failure is a series born at 1, and increase() over a born-at-1 series that never increments is 0. for: 0m because the two legs already carry the hysteresis; adding 30s only gives a scrape gap a chance to hide the first failure.

Plus LDRelayLogMetricShipperDown (absent(up{job="vector-ld-relay-log-metrics"} == 1), 15m): a log-derived counter's failure mode is that it stops existing, which no rule over the counter can see.

flush_period_secs: 3600, not 10 — the 3c B8b finding: it is an expiry, so at 10s an isolated counter can expire before the 30s scrape. Same silence.

No new CNP for the log source

The source is a node-local file source: /var/log/pods/ld-fleet_ld-runtime-*/runtime/*.log via a readOnly hostPath, in a DaemonSet in monitoring with automountServiceAccountToken: false and no RBAC. Vector opens no socket to read it, so there is no pod-to-X traffic for a policy to govern, and the namespace-scoped ld-fleet/ld-runtime CNP does not select this pod. The Prometheus scrape uses the same ServiceMonitor path vector-longhorn-log-metrics already uses, which needs no policy. No CNP in this PR; the egress rule in the prod PR is a different rule (the relay's own), not this shipper's.

Tests — run, outputs as recorded

$ promtool check rules ld-relay-alerts.yaml
  SUCCESS: 2 rules found

$ promtool test rules ld-relay-alerts.test.yaml        # prom/prometheus:v3.4.0, the cluster's version
  SUCCESS                                              # 6/6 cases

Cases: increment fires · flat silent · birth fires (born at 1, never increments) · birth-aged silent · no-shipper fires · shipper-ok silent.

Not vacuous — against a leg-(a)-only rule file:

$ promtool test rules ld-relay-alerts.test.yaml   # expr = increase(...) > 0 only
  FAILED: name: birth: a series born at 1 fires without ever incrementing,
          alertname: LDRelayAlertSendFailing, time: 20m, exp:[…], got:[]
  --- EXIT 1

Full suite (17 test files, 26 rule files) still green after this change: --- FULL SUITE EXIT 0.

End-to-end, the shipped config driven verbatim (ConfigMap payload extracted from the manifest, fixture mounted under the real /var/log/pods path shape, lines appended after the shipper is up so read_from: end is exercised):

$ ./ld-relay-vector-e2e.sh
baseline series named ld_relay_alert_send_failed_total: 0 (expect 0)
ld_relay_alert_send_failed_total{node="prod-wk-02",pod="ld-runtime-6d9c8f7b4-x2vqn"} 2
PASS counter=2: the two failure lines matched (send path + non-paging rail)
PASS healthy line 'result=sent' produced no series
PASS healthy line 'ACCEPTED by the alert manager' produced no series
PASS healthy line 'poll tick' produced no series
PASS the literal 'alert send failed' does not appear in the fixture at all
     (a filter on it would have counted 0 of the 2 real failures)
=== EXIT 0

This test caught a real defect in its own fixture first: a pod log path without the _<uid> suffix makes parse_regex! drop every event and the counter stays 0 (recorded in the script).

Contract gate and server-side validation:

$ python3 scripts/monitoring/check_alert_contracts.py
ok: every added or modified rule carries a complete ld_* disposition contract, and every
    rule label/annotation name is Prometheus-valid        # rules=124 changed=2 legacy=27

$ kubectl apply --dry-run=server -f core/manifests/monitoring/rules/ld-relay-alerts.yaml
prometheusrule.monitoring.coreos.com/ld-relay-alerts created (server dry run)

Stated limits (not papered over)

  1. The relay's Alertmanager destination is not configured on the live ld-runtime Deployment (no ALERT*/AM* env, read 2026-11-12). The prod PR admits the egress; it does not create the destination. Until that is set this counter can only report nothing — which is honest, and the rule's ld_note says so.
  2. No live scrape yet. The shipper has never run in this cluster; LDRelayLogMetricShipperDown exists for exactly that.
  3. The test engine is timberio/vector:0.61.0-distroless-static — the tag+digest pinned in the manifest — so the tested engine is the shipped one.

Not mine to run

Seats and the landing are a separate step (f1 reviews the build; a pair + GO follow). Lab-Run: IR-20261112-relayalert is the head commit's last trailer.

Refs F-RELAY-ALERT-SHIPPING.

## What this is F-RELAY-ALERT-SHIPPING, part 1 of 2 (the other half is Homelab/prod). The #955 verdict relay inside `ld-runtime` posts warning alerts to Alertmanager and **fails CLOSED**: a failed send leaves a line in its container log and nothing retries it. Nothing watched that line, so the relay's failure mode was silence. | artefact | file | |---|---| | Vector source → filter → counter | `core/manifests/monitoring/log-metrics/vector-ld-relay.yaml` | | PrometheusRule `LDRelayAlertSendFailing` (+ coverage rule) | `core/manifests/monitoring/rules/ld-relay-alerts.yaml` | | promtool cases | `core/manifests/monitoring/rule-tests/ld-relay-alerts.test.yaml` (generated by `ld-relay-alerts.gen.py`) | | end-to-end test of the shipped Vector config | `core/manifests/monitoring/rule-tests/ld-relay-vector-e2e.sh` + `ld-relay-log-fixtures.log` | The third artefact — the dead-man that does **not** route through the Alertmanager it watches — is in Homelab/prod (`hosts/mgmt/infra-watchers/ld-alertmanager-deadman.*`), together with the Alertmanager egress rule the runtime needed. Any one of the three alone leaves a silent failure mode, which is the thing being fixed. ## The filter matches the REAL log lines (the brief's string was wrong) The first brief for this work named the line `alert send failed`. **That string appears nowhere in the relay's format strings.** Source read at `infra-core 6de8ad695112a523afd11dabaf50bb938a6ab860`: - `cmd/ld-runtime/main.go:1279` — `LD-RELAY ALERT metric=ld_relay_alert_send_total result=sent|failed total_sent=<n> total_failed=<n> <detail> — the alert was ACCEPTED by the alert manager | the alert was NOT delivered` - `relay/poller.go:1406` — `LD-RELAY ALERT rail: the non-paging rail REFUSED the notification (<err>) — the alert was NOT delivered; the latch still stands and the next poll retries` So the config matches `contains(msg, "result=failed") || contains(msg, "the alert was NOT delivered")`. A filter on the brief's string would have shipped a counter that can never increment — a silent monitor on top of the silent monitor this change removes. The lane's rendered sample (`sample-send-failure.log`, copied to `rule-tests/ld-relay-log-fixtures.log`) proves it in both directions: old pattern 0 matches, these predicates 2, healthy lines 0. ## Naming choice (f1 item 5) The app's log line carries `metric=ld_relay_alert_send_total` with a `result` label. This exporter deliberately does **not** reuse that name; it emits **`ld_relay_alert_send_failed_total{node,pod}`**: - `ld_relay_alert_send_total` is the *application's* declared metric. The runtime exposes no `/metrics` and has no ServiceMonitor today, but the day it does, a log-derived series of the same name would merge with it and `result` would be a tag this exporter invented. Same collision-avoidance as the longhorn shipper's F7 note. - `…_send_failed_total` says what it counts, which is the only thing the rule selects on. Both rails (send path, non-paging-rail refusal) count into one series on purpose — `sum by (node)` would collapse a per-rail split anyway. Stated consequence: no denominator; sent-vs-failed lives in the log's `total_sent`/`total_failed`. ## The rule ```promql sum by (node) (increase(ld_relay_alert_send_failed_total[10m])) > 0 or sum by (node) (ld_relay_alert_send_failed_total unless ld_relay_alert_send_failed_total offset 10m) > 0 ``` `severity: warning`, `for: 0m`. The second leg is **#953's B8 fix (`f474c999`) reproduced, not re-derived**: Vector creates the counter at the first failure, so the realistic single failure is a series *born at 1*, and `increase()` over a born-at-1 series that never increments is 0. `for: 0m` because the two legs already carry the hysteresis; adding 30s only gives a scrape gap a chance to hide the first failure. Plus `LDRelayLogMetricShipperDown` (`absent(up{job="vector-ld-relay-log-metrics"} == 1)`, 15m): a log-derived counter's failure mode is that it stops existing, which no rule over the counter can see. `flush_period_secs: 3600`, not 10 — the 3c B8b finding: it is an **expiry**, so at 10s an isolated counter can expire before the 30s scrape. Same silence. ## No new CNP for the log source The source is a **node-local file source**: `/var/log/pods/ld-fleet_ld-runtime-*/runtime/*.log` via a readOnly `hostPath`, in a DaemonSet in `monitoring` with `automountServiceAccountToken: false` and no RBAC. Vector opens no socket to read it, so there is no pod-to-X traffic for a policy to govern, and the namespace-scoped `ld-fleet/ld-runtime` CNP does not select this pod. The Prometheus scrape uses the same ServiceMonitor path `vector-longhorn-log-metrics` already uses, which needs no policy. **No CNP in this PR**; the egress rule in the prod PR is a different rule (the relay's own), not this shipper's. ## Tests — run, outputs as recorded ``` $ promtool check rules ld-relay-alerts.yaml SUCCESS: 2 rules found $ promtool test rules ld-relay-alerts.test.yaml # prom/prometheus:v3.4.0, the cluster's version SUCCESS # 6/6 cases ``` Cases: `increment` fires · `flat` silent · `birth` fires (born at 1, never increments) · `birth-aged` silent · `no-shipper` fires · `shipper-ok` silent. **Not vacuous** — against a leg-(a)-only rule file: ``` $ promtool test rules ld-relay-alerts.test.yaml # expr = increase(...) > 0 only FAILED: name: birth: a series born at 1 fires without ever incrementing, alertname: LDRelayAlertSendFailing, time: 20m, exp:[…], got:[] --- EXIT 1 ``` Full suite (17 test files, 26 rule files) still green after this change: `--- FULL SUITE EXIT 0`. End-to-end, the shipped config driven verbatim (ConfigMap payload extracted from the manifest, fixture mounted under the real `/var/log/pods` path shape, lines appended after the shipper is up so `read_from: end` is exercised): ``` $ ./ld-relay-vector-e2e.sh baseline series named ld_relay_alert_send_failed_total: 0 (expect 0) ld_relay_alert_send_failed_total{node="prod-wk-02",pod="ld-runtime-6d9c8f7b4-x2vqn"} 2 PASS counter=2: the two failure lines matched (send path + non-paging rail) PASS healthy line 'result=sent' produced no series PASS healthy line 'ACCEPTED by the alert manager' produced no series PASS healthy line 'poll tick' produced no series PASS the literal 'alert send failed' does not appear in the fixture at all (a filter on it would have counted 0 of the 2 real failures) === EXIT 0 ``` This test caught a real defect in its own fixture first: a pod log path without the `_<uid>` suffix makes `parse_regex!` drop every event and the counter stays 0 (recorded in the script). Contract gate and server-side validation: ``` $ python3 scripts/monitoring/check_alert_contracts.py ok: every added or modified rule carries a complete ld_* disposition contract, and every rule label/annotation name is Prometheus-valid # rules=124 changed=2 legacy=27 $ kubectl apply --dry-run=server -f core/manifests/monitoring/rules/ld-relay-alerts.yaml prometheusrule.monitoring.coreos.com/ld-relay-alerts created (server dry run) ``` ## Stated limits (not papered over) 1. **The relay's Alertmanager destination is not configured** on the live `ld-runtime` Deployment (no `ALERT*`/`AM*` env, read 2026-11-12). The prod PR admits the egress; it does not create the destination. Until that is set this counter can only report nothing — which is honest, and the rule's `ld_note` says so. 2. **No live scrape yet.** The shipper has never run in this cluster; `LDRelayLogMetricShipperDown` exists for exactly that. 3. The test engine is `timberio/vector:0.61.0-distroless-static` — the tag+digest pinned in the manifest — so the tested engine is the shipped one. ## Not mine to run Seats and the landing are a separate step (f1 reviews the build; a pair + GO follow). `Lab-Run: IR-20261112-relayalert` is the head commit's last trailer. Refs F-RELAY-ALERT-SHIPPING.
IR-20261112: feat(monitoring): observe the ld-runtime relay's failed alert sends
All checks were successful
CI Review / runner-manifest-test (pull_request) Successful in 11s
Details
CI Review / helm-validate (pull_request) Successful in 12s
Details
Alert Disposition Contract / alert-contract (pull_request) Successful in 17s
Details
CI Review / receiver-test (pull_request) Successful in 10s
Details
CI Review / alert-receiver-test (pull_request) Successful in 37s
Details
Lint & Validate / yaml-lint (pull_request) Successful in 21s
Details
Lint & Validate / shellcheck (pull_request) Successful in 7s
Details
CI Review / record-sql-test (pull_request) Successful in 48s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 1m51s
Details
CI Review / cockpit-e2e (pull_request) Successful in 2m56s
Details
CI Review / cockpit-test (pull_request) Successful in 3m7s
Details
CI Review / dispatcher-test (pull_request) Successful in 3m17s
Details
945b702e16 
F-RELAY-ALERT-SHIPPING. The #955 verdict relay in ld-runtime posts warning alerts
to Alertmanager and FAILS CLOSED: a failed send leaves a line in its container log
and nothing retries it. Nothing watched that line, so the relay's failure mode was
silence.

Three artefacts; this commit is the two that live in the cluster:

1. log-metrics/vector-ld-relay.yaml — a Vector DaemonSet (a field-for-field copy of
   vector-longhorn-log-metrics) that tails ONLY the ld-runtime container log, filters
   the relay's real failure lines and exports ONE counter,
   ld_relay_alert_send_failed_total{node,pod}. No log lines are shipped; there is no
   sink other than the Prometheus exporter.

   THE FILTER IS THE POINT. The first brief for this work named the log line
   "alert send failed". That string appears NOWHERE in the relay's format strings
   (read at infra-core 6de8ad69: cmd/ld-runtime/main.go:1279,
   relay/poller.go:1406); the real ones are `result=failed` and the trailing
   `the alert was NOT delivered`. A filter on the brief's string would have shipped
   a counter that can never increment — a silent monitor built on top of the silent
   monitor this change removes. The config matches both real literal forms, and
   rule-tests/ld-relay-vector-e2e.sh drives the SHIPPED config against the lane's
   rendered sample and asserts both directions (2 failures counted; the sent/
   healthy lines count nothing; the fixture contains the old literal zero times).

   NAMING CHOICE, stated because the app names a metric in its own log line: the
   log says `metric=ld_relay_alert_send_total result=sent|failed`. This exporter
   deliberately does NOT reuse that name. It is the application's declared metric;
   the runtime exposes no /metrics today, but the day it does, a log-derived series
   of the same name would merge with it and `result` would be a tag this exporter
   invented. Same collision-avoidance the longhorn shipper records (F7).
   `ld_relay_alert_send_failed_total` says what it counts — the only thing the rule
   selects on — and the two rails (send path, non-paging-rail refusal) are counted
   into one series on purpose, because `sum by (node)` would collapse a per-rail
   split anyway. Consequence, stated: this counter has no denominator; sent-vs-failed
   lives in the log line's total_sent/total_failed fields.

2. rules/ld-relay-alerts.yaml — LDRelayAlertSendFailing, warning (non-paging):
       sum by (node) (increase(ld_relay_alert_send_failed_total[10m])) > 0
       or
       sum by (node) (ld_relay_alert_send_failed_total unless ld_relay_alert_send_failed_total offset 10m) > 0
   The second leg is #953's B8 fix (f474c999) reproduced, not re-derived: Vector
   creates the counter at the FIRST failure, so the realistic single failure is a
   series BORN at 1, and increase() over a born-at-1 series that never increments is
   0. Plus LDRelayLogMetricShipperDown, because a log-derived counter's failure mode
   is that it stops existing, which no rule over the counter can see.

   flush_period_secs: 3600 (not 10) — 3c's B8b finding: it is an EXPIRY, and at 10 s
   an isolated counter can expire before the 30 s scrape, which is the same silence.

Tests (all run; outputs in the PR body):
  * promtool 3.4.0 (the cluster's version): 6/6 cases pass — increment fires, flat
    silent, birth fires, birth-aged silent, shipper-absent fires, shipper-ok silent.
    The suite is NOT vacuous: against a leg-(a)-only rule file the `birth` case
    reports got:[] and promtool exits 1.
  * ld-relay-vector-e2e.sh: the shipped Vector config, driven against the real
    rendered lines, exports ld_relay_alert_send_failed_total 2 for the two failures
    and 0 for the healthy lines. It caught a defect in its own fixture first (a pod
    log path without the `_<uid>` suffix makes parse_regex! drop every event).
  * check_alert_contracts.py: green (124 rules, this file carries the full ld_*
    disposition contract). kubectl apply --dry-run=server accepts the
    PrometheusRule, so the ld_* annotation names are valid to the webhook.

The rule is covered by core/manifests/monitoring/rule-tests/ld-relay-alerts.test.yaml,
generated by ld-relay-alerts.gen.py so the annotation text cannot drift.

Lab-Run: IR-20261112-relayalert
All checks were successful
CI Review / runner-manifest-test (pull_request) Successful in 11s
Details
CI Review / helm-validate (pull_request) Successful in 12s
Required
Details
Alert Disposition Contract / alert-contract (pull_request) Successful in 17s
Details
CI Review / receiver-test (pull_request) Successful in 10s
Details
CI Review / alert-receiver-test (pull_request) Successful in 37s
Details
Lint & Validate / yaml-lint (pull_request) Successful in 21s
Required
Details
Lint & Validate / shellcheck (pull_request) Successful in 7s
Required
Details
CI Review / record-sql-test (pull_request) Successful in 48s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 1m51s
Required
Details
CI Review / cockpit-e2e (pull_request) Successful in 2m56s
Details
CI Review / cockpit-test (pull_request) Successful in 3m7s
Details
CI Review / dispatcher-test (pull_request) Successful in 3m17s
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin plane/IR-20261112-relayalert:plane/IR-20261112-relayalert
git switch plane/IR-20261112-relayalert
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
operator-review

Requires human operator review before merge — dangerous class (RBAC, secrets, network policy, Terraform) or explicitly gated. Never auto-merge.

No labels
operator-review
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Homelab/infra-core!964
No description provided.