#574 - DEMO ONLY — do not merge: contract-less rule must be refused (HOMELAB-1848 AC) - Homelab/infra-core

aaron commented

2026-10-02 04:05:12 +00:00

Owner

DEMO ONLY — DO NOT MERGE. This PR is closed without merging.

HOMELAB-1848 acceptance criterion: "alert-rule PR lacking a disposition contract is refused."

This PR adds one real rule (ContractDemoNoConsumer) to a real rule file with no ld-* annotations. The Alert Disposition Contract workflow (which lands in #573) must go red with ALERT-CONTRACT-001.

Local run of the same check:

$ python3 scripts/monitoring/check_alert_contracts.py --base origin/epic/LD4
alert-contract gate: base=origin/epic/LD4 · rules=57 · changed=25 · legacy=32

ALERT-CONTRACT-001: rule ContractDemoNoConsumer in core/manifests/monitoring/rules/node-alerts.yaml is missing ld-owner, ld-action, ld-frequency, ld-disposition, ld-silence-policy, ld-review-after — a rule without a consumer does not merge (see lab-director/alerts/README.md)

refused: 1 contract violation(s). The contract is documented in lab-director/alerts/README.md.
$ echo $?
1

CI link for this PR to be recorded on HOMELAB-1848. The fixture test that proves the same refusal survives the closed PR lives in scripts/monitoring/tests/test_alert_contracts.py::test_contract_less_added_rule_is_refused.

**DEMO ONLY — DO NOT MERGE. This PR is closed without merging.** HOMELAB-1848 acceptance criterion: *"alert-rule PR lacking a disposition contract is refused."* This PR adds one real rule (`ContractDemoNoConsumer`) to a real rule file with **no `ld-*` annotations**. The `Alert Disposition Contract` workflow (which lands in #573) must go **red** with `ALERT-CONTRACT-001`. Local run of the same check: ``` $ python3 scripts/monitoring/check_alert_contracts.py --base origin/epic/LD4 alert-contract gate: base=origin/epic/LD4 · rules=57 · changed=25 · legacy=32 ALERT-CONTRACT-001: rule ContractDemoNoConsumer in core/manifests/monitoring/rules/node-alerts.yaml is missing ld-owner, ld-action, ld-frequency, ld-disposition, ld-silence-policy, ld-review-after — a rule without a consumer does not merge (see lab-director/alerts/README.md) refused: 1 contract violation(s). The contract is documented in lab-director/alerts/README.md. $ echo $? 1 ``` CI link for this PR to be recorded on HOMELAB-1848. The fixture test that proves the same refusal survives the closed PR lives in `scripts/monitoring/tests/test_alert_contracts.py::test_contract_less_added_rule_is_refused`.

aaron added 6 commits

2026-10-02 04:05:12 +00:00

HOMELAB-1848: chore(monitoring): wip alert census generator + live probe findings d50f6970fe

WIP insurance commit. alert_census.py is functional and reproduces the brief's
repo-owned numbers exactly (63 rules / 27 page-eligible / 42.86%) against the
live retention-bounded 10.0d window; two pinned-window runs are byte-identical.

Also records the raw probe findings so a replacement builder can reconstruct the
derivations without re-discovering them: window measurement, the PromQL that
works, the route-tree walk, the 6 dead rules whose metrics do not exist, and the
open ControlPlaneStaticPodsCrashLooping anomaly (notified but no ALERTS samples).

Still WIP: tests, dispositions, silence reconciler, contract CI gate, PR bodies.

HOMELAB-1848: feat(monitoring): alert census + triage generator 15d030ab1d

The window is MEASURED at runtime, never assumed: retention here is size-bound
(storage.tsdb.retention.size=30GiB) even though retention.time reads 15d, so the
realised window was 10.00d. A census that assumed 30d would report every
slow-cadence alert as never-firing, and a delete disposition built on that would
be wrong. census.json therefore records the window, both retention flags, the
evaluation interval, and queries[] — every PromQL string with its step and bounds,
so a reviewer can re-derive every number by hand.

Measured 2026-10-02 over 10.00d:
  repo-owned (the denominator)  63 rules, 27 page-eligible = 42.9%
  all live rules (annex)        210 rules, 66 page-eligible instances (64 names)
  fired and page-eligible       9 repo-owned
The generator FAILS CLOSED below a 7-day window (--min-window-days): a short
window manufactures false never-firing rules.

Page-eligibility is a walk of the LIVE Alertmanager route tree (parsed from
/api/v2/status config.original), never a hard-coded list; repo attribution comes
from the PrometheusRule manifests. Two traps are fixed and regression-tested:
summing series for %window (348% on a three-series rule) and classifying by
substring on the rule file path (the mount directory contains
'kube-prometheus-stack').

Findings the census surfaced, in notes/2026-10-02-live-probe-findings.md:
  - 6 repo-owned page-eligible rules are DEAD: their metrics have no series at all
  - the route tree names 5 alertnames of which 4 have no live rule; one of those
    (EtcdHealthCritical) was the #2 source of notifications in the rail, i.e.
    Alertmanager re-notifying stale state for a rule that is not deployed
  - ControlPlaneStaticPodsCrashLooping was notified on 2026-09-30 with no ALERTS
    samples anywhere in the window: recorded as an open anomaly, not papered over

Also: --record appends a sampled daily line per alertname (the honest path to the
ticket's literal 30-day census, true from ~2026-11-01), and --severity-from
manifests computes the PROJECTED page-eligible count for a dispositions change
that is not deployed yet, using the same route walk.

HOMELAB-1848: feat(monitoring): silence register-as-code + flag-only reconciler 15bb5f354e

Before this change there were ZERO silences in git and 2 live in Alertmanager,
both created ad-hoc by 'pi-agent (incident-2026-10-01)' with no ticket and no
expiry review. A silence is a decision to stop telling a human about a condition;
a decision with no owner and no review date is indistinguishable from an outage
of the check. Both live silences are back-filled into silences.yaml, including
the finding that neither has a Plane ticket.

silence_reconcile.py compares the register with live Alertmanager and classifies:
unregistered-active/-expired, registered-expired, registered-absent, matcher-drift,
expiring-within-24h, over-max-active, ticket-missing, registered-closed.

FLAG-ONLY, by construction: the module contains no POST and no DELETE, asserted by
a unit test that walks its AST and requires every constructed HTTP request to be a
GET with no body (plus a grep in the alert-contract CI job). --fail-on-finding
changes only the exit code; it still writes nothing to Alertmanager. The ticket's
planted-silence criterion is met by a deterministic fixture test (an inert
LD45aPlantTest silence appears as unregistered-active) rather than by creating an
object in live Alertmanager.

Tier-2 interim home: systemd user units under deploy/ (LD4.4 has not landed, so
there is no routines registry). Those files are COPIES — the VM is authoritative —
and that is stated rather than pretended. Committing the daily outputs is
deliberately not automated (ALERT_CENSUS_AUTOPUSH is off by default).

HOMELAB-1848: feat(ci): alert disposition contract gate (diff-scoped + ratchet) cfcc3c2afc

Standing rule, enforced: every alert rule ships with a disposition contract —
ld-owner, ld-action, ld-frequency, ld-disposition (keep-page|keep-record),
ld-silence-policy, ld-review-after. A rule without a consumer does not merge.

Annotations, deliberately NOT labels: annotations do not participate in a series'
identity, so adding them creates no new ALERTS series, churns no fingerprints and
costs no cardinality. Labels would change the identity of every series — the
cheapest possible way to corrupt the census in the previous commit.

Two legs, because each alone has a hole:
  - diff-scoped refusal (ALERT-CONTRACT-001/002): rules added or modified by the
    PR must carry all six annotations with valid values
  - legacy ratchet (ALERT-CONTRACT-003/004): legacy-rules.txt lists the 32 rules
    that predate the contract; a listed rule that is MODIFIED without acquiring
    the contract is refused, and the file may only shrink (a new line is refused)

Stated limits, in the README rather than hidden: the gate can be bypassed by the
E-adj paths (Alertmanager route tree, defaultRules) which are protected-namespace
and controlled by review class; it refuses the ABSENCE of a contract, not a
dishonest one; and prod's 5 edge-probe rules carry the contract but are not yet
CI-gated (follow-up).

HOMELAB-1848: feat(monitoring): 15 demotions + ld-* contract on 24 rules

Plan in progress...

CI Review / receiver-test (pull_request) Successful in 4s

Details

CI Review / helm-validate (pull_request) Successful in 7s

Details

Alert Disposition Contract / alert-contract (pull_request) Successful in 7s

Details

CI Review / alert-receiver-test (pull_request) Successful in 20s

Details

CI Review / dispatcher-test (pull_request) Successful in 2m18s

Details

d760210dd4

Separate commit from the tooling so a revert of the paging change is one revert.

27 repo-owned rules were page-eligible (severity=critical reaches
pi-alert-receiver). 15 are demoted to severity=info with the ld-* contract, which
moves the repo-owned page fraction 42.9% -> 19.1% (27 -> 12 of 63), the ticket's
<20% target. Evidence per rule is in the ld-note annotation and in
lab-director/alerts/dispositions.yaml; every demotion is one of:

  (a) DEAD RULES (6) — the metric has no series anywhere in the TSDB, so the rule
      can never fire: VeleroBackupFailure, VeleroBackupMissing,
      LonghornBackupTargetMissing, LonghornVolumeReadOnly, PVEDiskFull,
      PVENodeUnreachable. Demotion is a no-op SAFETY step, not a paging
      reduction; deletion is proposed for the follow-up PR (demote first, delete
      after one clean cycle). Said plainly so the delta is not oversold.
  (b) CHRONIC STATE (3) — paging on a condition rather than an event, and
      duplicating a page-eligible chart rule: PersistentVolumeCritical (83.1% of
      the window at >90% PVC fill), NodeDiskFull (66 episodes around a flat 90%
      line), ClusterMemoryHeadroomCritical (fires below 2 GiB cluster-available
      while the measured minimum was 107 GiB — the threshold is ~54x below
      reality, with per-node usage peaking at 74% against NodeMemoryPressure's
      90% line).
  (c) FIRES CONSTANTLY, NEVER ACTED ON (5) — LonghornCSIAttacherLeaderLost (190
      episodes: the threshold is literally '> 0 restarts'), PodCrashLooping (99
      series, 44 notifications, for: 0m, duplicating the chart's damped
      KubePodCrashLooping), LonghornVolumeFaulted (47 volumes matched, consistent
      with detached/stale series), NodeMemoryOvercommit (55% observed against a
      200% line — a capacity signal, not an incident), LokiIngesterFailing
      (compares a raw counter to 0), TempoSpanPipelineStalled (an 'and' between
      differently-sampled counters: 22 pending episodes, 0 firing).

Deliberately NOT demoted, with reasons recorded in the annotations:
  - EdgeProbeDown (critical) — measured and found NOT to be a duplicate: it
    matches only the three self-hosted public services while the warning variant
    carries the other 25 edge hosts. Its flapping is a 'for:' problem, not a
    silencing problem.
  - ControlPlaneStaticPodsCrashLooping — a severity demotion CANNOT stop it
    paging: the Alertmanager route tree names it explicitly ahead of the
    severity=critical route.
  - CiliumEnvoyDown, NodeDown, PrometheusWALCorruption, LonghornDiskCritical,
    CNPGBackupStale — unique coverage, verified against the metric's observed
    range over the window (e.g. cilium-envoy up=1 and node-exporter up=1 for the
    whole window, Longhorn disk peaking at 70% against a 90% line).

No deletion disposition is proposed: the window is 10.0d, so a slow-cadence rule
would show zero evidence and a deletion would be window truncation, not proof.

Every annotated rule's expression was verified byte-identical (whitespace
normalised) to the running Prometheus revision, so the dispositions are authored
against what the cluster actually evaluates.

HOMELAB-1848: test(ci): DEMO ONLY — contract-less rule to prove the gate refuses

Plan in progress...

Alert Disposition Contract / alert-contract (pull_request) Failing after 5s

Details

CI Review / receiver-test (pull_request) Successful in 5s

Details

CI Review / helm-validate (pull_request) Successful in 7s

Details

CI Review / alert-receiver-test (pull_request) Successful in 22s

Details

CI Review / dispatcher-test (pull_request) Successful in 2m36s

Details

75d7e38e8b

Throwaway. This PR exists to demonstrate ALERT-CONTRACT-001 firing on a real rule
added by a real PR (the ticket's acceptance criterion). It is closed without
merging, and the workflow file that makes it red lands in PR #573.