HOMELAB-1848: LD4.5a alert audit + silence register (Stage 1) #573
No reviewers
Labels
No labels
operator-review
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Homelab/infra-core!573
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "plane/HOMELAB-1848-alert-audit"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Design brief:
.rpiv/artifacts/epic-ld4/design-LD4.5a-alert-audit.md. Orchestrator rulings on HOMELAB-1848 applied (30-day window not derivable → measure at runtime + recorder; denominator = repo-owned 63; annotations-only contract; diff-scoped gate + ratchet; silence reconciliation flag-only; inhibition/retention deferred).Class: C (rule YAML + scripts + CI). ArgoCD
monitoring-rulesauto-syncs the rules. Nomonitoring-namespace stack config is touched — no Alertmanager route, no inhibition, no retention change. Ready to land behind the normal review.The measured starting point
Window: 10.00 d (
2026-09-22T03:55Z → 2026-10-02T03:55Z), measured from the oldest sample in the TSDB — not fromretention.time, which reads 15d while the real bound isretention.size=30GiB. Prometheus 3.4.0, evaluation interval 30s.After this PR (projected)
The "after" number is computed, not asserted:
alert_census.py --severity-from manifestsreads each rule's severity out of the manifests and runs the same live route-tree walk, so the delta is derived. Artefacts for both runs are committed (triage/2026-10-02-W10d/andtriage/2026-10-02-W10d-projected-after/).What is in the PR
scripts/monitoring/alert_census.py— the census/triage generator. Window measured at runtime;census.json.queries[]carries every PromQL string with its step and bounds (Level-4 re-derivability); page-eligibility is a walk of the live Alertmanager route tree, never a hard-coded list; repo attribution comes from the PrometheusRule manifests. Fails closed below a 7-day window.lab-director/alerts/triage/2026-10-02-W10d/{census.json,triage.json,triage.md,index.md}+ the projected-after run +dispositions.yaml(read back out of the manifests, so table and code cannot drift).lab-director/alerts/silences.yaml— register-as-code, back-filled with the 2 live silences (both ad-hoc, neither with a Plane ticket — that absence is the finding).scripts/monitoring/silence_reconcile.py— flag-only reconciliation. No POST, no DELETE: asserted by an AST test that every constructed HTTP request is a GET with no body, plus a grep in the new CI job.scripts/monitoring/check_alert_contracts.py+.forgejo/workflows/alert-contract.yaml— the disposition-contract gate: diff-scoped refusal (001/002) + legacy ratchet (003/004), 32 ratcheted rules.critical → infowith theld-*contract, plus the contract on 9 rules that staykeep-page. Every demotion is evidenced in itsld-noteannotation.lab-director/alerts/history/2026-10-02.jsonl— the first recorder sample. The recorder is the honest path to the ticket's literal 30-day census (true from ~2026-11-01); it is sampled, and the README says so.deploy/— systemd user units (LD4.4 has not landed, so no routines registry). Copies, not the source; stated as such.Why 15 demotions, and what they are
> 0restarts), PodCrashLooping (99 series / 44 notifications,for: 0m), LonghornVolumeFaulted (47 series), NodeMemoryOvercommit (55 % observed vs a 200 % line), LokiIngesterFailing (raw counter== 0), TempoSpanPipelineStalled (22 pending, 0 firing — anandbetween differently-sampled counters)Claim honesty, stated up front: the 6 dead-rule demotions are a no-op safety step, not a paging reduction — nothing was paging because nothing could fire. Their deletion is proposed for the follow-up PR (demote first, delete after one clean cycle). 9 of the 15 are genuine paging reductions. This is said plainly so the delta is not oversold.
Deliberately NOT demoted, with the reason recorded in the annotation:
for:problem, not a silencing problem.severity="critical"route. Only a route change (E-adj, deferred) or deleting the rule would.up=1for the whole window; Longhorn disk peaked at 70 % against a 90 % line).No
deletedisposition is proposed. The window is 10.0 d, so a slow-cadence rule shows zero evidence and a deletion would be window truncation, not proof.Findings that are not dispositions
EtcdHealthCritical,TalosEtcdStatusError,TalosEtcdAlarmActive,ControlPlaneStaticPodMissing).EtcdHealthCriticalwas the #2 source of notifications in the rail over the spool (56 lines / ~2 days): Alertmanager re-notifying stale state for a rule that is not deployed. Fixing the route is E-adj → proposed, not landed.ControlPlaneStaticPodsCrashLoopingwas notified on 2026-09-30 yet has noALERTSsamples anywhere in the window. Recorded as an open anomaly innotes/2026-10-02-live-probe-findings.mdrather than papered over.origin/liveforcore/manifests/monitoring/rules/(148 lines: the Longhorn disk-scheduling group from HOMELAB-1854 and the HOMELAB-1835 storage-fraction fix). One live page-eligible rule (LonghornDiskSchedulingHeadroomCritical) therefore has no manifest on this base and could not be annotated from this branch. Not a defect of this PR, but the epic will need alivemerge before this can be the whole truth.Verify (Level-4)
Re-derivability — run on the same window:
Two runs with a pinned window are byte-identical (
triage.json,census.json,triage.md,index.md). The measured window, both retention flags, the evaluation interval and every PromQL string with its step and bounds are incensus.json.python3 -m unittest discover -s scripts/monitoring/tests— 48 tests, OK (route-walk semantics incl. first-match-wins andcontinue; the summed-percentage trap; episode derivation forfor:0vsfor>0; the contract enums, ratchet legs and the bootstrap case; the reconciler's no-write-path AST assertion; the planted unregistered silence).Not in this PR (deferred / E-adj)
inhibit_rulesand route changes — proposals only, in the triage table.defaultRules+ vendored copies) — audited in the annex, proposals listed, follow-up ticket.The window is MEASURED at runtime, never assumed: retention here is size-bound (storage.tsdb.retention.size=30GiB) even though retention.time reads 15d, so the realised window was 10.00d. A census that assumed 30d would report every slow-cadence alert as never-firing, and a delete disposition built on that would be wrong. census.json therefore records the window, both retention flags, the evaluation interval, and queries[] — every PromQL string with its step and bounds, so a reviewer can re-derive every number by hand. Measured 2026-10-02 over 10.00d: repo-owned (the denominator) 63 rules, 27 page-eligible = 42.9% all live rules (annex) 210 rules, 66 page-eligible instances (64 names) fired and page-eligible 9 repo-owned The generator FAILS CLOSED below a 7-day window (--min-window-days): a short window manufactures false never-firing rules. Page-eligibility is a walk of the LIVE Alertmanager route tree (parsed from /api/v2/status config.original), never a hard-coded list; repo attribution comes from the PrometheusRule manifests. Two traps are fixed and regression-tested: summing series for %window (348% on a three-series rule) and classifying by substring on the rule file path (the mount directory contains 'kube-prometheus-stack'). Findings the census surfaced, in notes/2026-10-02-live-probe-findings.md: - 6 repo-owned page-eligible rules are DEAD: their metrics have no series at all - the route tree names 5 alertnames of which 4 have no live rule; one of those (EtcdHealthCritical) was the #2 source of notifications in the rail, i.e. Alertmanager re-notifying stale state for a rule that is not deployed - ControlPlaneStaticPodsCrashLooping was notified on 2026-09-30 with no ALERTS samples anywhere in the window: recorded as an open anomaly, not papered over Also: --record appends a sampled daily line per alertname (the honest path to the ticket's literal 30-day census, true from ~2026-11-01), and --severity-from manifests computes the PROJECTED page-eligible count for a dispositions change that is not deployed yet, using the same route walk.Standing rule, enforced: every alert rule ships with a disposition contract — ld-owner, ld-action, ld-frequency, ld-disposition (keep-page|keep-record), ld-silence-policy, ld-review-after. A rule without a consumer does not merge. Annotations, deliberately NOT labels: annotations do not participate in a series' identity, so adding them creates no new ALERTS series, churns no fingerprints and costs no cardinality. Labels would change the identity of every series — the cheapest possible way to corrupt the census in the previous commit. Two legs, because each alone has a hole: - diff-scoped refusal (ALERT-CONTRACT-001/002): rules added or modified by the PR must carry all six annotations with valid values - legacy ratchet (ALERT-CONTRACT-003/004): legacy-rules.txt lists the 32 rules that predate the contract; a listed rule that is MODIFIED without acquiring the contract is refused, and the file may only shrink (a new line is refused) Stated limits, in the README rather than hidden: the gate can be bypassed by the E-adj paths (Alertmanager route tree, defaultRules) which are protected-namespace and controlled by review class; it refuses the ABSENCE of a contract, not a dishonest one; and prod's 5 edge-probe rules carry the contract but are not yet CI-gated (follow-up).Separate commit from the tooling so a revert of the paging change is one revert. 27 repo-owned rules were page-eligible (severity=critical reaches pi-alert-receiver). 15 are demoted to severity=info with the ld-* contract, which moves the repo-owned page fraction 42.9% -> 19.1% (27 -> 12 of 63), the ticket's <20% target. Evidence per rule is in the ld-note annotation and in lab-director/alerts/dispositions.yaml; every demotion is one of: (a) DEAD RULES (6) — the metric has no series anywhere in the TSDB, so the rule can never fire: VeleroBackupFailure, VeleroBackupMissing, LonghornBackupTargetMissing, LonghornVolumeReadOnly, PVEDiskFull, PVENodeUnreachable. Demotion is a no-op SAFETY step, not a paging reduction; deletion is proposed for the follow-up PR (demote first, delete after one clean cycle). Said plainly so the delta is not oversold. (b) CHRONIC STATE (3) — paging on a condition rather than an event, and duplicating a page-eligible chart rule: PersistentVolumeCritical (83.1% of the window at >90% PVC fill), NodeDiskFull (66 episodes around a flat 90% line), ClusterMemoryHeadroomCritical (fires below 2 GiB cluster-available while the measured minimum was 107 GiB — the threshold is ~54x below reality, with per-node usage peaking at 74% against NodeMemoryPressure's 90% line). (c) FIRES CONSTANTLY, NEVER ACTED ON (5) — LonghornCSIAttacherLeaderLost (190 episodes: the threshold is literally '> 0 restarts'), PodCrashLooping (99 series, 44 notifications, for: 0m, duplicating the chart's damped KubePodCrashLooping), LonghornVolumeFaulted (47 volumes matched, consistent with detached/stale series), NodeMemoryOvercommit (55% observed against a 200% line — a capacity signal, not an incident), LokiIngesterFailing (compares a raw counter to 0), TempoSpanPipelineStalled (an 'and' between differently-sampled counters: 22 pending episodes, 0 firing). Deliberately NOT demoted, with reasons recorded in the annotations: - EdgeProbeDown (critical) — measured and found NOT to be a duplicate: it matches only the three self-hosted public services while the warning variant carries the other 25 edge hosts. Its flapping is a 'for:' problem, not a silencing problem. - ControlPlaneStaticPodsCrashLooping — a severity demotion CANNOT stop it paging: the Alertmanager route tree names it explicitly ahead of the severity=critical route. - CiliumEnvoyDown, NodeDown, PrometheusWALCorruption, LonghornDiskCritical, CNPGBackupStale — unique coverage, verified against the metric's observed range over the window (e.g. cilium-envoy up=1 and node-exporter up=1 for the whole window, Longhorn disk peaking at 70% against a 90% line). No deletion disposition is proposed: the window is 10.0d, so a slow-cadence rule would show zero evidence and a deletion would be window truncation, not proof. Every annotated rule's expression was verified byte-identical (whitespace normalised) to the running Prometheus revision, so the dispositions are authored against what the cluster actually evaluates.Review —
0e5f950verdict: pass
Head reviewed:
0e5f950acfbd87b7a0014796bf9f67b4fe5e72c1· baseepic/LD4@1519dd2. CI on this head:alert-contractrun #1694 pass,ci.yamljobs run #1695 pass.Independent review in a fresh context, primary sources only (the diff, HOMELAB-1848's acceptance criteria, and
design-LD4.5a-alert-audit.md). No blocker.1 — Level-4 numbers re-derived, not re-read. My own census run on the pinned window (read-only via the existing 19090/19093 forwards, branch manifests + prod edge-probes): repo-owned 63 / 27 / 42.86 %, fired and page-eligible 9, all-rules 210 / 66 / 31.4 %; projected-after with
--severity-from manifests12 / 63 = 19.05 %, annex 52 / 210 = 24.76 %, fired and page-eligible 4. Every figure matches the committed artefacts. Two pinned runs are byte-identical on all four artefacts (census.json,triage.json,triage.md,index.md).--window 3dfails closed (exit 3). Page-eligibility is a live walk of the Alertmanager route tree fromconfig.original, not a hard-coded list.2 — Gate exercised with planted mutations, not just read. Real CLI, real git, base ref on disk: real PR shape (
rules=56,changed=24) → ok, exit 0. New contract-less rule →ALERT-CONTRACT-001, exit 1. Ratchet grows →ALERT-CONTRACT-003, exit 1. Ratchet shrinks → ok. Ratcheted rule modified without the contract →ALERT-CONTRACT-004, exit 1. Untouched legacy rules grandfathered. 48 unit tests OK. Backed by live CI: demo PR #574 run #1687alert-contractFAILED, closed unmerged.3 — Reconciler no-write path is a real assertion. The AST test requires every
urllib.request.Request(...)to bemethod="GET"with no body and forbids write verbs as string literals; proven by plantingRequest(url, method="POST")(test failed, 14 tests 1 failure) and reverting (48 OK). A live read-only run flagged both silences, verdictfindings, exit 0, no write to Alertmanager.4 — The 15 demotions and 6 dead-rule claims. Diff holds exactly 15
critical → infochanges and 24 rules carryingld-disposition(15 demoted + 9 keep-page). All six dead-rule metrics verified against the TSDB over the census window:velero_backup_failure_total,velero_backup_last_successful_timestamp,longhorn_backup_target_status,node_filesystem_readonly{mountpoint=~\"/var/mnt/u-longhorn/.*\"},pve_up,pve_storage_used_bytes→ 0 series / 0 points each, against controlcount(up)= 1 series / 1441 points. TheEdgeProbeDownrevert is in the branch (prod side): the critical variant iskeep-page, not demoted, with the per-series correction recorded in itsld-note; the warning variant correctly carries no contract.5 — Honesty and hygiene. The 6 dead-rule demotions are declared a no-op safety step, not a paging reduction, in the PR body, the ticket comment and each rule's own
ld-note. 9 of 15 are genuine paging reductions, arithmetically consistent with the measured delta. Nodeletedisposition is proposed, with the 10.0 d window-truncation reason stated. The recorder is documented as sampled, not raw.ControlPlaneStaticPodsCrashLooping's notified-but-never-firing anomaly is recorded rather than papered over. Zero credential-shaped additions in the diff.6 — Missing-manifest rule, named for the epic rebase.
LonghornDiskSchedulingHeadroomCritical—page_eligible: true,owner_repo: infra-core,proposed_disposition: nullin both my before and after runs. Its manifest is absent from this base (epic behind live); it arrives with HOMELAB-1854 and HOMELAB-1835, so the epic → live rebase must carry itsld-*contract. It is correctly counted inside the 63 today.Non-blocking concerns (no blocker is open):
rule_key()isfile:alertnamecollected into a dict andapply_dispositions()joins on alertname only. infra-core has no duplicate(file, alertname)key, so this PR's own coverage and CI are complete (the gate judged all 24 changed rules). Latent hole: a modified instance is invisible to the diff-scoped leg when one file holds two same-named rules. Already visible here: the committeddispositions.yamlcarries two identicalEdgeProbeDownrows both labelledseverity: critical, page_eligible: truewith the critical variant's contract text, while prod's warning instance is genuinely not under contract (27 rows vs 26 annotated rules). Key on(file, alertname, expr_sha256)— the census already computes that hash — before vendoring the gate into prod, and regenerate the artefact.triage/2026-10-02-W10d/index.md's re-derive command hard-codes the builder's worktree paths (/home/aaron/repos/homelab-wt/LD4/...), so it does not run as-is from another checkout. The PR body's$PWDform is the portable one.Merge note: the target is
epic/LD4and themonitoring-rulesArgoCD app syncs fromlive, so this merge deploys nothing. The paging effect (9 rules stop paging; 6 are dead no-ops) lands at theepic → livemerge, together with theLonghornDiskSchedulingHeadroomCriticalannotation.