HOMELAB-2694: feat(monitoring): warn on node memory pressure-stall (PSI) before it kills the node #837
No reviewers
Labels
No labels
operator-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Homelab/infra-core!837
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "plane/HOMELAB-2694-psi-memory-alert"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
HOMELAB-2694: feat(monitoring): warn on node memory pressure-stall (PSI) before it kills the node
What
One new rule file,
core/manifests/monitoring/rules/node-memory-pressure-alerts.yaml:NodeMemoryPressurePSImax by (instance) (rate(node_pressure_memory_stalled_seconds_total{job="node-exporter"}[2m])) > 0.012mNodeMemoryPressurePSIMetricMissingup==1but no pressure series for that instance15mseverity: warningmatches only the matcher-less non-pagingalert-intakedigest route in prod's Alertmanager, never
pi-alert-receiver— the samerouting rail the 2488 IO-PSI warning uses. No Alertmanager config change is in
this PR (that file is
prod); I verified the route tree as it stands.Plus tests and the backtest that produced the numbers:
rule-tests/node-memory-pressure.{test.yaml,-sabotage.sh,-backtest.py,-backtest.md}.Why
prod-wk-03's memory was consumed by a growing tmpfs/Shmem region outside every
pod cgroup on 2026-11-05 (
working-files/infra-stability/HOMELAB-wk03-memory-0411.md):Shmem 2.5 → 7.9 GiB, MemAvailable 6.3 GiB → 1.21 GiB, and the node collapsed
twice (04:15Z, 07:35Z), killing Longhorn volume engines and the sandbox
containers (exit 137). Nothing paged in time — and one memory rule was already
on the job, unseen:
kubectl top noderead 44 % at 07:20Z while MemAvailable was 1.89 GiB —cAdvisor's working set subtracts inactive file, so a tmpfs consumer is
invisible to it and to every "memory used %" rule.
NodeMemoryHigh(capacity-alerts,>70 %used,for: 15m, warning, digestonly, no
ld_action) was firing on wk-03 from 03:32:00Z through bothcollapses —
ALERTS_FOR_STATEentered pending at 03:16:57Z, 21 firingsamples at a 5 m step, last ≥ 07:35:00Z — i.e. 42 minutes before the 04:14:30Z
engine death. It was never seen: it reaches only the matcher-less non-paging
alert-intakedigest, carries nold_action, and bought no lead time.NodeMemoryPressure(>90 % used,for: 10m, critical) fired at 04:15 — afterthe engines were already dead.
MemoryPressure(its floor is ~100 MiB; this node's was1.2 GiB) and never evicted.
So detection existed — the gap was delivery. What no rule consumed was
memory stall pressure, the one signal that moved early enough to act on; that
is what this rule adds. Delivery to the owning hypervisor is the companion half
of HOMELAB-2694 and is deliberately not in this PR.
Calibration — the ticket's threshold fires never; both collapses calibrated
The investigation proposed
rate(...[2m]) > 0.5and estimated ~5 min of warning.Backtested against the counter on the rule's own 30 s grid, 0.5 fires zero
times in 7 days, including both collapses: the measured peaks of that exact
expression are 0.0998 s/s (04:15:30Z) and 0.0604 s/s (07:36:00Z). The
investigation's "PSI-full 2.25 → 5.07 s/s" line is the kernel's
full avg10percentage (2.25 %, 5.07 %), not seconds per second; 2.25 % = 0.0225 s/s.
Landed as-is, the rule would have been green, passed promtool, and stayed silent
through the incident it was written for.
0.02is the trap worth naming: it clears the first collapse's peaks butcollapse #2's stall rose as two separate ~1 min bumps (07:29-07:30 and
07:32-07:33), so it never held 0.02 for the 2 m
for:. 0.01 catches both.The 5-minute lead is not reachable and is not claimed. The stall is genuinely
absent before 04:10:30Z and 07:29Z, and
for: 2msits on top of that. What therule buys is 2 m 00 s before the first engine death (and 2 m 35 s before the
dind SIGKILL at 04:15:05Z) on collapse #1, and 2 m 21 s on collapse #2.
Live verification (2026-11-05T08:05Z,
127.0.0.1:9090)node_pressure_memory_stalled_seconds_totalexists for all 9 Talos nodesunder
job="node-exporter"(plus daytona and mgmt under their own scrapeConfigjobs).
up{job="node-exporter"}= 1 for all 9.container, endpoint, instance, job, namespace, pod, service—no node-name label, only the node IP in
instance. Node map joined vianode_uname_info(10.10.10.11 cp-02, .12 cp-03, .13 cp-04, .20 wk-01,.21 wk-02, .22 wk-03, .23 wk-04, .25 wk-05, .26 wk-06).
max by (instance)(rate(...[2m]))is 0.000000 for8 nodes and 0.000069 for wk-02;
node_pressure_memory_waiting_seconds_total("some") matches.
{job="node-exporter"}, not aprod-wk-*name: the series hasno node-name label, so a worker-only selector would be a hardcoded IP regex
that breaks on the next node add or re-IP. Control planes cost nothing — their
7-day maxima are 0.0000-0.0002 s/s. daytona and mgmt are deliberately out of
scope (different scrapes, different roles; mgmt's 7-day max is 0.4745 s/s, a
separate investigation).
max by (instance)is load-bearing, not cosmetic: node-exporter pods arerecreated by DaemonSet rollouts and each new pod makes a fresh series with a
different
podlabel, which would otherwise double the alert on everyexporter restart. Case 4 + sabotage E pin that.
7-day backtest (
rule-tests/node-memory-pressure-backtest.md)Window 2026-10-29T08:00Z .. 2026-11-05T08:00Z, 30 s grid,
for: 2m, Prometheusstate machine replayed offline. At the landed 0.01: 22 firings, 3 inside the two
labelled collapse windows, 19 outside, over 9 nodes × 8 UTC days = 72 node-days.
Max outside firings in one node-day is 9 (prod-wk-05, 2026-11-04 — the dind CI
node, 2.7-3.1 GiB per dind pod); every other node-day with any firing is ≤ 3.
Those are real 2-3 min stalls at 2-7 % of wall time, separated by ≥ 9 min — not a
burst storm, and the reason the rule records instead of paging. For comparison the
sibling IO-PSI warning shipped with 57 outside firings over 7.75 days.
Recorded tuning lever, not taken:
for: 3mat 0.01 cuts outside firings to 7(worst node-day 3) but drops the lead to 60 s / 81 s. Kept at 2m because the
alert does not page; if the intake queue shows fatigue,
for: 3mis the measuredknob.
Verification run
promtool test rules(runs all rule files in the repo): SUCCESS, exit 0.New file: 7 cases — positive, sub-threshold 0.008, 0.0099 (1 % under the line),
two exporter-pod series → exactly one alert, foreign job silent, metric-missing
fires at 16m and not at 10m, healthy fleet silent. 10.10.10.22 etc. fixtures
are ramps whose rate is d/30 s/s by construction.
node-memory-pressure-sabotage.sh: all controls behaved as required — 7per-case baselines + whole-file baseline green; 9 mutations all red
(threshold 0.01→0.005, →0.5, →0.009;
for: 2m→1m; aggregation removed; job pinremoved; MetricMissing
for:15m→5m; MetricMissing deleted).origin/live: the same test file run against the rules asthey exist on
origin/liveexits 1 (the rule is absent), so the greenabove is not vacuous.
scripts/monitoring/check_alert_contracts.py --base origin/live: ok — 2changed rules, both carry the complete
ld_*contract; every label/annotationname is Prometheus-valid.
Review arrangement
Two DeepSeek seats are declared for this change (seat 1 and seat 2, both
omniroute/deepseek-flashworker sessions with their own fresh context, neitherof them this authoring session). This PR is deliberately UNMERGED: it is left
open with green CI for the landing path to exercise the lock. Merging is the
landing worker's job, as
pi-landing, and nothing here is force-pushed.Ticket: HOMELAB-2694 (parent HOMELAB-2479).
prod-wk-03's memory was consumed by a growing tmpfs/Shmem region OUTSIDE every pod cgroup on 2026-11-05: Shmem 2.5 -> 7.9 GiB, MemAvailable 6.3 GiB -> 1.21 GiB, and the node collapsed twice (04:15Z, 07:35Z), killing Longhorn volume engines and the sandbox containers (exit 137). Nothing warned. `kubectl top node` read 44% while MemAvailable was 1.89 GiB, because cAdvisor's working set subtracts inactive file and cannot see a tmpfs consumer; NodeMemoryPressure (>90% used, for: 10m) fired at 04:15, after the engines were already dead; kubelet never entered MemoryPressure (its floor is ~100 MiB, this node's was 1.2 GiB). New rule file core/manifests/monitoring/rules/node-memory-pressure-alerts.yaml: NodeMemoryPressurePSI max by (instance) (rate(node_pressure_memory_stalled_seconds_total{job="node-exporter"}[2m])) > 0.01 for: 2m, severity: warning, keep-record -> alert-intake (non-paging) NodeMemoryPressurePSIMetricMissing (up==1 with no pressure series, for: 15m) The threshold is NOT the ticket's 0.5. Backtested against the counter on the rule's own 30 s grid, 0.5 fires ZERO times in 7 days, both collapses included: the measured peaks of that expression are 0.0998 s/s (04:15:30Z) and 0.0604 s/s (07:36:00Z). The investigation's "PSI-full 2.25 -> 5.07 s/s" line is the kernel's full avg10 PERCENTAGE, not seconds per second (2.25 % = 0.0225 s/s). 0.02, the next-safest-looking number, fires for collapse #1 but MISSES collapse #2 entirely (that stall rose as two separate ~1 min bumps, never holding 0.02 for 2 m). 0.01 catches both: fires 04:12:30Z, 2 m 00 s before the 04:14:30Z engine death and 2 m 35 s before the 04:15:05Z dind SIGKILL; and 07:33:00Z, 2 m 21 s before the 07:35:21Z death. The 5-minute lead the investigation asked for is not reachable on this counter and is not claimed - the stall is genuinely absent before 04:10:30Z and 07:29Z. Selector is {job="node-exporter"} (all 9 Talos nodes) and not prod-wk-*: the series carries no node-name label, only the node IP, so a worker-only selector would be a hardcoded IP regex that breaks on the next add or re-IP; the control planes' 7-day maxima are 0.0000-0.0002 s/s so including them costs nothing. daytona and mgmt are out of scope (different scrapes and roles; mgmt's 7-day max is 0.4745 s/s, a separate investigation). max by (instance) is load-bearing: node-exporter pods are recreated by DaemonSet rollouts and each new pod makes a fresh series, which would otherwise double the alert on every restart. Verification: - promtool test rules over the whole repo: SUCCESS, exit 0. New file has 7 cases (positive, 0.008 sub-threshold, 0.0099 just under the line, two exporter-pod series -> exactly one alert, foreign job silent, metric-missing fires at 16m not 10m, healthy fleet silent). - node-memory-pressure-sabotage.sh: per-case baselines + whole-file baseline green, 9 mutations all red (threshold 0.01->0.005 / ->0.5 / ->0.009, for: 2m->1m, aggregation removed, job pin removed, MetricMissing for: 15m->5m, MetricMissing deleted). No vacuous assertion. - fail-capable vs origin/live: the same test file against the rules as they are on origin/live exits 1 (rule absent), so the green is not vacuous. - check_alert_contracts.py --base origin/live: ok, 2 changed rules, both carry the complete ld_* contract, all names Prometheus-valid. - Live 08:05Z: series present for 9/9 Talos nodes (up=1), label shape container/endpoint/instance/job/namespace/pod/service (no node name), current rate 0.000000 except wk-02 0.000069. 7-day backtest (2026-10-29T08:00Z..2026-11-05T08:00Z, 30 s grid, for: 2m): 22 firings, 3 inside the two labelled collapse windows, 19 outside, 72 node-days. Outside per node: wk-05 15, wk-01 2, wk-02 2, others 0. Max outside in one node-day 9 (wk-05 2026-11-04, the dind CI node) - real 2-3 min stalls at 2-7 % of wall time, which is why the rule records instead of paging. Recorded, not-taken lever: for: 3m at 0.01 gives 7 outside firings (worst node-day 3) but only 60 s / 81 s of lead. Two DeepSeek review seats declared. Left UNMERGED on purpose for the landing path; only a reviewer posts verdicts, and only a landing worker merges as pi-landing.Verdict of record — ld-reviewer — head
adeba018…: APPROVEmodel: omniroute/deepseek-flash
session: homelab-2694-psi-seat2-adeba018
Seat 2 (independent DeepSeek session; seat 1's output not read). Angle: fatigue, interaction, blast radius. Head confirmed by API as
adeba01818b0dbd82b116b3928dc850745f9fa08, baselive@8e64b531, CI on head = success (13 contexts). All checks below were run against live Prometheus at 08:45Z.1. Fatigue reality-check — the resting place is right, and do NOT node-scope out wk-05
I replayed the landed expression (
max by (instance)(rate(node_pressure_memory_stalled_seconds_total{job="node-exporter"}[2m])) > 0.01,for: 2m) over the same 7-day window on a 60 s grid and applied thefor:gate myself. Sustained episodes: 16 total — wk-05 8, wk-03 4, wk-02 3, wk-01 1, wk-06/wk-04/cp-02/03/04 0. My grid is coarser than the author's 30 s replay so my count is a lower bound; the qualitative shape is the point, and it matches theirs: every wk-05 hit is an isolated ~2-minute blip, none clustered, no storm. At the 2 m grid the true-samples are 79 separate single-step hits — i.e. thefor: 2mgate is already doing most of the fatigue filtering, not the threshold.Is it CI noise the owners will mute? Yes, it is CI, and I confirmed the causal link rather than assuming it:
forgejo-runner/android-ci-01.72 cores,android-ci-10.79,sbx-lab/sbx-lab-00.97 on prod-wk-05.forgejo-runnerpod above 0.05 cores on that node.So the firings are dind/android CI memory, which is expected load and exactly what a CI owner would call normal. That is the case for excluding wk-05. The case against is stronger, and I pick keep the fleet-wide
{job="node-exporter"}selector:severity: warningreaches only the matcher-less non-pagingalert-intakedigest, neverpi-alert-receiver;ld_disposition: keep-record; and the measured knob (for: 3m→ 19 outside firings / worst node-day 3, at 60/81 s lead) is recorded in the file rather than silently taken. At ~0.26 outside firings per node-day fleet-wide, and worst-case 9 in one node-day, this cannot become a mute-and-forget rule unless the digest itself is ignored — which is a routing problem, not this rule's.No change requested on scope or threshold.
2. Interaction with existing rules — no double page, and one claim in the file is measurably wrong
NodeMemoryPressure(node-alerts.yaml,severity: critical,keep-page,MemAvailable > 90%used,for: 10m) is the only paging memory rule and it sits above this one, so the pair is an escalation ladder (digest warning at +120 s, page later) rather than two pages for one event.NodeMemoryHigh(capacity-alerts.yaml,severity: warning,(1 - MemAvailable/MemTotal)*100 > 70,for: 15m) — samealert-intakedigest rail, nold_*action. It overlaps this alert by design (same node, same digest, same cause).descriptionsays "no MemAvailable rule fired". That is false for the incident the rule cites:ALERTS{alertname="NodeMemoryHigh", instance="10.10.10.22:9100"}was firing from 03:32:00Z to at least 07:35:00Z on 2026-11-05 (21 firing samples at 5 m step;ALERTS_FOR_STATEentered pending at 03:16:57Z), i.e. 42 minutes before the 04:14:30Z engine death and through both collapses. wk-03 memory-used crossed 70 % at 03:18Z and stayed above it.TalosNodeIOPressureFullWarningwas firing on the same node in the same window.Consequence: the new alert will not be the first signal in that digest — it is the first actionable one. Fix the wording to something like "the only pre-collapse warning was capacity-alerts NodeMemoryHigh (>70 % used, no ld_action, no lead time); this rule buys the 2-minute stall lead and an actionable ld_action". Detection and threshold are unaffected, so this is text-only and not a blocker — but it ships into the digest as triage text, so correct it.
ld_action, not novelty.3. Metric-missing guard — verified, fires only on a real gap, but it is quiet by construction
up{job="node-exporter"}andnode_pressure_memory_stalled_seconds_total{job="node-exporter"}both carry exactly the same 9 instances (10.10.10.11/.12/.13/.20/.21/.22/.23/.25/.26:9100), sounless on(instance)cannot mis-join.count(...) < 9, which is the right call.severity: warning+keep-record→ digest only. A guard whose whole purpose is to catch silent absence of memory-stall alerting is itself non-paging and, withld_silence_policy: never, would sit firing indefinitely until someone reads the digest. It is identical in form toTalosNodeIOPressureMetricMissing, so it is consistent with the 2488 precedent and I am not blocking on it; worth revisiting in the 2479 review whether per-node alerting death deserveskeep-page.TalosNodeIOPressureMetricMissingfire for one cause — two digest entries, acceptable.4. Blast radius
Five added files, all under
core/manifests/monitoring/: the rule file,rule-tests/node-memory-pressure.test.yaml,-sabotage.sh, and the-backtest.py/-backtest.mdpair. Nothing outside monitoring rules; no Alertmanager config (correct — that lives in prod), no cluster, node or app manifests, no secret. The committed backtest artifacts match the existingnode-io-pressure-backtest.{py,md}precedent, so they are in-convention rather than stray working files. Tests pin the behaviours that matter for my angle: dedupe across exporter-pod series (exactly one alert),for:not met at 12 m before the ramp holds, sub-threshold 0.008/0.0099 silent, foreign job silent, metric-missing silent at 10 m and firing at 16 m.No blockers.
Verdict of record — ld-reviewer — head
adeba018…: APPROVEmodel: omniroute/deepseek-flash
session: homelab-2694-psi-seat1-adeba018
Fresh DeepSeek seat 1. Head confirmed from the API as
adeba01818b0dbd82b116b3928dc850745f9fa08(baselive8e64b531). Everything below was re-derived from primary sources: live Prometheus, the live Alertmanager route table, the rules/tests as they exist at this head, and the incident record inworking-files/infra-stability/HOMELAB-wk03-memory-0411.md. I did not read seat 2.1. Threshold calibration — independently reproduced, and it is right
I wrote my own replay (query_range on a 30 s grid, 2026-10-29T08:00Z..2026-11-05T08:00Z,
for: 2m= 5 consecutive true ticks, pending→firing state machine, resolve on the first non-true tick) instead of running the committed script, so the numbers are not the author's arithmetic repeated back.full avg10percentage — 2.25 % = 0.0225 s/s, five times under the proposed line. The calibration correction in the body is correct, not a rationalisation.for:.2. Series, shape, and the guard
node_pressure_memory_stalled_seconds_total{job="node-exporter"}returns 9 series, labelscontainer, endpoint, instance, job, namespace, pod, service— no node-name label, as documented.up{job="node-exporter"}==1= 9,up{...}==0= 0.max by (instance)returns 9 elements, all quiet (wk-02 0.000116, others 0/1e-08).podlabel changes on every DaemonSet rollout, so without it one node yields two alert identities. A stale series' rate decays to 0 while the fresh one carries the value, somax by (instance)selects the live pod — the right direction.NodeMemoryPressurePSIMetricMissing:count by (instance)(up{job="node-exporter"}==1) unless on(instance) count by (instance)(node_pressure_memory_stalled_seconds_total{job="node-exporter"}). Both sides reduce to{instance}so theunlessmatches correctly; it returns 0 elements live. It fail-louds after 15m and correctly stays silent forup==0(NodeDown's leg) — the two guards do not double-report.for: 15mis pinned by case 6 (silent at 10m, firing at 16m) and sabotage H.3. Receiver and severity — verified against the live route table, not just the config text
Live
amtool config routes test severity=warning alertname=NodeMemoryPressurePSI→alert-intakeonly. The same probe atseverity=critical→alert-intake,pi-alert-receiver. prod'svalues/kube-prometheus-stack/values.yamlhas the matcher-lessalert-intakeroute first withcontinue: true, ahead ofseverity = "critical"→pi-alert-receiver, so awarningcannot reach the Signal rail. The "no Alertmanager change in this PR" claim is true and the routing split is by severity, so not extending the routing test with this alert's name is defensible (the control case atalertmanager-routing.sh:121covers the split). Given 19 outside firings over 72 node-days on a non-paging digest,warning+keep-recordis the right disposition;criticalwould have paged the operator ~2.7 times a week for stalls that mostly resolve.4. Suite — re-run here, and it is fail-capable
promtool-test.sh(promtool 3.4.0 taken from the live Prometheus pod, matching the cluster): all 20 rule filescheck rulesclean and all 11 test files SUCCESS, exit 0.node-memory-pressure-sabotage.sh: 7 per-case baselines + whole-file baseline green, all 9 mutations red, exit 0 (threshold →0.005/0.5/0.009,for:→1m, aggregation dropped, job pin dropped, MetricMissingfor:→5m and deletion).origin/live(8e64b531): the referenced rule file does not exist there, so this test cannot pass on the base — the green is not vacuous.for:boundary:rate[2m]reads 0.005 at 30 s (window dilution), crosses 0.01 only at 1m30s → no alert at 3m, alert at 3m30s. The boundary is genuinely one tick on each side, not a wide margin that would pass under a shortenedfor:.check_alert_contracts.py --base 8e64b531: ok,changed=2, both rules carry the completeld_*contract, all label/annotation names Prometheus-valid.Concerns, non-blocking:
wk-05 carries 15 of the 19 outside firings, with 9 in one node-day, on the dind CI node. Non-paging so I would not change the threshold today, and the
for: 3mlever is pre-measured (7 outside, worst node-day 3, at the cost of 60 s/81 s of lead). Worth an explicit look atld_review_after2027-02-05 rather than an assumption that the digest queue absorbs it.The alert surfaces only
instance(node IP), and the operator has ~2 minutes. The IP→node map lives in the file header and in the ticket, not in the annotation. Anode_uname_infojoin would put the node name in the alert, but it would change the selector shape the sibling IO-PSI rules pin, so deferring it is reasonable — just record it as the known ergonomic cost of a 2-minute window.The calibration table lists one firing at 0.005 for collapse #2; a live Alertmanager resolves and re-fires across that gap (I see 07:30:00Z and 07:33:00Z). The earliest-firing claim the row is used for is correct — noting it so the number is not later read as an activation count.
No blockers.