Homelab/infra-core
6
0
Fork 2
Code Issues Pull requests 35 Projects Releases 1 Packages Wiki Activity Actions

HOMELAB-2694: page on sandbox-node memory headroom (El Director + owning hypervisor; PSI stays diagnostic) #841

Open
pi-landing wants to merge 1 commit from plane/HOMELAB-2694-sandbox-mem-page into live
pull from: plane/HOMELAB-2694-sandbox-mem-page
merge into: Homelab:live
Conversation 0 Commits 1 Files changed 3 +390 -4
pi-landing commented 2026-11-05 10:20:14 +00:00
Collaborator
Copy link

What fires, and when

One new rule, SandboxNodeMemoryHeadroomLow, in core/manifests/monitoring/rules/sandbox-node-memory-alerts.yaml:

expr: |
  min by (nodename) (
    node_memory_MemAvailable_bytes{job="node-exporter"}
    * on (instance) group_left (nodename)
    node_uname_info{job="node-exporter", nodename=~"prod-wk-01|prod-wk-02|prod-wk-03"}
  ) < 3.5 * 1024^3
for: 5m
labels: {severity: warning, component: node-memory}

It pages when a sandbox node has less than 3.5 GiB of MemAvailable for 5 minutes. prod-wk-01 hosts sbx-gits, prod-wk-02 hosts sbx-lab, prod-wk-03 is the node all three 2026-11-05 collapses happened on.

The PSI rule (NodeMemoryPressurePSI, #837, merged 8c336d96) is not touched — it stays a diagnostic, as ruled. Nothing else in monitoring changes.

Which rule actually fired, and why this is a new rule rather than an edit

Two rule groups name a memory-headroom rule. Read live from /api/v1/rules:

group alert expr for state
capacity.rules NodeMemoryHigh (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 70 15m firing (instance daytona)
node-exporter NodeMemoryHighUtilization (not NodeMemoryHigh) 100 - (node_memory_MemAvailable_bytes{job="node-exporter"} / node_memory_MemTotal_bytes{job="node-exporter"} * 100) > 90 15m inactive

Only capacity.rules/NodeMemoryHigh is literally NodeMemoryHigh; the second is the chart's own rule with a different name and a >90 % threshold, and it did not fire on this event. NodeMemoryHigh is the signal being promoted, and its history proves it:

ALERTS{alertname="NodeMemoryHigh"} firing, instance 10.10.10.22:9100 (= prod-wk-03):
  11-05 03:32Z .. 04:14Z      (collapse #1 at 04:16Z)
  11-05 06:40Z .. 07:34Z      (collapse #2 at 07:36Z)
  11-05 08:48Z .. 09:36Z      (collapse #3 at 09:36Z)   <- 48 min of notice

It is not promoted in place, for two measured reasons:

  1. It is not sandbox-scoped. It evaluates every node and both scraped Proxmox hosts. daytona sat above 70 % for the whole 24 h to 2026-11-05T10:08Z, so promoting it in place would page on a non-sandbox host continuously.
  2. Its percentage is not comparable across the three nodes. 70 % used leaves 8.2 GiB on prod-wk-01 (27.4 GiB RAM) and 4.7 GiB on prod-wk-02/03 (15.6 GiB each). For a fault that runs away with absolute host RAM, the percentage hides which node is actually near stalling.

Proof it would have fired on this event

Same expression, evaluated against live Prometheus (1-minute samples), for the third episode:

prod-wk-03   below 3.5 GiB continuously   08:48:xx Z .. 09:36:xx Z
prod-wk-03   value at 09:34:00Z          1.358 GiB   (< 3.5 GiB: TRUE)
prod-wk-01   value at 09:34:00Z         11.077 GiB   (silent)
prod-wk-02   value at 09:34:00Z          7.075 GiB   (silent)

With for: 5m that is firing at 08:53Z, 43 minutes before the 09:36Z collapse — the same minute-range the existing warning rule reached pending, i.e. the notice was always there and only the route was missing. The other two episodes fire at 03:59Z and 06:55Z, before the 04:16Z and 07:36Z collapses.

Honest caveat, also in the rule's comments: the alert resolves just before the collapse. The node stalls, node-exporter cannot be scraped, the series disappears and the alert clears. The page happens on the way down, which is the point, but during a sandbox incident the recovery of this alert is a symptom, not recovery.

The threshold, from 7 days of data

min by (nodename) node_memory_MemAvailable_bytes, 2026-10-29T10:11Z .. 2026-11-05T10:11Z, 1-minute samples:

node RAM 7d min MemAvailable episodes < 3.5 GiB minutes < 3.5 GiB
prod-wk-01 27.4 GiB 3.23 GiB @11-04 18:51 5 (all on 11-04) 12
prod-wk-02 15.6 GiB 4.65 GiB @11-04 23:44 0 0
prod-wk-03 15.6 GiB 1.14 GiB @11-05 07:28 5 117

prod-wk-03's incident episodes are 03:54-04:14Z, 06:50-07:35Z, 08:48-09:36Z — 3/3 recall on the collapse class. prod-wk-02 is silent for the week.

The cost, stated rather than hidden. prod-wk-01 has 5 short episodes, all on 11-04 evening (18:02-18:56Z, 22:29-22:35Z), 12 minutes total; two of them (18:13-18:18Z, 18:51-18:56Z) last exactly 5 minutes, so the for: hold may page twice that evening. 3 GiB had zero false positives but fired at 09:14Z (17 min of notice); 3.5 GiB buys 31 more minutes of warning for at most two pages in the week. 4 GiB is disqualified (12 episodes / 156 min on prod-wk-01). El Director asked for notice, so the lead time wins — and it is one number, reversible on this table.

Scope, and how drift is handled (no labels added)

Matched by node name, joined via node_uname_info because node-exporter's instance is the node IP and carries no node-name label.

A static node list is a drift risk: move a sandbox to prod-wk-04 and this rule watches the node it left. The robust form is a node label (sandbox-host=true) joined through kube_node_labels, not added here (cluster-configuration change, own review). Checked for an existing label to use instead (kubectl get nodes --show-labels): the nodes carry workload-class (prod-wk-05/06 = ci), topology.kubernetes.io/zone, node.longhorn.io/create-default-disk and homelab.aaron.reynoza.org/gpu — nothing that identifies a sandbox host. So the three names in nodename=~ are the single place to edit, and a sandbox move must edit it in the same change that moves the sandbox.

Who is woken — and who explicitly is not

Read from the LIVE Alertmanager config (rendered alertmanager.yaml, three receivers, three routes of interest):

receiver what it is
null default route; discards
pi-alert-receiver Aaron's phone/Signal. webhook http://10.10.10.5:8089/alertmanager → pi-alert-receiver (systemd user unit on the mgmt VM, SIGNAL_BRIDGE_SESSION=signal-bridge) → pi-signal-bridge → Aaron's Signal. Reached by the severity = "critical" catch-all.
alert-intake http://10.10.10.5:8091/alertmanager, the matcher-less P1-P4 episode ledger (HOMELAB-1847). Records; does not reach a person on its own.

This alert reaches pi-alert-receiver and therefore Aaron's phone NEVER, and the mechanism is severity: warning, not a comment: the only route that reaches him for node alerts is severity = "critical", and this rule is warning. Moving it to critical would silently re-point it at his phone, which is the one thing the ruling forbids at a 2-hour cycle. It will reach alert-intake (the ledger) — that is the existing matcher-less rail, unchanged.

The El Director / owning-hypervisor receivers do not exist yet, and this PR does not invent them. Verified on the mgmt VM today: live listeners are 8089 (pi-alert-receiver), 8091 (alert-intake), 8088 (pi-review-receiver), 8081 (lab-director-operator), 8096 (jellyfin). There is no El Director receiver and no sbx-*-hypervisor receiver, and homelab-b1 (El Director) is reached today only by a hypervisor session relaying to it. A route to a receiver with no instance is silent, which is exactly the failure this ticket exists to fix — so the route half is not written here as a fake.

The route, and the dependency on prod#1127

Homelab/prod#1127 (open, panel-complete, NOT merged — awaiting El Director's go) already adds everything the owner half needs: receivers sbx-gits-hypervisor (:8097, session homelab-33), sbx-lab-hypervisor (:8098, homelab-15), sbx-sendrast-hypervisor (:8099, homelab-ba, relays to El Director only), and six continue: true child routes.

Relationship: independent of its landing order, and it needs one addition.

  • No textual conflict. #1127 touches only values/kube-prometheus-stack/values.yaml (+172 lines). This PR touches only core/manifests/monitoring/rules/ and scripts/monitoring/promtool/ in a different repo. Either can land first.

  • No behavioural conflict. #1127 routes NodeMemoryPressurePSI|NodeMemoryPressurePSIMetricMissing and the three Sandbox* alerts. This rule has a new alertname, so it matches none of #1127's children and falls through them unchanged to alert-intake.

  • What it needs from prod, once #1127 lands, is one route child above the noise suppressions:

    - matchers:
    - alertname = "SandboxNodeMemoryHeadroomLow"
  receiver: "sbx-lab-hypervisor"   # prod-wk-01 -> sbx-gits-hypervisor, prod-wk-02 -> sbx-lab-hypervisor
  continue: true

    Alertmanager cannot compute which hypervisor owns the node, because the alert carries only nodename — so, exactly as #1127 does for PSI, that child fans out to the three owner rails, or is written per-node once a node label exists. A direct El Director receiver still does not exist: adding one means a fourth pi-alert-receiver instance with SIGNAL_BRIDGE_SESSION=homelab-b1, a port, a ufw rule and a token — host work that would be silent if only the receiver were declared. Recorded as the follow-up this PR does not fake.

So, plainly: this PR makes the signal fire and makes it untouchable by Aaron's pager. The delivery to El Director + the owning hypervisor is completed by a prod route change that depends on #1127 landing and on its receiver instances existing; until then this alert lands in the alert-intake ledger only. Nothing here is worse than today, and the rule is ready the moment the rail is.

Tests — promtool, and they can fail

core/manifests/monitoring/rule-tests/sandbox-node-memory.test.yaml, run through the repo harness:

$ PROMTOOL=/path/to/promtool scripts/monitoring/promtool/run.sh \
    core/manifests/monitoring/rule-tests/sandbox-node-memory.test.yaml
==> test:     core/manifests/monitoring/rule-tests/sandbox-node-memory.test.yaml
  SUCCESS
# case asserts
T1 2.5 GiB on prod-wk-03 quiet at 4m, FIRES at 10m (the for: 5m hold)
T2 every sandbox node above the floor silent at 60m
T3 negative control: prod-wk-04 at 1 GiB silent — fails if the nodename=~ scope is dropped
T4 negative control: a 4m dip under the floor never reaches the hold
T5 negative control: exactly 3.5 GiB not "under 3.5 GiB"
T6 duplicate node-exporter series for one node one alert, not two (the min by collapse)
T7 two sandbox nodes low at once two alerts, one per nodename

The alert-contract gate also passes:

$ python3 scripts/monitoring/check_alert_contracts.py --repo-root . --base origin/live
alert-contract gate: base=origin/live (merge-base 3e5572e67f7c) · rules=104 · changed=1 · legacy=27 · name-surfaces=108
ok: every added or modified rule carries a complete ld_* disposition contract, and every rule label/annotation name is Prometheus-valid

scripts/monitoring/promtool/run.sh gains one backwards-compatible feature: a test may name the CRD it exercises with a top-level rule_crd: key (the runner previously only ever rendered backup-alerts.yaml). Tests without the key behave exactly as before.

Not in this PR

  • No Alertmanager route, receiver or inhibit_rules change (that is prod; see above).
  • No change to NodeMemoryPressurePSI or NodeMemoryPressurePSIMetricMissing. The PSI rule is left exactly as merged in 8c336d96.
  • No change to capacity-alerts.yaml / NodeMemoryHigh, no node labels, no cluster writes.
## What fires, and when One new rule, `SandboxNodeMemoryHeadroomLow`, in `core/manifests/monitoring/rules/sandbox-node-memory-alerts.yaml`: ```yaml expr: | min by (nodename) ( node_memory_MemAvailable_bytes{job="node-exporter"} * on (instance) group_left (nodename) node_uname_info{job="node-exporter", nodename=~"prod-wk-01|prod-wk-02|prod-wk-03"} ) < 3.5 * 1024^3 for: 5m labels: {severity: warning, component: node-memory} ``` It pages when a **sandbox node** has less than **3.5 GiB** of `MemAvailable` for 5 minutes. `prod-wk-01` hosts `sbx-gits`, `prod-wk-02` hosts `sbx-lab`, `prod-wk-03` is the node all three 2026-11-05 collapses happened on. The PSI rule (`NodeMemoryPressurePSI`, #837, merged `8c336d96`) is **not touched** — it stays a diagnostic, as ruled. Nothing else in `monitoring` changes. ## Which rule actually fired, and why this is a new rule rather than an edit Two rule groups name a memory-headroom rule. Read live from `/api/v1/rules`: | group | alert | expr | for | state | |---|---|---|---|---| | `capacity.rules` | **`NodeMemoryHigh`** | `(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 70` | 15m | **firing** (instance `daytona`) | | `node-exporter` | `NodeMemoryHighUtilization` (not `NodeMemoryHigh`) | `100 - (node_memory_MemAvailable_bytes{job="node-exporter"} / node_memory_MemTotal_bytes{job="node-exporter"} * 100) > 90` | 15m | inactive | Only `capacity.rules/NodeMemoryHigh` is *literally* `NodeMemoryHigh`; the second is the chart's own rule with a different name and a >90 % threshold, and it did not fire on this event. `NodeMemoryHigh` is the signal being promoted, and its history proves it: ``` ALERTS{alertname="NodeMemoryHigh"} firing, instance 10.10.10.22:9100 (= prod-wk-03): 11-05 03:32Z .. 04:14Z (collapse #1 at 04:16Z) 11-05 06:40Z .. 07:34Z (collapse #2 at 07:36Z) 11-05 08:48Z .. 09:36Z (collapse #3 at 09:36Z) <- 48 min of notice ``` It is **not** promoted in place, for two measured reasons: 1. **It is not sandbox-scoped.** It evaluates every node and both scraped Proxmox hosts. `daytona` sat above 70 % for the whole 24 h to 2026-11-05T10:08Z, so promoting it in place would page on a non-sandbox host continuously. 2. **Its percentage is not comparable across the three nodes.** 70 % used leaves 8.2 GiB on `prod-wk-01` (27.4 GiB RAM) and 4.7 GiB on `prod-wk-02`/`03` (15.6 GiB each). For a fault that runs away with *absolute* host RAM, the percentage hides which node is actually near stalling. ## Proof it would have fired on this event Same expression, evaluated against live Prometheus (1-minute samples), for the third episode: ``` prod-wk-03 below 3.5 GiB continuously 08:48:xx Z .. 09:36:xx Z prod-wk-03 value at 09:34:00Z 1.358 GiB (< 3.5 GiB: TRUE) prod-wk-01 value at 09:34:00Z 11.077 GiB (silent) prod-wk-02 value at 09:34:00Z 7.075 GiB (silent) ``` With `for: 5m` that is **firing at 08:53Z, 43 minutes before the 09:36Z collapse** — the same minute-range the existing warning rule reached `pending`, i.e. the notice was always there and only the route was missing. The other two episodes fire at 03:59Z and 06:55Z, before the 04:16Z and 07:36Z collapses. Honest caveat, also in the rule's comments: the alert **resolves just before the collapse**. The node stalls, node-exporter cannot be scraped, the series disappears and the alert clears. The page happens on the way down, which is the point, but during a sandbox incident the recovery of this alert is a symptom, not recovery. ## The threshold, from 7 days of data `min by (nodename) node_memory_MemAvailable_bytes`, 2026-10-29T10:11Z .. 2026-11-05T10:11Z, 1-minute samples: | node | RAM | 7d min MemAvailable | episodes < 3.5 GiB | minutes < 3.5 GiB | |---|---|---|---|---| | prod-wk-01 | 27.4 GiB | 3.23 GiB @11-04 18:51 | 5 (all on 11-04) | 12 | | prod-wk-02 | 15.6 GiB | 4.65 GiB @11-04 23:44 | 0 | 0 | | prod-wk-03 | 15.6 GiB | 1.14 GiB @11-05 07:28 | 5 | 117 | `prod-wk-03`'s incident episodes are `03:54-04:14Z`, `06:50-07:35Z`, `08:48-09:36Z` — 3/3 recall on the collapse class. `prod-wk-02` is silent for the week. **The cost, stated rather than hidden.** `prod-wk-01` has 5 short episodes, all on 11-04 evening (18:02-18:56Z, 22:29-22:35Z), 12 minutes total; two of them (18:13-18:18Z, 18:51-18:56Z) last exactly 5 minutes, so the `for:` hold may page twice that evening. `3 GiB` had **zero** false positives but fired at 09:14Z (17 min of notice); `3.5 GiB` buys 31 more minutes of warning for at most two pages in the week. `4 GiB` is disqualified (12 episodes / 156 min on `prod-wk-01`). El Director asked for notice, so the lead time wins — and it is one number, reversible on this table. ## Scope, and how drift is handled (no labels added) Matched **by node name**, joined via `node_uname_info` because node-exporter's `instance` is the node IP and carries no node-name label. A static node list **is** a drift risk: move a sandbox to `prod-wk-04` and this rule watches the node it left. The robust form is a node label (`sandbox-host=true`) joined through `kube_node_labels`, **not added here** (cluster-configuration change, own review). Checked for an existing label to use instead (`kubectl get nodes --show-labels`): the nodes carry `workload-class` (`prod-wk-05/06 = ci`), `topology.kubernetes.io/zone`, `node.longhorn.io/create-default-disk` and `homelab.aaron.reynoza.org/gpu` — **nothing** that identifies a sandbox host. So the three names in `nodename=~` are the single place to edit, and a sandbox move must edit it in the same change that moves the sandbox. ## Who is woken — and who explicitly is not Read from the LIVE Alertmanager config (rendered `alertmanager.yaml`, three receivers, three routes of interest): | receiver | what it is | |---|---| | `null` | default route; discards | | **`pi-alert-receiver`** | **Aaron's phone/Signal.** webhook `http://10.10.10.5:8089/alertmanager` → `pi-alert-receiver` (systemd user unit on the mgmt VM, `SIGNAL_BRIDGE_SESSION=signal-bridge`) → `pi-signal-bridge` → Aaron's Signal. Reached by the `severity = "critical"` catch-all. | | `alert-intake` | `http://10.10.10.5:8091/alertmanager`, the matcher-less P1-P4 episode ledger (HOMELAB-1847). Records; does not reach a person on its own. | **This alert reaches `pi-alert-receiver` and therefore Aaron's phone NEVER**, and the mechanism is `severity: warning`, not a comment: the only route that reaches him for node alerts is `severity = "critical"`, and this rule is `warning`. Moving it to `critical` would silently re-point it at his phone, which is the one thing the ruling forbids at a 2-hour cycle. It will reach `alert-intake` (the ledger) — that is the existing matcher-less rail, unchanged. **The El Director / owning-hypervisor receivers do not exist yet, and this PR does not invent them.** Verified on the mgmt VM today: live listeners are `8089` (`pi-alert-receiver`), `8091` (`alert-intake`), `8088` (`pi-review-receiver`), `8081` (`lab-director-operator`), `8096` (jellyfin). There is **no** El Director receiver and no `sbx-*-hypervisor` receiver, and `homelab-b1` (El Director) is reached today only by a hypervisor session relaying to it. A route to a receiver with no instance is silent, which is exactly the failure this ticket exists to fix — so the route half is **not** written here as a fake. ## The route, and the dependency on prod#1127 `Homelab/prod#1127` (**open, panel-complete, NOT merged** — awaiting El Director's go) already adds everything the owner half needs: receivers `sbx-gits-hypervisor` (:8097, session `homelab-33`), `sbx-lab-hypervisor` (:8098, `homelab-15`), `sbx-sendrast-hypervisor` (:8099, `homelab-ba`, relays to El Director only), and six `continue: true` child routes. **Relationship: independent of its landing order, and it needs one addition.** * **No textual conflict.** #1127 touches only `values/kube-prometheus-stack/values.yaml` (+172 lines). This PR touches only `core/manifests/monitoring/rules/` and `scripts/monitoring/promtool/` in a *different repo*. Either can land first. * **No behavioural conflict.** #1127 routes `NodeMemoryPressurePSI|NodeMemoryPressurePSIMetricMissing` and the three `Sandbox*` alerts. This rule has a new alertname, so it matches none of #1127's children and falls through them unchanged to `alert-intake`. * **What it needs from prod, once #1127 lands**, is one route child above the noise suppressions: ```yaml - matchers: - alertname = "SandboxNodeMemoryHeadroomLow" receiver: "sbx-lab-hypervisor" # prod-wk-01 -> sbx-gits-hypervisor, prod-wk-02 -> sbx-lab-hypervisor continue: true ``` Alertmanager cannot compute *which* hypervisor owns the node, because the alert carries only `nodename` — so, exactly as #1127 does for PSI, that child fans out to the three owner rails, or is written per-node once a node label exists. **A direct El Director receiver still does not exist**: adding one means a fourth `pi-alert-receiver` instance with `SIGNAL_BRIDGE_SESSION=homelab-b1`, a port, a ufw rule and a token — host work that would be silent if only the receiver were declared. Recorded as the follow-up this PR does not fake. **So, plainly: this PR makes the signal fire and makes it untouchable by Aaron's pager. The delivery to El Director + the owning hypervisor is completed by a prod route change that depends on #1127 landing and on its receiver instances existing; until then this alert lands in the `alert-intake` ledger only.** Nothing here is worse than today, and the rule is ready the moment the rail is. ## Tests — promtool, and they can fail `core/manifests/monitoring/rule-tests/sandbox-node-memory.test.yaml`, run through the repo harness: ``` $ PROMTOOL=/path/to/promtool scripts/monitoring/promtool/run.sh \ core/manifests/monitoring/rule-tests/sandbox-node-memory.test.yaml ==> test: core/manifests/monitoring/rule-tests/sandbox-node-memory.test.yaml SUCCESS ``` | # | case | asserts | |---|---|---| | T1 | 2.5 GiB on `prod-wk-03` | quiet at 4m, **FIRES** at 10m (the `for: 5m` hold) | | T2 | every sandbox node above the floor | silent at 60m | | **T3** | **negative control: `prod-wk-04` at 1 GiB** | **silent** — fails if the `nodename=~` scope is dropped | | T4 | negative control: a 4m dip under the floor | never reaches the hold | | T5 | negative control: exactly 3.5 GiB | not "under 3.5 GiB" | | T6 | duplicate node-exporter series for one node | **one** alert, not two (the `min by` collapse) | | T7 | two sandbox nodes low at once | two alerts, one per `nodename` | The alert-contract gate also passes: ``` $ python3 scripts/monitoring/check_alert_contracts.py --repo-root . --base origin/live alert-contract gate: base=origin/live (merge-base 3e5572e67f7c) · rules=104 · changed=1 · legacy=27 · name-surfaces=108 ok: every added or modified rule carries a complete ld_* disposition contract, and every rule label/annotation name is Prometheus-valid ``` `scripts/monitoring/promtool/run.sh` gains one backwards-compatible feature: a test may name the CRD it exercises with a top-level `rule_crd:` key (the runner previously only ever rendered `backup-alerts.yaml`). Tests without the key behave exactly as before. ## Not in this PR - No Alertmanager route, receiver or `inhibit_rules` change (that is `prod`; see above). - **No change to `NodeMemoryPressurePSI` or `NodeMemoryPressurePSIMetricMissing`.** The PSI rule is left exactly as merged in `8c336d96`. - No change to `capacity-alerts.yaml` / `NodeMemoryHigh`, no node labels, no cluster writes.
HOMELAB-2694: page on sandbox-node memory headroom (rule + promtool cases)
All checks were successful
0/0 projects applied successfully.
CI Review / receiver-test (pull_request) Successful in 23s
Details
CI Review / runner-manifest-test (pull_request) Successful in 24s
Details
CI Review / helm-validate (pull_request) Successful in 29s
Details
Alert Disposition Contract / alert-contract (pull_request) Successful in 34s
Details
Lint & Validate / shellcheck (pull_request) Successful in 28s
Details
CI Review / alert-receiver-test (pull_request) Successful in 35s
Details
Lint & Validate / yaml-lint (pull_request) Successful in 36s
Details
CI Review / record-sql-test (pull_request) Successful in 1m41s
Details
CI Review / cockpit-test (pull_request) Successful in 3m23s
Details
CI Review / dispatcher-test (pull_request) Successful in 4m46s
Details
CI Review / cockpit-e2e (pull_request) Successful in 7m5s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 7m50s
Details
ab3d144989 
Promotes the memory-HEADROOM signal to a paging alert scoped to the three
sandbox nodes (prod-wk-01 gits, prod-wk-02 lab, prod-wk-03), on an absolute
3.5 GiB MemAvailable floor picked from 7 days of data.

The rule that already fired on this event is capacity.rules/NodeMemoryHigh
(>70% used, for 15m, severity warning), firing in three bursts on
10.10.10.22: 03:32-04:14Z, 06:40-07:34Z, 08:48-09:36Z. It is not promoted in
place because it is not sandbox-scoped (it fires for daytona and every node)
and its percentage threshold means different absolute headroom on 27.4 GiB
and 15.6 GiB nodes.

At 3.5 GiB the new expression is true on prod-wk-03 from 08:48Z and reads
1.358 GiB at 09:34Z, 43 minutes of notice before the 09:36Z collapse; the
same threshold is silent on prod-wk-02 and reached on prod-wk-01 only in
12 short minutes on 11-04.

severity stays warning on purpose: severity=critical is the catch-all to
pi-alert-receiver (Aaron's Signal), which the ruling excludes.

promtool: 7 cases, 3 of them negative controls; SUCCESS.
PSI rule untouched.
All checks were successful
0/0 projects applied successfully.
CI Review / receiver-test (pull_request) Successful in 23s
Details
CI Review / runner-manifest-test (pull_request) Successful in 24s
Details
CI Review / helm-validate (pull_request) Successful in 29s
Required
Details
Alert Disposition Contract / alert-contract (pull_request) Successful in 34s
Details
Lint & Validate / shellcheck (pull_request) Successful in 28s
Required
Details
CI Review / alert-receiver-test (pull_request) Successful in 35s
Details
Lint & Validate / yaml-lint (pull_request) Successful in 36s
Required
Details
CI Review / record-sql-test (pull_request) Successful in 1m41s
Details
CI Review / cockpit-test (pull_request) Successful in 3m23s
Details
CI Review / dispatcher-test (pull_request) Successful in 4m46s
Details
CI Review / cockpit-e2e (pull_request) Successful in 7m5s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 7m50s
Required
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin plane/HOMELAB-2694-sandbox-mem-page:plane/HOMELAB-2694-sandbox-mem-page
git switch plane/HOMELAB-2694-sandbox-mem-page
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
operator-review

Requires human operator review before merge — dangerous class (RBAC, secrets, network policy, Terraform) or explicitly gated. Never auto-merge.

No labels
operator-review
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Homelab/infra-core!841
No description provided.