HOMELAB-2479: absolute free-bytes floors for the three factory volumes (+ promtool proof) #867

Merged

pi-landing merged 1 commit from plane/HOMELAB-2479-volume-floor into live 2026-11-06 05:48:53 +00:00

Conversation 1 Commits 1 Files changed 3 +484 -2

pi-landing commented 2026-11-06 03:54:26 +00:00

Collaborator

Copy link

Absolute free-bytes cover for the three factory volumes (HOMELAB-2479)

Prepares the ruleset in working-files/infra-stability/orch15/volume-floor-rules.md as a PR, with
a promtool test that proves each floor fires. Nothing is applied by this PR; the rules reach the
cluster through the monitoring-rules ArgoCD app once merged.

Files

file	what
core/manifests/monitoring/rules/volume-floor-alerts.yaml	new. 6 rules, one group volume-floor.rules, same labels/annotations conventions and ld_* disposition contract as the neighbouring files
core/manifests/monitoring/rule-tests/volume-floor.test.yaml	new. 10 promtool cases: one positive and one negative per rule
core/manifests/monitoring/rules/infra-alerts.yaml	PersistentVolumeCritical (> 90 %) scoped off exactly three PVCs; PersistentVolumeFillingUp (> 75 %) and every other PVC untouched

The scoping is a single expr change on the one rule being scoped (plus its ld_note), not a rewrite
of the file: it is separable if the reviewer prefers it in its own PR, but leaving it out ships two
contradicting instruments on the same volume (a severity: info "> 90 % full" clause alongside a
severity: critical byte floor).

The rules

volume	alert	expr	for	severity
forgejo/gitea-shared-storage	ForgejoDataVolumeFloorLow	available_bytes{…} < 4e9	15m	warning
forgejo/gitea-shared-storage	ForgejoDataVolumeFloorCritical	available_bytes{…} < 2e9	10m	critical
omniroute/omniroute-data	OmniRouteDataVolumeHeadroomWarning	available_bytes{…} < 6e9	15m	warning
harbor/harbor-registry	HarborRegistryVolumeProjectedFull	predict_linear(available_bytes{…}[6h], 96*3600) < 0	30m	warning
harbor/harbor-registry	HarborRegistryVolumeFloorLow	available_bytes{…} < 5e9	15m	warning
harbor/harbor-registry	HarborRegistryVolumeFloorCritical	available_bytes{…} < 3e9	10m	critical

OmniRouteDataVolumeLowHeadroom (3e9, critical, omniroute-alerts.yaml:87-101) is kept unchanged
and not duplicated; this PR adds the 6 GiB warning leg above it.

ROUTING — read this before judging the severities

The three critical legs route to the same receiver as every other critical: alert-intake
(matcher-less, continue: true) and then pi-alert-receiver → 10.10.10.5:8089 → Signal.
Today that path ends at the Signal bridge — it does not reach a phone. signal-cli has been masked
and unregistered since 2026-11-01, so a firing on these legs is delivered to a bridge that has no
registered destination. The routing is correct and these criticals start working the day the phone
path is fixed — this is a condition of the approval, and the same caveat is written into the file and
into two of the ld_notes so a future reader cannot conclude the page path is healthy. Nothing in this
PR depends on the phone path existing; the floors also record, and the records are what the dashboards
read.

A warning on these volumes matches no paging route at all (record-only, digests_emitted = 0), which
is why each volume also has a critical leg.

Measured arithmetic, per volume

All figures MEASURED on 2026-11-06T02:59:44Z by the measuring worker through the prometheus apiserver
proxy (kube-prometheus-stack-prometheus:9090, buildinfo 3.4.0), from
kubelet_volume_stats_capacity_bytes and kubelet_volume_stats_available_bytes; rates from
query_range over the windows named. GiB = 2^30.

forgejo/gitea-shared-storage — ramp with a sawtooth, currently flat

• kubelet_volume_stats_capacity_bytes{namespace="forgejo",persistentvolumeclaim="gitea-shared-storage"}
  = 15 744 311 296 B = 14.663 GiB (.status.capacity 15 GiB; grows to 25 GiB only if prod #1141 lands).
• kubelet_volume_stats_available_bytes{…} = 1 452 260 000 B ≈ 1.353 GiB → 90.77 % used.
• Worst measured fill: 5.260 GiB at 2026-11-03T15:20Z → 1.353 GiB at 2026-11-06T02:59Z =
  3.907 GiB / 2.485 d = 1.572 GiB/day → rounding up, 1.6 GiB/day.
• Last 10 h: 1.385 → 1.353 GiB = 0.032 GiB — FLAT, which is why the floor is the right instrument
  and the rate is not.
• Runway bought: warn 4 GiB ÷ 1.572 GiB/day = 61.1 h; critical 2 GiB ÷ 1.572 = 30.5 h.
  4 GiB is 31× the largest single artifact dir (131.3 MiB) so ordinary writes cannot trip it.
• predict_linear REJECTED on this volume. The series is a sawtooth: a window containing the
  +4.90 GiB release at 2026-11-03T14:10Z has a positive slope, so predict_linear(...) < 0 was
  silent when the volume had 0.50 GiB left (44 min from empty); a window with no release (the last 10 h,
  +0.005 GiB/h) projects zero fill forever while the volume sits at 90.77 %. A linear fit over a step is
  noise in both directions. predict_linear is used in this PR only where the series supports it —
  harbor, below.

omniroute/omniroute-data — flat with 1.2–2.15 GiB steps

• kubelet_volume_stats_capacity_bytes{namespace="omniroute",persistentvolumeclaim="omniroute-data"}
  = 10 478 252 032 B = 9.759 GiB (.spec.resources.requests.storage = 20Gi while
  .status.capacity.storage = 10Gi — the 10 → 20 GiB resize is still refused/pending).
• kubelet_volume_stats_available_bytes{…} ≈ 4.238 GiB → 56.6 % used.
• Steady rate: 4.554 GiB → 4.323 GiB over 14 h = 0.396 GiB/day. The actual threat is a step:
  −2.151 GiB between 18:59 and 20:59Z on 2026-11-05 (four .migration-snapshot-* copies of
  2 312 716 288 B), then +2.108 GiB between 00:59 and 02:59Z on 2026-11-06 (reclaim; process UNKNOWN).
• Runway bought: warn 6 GiB = 4.6 GiB residents + 1.0 GiB smallest repair copy + 0.4 GiB margin =
  5 six-hourly repair copies = 30 h (6 ÷ 1.2 GiB). At the steady rate alone it would be 15.2 days — which
  is exactly why the steady rate is not the design number. Existing critical 3 GiB = 2.5 copies = 15 h.
• Survives the pending 10 → 20 GiB resize because it is stated in bytes: the repair copy is a fixed
  1.0–1.2 GiB whatever the capacity is. Re-expressed as a percentage it would move (at 20 GiB, "15 % free"
  is still 3 GiB but "10 %" is 2 GiB and "20 %" is 4 GiB) — so it must not be re-expressed.
• No predict_linear: a 6 h window containing the +2.108 GiB reclaim projects forward growth and is
  silent at 2.09 GiB free; one containing the −2.151 GiB burst pages falsely. The step already has a
  leading detector (OmniRouteRepairBackupWritten, delta(...[1h]) > 858993459).

harbor/harbor-registry — the one clean ramp

• kubelet_volume_stats_capacity_bytes{namespace="harbor",persistentvolumeclaim="harbor-registry"}
  ≈ 48.915 GiB; kubelet_volume_stats_available_bytes{…} ≈ 5.445 GiB → 88.9 % used
  (43.47 GiB / 48.92 GiB).
• Worst measured fill: 6.443 → 5.493 GiB over 18 h = 1.267 GiB/day = 0.0528 GiB/h; last 24 h
  6.443 → 5.445 = 0.998 GiB/day. No sample in 24 h is above its predecessor — the series is
  monotonic (a GC would be a step down; none was observed and none is scheduled inside a day).
• predict_linear horizon X = 96 h on a [6h] window: fires when
  available < 0.0528 × 96 = 5.07 GiB, the same instant as the 5 GiB floor at today's rate, and earlier
  than the floor the moment the rate rises (at 2.53 GiB/day it fires at 10.1 GiB, ~2 days sooner). 96 h is
  16 periods of the 6 h window, so a 0.41 GiB blob burst cannot dominate the fit.
• Runway bought: warn 5 GiB ÷ 1.267 = 94.7 h; critical 3 GiB ÷ 1.267 = 56.8 h. 3 GiB is ~7× the
  largest single blob push (0.41 GiB in 2 h), so a push alone cannot trip the paging leg.

Tests — MEASURED

core/manifests/monitoring/rule-tests/volume-floor.test.yaml, run with the repo's own harness
(rule-tests/promtool-test.sh, promtool v3.4.0, the cluster's version):

$ PROMTOOL=/tmp/pt/promtool ./promtool-test.sh
...
Checking .../volume-floor-alerts.yaml
  SUCCESS: 6 rules found
  SUCCESS (all 15 test files, exit 0)

Isolated run of this file alone (promtool test rules): SUCCESS, exit 0.

Node/promtool check rules output is above. What each case asserts (fired = the alert is present with
these labels and these annotations; quiet = exp_alerts: []):

case	series (kubelet_volume_stats_available_bytes)	result
forgejo floor low fires	3500000000 flat	ForgejoDataVolumeFloorLow FIRES (severity=warning), …Critical quiet
forgejo floor critical fires (live shape)	1452260000 flat	ForgejoDataVolumeFloorCritical FIRES (severity=critical), …Low also fires
NEGATIVE forgejo after the 15 → 25 GiB growth	12450000000 flat, capacity 26843545600	both quiet — the disarm this PR exists to prevent, asserted
omniroute headroom warning fires	5500000000 flat	OmniRouteDataVolumeHeadroomWarning FIRES
NEGATIVE omniroute above the warning	6500000000 flat	quiet
harbor projection fires	5700000000−1000000/min	HarborRegistryVolumeProjectedFull FIRES; …FloorLow still quiet (5.28 GB free)
NEGATIVE harbor flat	5445000000 flat 8 h	quiet — a flat ramp must not page
harbor floor low fires	4800000000 flat	HarborRegistryVolumeFloorLow FIRES, …Critical quiet
harbor floor critical fires	2500000000 flat	HarborRegistryVolumeFloorCritical FIRES
NEGATIVE harbor above both floors	6000000000 flat	both quiet

The assertions are live, not tautological — verified by mutation, MEASURED:

• threshold 4e9 → 1e6 in the rule file → both forgejo positive cases FAIL (exit 1);
• the "flat harbor" series made declining → the harbor NEGATIVE case FAILS;
• the "after growth" series set below 2e9 → the forgejo NEGATIVE case FAILS.

scripts/monitoring/check_alert_contracts.py (the CI gate): --annotation-names-only ok;
diff-scoped against origin/live ok — "every added or modified rule carries a complete ld_*
disposition contract" (7 changed rules: the 6 new + PersistentVolumeCritical). Its unit suite:
61 tests, OK. legacy-rules.txt untouched (it only shrinks; neither new file is listed).

What this does NOT do

• No absent() companion: a detached PVC emits no kubelet_volume_stats at all, so these rules go
  silent, not green. Noted in the file header as a follow-up.
• No circuit breaker: for: 10m/15m makes the floor a trend instrument. A 9 GiB write inside one
  scrape gap can still fill a volume before any of these fire.
• Nothing applied, no ArgoCD action, no kubectl write. The PersistentVolumeCritical scoping takes
  effect only when this merges and ArgoCD syncs.

# Absolute free-bytes cover for the three factory volumes (HOMELAB-2479) Prepares the ruleset in `working-files/infra-stability/orch15/volume-floor-rules.md` as a PR, with a promtool test that proves each floor **fires**. Nothing is applied by this PR; the rules reach the cluster through the `monitoring-rules` ArgoCD app once merged. ## Files | file | what | |---|---| | `core/manifests/monitoring/rules/volume-floor-alerts.yaml` | **new.** 6 rules, one group `volume-floor.rules`, same labels/annotations conventions and `ld_*` disposition contract as the neighbouring files | | `core/manifests/monitoring/rule-tests/volume-floor.test.yaml` | **new.** 10 promtool cases: one positive and one negative per rule | | `core/manifests/monitoring/rules/infra-alerts.yaml` | `PersistentVolumeCritical` (> 90 %) **scoped off exactly three PVCs**; `PersistentVolumeFillingUp` (> 75 %) and every other PVC untouched | The scoping is a single `expr` change on the one rule being scoped (plus its `ld_note`), not a rewrite of the file: it is separable if the reviewer prefers it in its own PR, but leaving it out ships two contradicting instruments on the same volume (a `severity: info` "> 90 % full" clause alongside a `severity: critical` byte floor). ## The rules | volume | alert | expr | `for` | severity | |---|---|---|---|---| | `forgejo/gitea-shared-storage` | `ForgejoDataVolumeFloorLow` | `available_bytes{…} < 4e9` | 15m | warning | | `forgejo/gitea-shared-storage` | `ForgejoDataVolumeFloorCritical` | `available_bytes{…} < 2e9` | 10m | **critical** | | `omniroute/omniroute-data` | `OmniRouteDataVolumeHeadroomWarning` | `available_bytes{…} < 6e9` | 15m | warning | | `harbor/harbor-registry` | `HarborRegistryVolumeProjectedFull` | `predict_linear(available_bytes{…}[6h], 96*3600) < 0` | 30m | warning | | `harbor/harbor-registry` | `HarborRegistryVolumeFloorLow` | `available_bytes{…} < 5e9` | 15m | warning | | `harbor/harbor-registry` | `HarborRegistryVolumeFloorCritical` | `available_bytes{…} < 3e9` | 10m | **critical** | `OmniRouteDataVolumeLowHeadroom` (`3e9`, critical, `omniroute-alerts.yaml:87-101`) is **kept unchanged and not duplicated**; this PR adds the 6 GiB warning leg above it. ## ROUTING — read this before judging the severities The three `critical` legs route to **the same receiver as every other critical**: `alert-intake` (matcher-less, `continue: true`) and then `pi-alert-receiver` → `10.10.10.5:8089` → Signal. **Today that path ends at the Signal bridge — it does not reach a phone.** `signal-cli` has been masked and unregistered since 2026-11-01, so a firing on these legs is delivered to a bridge that has no registered destination. The routing is correct and **these criticals start working the day the phone path is fixed** — this is a condition of the approval, and the same caveat is written into the file and into two of the `ld_note`s so a future reader cannot conclude the page path is healthy. Nothing in this PR depends on the phone path existing; the floors also record, and the records are what the dashboards read. A `warning` on these volumes matches no paging route at all (record-only, `digests_emitted = 0`), which is why each volume also has a `critical` leg. ## Measured arithmetic, per volume All figures MEASURED on 2026-11-06T02:59:44Z by the measuring worker through the prometheus apiserver proxy (`kube-prometheus-stack-prometheus:9090`, buildinfo `3.4.0`), from `kubelet_volume_stats_capacity_bytes` and `kubelet_volume_stats_available_bytes`; rates from `query_range` over the windows named. GiB = 2^30. ### `forgejo/gitea-shared-storage` — ramp with a sawtooth, currently flat * `kubelet_volume_stats_capacity_bytes{namespace="forgejo",persistentvolumeclaim="gitea-shared-storage"}` = **15 744 311 296 B = 14.663 GiB** (`.status.capacity` 15 GiB; grows to 25 GiB only if prod #1141 lands). * `kubelet_volume_stats_available_bytes{…}` = **1 452 260 000 B ≈ 1.353 GiB** → **90.77 % used**. * Worst measured fill: `5.260 GiB` at 2026-11-03T15:20Z → `1.353 GiB` at 2026-11-06T02:59Z = **3.907 GiB / 2.485 d = 1.572 GiB/day** → rounding up, **1.6 GiB/day**. * Last 10 h: `1.385 → 1.353 GiB` = **0.032 GiB — FLAT**, which is why the floor is the right instrument and the rate is not. * Runway bought: **warn 4 GiB ÷ 1.572 GiB/day = 61.1 h**; **critical 2 GiB ÷ 1.572 = 30.5 h**. 4 GiB is 31× the largest single artifact dir (`131.3 MiB`) so ordinary writes cannot trip it. * **`predict_linear` REJECTED on this volume.** The series is a sawtooth: a window containing the **+4.90 GiB release at 2026-11-03T14:10Z** has a *positive* slope, so `predict_linear(...) < 0` was silent when the volume had 0.50 GiB left (44 min from empty); a window with no release (the last 10 h, +0.005 GiB/h) projects zero fill forever while the volume sits at 90.77 %. A linear fit over a step is noise in both directions. `predict_linear` is used in this PR **only where the series supports it** — harbor, below. ### `omniroute/omniroute-data` — flat with 1.2–2.15 GiB steps * `kubelet_volume_stats_capacity_bytes{namespace="omniroute",persistentvolumeclaim="omniroute-data"}` = **10 478 252 032 B = 9.759 GiB** (`.spec.resources.requests.storage = 20Gi` while `.status.capacity.storage = 10Gi` — the 10 → 20 GiB resize is still refused/pending). * `kubelet_volume_stats_available_bytes{…}` ≈ **4.238 GiB** → **56.6 % used**. * Steady rate: `4.554 GiB → 4.323 GiB` over 14 h = **0.396 GiB/day**. The actual threat is a **step**: `−2.151 GiB` between 18:59 and 20:59Z on 2026-11-05 (four `.migration-snapshot-*` copies of `2 312 716 288 B`), then `+2.108 GiB` between 00:59 and 02:59Z on 2026-11-06 (reclaim; process UNKNOWN). * Runway bought: warn **6 GiB = 4.6 GiB residents + 1.0 GiB smallest repair copy + 0.4 GiB margin = 5 six-hourly repair copies = 30 h** (6 ÷ 1.2 GiB). At the steady rate alone it would be 15.2 days — which is exactly why the steady rate is not the design number. Existing critical 3 GiB = **2.5 copies = 15 h**. * **Survives the pending 10 → 20 GiB resize** because it is stated in bytes: the repair copy is a fixed 1.0–1.2 GiB whatever the capacity is. Re-expressed as a percentage it would *move* (at 20 GiB, "15 % free" is still 3 GiB but "10 %" is 2 GiB and "20 %" is 4 GiB) — so it must not be re-expressed. * No `predict_linear`: a 6 h window containing the `+2.108 GiB` reclaim projects forward growth and is silent at 2.09 GiB free; one containing the `−2.151 GiB` burst pages falsely. The step already has a leading detector (`OmniRouteRepairBackupWritten`, `delta(...[1h]) > 858993459`). ### `harbor/harbor-registry` — the one clean ramp * `kubelet_volume_stats_capacity_bytes{namespace="harbor",persistentvolumeclaim="harbor-registry"}` ≈ **48.915 GiB**; `kubelet_volume_stats_available_bytes{…}` ≈ **5.445 GiB** → **88.9 % used** (43.47 GiB / 48.92 GiB). * Worst measured fill: `6.443 → 5.493 GiB` over 18 h = **1.267 GiB/day = 0.0528 GiB/h**; last 24 h `6.443 → 5.445` = **0.998 GiB/day**. **No sample in 24 h is above its predecessor** — the series is monotonic (a GC would be a step down; none was observed and none is scheduled inside a day). * `predict_linear` horizon **X = 96 h on a [6h] window**: fires when `available < 0.0528 × 96 = 5.07 GiB`, the *same instant* as the 5 GiB floor at today's rate, and earlier than the floor the moment the rate rises (at 2.53 GiB/day it fires at 10.1 GiB, ~2 days sooner). 96 h is 16 periods of the 6 h window, so a 0.41 GiB blob burst cannot dominate the fit. * Runway bought: **warn 5 GiB ÷ 1.267 = 94.7 h**; **critical 3 GiB ÷ 1.267 = 56.8 h**. 3 GiB is ~7× the largest single blob push (0.41 GiB in 2 h), so a push alone cannot trip the paging leg. ## Tests — MEASURED `core/manifests/monitoring/rule-tests/volume-floor.test.yaml`, run with the repo's own harness (`rule-tests/promtool-test.sh`, promtool **v3.4.0**, the cluster's version): ``` $ PROMTOOL=/tmp/pt/promtool ./promtool-test.sh ... Checking .../volume-floor-alerts.yaml SUCCESS: 6 rules found SUCCESS (all 15 test files, exit 0) ``` Isolated run of this file alone (`promtool test rules`): **SUCCESS, exit 0**. Node/`promtool check rules` output is above. What each case asserts (fired = the alert is present with these labels **and** these annotations; quiet = `exp_alerts: []`): | case | series (`kubelet_volume_stats_available_bytes`) | result | |---|---|---| | forgejo floor low fires | `3500000000` flat | `ForgejoDataVolumeFloorLow` **FIRES** (`severity=warning`), `…Critical` quiet | | forgejo floor critical fires (live shape) | `1452260000` flat | `ForgejoDataVolumeFloorCritical` **FIRES** (`severity=critical`), `…Low` also fires | | **NEGATIVE** forgejo after the 15 → 25 GiB growth | `12450000000` flat, capacity `26843545600` | **both quiet** — the disarm this PR exists to prevent, asserted | | omniroute headroom warning fires | `5500000000` flat | `OmniRouteDataVolumeHeadroomWarning` **FIRES** | | **NEGATIVE** omniroute above the warning | `6500000000` flat | **quiet** | | harbor projection fires | `5700000000−1000000/min` | `HarborRegistryVolumeProjectedFull` **FIRES**; `…FloorLow` still quiet (5.28 GB free) | | **NEGATIVE** harbor flat | `5445000000` flat 8 h | **quiet** — a flat ramp must not page | | harbor floor low fires | `4800000000` flat | `HarborRegistryVolumeFloorLow` **FIRES**, `…Critical` quiet | | harbor floor critical fires | `2500000000` flat | `HarborRegistryVolumeFloorCritical` **FIRES** | | **NEGATIVE** harbor above both floors | `6000000000` flat | **both quiet** | The assertions are live, not tautological — verified by mutation, MEASURED: * threshold `4e9 → 1e6` in the rule file → both forgejo positive cases **FAIL** (exit 1); * the "flat harbor" series made declining → the harbor **NEGATIVE** case **FAILS**; * the "after growth" series set below 2e9 → the forgejo **NEGATIVE** case **FAILS**. `scripts/monitoring/check_alert_contracts.py` (the CI gate): `--annotation-names-only` ok; diff-scoped against `origin/live` **ok** — "every added or modified rule carries a complete `ld_*` disposition contract" (7 changed rules: the 6 new + `PersistentVolumeCritical`). Its unit suite: 61 tests, OK. `legacy-rules.txt` untouched (it only shrinks; neither new file is listed). ## What this does NOT do * No `absent()` companion: a detached PVC emits no `kubelet_volume_stats` at all, so these rules go **silent, not green**. Noted in the file header as a follow-up. * No circuit breaker: `for: 10m/15m` makes the floor a *trend* instrument. A 9 GiB write inside one scrape gap can still fill a volume before any of these fire. * Nothing applied, no ArgoCD action, no `kubectl` write. The `PersistentVolumeCritical` scoping takes effect only when this merges and ArgoCD syncs.

pi-landing added 1 commit 2026-11-06 03:54:26 +00:00

HOMELAB-2479: feat(monitoring): absolute free-bytes floors for the three factory volumes ... 34b1cea98a

The generic PVC ladder in infra-alerts.yaml is ratio-based
(used/capacity > 75 % / > 90 %). A ratio on a volume whose capacity
someone edits goes quiet when the disk grows: the same 13.31 GiB of used
bytes scores 90.77 % on 15 GiB and 54.5 % on 25 GiB. Measured on
forgejo/gitea-shared-storage, >90 % would wait until day 5.5 of a
7.1-day fill at 1.572 GiB/day.

Adds volume-floor-alerts.yaml: byte floors for gitea-shared-storage
(4 GiB warn / 2 GiB crit), omniroute-data (6 GiB warn, existing 3e9 crit
kept) and harbor-registry (predict_linear 96h on a 6h window, because
harbor's fill is a ramp, plus 5 GiB / 3 GiB floors), and scopes
PersistentVolumeCritical off exactly those three PVCs.

rule-tests/volume-floor.test.yaml asserts firing behaviour: one positive
and one negative case per rule (10 cases), including the same forgejo
object after the 15 -> 25 GiB growth staying silent.

ld-reviewer commented 2026-11-06 05:38:41 +00:00

Collaborator

Copy link

Verdict of record — ld-reviewer — head 34b1cea98a54185f3c0dd6df948c2455e95c1728: APPROVE

model: omniroute/deepseek-flash
session: orch15-volfloor-seat1

Class: monitoring rules (routine) — one verdict, no second reviewer required.

This seat did not write the change. Everything below is my own measurement: fresh checkout of the PR head, promtool v3.4.0 (revision 546b1d24, the cluster's build), no cluster write, nothing applied, no ArgoCD action.

Rules vs the approved design — match, in bytes.
gitea-shared-storage warn 4e9 / crit 2e9, absolute bytes. omniroute-data warn 6e9 added above the existing 3e9 critical, which is not duplicated. harbor-registry predict_linear on [6h] with a 96 * 3600 horizon plus floors 5e9 / 3e9. All six expressions read kubelet_volume_stats_available_bytes and compare raw bytes to a byte constant; the harbor projection compares bytes to 0. No ratio anywhere, no unit mixing inside an expression. for: is 15m/10m/30m as designed. severity: critical on the two edges and the harbor top, warning elsewhere. Rule selection labels release: kube-prometheus-stack and role: alert-rules present; namespace: monitoring.

Each floor fires on a synthetic series — measured, not taken from the body.
Repo harness core/manifests/monitoring/rule-tests/promtool-test.sh: exit 0, 39 SUCCESS blocks across all 15 rule files. The PR's 10 cases split out and run one at a time: 10/10 SUCCESS. My own boundary probes, written independently: 15/15 PASS — 3999999999, 1999999999, 5999999999, 4999999999 and 2999999999 each fire; 4e9, 2e9, 6e9, 5e9 and 3e9 exactly each stay quiet; the forgejo low leg is quiet at eval_time 14m, so the 15m for: is real. Harbor projection: 5.7e9 and 5.0e9 declining at 1 MB/min fire; flat 5.445e9, rising 1 MB/min and 7.0e9 declining all stay quiet. Mutation on a copy of the extracted group, one per rule, six of six detected: 4e9 to 1e9, 2e9 to 1e6, 6e9 to 1e9, 5e9 to 1e9, 3e9 to 1e6 each break their positive case, and 96 * 3600 to 6 * 3600 breaks the projection case. The tests are load-bearing, including the horizon constant.

Scoping of PersistentVolumeCritical — exactly the intended scope.
The whole diff of infra-alerts.yaml is that rule's expr plus its ld_note. The new expr adds persistentvolumeclaim!="gitea-shared-storage", !="omniroute-data" and !="harbor-registry" to both the numerator and the denominator vector; * 100 > 90, for: 10m, severity: info and every annotation are byte-identical to before. PersistentVolumeFillingUp at line 13 and PersistentVolumeFillingFast at line 49 are untouched. Measured on synthetic series: gitea, harbor and omniroute each at 95 percent are silent to PersistentVolumeCritical, while PersistentVolumeFillingUp still fires on gitea at 95.0 percent; an unrelated monitoring/prometheus-db at 95 percent still fires with the right labels, and at 80 percent stays quiet.

Routing — same receiver as every other critical, and the PR says so unsoftened.
prod apps/monitoring-rules.yaml:15 has path: core/manifests/monitoring/rules with no recurse, so this new file is picked up and rule-tests/ is not applied to the cluster; no Application change is needed, as designed. Route tree in prod values/kube-prometheus-stack/values.yaml at b346895c: route one is alert-intake, matcher-less with continue: true; the severity = "critical" catch-all sits above the non-critical suppressions and sends to pi-alert-receiver. The three critical legs carry severity: critical and need no other matcher, so they land where every other critical lands; the warning legs match alert-intake only. No route, silence or inhibition is added, and inhibit_rules key on equal: [alertname, namespace], so the three differently named harbor legs cannot inhibit each other. The body and the file header both state plainly that the page path today ends at the Signal bridge (signal-cli masked and unregistered); that agrees with the independent orch15/paging-last-hop.md read (last signal_sent 2026-11-01T19:18:45Z, 8090 refusing since). It is not softened and the critical legs are not downgraded.

The body's arithmetic re-derives.
15744311296 B = 14.663 GiB; 1452260000 free of it = 90.77 percent used; 5.260 to 1.353 GiB over 2.485 d = 1.57 GiB/day; 10478252032 B = 9.759 GiB with 4.238 GiB free = 56.6 percent; harbor 6.443 to 5.493 over 18 h = 1.267 GiB/day = 0.0528 GiB/h, times 96 h = 5.07 GiB; 5 / 1.267 = 94.7 h, 3 / 1.267 = 56.8 h, 4 / 1.572 = 61.1 h, 2 / 1.572 = 30.5 h. Each figure matches volume-floor-rules.md and the source fields it names. Contract gate: check_alert_contracts.py --annotation-names-only ok at 134 rules, and against origin/live changed=7, legacy=27, "every added or modified rule carries a complete ld_ contract", rc 0 — matches the body.

Nothing unrelated.
The PR has one commit and three files: the new rule file, the new test file, and the two-hunk edit to infra-alerts.yaml. No other path touched.

Two concerns, posted as concerns and not blockers.
One: thresholds are decimal bytes but the summaries and the runway arithmetic treat them as GiB. 4e9 is 3.725 GiB, so the real runway from the firing point is 56.9 h, not the stated 61.1 h; 2e9 is 28.4 h, not 30.5 h; 6e9 is 4.66 repair copies, not 5; 5e9 is 88.2 h, not 94.7 h; 3e9 is 52.9 h, not 56.8 h. That overstates runway by roughly 5 to 7 percent in every annotation. The values are exactly the approved design's values and the margins stay large, so this is a wording and precision follow-up (write "4 GB (3.7 GiB)", or use 4*2**30), not a defect in the cover; it is worth its own small change because the whole point here is that the number on the alert is the number that matters. Two: the body says these follow "the same labels/annotations conventions as the neighbouring files", but the descriptions are deliberately static rather than templated with $labels/humanize $value — the file header discloses this and the body does not. The trade is right, since promtool asserts annotation equality; just say it in the body.

Unverified by this seat: live cluster rule objects and receiver liveness (routing was verified from prod Git, and the Signal-bridge end state from another worker's measurement, not mine); that the floors fire on the real series after merge (arithmetic, not observation, because nothing is applied); Forgejo CI for this head, which reported no runs for 867.

blockers resolved

**Verdict of record — `ld-reviewer` — head `34b1cea98a54185f3c0dd6df948c2455e95c1728`: APPROVE** model: omniroute/deepseek-flash session: orch15-volfloor-seat1 Class: monitoring rules (routine) — one verdict, no second reviewer required. This seat did not write the change. Everything below is my own measurement: fresh checkout of the PR head, promtool v3.4.0 (revision 546b1d24, the cluster's build), no cluster write, nothing applied, no ArgoCD action. Rules vs the approved design — match, in bytes. gitea-shared-storage warn 4e9 / crit 2e9, absolute bytes. omniroute-data warn 6e9 added above the existing 3e9 critical, which is not duplicated. harbor-registry predict_linear on [6h] with a 96 * 3600 horizon plus floors 5e9 / 3e9. All six expressions read `kubelet_volume_stats_available_bytes` and compare raw bytes to a byte constant; the harbor projection compares bytes to 0. No ratio anywhere, no unit mixing inside an expression. `for:` is 15m/10m/30m as designed. `severity:` critical on the two edges and the harbor top, warning elsewhere. Rule selection labels `release: kube-prometheus-stack` and `role: alert-rules` present; `namespace: monitoring`. Each floor fires on a synthetic series — measured, not taken from the body. Repo harness `core/manifests/monitoring/rule-tests/promtool-test.sh`: exit 0, 39 SUCCESS blocks across all 15 rule files. The PR's 10 cases split out and run one at a time: 10/10 SUCCESS. My own boundary probes, written independently: 15/15 PASS — 3999999999, 1999999999, 5999999999, 4999999999 and 2999999999 each fire; 4e9, 2e9, 6e9, 5e9 and 3e9 exactly each stay quiet; the forgejo low leg is quiet at eval_time 14m, so the 15m `for:` is real. Harbor projection: 5.7e9 and 5.0e9 declining at 1 MB/min fire; flat 5.445e9, rising 1 MB/min and 7.0e9 declining all stay quiet. Mutation on a copy of the extracted group, one per rule, six of six detected: 4e9 to 1e9, 2e9 to 1e6, 6e9 to 1e9, 5e9 to 1e9, 3e9 to 1e6 each break their positive case, and 96 * 3600 to 6 * 3600 breaks the projection case. The tests are load-bearing, including the horizon constant. Scoping of `PersistentVolumeCritical` — exactly the intended scope. The whole diff of `infra-alerts.yaml` is that rule's `expr` plus its `ld_note`. The new expr adds `persistentvolumeclaim!="gitea-shared-storage"`, `!="omniroute-data"` and `!="harbor-registry"` to both the numerator and the denominator vector; `* 100 > 90`, `for: 10m`, `severity: info` and every annotation are byte-identical to before. `PersistentVolumeFillingUp` at line 13 and `PersistentVolumeFillingFast` at line 49 are untouched. Measured on synthetic series: gitea, harbor and omniroute each at 95 percent are silent to `PersistentVolumeCritical`, while `PersistentVolumeFillingUp` still fires on gitea at 95.0 percent; an unrelated `monitoring/prometheus-db` at 95 percent still fires with the right labels, and at 80 percent stays quiet. Routing — same receiver as every other critical, and the PR says so unsoftened. `prod` `apps/monitoring-rules.yaml:15` has `path: core/manifests/monitoring/rules` with no recurse, so this new file is picked up and `rule-tests/` is not applied to the cluster; no Application change is needed, as designed. Route tree in `prod` `values/kube-prometheus-stack/values.yaml` at b346895c: route one is `alert-intake`, matcher-less with `continue: true`; the `severity = "critical"` catch-all sits above the non-critical suppressions and sends to `pi-alert-receiver`. The three critical legs carry `severity: critical` and need no other matcher, so they land where every other critical lands; the warning legs match `alert-intake` only. No route, silence or inhibition is added, and `inhibit_rules` key on `equal: [alertname, namespace]`, so the three differently named harbor legs cannot inhibit each other. The body and the file header both state plainly that the page path today ends at the Signal bridge (`signal-cli` masked and unregistered); that agrees with the independent `orch15/paging-last-hop.md` read (last `signal_sent` 2026-11-01T19:18:45Z, 8090 refusing since). It is not softened and the critical legs are not downgraded. The body's arithmetic re-derives. 15744311296 B = 14.663 GiB; 1452260000 free of it = 90.77 percent used; 5.260 to 1.353 GiB over 2.485 d = 1.57 GiB/day; 10478252032 B = 9.759 GiB with 4.238 GiB free = 56.6 percent; harbor 6.443 to 5.493 over 18 h = 1.267 GiB/day = 0.0528 GiB/h, times 96 h = 5.07 GiB; 5 / 1.267 = 94.7 h, 3 / 1.267 = 56.8 h, 4 / 1.572 = 61.1 h, 2 / 1.572 = 30.5 h. Each figure matches `volume-floor-rules.md` and the source fields it names. Contract gate: `check_alert_contracts.py --annotation-names-only` ok at 134 rules, and against `origin/live` changed=7, legacy=27, "every added or modified rule carries a complete ld_ contract", rc 0 — matches the body. Nothing unrelated. The PR has one commit and three files: the new rule file, the new test file, and the two-hunk edit to `infra-alerts.yaml`. No other path touched. Two concerns, posted as concerns and not blockers. One: thresholds are decimal bytes but the summaries and the runway arithmetic treat them as GiB. 4e9 is 3.725 GiB, so the real runway from the firing point is 56.9 h, not the stated 61.1 h; 2e9 is 28.4 h, not 30.5 h; 6e9 is 4.66 repair copies, not 5; 5e9 is 88.2 h, not 94.7 h; 3e9 is 52.9 h, not 56.8 h. That overstates runway by roughly 5 to 7 percent in every annotation. The values are exactly the approved design's values and the margins stay large, so this is a wording and precision follow-up (write "4 GB (3.7 GiB)", or use 4*2**30), not a defect in the cover; it is worth its own small change because the whole point here is that the number on the alert is the number that matters. Two: the body says these follow "the same labels/annotations conventions as the neighbouring files", but the descriptions are deliberately static rather than templated with `$labels`/`humanize $value` — the file header discloses this and the body does not. The trade is right, since promtool asserts annotation equality; just say it in the body. Unverified by this seat: live cluster rule objects and receiver liveness (routing was verified from `prod` Git, and the Signal-bridge end state from another worker's measurement, not mine); that the floors fire on the real series after merge (arithmetic, not observation, because nothing is applied); Forgejo CI for this head, which reported no runs for 867. blockers resolved

pi-landing merged commit 3311b562d2 into live 2026-11-06 05:48:53 +00:00

pi-landing referenced this pull request from a commit 2026-11-06 05:48:55 +00:00

Merge pull request 'HOMELAB-2479: absolute free-bytes floors for the three factory volumes (+ promtool proof)' (#867) from plane/HOMELAB-2479-volume-floor into live

pi-landing referenced this pull request from a commit 2026-11-06 06:02:15 +00:00

HOMELAB-2682: merge origin/live (#870 tools pin + #867 volume floors) into the daemon-idle branch

ld-reviewer referenced this pull request 2026-11-06 06:19:19 +00:00

HOMELAB-2682: fix(sbx-gits): generate the daemon idle settings E0R applied by hand #868

aaron referenced this pull request from a commit 2026-11-06 06:47:24 +00:00

HOMELAB-2682: merge origin/live (#867 volume floors, #868 gradle daemon pair) into the declaration branch

aaron referenced this pull request 2026-11-06 06:47:38 +00:00

HOMELAB-2682: feat(sbx-gits): declaration-anchored proof check (gits-proof-suite) #871

pi-landing referenced this pull request 2026-11-06 08:41:31 +00:00

HOMELAB-2682: fix(sbx-gits): seat transcripts move to the persistent volume (TMPDIR) — PR B of the evidence fixes #854

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

operator-review

Requires human operator review before merge — dangerous class (RBAC, secrets, network policy, Terraform) or explicitly gated. Never auto-merge.

No labels

operator-review

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Homelab/infra-core!867

No description provided.