Homelab/infra-core
6
0
Fork 2
Code Issues Pull requests 35 Projects Releases 1 Packages Wiki Activity Actions

IR-20261112: alert on sandbox work-volume fill, cap GOCACHE (NOT FOR LANDING) #963

Open
pi-landing wants to merge 1 commit from plane/IR-20261112-sbxdisk into live
pull from: plane/IR-20261112-sbxdisk
merge into: Homelab:live
Conversation 0 Commits 1 Files changed 5 +482 -15
pi-landing commented 2026-11-12 14:20:49 +00:00
Collaborator
Copy link

NOT FOR LANDING

Build-only PR (Lab-Run: IR-20261112-sbxdisk). f1 one-look first, then a clean seat pair + El Director's GO. Nothing here was applied to the cluster and no pod was touched.

The incident this answers

sbx-lab-0's 60 Gi work PVC (work-sbx-lab-0, class longhorn-fleet-work) reached 100 % at 2026-11-12T12:38Z and writes failed. A live orchestration session lost its scratch, its builds and its git worktrees in one step, and no alarm told anyone — the session found it itself. The same day the volume was measured climbing again at ~7 G/h, from scratch directories. This PR makes that condition loud and removes one of its unbounded residents from the volume.

1. PrometheusRule sandbox-work-volume-alerts (namespace monitoring)

One selector, two legs:

kubelet_volume_stats_used_bytes{namespace=~"sbx-.*|ld-fleet"}
/
kubelet_volume_stats_capacity_bytes{namespace=~"sbx-.*|ld-fleet"}
alertname threshold for: severity route
SandboxWorkVolumeFilling > 0.80 10m warning alert-intake (non-paging)
SandboxWorkVolumeFillingCritical > 0.92 10m warning alert-intake (non-paging)

Both carry the repo's full ld_* disposition contract (ld_owner, ld_action, ld_frequency, ld_disposition, ld_silence_policy, ld_review_after, ld_note).

Routing, why warning on both. The route tree (live Alertmanager config.original, recorded in rules/sbx-gits-alerts.yaml): the matcher-less first child is receiver: alert-intake with continue: true, so every alert reaches the non-paging ledger; pi-alert-receiver (the paging webhook → pi-signal-bridge → Signal) is reached only by severity="critical" and the named disk-pressure routes. Per the inbox routing, a sandbox scratch fill stays on the non-paging tier, so the second leg carries its urgency in the name (…Critical) and not in the severity. Flipping either to critical is a one-line change in the rule and needs no route edit.

The selector is namespace-scoped on purpose — a per-PVC list goes stale when a sandbox is added, and a stale list is silent, which is the exact failure this closes.

The selector is not vacuous — measured, read-only, 13:56Z

/api/v1/nodes/<node>/proxy/stats/summary (the same kubelet series these rules evaluate) plus kubectl get pvc -A match five claims today:

namespace claim used / capacity % legs firing
sbx-lab work-sbx-lab-0 (60Gi) 32 506 445 824 / 63 090 503 680 51.5 % none
ld-fleet ld-record-1 pgdata (10Gi) 295 796 736 / 10 464 022 528 2.8 % none
ld-fleet ld-record-1 pg-wal (10Gi) 604 061 696 / 10 464 022 528 5.8 % none
ld-fleet ld-record-2 pgdata (10Gi) 296 009 728 / 10 464 022 528 2.8 % none
ld-fleet ld-record-2 pg-wal (10Gi) 604 061 696 / 10 464 022 528 5.8 % none

Both legs are silent on today's healthy state. (For the record: the 74 % figure in the ticket brief is not what the live series said at 13:56Z — the volume measured 51.5 % then, so something freed space in between; the rule does not depend on either number.)

Stated blind spot: sbx-gits and sbx-sendrast mount work as a node hostPath (/var/mnt/u-longhorn/sbx-gits/work). A hostPath has no kubelet_volume_stats series at all, so this rule cannot see it. That is said in the rule file's header too, not just here; the hostPath's resident that could be capped (GOCACHE) is capped by this same PR, and the rest is node-level rule territory.

Honesty note on ordering

PersistentVolumeFillingUp (> 75 % for 15m, warning) already covers these volumes and fires first on a slow fill. This file is not a threshold improvement: it adds the sandbox scope, the name, the disposition contract and the 0.92 leg. That sentence is in the rule's own description as well, so nobody reads it later as a better alarm.

2. GOCACHE cap — decision and trade-offs

GOCACHE moves from /home/ld/work/go/cache (on the volume that filled) to /home/ld/go-cache, a new emptyDir with sizeLimit: 8Gi in sbx-lab, 4Gi in sbx-gits. GOMODCACHE is deliberately untouched in both and stays at /home/ld/work/go/pkg/mod (small, mostly read-only per build, and expensive to re-download).

Why sizeLimit and not a pre-start go clean -cache at N GB:

  • A sizeLimit binds continuously. Exceeding it evicts this one pod — loud, immediate, and self-healing: the restarted pod starts with an empty cache. A pre-start go clean fires only at a pod start, so a sandbox that runs for days between restarts (both pods are OnDelete; sbx-lab is restarted only with the supervisor) can fill the volume between restarts and never run the clean at all. The cap has to bind between restarts, which is exactly what a pre-start check cannot do.
  • Effect on build times: the cache does not survive a recreate, so the first go build afterwards is cold. That is the right thing to lose — it is a cache; the git work tree survives on the PVC, and the module cache (the part that costs network) stays.
  • Ephemeral-storage arithmetic, against each container's own limit: sbx-lab's sandbox mounts become tmp 8Gi + go-cache 8Gi + herdr-sock 64Mi = 16.06Gi against its 20Gi limit, so the volume caps still bind before the container limit. sbx-gits's sandbox becomes 8 + 4 + 0.06 = 12.06Gi against a 10Gi limit — which is why it gets 4Gi and not 8Gi: the cache alone (4Gi) and /tmp alone (8.06Gi) can each never reach the limit, only their sum can, and both are bounded. Raising that container limit was the alternative and is rejected: 10Gi was chosen so the volume cap binds first, and the node has ~31 GiB free for the pod's whole ceiling.

prepare-work in sbx-gits still creates $DST/go/cache (empty, now unused) because the epic suite's check [15] pins that exact install -d line as its proof that go/pkg is handed to 1000:1000. Removing the name would turn a real check red for a cosmetic tidy.

3. promtool evidence (promtool v3.4.0, the version the cluster runs)

$ promtool check rules <extracted rules>
Checking /tmp/sbxdisk-rules.yaml
  SUCCESS: 2 rules found

$ promtool test rules rule-tests/sandbox-work-volume.test.yaml
  SUCCESS

$ rule-tests/promtool-test.sh          # the whole suite
  ... SUCCESS (all rule files + all test files)

The test file asserts firing decisions, not existence:

  • 0.81 → SandboxWorkVolumeFilling fires, …Critical silent, and it is still silent at 5m (so for: 10m is proven to be 10m);
  • 0.79 → both silent;
  • 0.93 → …Critical fires (and so does the 0.80 leg);
  • 0.91 → only the 0.80 leg fires (one point of margin, pinned on which side each leg sits);
  • the real capacity of the volume that filled (63090503680 B, 0.810 used) → fires;
  • an ld-fleet record volume is reached by the same selector;
  • a 99 % volume in media (outside sbx-.*|ld-fleet) → silent, so the selector's scope is asserted too.

Annotations are copied from the rule by a generator, because promtool compares annotations for equality and a hand-retyped copy drifts into asserting formatting.

4. Which statefulsets change

  • core/manifests/apps/sbx-lab/statefulset.yaml — GOCACHE value, the go-cache mount, the go-cache volume (sizeLimit: 8Gi), plus the in-file comments that used to say GOCACHE lives on the PVC.
  • core/manifests/apps/sbx-gits/statefulset.yaml — same three, sizeLimit: 4Gi.
  • core/manifests/monitoring/rules/sandbox-work-volume-alerts.yaml (new) + rule-tests/sandbox-work-volume.test.yaml (new).
  • scripts/gits-sbx-epic-test.sh — the two literals this branch's delta is pinned by: p1_lab_sts_bad's allowed-addition allowlist gains go-cache / /home/ld/go-cache, and SBX_GITS_DIFF_SET is re-measured to M statefulset.yaml. Nothing is loosened: the allowlist is still an allowlist and its mutation legs still bite.

Both statefulsets are updateStrategy: OnDelete, so this template change restarts nothing; GOCACHE and the new volume take effect at each pod's next recreate.

5. Not folded into EXTDEL v2 (#958)

#958 is open (two commits, head 0c72e058, NOT FOR LANDING) and its landing gate pins that head for a seat pair plus a recreate choreography with 4a and homelab-33. Folding this in would move its head (voiding its seats), give one PR two Lab-Run ids, and force a re-measure of the same allowed-change sets it pins (T7[diff], 17[R159](d)) — while touching the same two files. So: its own PR.

CI

Run ids are reported as comments on this PR as they land.

Lab-Run: IR-20261112-sbxdisk

## NOT FOR LANDING Build-only PR (`Lab-Run: IR-20261112-sbxdisk`). f1 one-look first, then a clean seat pair + El Director's GO. Nothing here was applied to the cluster and no pod was touched. ## The incident this answers `sbx-lab-0`'s 60 Gi `work` PVC (`work-sbx-lab-0`, class `longhorn-fleet-work`) reached **100 % at 2026-11-12T12:38Z and writes failed.** A live orchestration session lost its scratch, its builds and its git worktrees in one step, and **no alarm told anyone** — the session found it itself. The same day the volume was measured climbing again at **~7 G/h**, from scratch directories. This PR makes that condition loud and removes one of its unbounded residents from the volume. ## 1. PrometheusRule `sandbox-work-volume-alerts` (namespace `monitoring`) One selector, two legs: ```promql kubelet_volume_stats_used_bytes{namespace=~"sbx-.*|ld-fleet"} / kubelet_volume_stats_capacity_bytes{namespace=~"sbx-.*|ld-fleet"} ``` | alertname | threshold | `for:` | severity | route | |---|---|---|---|---| | `SandboxWorkVolumeFilling` | `> 0.80` | `10m` | `warning` | `alert-intake` (non-paging) | | `SandboxWorkVolumeFillingCritical` | `> 0.92` | `10m` | `warning` | `alert-intake` (non-paging) | Both carry the repo's full `ld_*` disposition contract (`ld_owner`, `ld_action`, `ld_frequency`, `ld_disposition`, `ld_silence_policy`, `ld_review_after`, `ld_note`). **Routing, why `warning` on both.** The route tree (live Alertmanager `config.original`, recorded in `rules/sbx-gits-alerts.yaml`): the matcher-less first child is `receiver: alert-intake` with `continue: true`, so every alert reaches the non-paging ledger; `pi-alert-receiver` (the paging webhook → pi-signal-bridge → Signal) is reached **only** by `severity="critical"` and the named disk-pressure routes. Per the inbox routing, a sandbox scratch fill stays on the **non-paging** tier, so the second leg carries its urgency in the **name** (`…Critical`) and not in the severity. Flipping either to `critical` is a one-line change in the rule and needs no route edit. **The selector is namespace-scoped on purpose** — a per-PVC list goes stale when a sandbox is added, and a stale list is *silent*, which is the exact failure this closes. ### The selector is not vacuous — measured, read-only, 13:56Z `/api/v1/nodes/<node>/proxy/stats/summary` (the same kubelet series these rules evaluate) plus `kubectl get pvc -A` match **five claims today**: | namespace | claim | used / capacity | % | legs firing | |---|---|---|---|---| | sbx-lab | `work-sbx-lab-0` (60Gi) | 32 506 445 824 / 63 090 503 680 | **51.5 %** | none | | ld-fleet | `ld-record-1` pgdata (10Gi) | 295 796 736 / 10 464 022 528 | 2.8 % | none | | ld-fleet | `ld-record-1` pg-wal (10Gi) | 604 061 696 / 10 464 022 528 | 5.8 % | none | | ld-fleet | `ld-record-2` pgdata (10Gi) | 296 009 728 / 10 464 022 528 | 2.8 % | none | | ld-fleet | `ld-record-2` pg-wal (10Gi) | 604 061 696 / 10 464 022 528 | 5.8 % | none | Both legs are **silent on today's healthy state**. (For the record: the 74 % figure in the ticket brief is not what the live series said at 13:56Z — the volume measured 51.5 % then, so something freed space in between; the rule does not depend on either number.) **Stated blind spot:** `sbx-gits` and `sbx-sendrast` mount `work` as a node **hostPath** (`/var/mnt/u-longhorn/sbx-gits/work`). A hostPath has **no** `kubelet_volume_stats` series at all, so this rule cannot see it. That is said in the rule file's header too, not just here; the hostPath's resident that *could* be capped (GOCACHE) is capped by this same PR, and the rest is node-level rule territory. ### Honesty note on ordering `PersistentVolumeFillingUp` (`> 75 %` for `15m`, warning) **already** covers these volumes and **fires first** on a slow fill. This file is *not* a threshold improvement: it adds the sandbox scope, the name, the disposition contract and the 0.92 leg. That sentence is in the rule's own `description` as well, so nobody reads it later as a better alarm. ## 2. GOCACHE cap — decision and trade-offs `GOCACHE` moves from `/home/ld/work/go/cache` (**on the volume that filled**) to `/home/ld/go-cache`, a **new `emptyDir` with `sizeLimit`**: **8Gi in `sbx-lab`**, **4Gi in `sbx-gits`**. `GOMODCACHE` **is deliberately untouched** in both and stays at `/home/ld/work/go/pkg/mod` (small, mostly read-only per build, and expensive to re-download). **Why `sizeLimit` and not a pre-start `go clean -cache` at N GB:** * A `sizeLimit` binds **continuously**. Exceeding it **evicts this one pod** — loud, immediate, and self-healing: the restarted pod starts with an empty cache. A pre-start `go clean` fires **only at a pod start**, so a sandbox that runs for days between restarts (both pods are `OnDelete`; `sbx-lab` is restarted only with the supervisor) can fill the volume between restarts and **never run the clean at all**. The cap has to bind between restarts, which is exactly what a pre-start check cannot do. * **Effect on build times:** the cache does not survive a recreate, so the first `go build` afterwards is cold. That is the right thing to lose — it is a *cache*; the git work tree survives on the PVC, and the module cache (the part that costs network) stays. * **Ephemeral-storage arithmetic**, against each container's own limit: `sbx-lab`'s `sandbox` mounts become `tmp 8Gi + go-cache 8Gi + herdr-sock 64Mi = 16.06Gi` against its **20Gi** limit, so the volume caps still bind before the container limit. `sbx-gits`'s `sandbox` becomes `8 + 4 + 0.06 = 12.06Gi` against a **10Gi** limit — which is why it gets 4Gi and not 8Gi: the cache alone (4Gi) and `/tmp` alone (8.06Gi) can each **never** reach the limit, only their sum can, and both are bounded. Raising that container limit was the alternative and is rejected: 10Gi was chosen so the volume cap binds first, and the node has ~31 GiB free for the pod's whole ceiling. `prepare-work` in `sbx-gits` still creates `$DST/go/cache` (empty, now unused) because the epic suite's check `[15]` pins that exact `install -d` line as its proof that `go/pkg` is handed to `1000:1000`. Removing the name would turn a real check red for a cosmetic tidy. ## 3. promtool evidence (promtool v3.4.0, the version the cluster runs) ``` $ promtool check rules <extracted rules> Checking /tmp/sbxdisk-rules.yaml SUCCESS: 2 rules found $ promtool test rules rule-tests/sandbox-work-volume.test.yaml SUCCESS $ rule-tests/promtool-test.sh # the whole suite ... SUCCESS (all rule files + all test files) ``` The test file asserts **firing decisions**, not existence: * `0.81` → `SandboxWorkVolumeFilling` **fires**, `…Critical` silent, and it is **still silent at 5m** (so `for: 10m` is proven to be 10m); * `0.79` → **both** silent; * `0.93` → `…Critical` **fires** (and so does the 0.80 leg); * `0.91` → only the **0.80** leg fires (one point of margin, pinned on which side each leg sits); * the **real capacity** of the volume that filled (`63090503680` B, `0.810` used) → fires; * an **`ld-fleet` record volume** is reached by the same selector; * a **99 %** volume in `media` (outside `sbx-.*|ld-fleet`) → **silent**, so the selector's *scope* is asserted too. Annotations are copied from the rule by a generator, because promtool compares annotations for equality and a hand-retyped copy drifts into asserting formatting. ## 4. Which statefulsets change * `core/manifests/apps/sbx-lab/statefulset.yaml` — `GOCACHE` value, the `go-cache` mount, the `go-cache` volume (`sizeLimit: 8Gi`), plus the in-file comments that used to say GOCACHE lives on the PVC. * `core/manifests/apps/sbx-gits/statefulset.yaml` — same three, `sizeLimit: 4Gi`. * `core/manifests/monitoring/rules/sandbox-work-volume-alerts.yaml` (new) + `rule-tests/sandbox-work-volume.test.yaml` (new). * `scripts/gits-sbx-epic-test.sh` — the two literals this branch's delta is pinned by: `p1_lab_sts_bad`'s allowed-addition allowlist gains `go-cache` / `/home/ld/go-cache`, and `SBX_GITS_DIFF_SET` is **re-measured** to `M statefulset.yaml`. Nothing is loosened: the allowlist is still an allowlist and its mutation legs still bite. Both statefulsets are `updateStrategy: OnDelete`, so this template change **restarts nothing**; GOCACHE and the new volume take effect at each pod's next recreate. ## 5. Not folded into EXTDEL v2 (#958) #958 is open (two commits, head `0c72e058`, NOT FOR LANDING) and its landing gate pins **that head** for a seat pair plus a recreate choreography with 4a and homelab-33. Folding this in would move its head (**voiding its seats**), give one PR **two `Lab-Run` ids**, and force a re-measure of the same allowed-change sets it pins (`T7[diff]`, `17[R159](d)`) — while touching the same two files. So: its own PR. ## CI Run ids are reported as comments on this PR as they land. Lab-Run: IR-20261112-sbxdisk
IR-20261112: feat(sbx,monitoring): alert on sandbox work-volume fill, cap GOCACHE
All checks were successful
CI Review / helm-validate (pull_request) Successful in 11s
Details
CI Review / runner-manifest-test (pull_request) Successful in 11s
Details
Alert Disposition Contract / alert-contract (pull_request) Successful in 12s
Details
CI Review / receiver-test (pull_request) Successful in 5s
Details
CI Review / alert-receiver-test (pull_request) Successful in 21s
Details
Lint & Validate / shellcheck (pull_request) Successful in 5s
Details
Lint & Validate / yaml-lint (pull_request) Successful in 20s
Details
CI Review / record-sql-test (pull_request) Successful in 56s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 1m53s
Details
CI Review / cockpit-e2e (pull_request) Successful in 2m41s
Details
CI Review / cockpit-test (pull_request) Successful in 2m50s
Details
CI Review / dispatcher-test (pull_request) Successful in 3m6s
Details
13b5619a9d 
sbx-lab-0's 60Gi `work` PVC (longhorn-fleet-work, claim work-sbx-lab-0)
reached 100% at 2026-11-12T12:38Z and WRITES FAILED: a live orchestration
session lost its scratch, its builds and its git worktrees, and no alarm
told anyone. The same day it was measured climbing again at ~7 G/h from
scratch directories. This commit makes that condition loud and takes one
of its unbounded residents off the volume.

1. PrometheusRule `sandbox-work-volume-alerts` (namespace monitoring), two
   legs, one selector:

     kubelet_volume_stats_used_bytes{namespace=~"sbx-.*|ld-fleet"}
     / kubelet_volume_stats_capacity_bytes{namespace=~"sbx-.*|ld-fleet"}

   * SandboxWorkVolumeFilling          > 0.80 for 10m, severity warning
   * SandboxWorkVolumeFillingCritical  > 0.92 for 10m, severity warning
     (the NAME carries the urgency; the severity deliberately does not,
      because the operator's inbox routing keeps a sandbox scratch fill on
      the non-paging `alert-intake` tier. A warning also never reaches
      `pi-alert-receiver`, which only `severity="critical"` and the named
      disk-pressure routes do.)

   The selector is namespace-scoped on purpose: a per-PVC list goes stale
   when a sandbox is added, and a stale list is silent -- the exact failure
   this closes. It matches FIVE claims today (read-only, 13:56Z):
   sbx-lab/work-sbx-lab-0 at 51.5% (32 506 445 824 / 63 090 503 680) and the
   four ld-fleet record volumes at 2.8-5.8%, so it is not vacuous and it is
   silent on today's healthy state. sbx-gits and sbx-sendrast mount `work`
   as a node hostPath, which has no kubelet_volume_stats series at all; that
   blind spot is stated in the file's header rather than papered over.
   Both rules carry the repo's full ld_* disposition contract
   (owner/action/frequency/disposition/silence/review/note).

   HONESTY NOTE: PersistentVolumeFillingUp (>75%, 15m, warning) already
   covers these volumes and fires FIRST on a slow fill. This file is not a
   threshold improvement; it adds the sandbox scope, the name, the
   disposition and the 0.92 leg. Said in the rule's own description too, so
   nobody reads it later as a better alarm.

2. GOCACHE off the shared volume onto a capped emptyDir, in BOTH sandboxes.
   /home/ld/work/go/cache -> /home/ld/go-cache (volume `go-cache`):
   sizeLimit 8Gi in sbx-lab, 4Gi in sbx-gits.

   CHOSEN: emptyDir + sizeLimit, NOT a pre-start `go clean -cache` at N GB.
   * A sizeLimit binds CONTINUOUSLY; exceeding it evicts THIS pod -- loud,
     immediate and self-healing, and the restarted pod starts with an empty
     cache. A pre-start clean only ever fires at a pod START, so a sandbox
     that runs for days between restarts (sbx-lab is OnDelete and is only
     restarted with the supervisor) can fill the volume between restarts and
     never run the clean at all.
   * Cost: the build cache does not survive a recreate, so the first
     `go build` afterwards is cold. That is the right thing to lose -- it is
     a cache; the git work tree lives on the PVC, and the module cache is
     what actually costs network time to rebuild.
   * Ephemeral-storage arithmetic, measured against the containers' own
     limits: sbx-lab `sandbox` mounts become tmp 8Gi + go-cache 8Gi +
     herdr-sock 64Mi = 16.06Gi against its 20Gi limit, so the volume caps
     still bind before the container limit. sbx-gits `sandbox` is 8+4+0.06 =
     12.06Gi against a 10Gi limit: the cache alone (4Gi) and /tmp alone
     (8.06Gi) can each never reach it, only their sum can, and sbx-gits is
     capped at 4Gi rather than 8Gi for exactly that reason. Raising that
     container limit was the alternative and is rejected: 10Gi was chosen so
     the volume cap binds first, and the node has 31 GiB free for the pod's
     whole ceiling.
   GOMODCACHE IS DELIBERATELY UNTOUCHED in both sandboxes and stays on the
   work volume (/home/ld/work/go/pkg/mod): it is small, mostly read-only per
   build and expensive to re-download.

3. promtool unit tests (rule-tests/sandbox-work-volume.test.yaml), run with
   promtool v3.4.0 -- the version the cluster runs:
     * 0.81 fires the 0.80 leg, does NOT fire the 0.92 leg, and is still
       silent at 5m (so `for: 10m` is proven to be 10m);
     * 0.79 is silent on both legs;
     * 0.93 fires the 0.92 leg;
     * 0.91 fires only the 0.80 leg;
     * the real capacity of the volume that filled (63090503680 B, 0.810)
       fires;
     * an ld-fleet record volume is reached by the same selector;
     * a 99% volume in a namespace OUTSIDE sbx-.*|ld-fleet stays silent, so
       the selector's scope is asserted, not just described.
   `promtool check rules` -> SUCCESS: 2 rules found.
   `promtool test rules` -> SUCCESS. The whole rule-tests suite
   (rule-tests/promtool-test.sh) is green.

NOT FOLDED INTO EXTDEL v2 (infra-core #958): that PR is open, two commits,
NOT FOR LANDING, and its landing gate pins a head (`0c72e058`) for a seat
pair plus a recreate choreography with 4a and homelab-33. Adding commits
there would move its head (voiding its seats), give one PR two Lab-Run ids,
and force a re-measure of the allowed-change sets it pins (T7[diff],
17[R159](d)); this change touches the same two files. So it is its own PR.
This branch still changes exactly `M statefulset.yaml` under
core/manifests/apps/sbx-gits, which is what T7[diff] permits.

Both statefulsets are OnDelete: this changes the template and restarts
NOTHING. GOCACHE takes effect at each pod's next recreate, and so does the
new volume. No pod was touched, no cluster write was made by this commit.

4. The repo's own epic suite (scripts/gits-sbx-epic-test.sh) is updated where
   this branch's delta is the thing it pins, and nowhere else:
   * p1_lab_sts_bad's allowed-addition literals gain `go-cache` and
     /home/ld/go-cache (the same one-place allowlist every wave that adds a
     volume extends; the check stays an allowlist, not a blanket "it changed");
   * SBX_GITS_DIFF_SET is RE-MEASURED for this branch to `M statefulset.yaml`
     (the previous literal was the measured delta of a branch already in live).
   MEASURED: this branch 16 of 701 checks fail; a clean origin/live worktree
   fails the SAME 16 of 700 (diff of the two FAIL lists is empty, and it is
   reported as 16 pre-existing environment failures, not as green CI).
   The new checks here are green: H8[8][sbx-gits] PASS,
   H8[8][sbx-gits][mut] baseline PASS, 10b[R99][sbx-sendrast] PASS,
   T7[diff] PASS with its red mutation PASS.

NOT FOR LANDING: f1 one-look, then a seat pair + GO.

Lab-Run: IR-20261112-sbxdisk
All checks were successful
CI Review / helm-validate (pull_request) Successful in 11s
Required
Details
CI Review / runner-manifest-test (pull_request) Successful in 11s
Details
Alert Disposition Contract / alert-contract (pull_request) Successful in 12s
Details
CI Review / receiver-test (pull_request) Successful in 5s
Details
CI Review / alert-receiver-test (pull_request) Successful in 21s
Details
Lint & Validate / shellcheck (pull_request) Successful in 5s
Required
Details
Lint & Validate / yaml-lint (pull_request) Successful in 20s
Required
Details
CI Review / record-sql-test (pull_request) Successful in 56s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 1m53s
Required
Details
CI Review / cockpit-e2e (pull_request) Successful in 2m41s
Details
CI Review / cockpit-test (pull_request) Successful in 2m50s
Details
CI Review / dispatcher-test (pull_request) Successful in 3m6s
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin plane/IR-20261112-sbxdisk:plane/IR-20261112-sbxdisk
git switch plane/IR-20261112-sbxdisk
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
operator-review

Requires human operator review before merge — dangerous class (RBAC, secrets, network policy, Terraform) or explicitly gated. Never auto-merge.

No labels
operator-review
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Homelab/infra-core!963
No description provided.