IR-20261111: chore(forgejo-runner): replicas 8 -> 10 (restore CI slots after capacity 1) #949

Merged

pi-landing merged 2 commits from plane/IR-20261111-runners10 into live 2026-11-12 03:45:41 +00:00

Conversation 2 Commits 2 Files changed 4 +123 -12

pi-landing commented 2026-11-11 21:50:20 +00:00

Collaborator

Copy link

Why

homelab-3c's PR takes the runner pool's capacity from 2 to 1 (two host-mode jobs were sharing one runner container's 2-CPU limit, and the Sendrast e2e timed out). At 9 pods that is 16 → 9 concurrent job slots if nothing else changes. This PR restores part of that loss by adding pods: 8 → 9 replicas.

Why 9 and not 10 (round 2, after prod-wk-01 came off the pool). Each runner pod requests 14Gi of ephemeral storage (10Gi runner + 4Gi dind). With wk-01 excluded the eligible nodes are wk-02, wk-05, wk-06, and 14Gi is the binding leg:

node	eph allocatable	eph requested now	+1 pod	+2 pods
prod-wk-02	92.4Gi	12.0Gi (13.0%)	26.0Gi (28.1%)	40.0Gi (43.3%)
prod-wk-05	87.8Gi	62.1Gi (70.8%)	76.1Gi (86.7%) — its last pod	90.1Gi (102.7% — unschedulable)
prod-wk-06	87.8Gi	56.0Gi (63.8%)	70.0Gi (79.8%)	84.0Gi (95.7% — over the 90% bar)

Nine pods means one new ordinal: it can land on any eligible node and every case stays ≤86.7% of allocatable ephemeral. Ten means two, and the ceiling the fleet sizes ephemeral against (≤90% requested per node) then holds only in some placements: wk-05 physically cannot take a second (102.7%), and a second on wk-06 is 95.7%. wk-02 is not a dependable safety valve — it is at 99% actual CPU (kubectl top node: 7929m of 7950m) and its Longhorn disk is already over its booking limit (below), so a soft spread that points both new pods there is not a placement anyone can rely on. 10 does not certify ≤90% without wk-01; 9 does. 9 is the max that does, and the overseer's brief explicitly allows it. 8 → 9 costs one slot against the 10 the first round of this PR assumed.

The change — three files, all saying the same number

 core/manifests/apps/forgejo-runner/statefulset.yaml
-  replicas: 10            # (round 1 said 10; see the headroom table above)
+  replicas: 9
   affinity:
     nodeAffinity:
       requiredDuringSchedulingIgnoredDuringExecution:
         nodeSelectorTerms:
-          - matchExpressions:
-              - key: kubernetes.io/hostname
-                operator: In
-                values: [prod-wk-01, prod-wk-02, prod-wk-05, prod-wk-06]
+          - matchExpressions:
+              - key: kubernetes.io/hostname
+                operator: In
+                values: [prod-wk-02, prod-wk-05, prod-wk-06]
+              - key: kubernetes.io/hostname
+                operator: NotIn
+                values: [prod-wk-01]

 core/manifests/apps/forgejo-runner/dind-gc-cronjob.yaml   RUNNER_COUNT "8" -> "9"
 core/manifests/apps/forgejo-runner/dind-gc-script.yaml    default 8 -> 9, loop guard
 scripts/manifests/tests/test_runner_dind_gc_tightening.py sweep width now read from spec.replicas

No requests, no limits, no images, no storage class, no runtime config, no RBAC, no NetworkPolicy, no .forgejo/**, no Terraform, no DB. configmap.yaml's stale "capacity 2 x 8 replicas" comment is deliberately untouched: it is the act_runner runtime config, and a comment there belongs in its own change, not in a replica-count fix.

prod-wk-01 is off the pool (required, not preferred)

prod-wk-01 hosts sbx-gits, whose Android emulator was OOMKilled at 20:20Z on 2026-11-11; with one more runner its memory requests would reach 86.9%. The exclusion is a required NotIn [prod-wk-01], plus wk-01 removed from the In list, so a future widening of that list cannot silently re-admit it.

It is in the SAME nodeSelectorTerm on purpose. nodeSelectorTerms are OR'd; matchExpressions inside a term are AND'd. A NotIn in its own term would widen the eligible set to "every node except wk-01" and let pods onto wk-03 (85% memory requests) and wk-04 (3.8Gi memory allocatable) — the two nodes this affinity exists to exclude. The soft topology spread (maxSkew: 1, whenUnsatisfiable: ScheduleAnyway) is untouched, for the HOMELAB-2418 reason: existing pods are volume-pinned by longhorn-runner (strict-local), so a hard skew constraint would refuse to place restarts.

Today's eight pods are unchanged by the affinity edit and stay where their volumes are: wk-05 = 4 (ordinals 0,2,3,6), wk-06 = 4 (1,4,5,7), wk-01 = 0, wk-02 = 0. The new ordinal 8 has no PVC yet (WaitForFirstConsumer), so the scheduler is free to choose. wk-02 is the spread's first choice (0 runners) and wk-05/wk-06 the storage-steered one; both placements are sized below.

Per-node headroom — BEFORE (live, 8 pods, 2026-11-11 ~22:5xZ)

kubectl describe node (allocatable / requests) + kubectl top node (actual working set) + Prometheus node_memory_MemAvailable_bytes.

node	mem alloc	mem req	free mem (top: alloc − working set)	MemAvailable	eph alloc	eph req	cpu req	actual mem
prod-wk-01 (now EXCLUDED)	32.8Gi	27126Mi (80.8%)	14.4Gi	14.41Gi	92.4Gi	9.0Gi (9.7%)	10499m (58.5%)	18376Mi (54%)
prod-wk-02	23.0Gi	9245Mi (39.3%)	13.5Gi	14.45Gi	92.4Gi	12.0Gi (13.0%)	4844m (60.9%)	9733Mi (41%)
prod-wk-05	58.4Gi	34883Mi (58.4%)	41.5Gi	41.26Gi	87.8Gi	62.1Gi (70.8%)	29159m (60.8%)	17243Mi (28%)
prod-wk-06	23.0Gi	15249Mi (64.8%)	16.4Gi	15.94Gi	87.8Gi	56.0Gi (63.8%)	11064m (27.7%)	6763Mi (28%)

top is the working-set basis the kubelet's own memory.available signal uses, so "free mem" here is the number its eviction threshold reads (it sits 2.1-2.3GiB above /proc MemAvailable — capacity-policy-20261110/remodel-c.md).

Per-node headroom — AFTER (+1 pod = 2048Mi / 1500m / 14Gi requests; 10.37Gi / 10c / 24Gi limits)

where the new pod lands	mem req	eph req	cpu req	headroom left (mem / eph / cpu)
prod-wk-02	11293Mi (48.0%)	26.0Gi (28.1%)	6344m (79.8%)	12.5Gi / 66.4Gi / 1606m
prod-wk-05	36931Mi (61.8%)	76.1Gi (86.7%)	30659m (63.9%)	20.4Gi / 11.7Gi / 17291m
prod-wk-06	17297Mi (73.6%)	70.0Gi (79.8%)	12564m (31.4%)	5.7Gi / 17.8Gi / 27386m

Every eligible node stays inside allocatable and inside the ≤90% ephemeral bar. wk-05 is now at its last pod (one more after this is 102.7%, unschedulable), which is the constraint that sets the count at 9.

The Talos OOM question — can +1 pod push a worker into the OOM zone?

No. One new pod adds 2048Mi of requests and, at the pool's measured worst pod-cgroup peak of 2258.79Mi (7d), ~2.2Gi of real memory — against 13.5-41.5Gi of free memory on every eligible node.

1. A runner cannot starve a node — it is a bounded cgroup. Both containers declare limits, so the pod is Burstable with memory.max set: 2560Mi (runner) + 8Gi (dind) = a 10.37Gi pod ceiling. A job that tries to take 20Gi is OOM-killed inside its own pod cgroup, which costs one CI slot and touches nothing else.
2. The Talos OOM zone needs node-wide stall, not a big pod. The live trigger on all six workers (FIX-B: wk02-oom-20261110/, talos-oom/trigger-proposal.md) is memory_full_avg10 > 75 && time_since_trigger > duration("10s") (or the QoS-full-vector clause) — memory PSI full ≥75% for 10s, which on a 24GiB node lands at MemAvailable ≈1.2-1.4Gi (capacity-policy-20261110/remodel-c.md; measured there: signal 9.5%, MemAvailable 0.68Gi when it fired). After this change the smallest eligible node still has ~5.7Gi of memory headroom in the worst placement — roughly 4x above the trigger depth — and 13.5Gi is the free-memory number on wk-02 today.
3. If a runner ever does misbehave, Talos kills it first. Under FIX-B a bounded cgroup's score is multiplied by ×4 (memory_max.hasValue() ? 4.0 : 1.0) while Guaranteed/Podruntime/System rank 0.0, so the runner is at the front of the kill queue and Longhorn's instance-manager (Guaranteed, system-node-critical) at the back — the ordering capacity-policy-20261110/brief.md §2/§3 requires.
4. The worst case is bounded and stated: the new pod's own 10.37Gi limit stops it long before the node; +1 pod cannot stack two dind ceilings on one node, which is exactly what the 9-not-10 decision above buys.

Longhorn booking for the new ordinal (round-2 confirmation)

The new ordinal provisions runner-data 1Gi + docker-data 30Gi on longhorn-runner, which is strict-local, 1 replica — so the booking cost is 31Gi on one node, not 62Gi. Live read (nodes.longhorn.io status + the LonghornDiskSchedulingHeadroom expression from core/manifests/monitoring/rules/longhorn-alerts.yaml, bookable = (storageMaximum − storageReserved) × 110% − storageScheduled):

node	physical available	bookable headroom	verdict for a new 31Gi strict-local volume
prod-wk-01 (excluded)	294.9Gi	+5.60Gi	would not fit either
prod-wk-02	346.8Gi	−18.52Gi	disk Schedulable=False right now: "not schedulable for more replica; ScheduledTotal = 492042190848 (Size + StorageScheduled) is greater than ProvisionedLimit = 472155741798"
prod-wk-05	116.7Gi	+1.66Gi	31Gi exceeds the 110% provisioned limit (350.1Gi booked of 351.8Gi)
prod-wk-06	179.9Gi	+8.93Gi	31Gi exceeds the 110% provisioned limit (325.2Gi booked of 334.2Gi)
prod-wk-03 (not runner-eligible)	298.2Gi	+161.81Gi	the only disk with room

Confirmation, stated plainly: physically there is ample space on wk-05 (116.7Gi) and wk-06 (179.9Gi) for one 31Gi volume, so Longhorn can place it and this PR does not risk data. But the cluster is booked to its over-provisioning ceiling (HOMELAB-2249 records this as by-design: 0/6 disks meaningfully bookable), and wk-02 has actually tripped the guard. So the one new PVC may be refused or delayed by Longhorn's provisioned-limit check, and the symptom would be ordinal 8 sitting Pending — i.e. the pool stays at 8 slots — not a memory, OOM or data event. This changes nothing about the replica arithmetic above (9 needs one 31Gi booking, not two), and it is pre-existing cluster state, not something this PR creates. Levers, per HOMELAB-1852, to decide in the gap if ordinal 8 goes Pending: lower storageReserved on wk-05 (80Gi) / wk-06 (96Gi), grow the disk, or place the new pod on wk-03 (161.8Gi bookable, excluded here only by the 85%-memory-request rule). Per storageclass.yaml's own documented step, the new cache volume should be labelled recurring-job-group.longhorn.io/default=disabled after it provisions, so 30Gi of regenerable cache stays out of the B2 backup set.

dind-gc now follows the replica count (closes round 1's known coupling)

Round 1 left it broken on purpose and flagged it. This head fixes it:

• dind-gc-cronjob.yaml — RUNNER_COUNT: "8" → "9", with the reason written next to it (the sweep probes ordinals 0..RUNNER_COUNT-1 over pod DNS, so a pod above that number is never visited and its docker-data grows without bound — the HOMELAB-2481 leak class).
• dind-gc-script.yaml — the shell default moves 8 → 9 with it, the ordinal loop is documented as RUNNER_COUNT-driven (it already was, while [ "$ordinal" -lt "$RUNNER_COUNT" ] — no hard-coded 0..7 range existed at this head), and main() now rejects a non-numeric RUNNER_COUNT (exit 2) instead of silently sweeping a garbage range.
• test_runner_dind_gc_tightening.py (A5) — the assertion used to hardcode RUNNER_COUNT = "8", which is why a replica bump could leave the CronJob behind and still pass CI. It now reads statefulset.yaml spec.replicas and fails if the CronJob env or the script default disagree with it, and if the ordinal loop stops referencing RUNNER_COUNT. Two new --self-check cases prove both legs can fail (CronJob left at 8; loop hard-coded to -lt 8). --self-check rejects 31 bad inputs; the suite passes.

The GC job's node affinity (In [prod-wk-05, prod-wk-06]) is unchanged and still correct: it reaches every ordinal over pod DNS, wherever the pod runs.

Rollout — this one DOES bounce the pool

• Merge alone changes nothing. No kubectl apply/scale, no cluster change from this branch; ArgoCD applies it in the runner gap (homelab-a9 gets the 30-minute notice).
• Read this before scheduling the gap: round 1's diff was the replicas field only, so it created ordinals 8-9 without touching 0-7. This head also changes the pod template (the affinity), so the StatefulSet controller replaces every replica — reverse ordinal order, and because podManagementPolicy: Parallel the update pass is not gated on each pod becoming Ready. Expect the whole pool to bounce together, in-flight jobs killed, and pods returning to their own nodes (strict-local PVCs pin them). That is the cost of the wk-01 exclusion and it belongs in the gap, next to 3c's capacity change.
• Existing volumes stay pinned where they are; only the new ordinal picks a node.

Class and landing

• Routine class, not dangerous: one replica count, one affinity expression, a GC count + script guard + its CI assertion. No RBAC, no secrets, no NetworkPolicy, no Terraform, no DB schema, no .forgejo/**, no runtime config, no PVC delete.
• CI at this head: see the checks on the commit. The three offline runner suites pass locally (test_runner_dind_gc_tightening.py + --self-check, test_runner_placement_prep.py, test_runner_ephemeral_ceilings.py), and shellcheck is clean on the embedded GC script.
• No seats, no merge: landing-lock wants two independent verdicts at the head (verdict comments carrying model: and a bare session:), then El Director's LD-GO. The head commit's last trailer is Lab-Run: IR-20261111-runners10.

## Why homelab-3c's PR takes the runner pool's `capacity` from 2 to 1 (two host-mode jobs were sharing one runner container's 2-CPU limit, and the Sendrast e2e timed out). At 9 pods that is **16 → 9 concurrent job slots** if nothing else changes. This PR restores part of that loss by adding pods: **8 → 9 replicas**. **Why 9 and not 10 (round 2, after prod-wk-01 came off the pool).** Each runner pod *requests* 14Gi of ephemeral storage (10Gi runner + 4Gi dind). With wk-01 excluded the eligible nodes are wk-02, wk-05, wk-06, and 14Gi is the binding leg: | node | eph allocatable | eph requested now | +1 pod | +2 pods | |---|---|---|---|---| | prod-wk-02 | 92.4Gi | 12.0Gi (13.0%) | 26.0Gi (28.1%) | 40.0Gi (43.3%) | | prod-wk-05 | 87.8Gi | 62.1Gi (**70.8%**) | **76.1Gi (86.7%) — its last pod** | 90.1Gi (**102.7% — unschedulable**) | | prod-wk-06 | 87.8Gi | 56.0Gi (63.8%) | 70.0Gi (79.8%) | 84.0Gi (**95.7% — over the 90% bar**) | Nine pods means **one** new ordinal: it can land on any eligible node and every case stays ≤86.7% of allocatable ephemeral. Ten means two, and the ceiling the fleet sizes ephemeral against (≤90% requested per node) then holds only in *some* placements: wk-05 physically cannot take a second (102.7%), and a second on wk-06 is 95.7%. wk-02 is not a dependable safety valve — it is at **99% actual CPU** (`kubectl top node`: 7929m of 7950m) and its Longhorn disk is already over its booking limit (below), so a soft spread that points both new pods there is not a placement anyone can rely on. **10 does not certify ≤90% without wk-01; 9 does. 9 is the max that does, and the overseer's brief explicitly allows it.** 8 → 9 costs one slot against the 10 the first round of this PR assumed. ## The change — three files, all saying the same number ```diff core/manifests/apps/forgejo-runner/statefulset.yaml - replicas: 10 # (round 1 said 10; see the headroom table above) + replicas: 9 affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - - matchExpressions: - - key: kubernetes.io/hostname - operator: In - values: [prod-wk-01, prod-wk-02, prod-wk-05, prod-wk-06] + - matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: [prod-wk-02, prod-wk-05, prod-wk-06] + - key: kubernetes.io/hostname + operator: NotIn + values: [prod-wk-01] ``` ``` core/manifests/apps/forgejo-runner/dind-gc-cronjob.yaml RUNNER_COUNT "8" -> "9" core/manifests/apps/forgejo-runner/dind-gc-script.yaml default 8 -> 9, loop guard scripts/manifests/tests/test_runner_dind_gc_tightening.py sweep width now read from spec.replicas ``` No requests, no limits, no images, no storage class, no runtime config, no RBAC, no NetworkPolicy, no `.forgejo/**`, no Terraform, no DB. `configmap.yaml`'s stale "capacity 2 x 8 replicas" comment is deliberately untouched: it is the act_runner runtime config, and a comment there belongs in its own change, not in a replica-count fix. ## prod-wk-01 is off the pool (required, not preferred) `prod-wk-01` hosts **sbx-gits**, whose Android emulator was **OOMKilled at 20:20Z on 2026-11-11**; with one more runner its memory requests would reach 86.9%. The exclusion is a **required** `NotIn [prod-wk-01]`, plus wk-01 removed from the `In` list, so a future widening of that list cannot silently re-admit it. **It is in the SAME `nodeSelectorTerm` on purpose.** `nodeSelectorTerms` are OR'd; `matchExpressions` inside a term are AND'd. A `NotIn` in its own term would *widen* the eligible set to "every node except wk-01" and let pods onto wk-03 (85% memory requests) and wk-04 (3.8Gi memory allocatable) — the two nodes this affinity exists to exclude. The soft topology spread (`maxSkew: 1`, `whenUnsatisfiable: ScheduleAnyway`) is untouched, for the HOMELAB-2418 reason: existing pods are volume-pinned by `longhorn-runner` (strict-local), so a hard skew constraint would refuse to place restarts. Today's eight pods are unchanged by the affinity edit and stay where their volumes are: **wk-05 = 4** (ordinals 0,2,3,6), **wk-06 = 4** (1,4,5,7), **wk-01 = 0, wk-02 = 0**. The new ordinal 8 has no PVC yet (`WaitForFirstConsumer`), so the scheduler is free to choose. wk-02 is the spread's first choice (0 runners) and wk-05/wk-06 the storage-steered one; both placements are sized below. ## Per-node headroom — BEFORE (live, 8 pods, 2026-11-11 ~22:5xZ) `kubectl describe node` (allocatable / requests) + `kubectl top node` (actual working set) + Prometheus `node_memory_MemAvailable_bytes`. | node | mem alloc | mem req | free mem (`top`: alloc − working set) | MemAvailable | eph alloc | eph req | cpu req | actual mem | |---|---|---|---|---|---|---|---|---| | prod-wk-01 *(now EXCLUDED)* | 32.8Gi | 27126Mi (80.8%) | 14.4Gi | 14.41Gi | 92.4Gi | 9.0Gi (9.7%) | 10499m (58.5%) | 18376Mi (54%) | | prod-wk-02 | 23.0Gi | 9245Mi (39.3%) | 13.5Gi | 14.45Gi | 92.4Gi | 12.0Gi (13.0%) | 4844m (60.9%) | 9733Mi (41%) | | prod-wk-05 | 58.4Gi | 34883Mi (58.4%) | 41.5Gi | 41.26Gi | 87.8Gi | 62.1Gi (**70.8%**) | 29159m (60.8%) | 17243Mi (28%) | | prod-wk-06 | 23.0Gi | 15249Mi (64.8%) | 16.4Gi | 15.94Gi | 87.8Gi | 56.0Gi (63.8%) | 11064m (27.7%) | 6763Mi (28%) | `top` is the working-set basis the kubelet's own `memory.available` signal uses, so "free mem" here is the number its eviction threshold reads (it sits 2.1-2.3GiB above `/proc MemAvailable` — `capacity-policy-20261110/remodel-c.md`). ## Per-node headroom — AFTER (+1 pod = 2048Mi / 1500m / 14Gi requests; 10.37Gi / 10c / 24Gi limits) | where the new pod lands | mem req | eph req | cpu req | headroom left (mem / eph / cpu) | |---|---|---|---|---| | prod-wk-02 | 11293Mi (48.0%) | 26.0Gi (28.1%) | 6344m (79.8%) | 12.5Gi / 66.4Gi / 1606m | | prod-wk-05 | 36931Mi (61.8%) | **76.1Gi (86.7%)** | 30659m (63.9%) | **20.4Gi / 11.7Gi / 17291m** | | prod-wk-06 | 17297Mi (73.6%) | **70.0Gi (79.8%)** | 12564m (31.4%) | 5.7Gi / 17.8Gi / 27386m | Every eligible node stays inside allocatable and inside the ≤90% ephemeral bar. **wk-05 is now at its last pod** (one more after this is 102.7%, unschedulable), which is the constraint that sets the count at 9. ## The Talos OOM question — can +1 pod push a worker into the OOM zone? **No.** One new pod adds **2048Mi of requests** and, at the pool's measured worst pod-cgroup peak of 2258.79Mi (7d), ~2.2Gi of real memory — against 13.5-41.5Gi of free memory on every eligible node. 1. **A runner cannot starve a node — it is a bounded cgroup.** Both containers declare limits, so the pod is Burstable with `memory.max` set: 2560Mi (runner) + 8Gi (dind) = a **10.37Gi pod ceiling**. A job that tries to take 20Gi is OOM-killed *inside its own pod cgroup*, which costs one CI slot and touches nothing else. 2. **The Talos OOM zone needs node-wide stall, not a big pod.** The live trigger on all six workers (FIX-B: `wk02-oom-20261110/`, `talos-oom/trigger-proposal.md`) is `memory_full_avg10 > 75 && time_since_trigger > duration("10s")` (or the QoS-full-vector clause) — memory PSI full ≥75% for 10s, which on a 24GiB node lands at MemAvailable ≈1.2-1.4Gi (`capacity-policy-20261110/remodel-c.md`; measured there: signal 9.5%, MemAvailable 0.68Gi when it fired). After this change the smallest eligible node still has ~5.7Gi of memory headroom in the worst placement — **roughly 4x above the trigger depth** — and 13.5Gi is the free-memory number on wk-02 today. 3. **If a runner ever does misbehave, Talos kills it first.** Under FIX-B a bounded cgroup's score is multiplied by **×4** (`memory_max.hasValue() ? 4.0 : 1.0`) while Guaranteed/Podruntime/System rank 0.0, so the runner is at the front of the kill queue and Longhorn's instance-manager (Guaranteed, system-node-critical) at the back — the ordering `capacity-policy-20261110/brief.md` §2/§3 requires. 4. **The worst case is bounded and stated:** the new pod's own 10.37Gi limit stops it long before the node; +1 pod cannot stack two dind ceilings on one node, which is exactly what the 9-not-10 decision above buys. ## Longhorn booking for the new ordinal (round-2 confirmation) The new ordinal provisions `runner-data` 1Gi + `docker-data` 30Gi on `longhorn-runner`, which is **strict-local, 1 replica** — so the booking cost is **31Gi on one node, not 62Gi**. Live read (`nodes.longhorn.io` status + the `LonghornDiskSchedulingHeadroom` expression from `core/manifests/monitoring/rules/longhorn-alerts.yaml`, bookable = `(storageMaximum − storageReserved) × 110% − storageScheduled`): | node | physical available | bookable headroom | verdict for a new 31Gi strict-local volume | |---|---|---|---| | prod-wk-01 *(excluded)* | 294.9Gi | **+5.60Gi** | would not fit either | | prod-wk-02 | 346.8Gi | **−18.52Gi** | disk `Schedulable=False` right now: "not schedulable for more replica; ScheduledTotal = 492042190848 (Size + StorageScheduled) is greater than ProvisionedLimit = 472155741798" | | prod-wk-05 | 116.7Gi | **+1.66Gi** | 31Gi exceeds the 110% provisioned limit (350.1Gi booked of 351.8Gi) | | prod-wk-06 | 179.9Gi | **+8.93Gi** | 31Gi exceeds the 110% provisioned limit (325.2Gi booked of 334.2Gi) | | prod-wk-03 *(not runner-eligible)* | 298.2Gi | +161.81Gi | the only disk with room | **Confirmation, stated plainly:** *physically* there is ample space on wk-05 (116.7Gi) and wk-06 (179.9Gi) for one 31Gi volume, so Longhorn can place it and this PR does not risk data. But the cluster is booked to its over-provisioning ceiling (HOMELAB-2249 records this as by-design: 0/6 disks meaningfully bookable), and wk-02 has actually tripped the guard. So the one new PVC may be **refused or delayed** by Longhorn's provisioned-limit check, and the symptom would be ordinal 8 sitting Pending — i.e. the pool stays at 8 slots — **not** a memory, OOM or data event. This changes nothing about the replica arithmetic above (9 needs one 31Gi booking, not two), and it is pre-existing cluster state, not something this PR creates. Levers, per HOMELAB-1852, to decide in the gap if ordinal 8 goes Pending: lower `storageReserved` on wk-05 (80Gi) / wk-06 (96Gi), grow the disk, or place the new pod on wk-03 (161.8Gi bookable, excluded here only by the 85%-memory-request rule). Per `storageclass.yaml`'s own documented step, the new cache volume should be labelled `recurring-job-group.longhorn.io/default=disabled` after it provisions, so 30Gi of regenerable cache stays out of the B2 backup set. ## dind-gc now follows the replica count (closes round 1's known coupling) Round 1 left it broken on purpose and flagged it. This head fixes it: - `dind-gc-cronjob.yaml` — `RUNNER_COUNT: "8"` → **`"9"`**, with the reason written next to it (the sweep probes ordinals `0..RUNNER_COUNT-1` over pod DNS, so a pod above that number is never visited and its `docker-data` grows without bound — the HOMELAB-2481 leak class). - `dind-gc-script.yaml` — the shell default moves 8 → 9 with it, the ordinal loop is documented as `RUNNER_COUNT`-driven (**it already was**, `while [ "$ordinal" -lt "$RUNNER_COUNT" ]` — no hard-coded 0..7 range existed at this head), and `main()` now rejects a non-numeric `RUNNER_COUNT` (`exit 2`) instead of silently sweeping a garbage range. - `test_runner_dind_gc_tightening.py` (A5) — the assertion **used to hardcode `RUNNER_COUNT = "8"`**, which is why a replica bump could leave the CronJob behind and still pass CI. It now reads `statefulset.yaml spec.replicas` and fails if the CronJob env or the script default disagree with it, and if the ordinal loop stops referencing `RUNNER_COUNT`. Two new `--self-check` cases prove both legs can fail (CronJob left at 8; loop hard-coded to `-lt 8`). `--self-check` rejects 31 bad inputs; the suite passes. The GC job's node affinity (`In [prod-wk-05, prod-wk-06]`) is unchanged and still correct: it reaches every ordinal over pod DNS, wherever the pod runs. ## Rollout — this one DOES bounce the pool - **Merge alone changes nothing.** No `kubectl apply/scale`, no cluster change from this branch; ArgoCD applies it in the runner gap (homelab-a9 gets the 30-minute notice). - **Read this before scheduling the gap:** round 1's diff was the replicas field only, so it created ordinals 8-9 without touching 0-7. **This head also changes the pod template** (the affinity), so the StatefulSet controller replaces **every** replica — reverse ordinal order, and because `podManagementPolicy: Parallel` the update pass is not gated on each pod becoming Ready. Expect the whole pool to bounce together, in-flight jobs killed, and pods returning to their own nodes (strict-local PVCs pin them). That is the cost of the wk-01 exclusion and it belongs in the gap, next to 3c's capacity change. - Existing volumes stay pinned where they are; only the new ordinal picks a node. ## Class and landing - **Routine class**, not dangerous: one replica count, one affinity expression, a GC count + script guard + its CI assertion. No RBAC, no secrets, no NetworkPolicy, no Terraform, no DB schema, no `.forgejo/**`, no runtime config, no PVC delete. - CI at this head: see the checks on the commit. The three offline runner suites pass locally (`test_runner_dind_gc_tightening.py` + `--self-check`, `test_runner_placement_prep.py`, `test_runner_ephemeral_ceilings.py`), and `shellcheck` is clean on the embedded GC script. - No seats, no merge: landing-lock wants two independent verdicts at the head (verdict comments carrying `model:` and a bare `session:`), then El Director's `LD-GO`. The head commit's last trailer is `Lab-Run: IR-20261111-runners10`.

pi-landing added 1 commit 2026-11-11 21:50:20 +00:00

IR-20261111: chore(forgejo-runner): replicas 8 -> 10 ... d77bd1a756

3c's capacity 2 -> 1 halves the pool's concurrent job slots 16 -> 8.
One more pod per runner node restores 10 slots with no per-pod resource
change: the pod's requests (1.5c / 2048Mi / 14Gi) and limits (10c /
10.37Gi / 24Gi) are unchanged.

Ephemeral storage is what caps this at 10, not 12: 14Gi of requests per
pod against prod-wk-05's 63616Mi of 89863Mi allocatable (70.8%) -> one
more pod is 86.7%, two are 102.7% and the second is unschedulable there.
wk-06 goes 63.8% -> 79.8%. Full per-node before/after headroom, the
placement arithmetic (soft spread puts the two new ordinals on wk-01 /
wk-02) and the Talos OOM statement are in the PR body.

The diff is the replicas field only. dind-gc's RUNNER_COUNT (=8, "must
match spec.replicas") has to reach 10 in the same rollout gap or
ordinals 8/9 go unswept; that one-line follow-up is flagged on the PR and
deliberately kept out of this commit, per the brief's scope.

Lab-Run: IR-20261111-runners10

pi-landing referenced this pull request from a commit 2026-11-11 22:10:27 +00:00

IR-20261111: fix(forgejo-runner): replicas 9, wk-01 off the pool, dind-gc follows it

pi-landing added 1 commit 2026-11-11 22:10:27 +00:00

IR-20261111: fix(forgejo-runner): replicas 9, wk-01 off the pool, dind-gc follows it ... a290cc94d6

Round 2 on #949. Three changes, all consistent with each other:

1. replicas 8 -> 9, not 10. prod-wk-01 is out of the eligible set (it
   hosts sbx-gits, whose emulator was OOMKilled at 20:20Z on 11-11), and
   without it 10 does not clear the <=90% ephemeral-storage bar the fleet
   sizes against: one more pod takes wk-05 to 86.7% and is its last (two
   are 102.7%, unschedulable), wk-06 goes 63.8% -> 79.8% for one and
   95.7% for two. 9 clears the bar in every reachable placement; 10 only
   clears it in some. So 9 (one new ordinal, slot count 8 -> 9).

2. prod-wk-01 is removed from the runner affinity AND named in an
   explicit required NotIn, in the SAME nodeSelectorTerm (terms are OR'd,
   expressions AND'd, so a NotIn in its own term would widen the set to
   "everything except wk-01"). The soft topology spread is untouched.

3. dind-gc keeps up: RUNNER_COUNT 8 -> 9 in the CronJob, the script
   default moves with it, and the ordinal loop (already RUNNER_COUNT-
   driven) now has a guard that fails loudly on a non-numeric count. The
   CI assertion that used to hardcode "8" now READS spec.replicas, so a
   future replica change cannot leave the sweep behind again.

A pod-template change also rolls every existing replica (StatefulSet
RollingUpdate, reverse ordinal order, and with podManagementPolicy:
Parallel the controller does not gate on readiness), so unlike the
round-1 diff this one bounces the whole pool: it belongs in the runner
gap and it interrupts in-flight jobs. Merge alone changes nothing in the
cluster until ArgoCD syncs.

Per-node headroom without wk-01, the Talos OOM statement, and the
Longhorn booking confirmation (a new 31Gi strict-local volume has 1.66Gi
of bookable headroom on wk-05 and 8.93Gi on wk-06) are below.

Lab-Run: IR-20261111-runners10

ld-reviewer commented 2026-11-12 02:28:50 +00:00

Collaborator

Copy link

APPROVE — ld-reviewer — head a290cc94d6

Reviewed the diff, the head files and the commit statuses independently at a290cc94d6 (branch plane/IR-20261111-runners10, two commits: d77bd1a756 round 1, a290cc94d6 head). I did not read any existing comment or review on this PR; this verdict comes only from the diff, the statuses and the body.

1. Scope — exactly four files, nothing else. The files listing and the raw diff agree: core/manifests/apps/forgejo-runner/dind-gc-cronjob.yaml (+10/−3), dind-gc-script.yaml (+18/−1), statefulset.yaml (+36/−2), scripts/manifests/tests/test_runner_dind_gc_tightening.py (+59/−6). changed_files = 4, and stripping comment lines leaves only: replicas: 8 → 9, the two affinity expressions, RUNNER_COUNT "8" → "9" in the CronJob env and in the script default, the case "$RUNNER_COUNT" guard in main(), and the test rewrite. Pass.

2. No changes to the prohibited areas. No hunk touches resources, images, storage, RBAC, NetworkPolicy, .forgejo/, or terraform. At head the container resources are unchanged and still sum to the documented pod request/limits: runner 500m / 1003Mi / 10Gi requests, 2c / 2560Mi / 14Gi limits; dind 200m / 1045Mi / 4Gi requests, 8c / 8Gi / 10Gi limits; both images remain digest-pinned; both volumeClaimTemplates keep storageClassName: longhorn-runner (1Gi runner-data, 30Gi docker-data); the NotIn sits inside the same single nodeSelectorTerm as the In list, so it narrows rather than widens, and the soft topologySpreadConstraints (maxSkew: 1, ScheduleAnyway) is byte-for-byte untouched. Pass.

3. Headroom arithmetic — recomputed, and the ephemeral leg that decides 9 is consistent. wk-02: 12.0/92.4 = 13.0%, +1 = 26.0/92.4 = 28.1%, +2 = 40.0/92.4 = 43.3%. wk-05: 62.1/87.8 = 70.7%, +1 = 76.1/87.8 = 86.7%, +2 = 90.1/87.8 = 102.6% (above allocatable, i.e. unschedulable there). wk-06: 56.0/87.8 = 63.8%, +1 = 70.0/87.8 = 79.7%, +2 = 84.0/87.8 = 95.7%. So after +1 no eligible node exceeds 86.7% and wk-05 is indeed on its last slot; +2 fails the ≤90% bar on both wk-05 and wk-06. The 9-not-10 conclusion holds on the numbers. One column is wrong, in the safe direction: the "+1 pod = 2048Mi / 1500m / 14Gi requests" line disagrees with the manifest, which sums to 700m cpu requests (500m runner + 200m dind); memory (2048Mi) and ephemeral (14Gi) match exactly. Because the stated cpu delta is larger than the real one, that column overstates consumption and errs toward refusing the 10th pod — it cannot have let a bad placement through, and it does not move the ephemeral leg that sets the count. Worth correcting in the merged record.

4. The pool bounce is disclosed. The "Rollout — this one DOES bounce the pool" section and the commit message both say plainly that the pod-template change replaces every replica, kills in-flight jobs, and belongs inside the runner gap, and that merging alone changes nothing until ArgoCD syncs. One mechanism refinement: podManagementPolicy: Parallel upstream "only affects the behavior for scaling operations. Updates are not affected", so the replacement is reverse-ordinal and readiness-gated per pod rather than one simultaneous swap. That changes the shape of the outage, not its operational meaning — every replica is still replaced and every in-flight job is still killed inside the gap, which is the part the plan depends on and the part that is disclosed. Pass.

5. CI on the exact head: green. GET /repos/Homelab/infra-core/commits/a290cc94d6efeb71de52860213a150b7c544c0c2/status returns state=success, total_count=11, with all eleven contexts success: runner-manifest-test, helm-validate, receiver-test, alert-receiver-test, shellcheck, yaml-lint, record-sql-test, terraform-validate, cockpit-e2e, cockpit-test, dispatcher-test. The statuses list for that same sha returned 33 rows whose latest entry per context is success, finishing 2026-11-11T22:11Z–22:14Z. No pending or failing context remains.

6. Class claim — routine, agreed. The diff is a replica count, one affinity term, a GC count plus a numeric guard, and a CI assertion. No RBAC, no secrets, no NetworkPolicy, no Terraform, no DB/schema, no .forgejo/**, no PVC delete, no image or resource change. Routine fits.

Two small things to pick up when convenient, neither affecting this verdict: the PR title still reads "replicas 8 -> 10" from round 1 while the diff is 9 (the body is explicit, but the title is what the merged history will show), and _ordinal_loop_block proves the intent by matching any while/for line containing RUNNER_COUNT, which is coarse — it is tight enough today only because the new self-check case pins the hard-coded-8 mutation, and that coupling deserves a comment before someone adds a second RUNNER_COUNT-driven loop.

Verdict stands at this exact head: approve.

model: omniroute/deepseek-flash
session: IR-20261111-runners10-seatA-20261112T0226Z

**APPROVE — ld-reviewer — head a290cc94d6efeb71de52860213a150b7c544c0c2** Reviewed the diff, the head files and the commit statuses independently at a290cc94d6efeb71de52860213a150b7c544c0c2 (branch `plane/IR-20261111-runners10`, two commits: d77bd1a756c9 round 1, a290cc94d6ef head). I did not read any existing comment or review on this PR; this verdict comes only from the diff, the statuses and the body. **1. Scope — exactly four files, nothing else.** The files listing and the raw diff agree: `core/manifests/apps/forgejo-runner/dind-gc-cronjob.yaml` (+10/−3), `dind-gc-script.yaml` (+18/−1), `statefulset.yaml` (+36/−2), `scripts/manifests/tests/test_runner_dind_gc_tightening.py` (+59/−6). changed_files = 4, and stripping comment lines leaves only: `replicas: 8` → `9`, the two affinity expressions, `RUNNER_COUNT` `"8"` → `"9"` in the CronJob env and in the script default, the `case "$RUNNER_COUNT"` guard in `main()`, and the test rewrite. Pass. **2. No changes to the prohibited areas.** No hunk touches resources, images, storage, RBAC, NetworkPolicy, `.forgejo/`, or terraform. At head the container resources are unchanged and still sum to the documented pod request/limits: runner 500m / 1003Mi / 10Gi requests, 2c / 2560Mi / 14Gi limits; dind 200m / 1045Mi / 4Gi requests, 8c / 8Gi / 10Gi limits; both images remain digest-pinned; both `volumeClaimTemplates` keep `storageClassName: longhorn-runner` (1Gi runner-data, 30Gi docker-data); the `NotIn` sits inside the same single `nodeSelectorTerm` as the `In` list, so it narrows rather than widens, and the soft `topologySpreadConstraints` (`maxSkew: 1`, `ScheduleAnyway`) is byte-for-byte untouched. Pass. **3. Headroom arithmetic — recomputed, and the ephemeral leg that decides 9 is consistent.** wk-02: 12.0/92.4 = 13.0%, +1 = 26.0/92.4 = 28.1%, +2 = 40.0/92.4 = 43.3%. wk-05: 62.1/87.8 = 70.7%, +1 = 76.1/87.8 = **86.7%**, +2 = 90.1/87.8 = 102.6% (above allocatable, i.e. unschedulable there). wk-06: 56.0/87.8 = 63.8%, +1 = 70.0/87.8 = 79.7%, +2 = 84.0/87.8 = 95.7%. So after +1 no eligible node exceeds 86.7% and wk-05 is indeed on its last slot; +2 fails the ≤90% bar on both wk-05 and wk-06. The 9-not-10 conclusion holds on the numbers. One column is wrong, in the safe direction: the "+1 pod = 2048Mi / **1500m** / 14Gi requests" line disagrees with the manifest, which sums to **700m** cpu requests (500m runner + 200m dind); memory (2048Mi) and ephemeral (14Gi) match exactly. Because the stated cpu delta is larger than the real one, that column overstates consumption and errs toward refusing the 10th pod — it cannot have let a bad placement through, and it does not move the ephemeral leg that sets the count. Worth correcting in the merged record. **4. The pool bounce is disclosed.** The "Rollout — this one DOES bounce the pool" section and the commit message both say plainly that the pod-template change replaces every replica, kills in-flight jobs, and belongs inside the runner gap, and that merging alone changes nothing until ArgoCD syncs. One mechanism refinement: `podManagementPolicy: Parallel` upstream "only affects the behavior for scaling operations. Updates are not affected", so the replacement is reverse-ordinal and readiness-gated per pod rather than one simultaneous swap. That changes the shape of the outage, not its operational meaning — every replica is still replaced and every in-flight job is still killed inside the gap, which is the part the plan depends on and the part that is disclosed. Pass. **5. CI on the exact head: green.** `GET /repos/Homelab/infra-core/commits/a290cc94d6efeb71de52860213a150b7c544c0c2/status` returns `state=success`, `total_count=11`, with all eleven contexts success: runner-manifest-test, helm-validate, receiver-test, alert-receiver-test, shellcheck, yaml-lint, record-sql-test, terraform-validate, cockpit-e2e, cockpit-test, dispatcher-test. The statuses list for that same sha returned 33 rows whose latest entry per context is success, finishing 2026-11-11T22:11Z–22:14Z. No pending or failing context remains. **6. Class claim — routine, agreed.** The diff is a replica count, one affinity term, a GC count plus a numeric guard, and a CI assertion. No RBAC, no secrets, no NetworkPolicy, no Terraform, no DB/schema, no `.forgejo/**`, no PVC delete, no image or resource change. Routine fits. Two small things to pick up when convenient, neither affecting this verdict: the PR title still reads "replicas 8 -> 10" from round 1 while the diff is 9 (the body is explicit, but the title is what the merged history will show), and `_ordinal_loop_block` proves the intent by matching any `while`/`for` line containing `RUNNER_COUNT`, which is coarse — it is tight enough today only because the new self-check case pins the hard-coded-8 mutation, and that coupling deserves a comment before someone adds a second RUNNER_COUNT-driven loop. Verdict stands at this exact head: approve. model: omniroute/deepseek-flash session: IR-20261111-runners10-seatA-20261112T0226Z

ld-reviewer commented 2026-11-12 02:29:41 +00:00

Collaborator

Copy link

APPROVE — ld-reviewer — head a290cc94d6

Seat B, independent read at the pinned head. The head I read equals the pin exactly (a290cc94d6efeb71de52860213a150b7c544c0c2), base live@3563ec7. Two commits on the branch (d77bd1a7 round 1, a290cc94 round 2); PR diff is +123/−12.

1. Diff is exactly the four files, nothing else. core/manifests/apps/forgejo-runner/dind-gc-cronjob.yaml (+10/−3), core/manifests/apps/forgejo-runner/dind-gc-script.yaml (+18/−1), core/manifests/apps/forgejo-runner/statefulset.yaml (+36/−2), scripts/manifests/tests/test_runner_dind_gc_tightening.py (+59/−6). The body's "three files" heading undercounts; the fourth is the CI assertion, and the file list has four.

2. No protected surfaces touched. No requests/limits change, no image change (both container digests identical before and after), no storageClass or volumeClaimTemplates change, no RBAC, no NetworkPolicy, no .forgejo/**, no Terraform, no DB. configmap.yaml is untouched, as the body states. In the StatefulSet the only spec edits are replicas: 8 -> 9 and the affinity block.

3. Affinity shape is the safe one. I read the whole file at the head, not just the diff: a single nodeSelectorTerm carries In [prod-wk-02, prod-wk-05, prod-wk-06] and NotIn [prod-wk-01] as two matchExpressions at the same level, so they AND. topologySpreadConstraints is unchanged (maxSkew: 1, ScheduleAnyway). A NotIn in its own term would indeed have widened the set, and the in-file comments say so.

4. Headroom arithmetic. The ephemeral leg — the one that decides 9 against 10 — recomputes exactly on every eligible node: at 14Gi per pod and the allocatables in the table, +1 pod is wk-02 26.0/92.4Gi = 28.1%, wk-05 76.1/87.8 = 86.7%, wk-06 70.0/87.8 = 79.8%; +2 is wk-05 102.6% (body says 102.7%, rounding — over 100% either way, i.e. unschedulable) and wk-06 95.7%. So every eligible node stays ≤86.7% after one pod and wk-05 is at its last. The memory columns recompute exactly too (48.0% / 61.8% / 73.6%). One derived column does not: "headroom left (mem)" of 12.5 / 20.4 / 5.7Gi is not allocatable−requests, which I compute as ~12.0 / 22.3 / 6.1Gi. The deviation is sub-GiB to ~1.9Gi and runs in both directions, and it crosses no bar — the smallest residual (~6Gi on wk-06) is still several times the Talos OOM trigger depth the body quotes. Worth recomputing if the body is reprinted; no claim that depends on it changes.

5. Bounce disclosure is honest and specific. The body states plainly that the pod-template change (plus podManagementPolicy: Parallel, not gated on readiness) replaces every replica in reverse ordinal order, kills in-flight jobs, and belongs in the runner gap; the commit message repeats it. Merge alone changes nothing until ArgoCD syncs. It is not understated anywhere.

6. dind-gc follows the replica count. At this head: CronJob RUNNER_COUNT: "9" == script default ${RUNNER_COUNT:-9} == spec.replicas: 9. The ordinal loop is while [ "$ordinal" -lt "$RUNNER_COUNT" ] (it was already count-driven — the diff documents that rather than changing it), main() now exits 2 on a non-numeric count before any sweep, and the assertion derives the width from spec.replicas instead of a literal. The old module-level RUNNER_COUNT = "8" is gone, and nothing in scripts/manifests/tests/ (three files there) imports or references it, so no dangling NameError. Both new --self-check cases genuinely fail their mutation: CronJob env 8 against replicas 9, and a loop rewritten to -lt 8, which no longer references RUNNER_COUNT.

7. CI on the exact head. I read the commit-statuses API for a290cc94d6efeb71de52860213a150b7c544c0c2: combined state success, 11 of 11 latest contexts success — CI Review: runner-manifest-test, helm-validate, receiver-test, alert-receiver-test, record-sql-test, dispatcher-test, cockpit-test, cockpit-e2e; Lint & Validate: shellcheck, yaml-lint, terraform-validate. The pending rows visible per context are older than that context's success row. I did not run the suites inside this pod (no PyYAML here); the runner-manifest-test job at this head is the executed evidence, and the static reading above is separate.

8. Class claim matches the diff. Routine is right: one integer, one required-affinity expression pair, a GC count plus default plus input guard, and its CI assertion. The only material risk is the required-affinity narrowing, and it is disclosed with failure mode and levers (ordinal 8 can sit Pending on Longhorn's provisioned-limit check, pool stays at 8 slots, no memory/OOM/data event). The whole-pool restart is operational and is placed in the gap. Nothing here is the dangerous class (no authz, secrets, data surface, or Terraform).

Approving at this head. No seats and no merge from me.

model: omniroute/deepseek-flash
session: IR-20261111-runners10-seatB-20261112T0226Z

**APPROVE — ld-reviewer — head a290cc94d6efeb71de52860213a150b7c544c0c2** Seat B, independent read at the pinned head. The head I read equals the pin exactly (`a290cc94d6efeb71de52860213a150b7c544c0c2`), base `live@3563ec7`. Two commits on the branch (d77bd1a7 round 1, a290cc94 round 2); PR diff is +123/−12. 1. **Diff is exactly the four files, nothing else.** `core/manifests/apps/forgejo-runner/dind-gc-cronjob.yaml` (+10/−3), `core/manifests/apps/forgejo-runner/dind-gc-script.yaml` (+18/−1), `core/manifests/apps/forgejo-runner/statefulset.yaml` (+36/−2), `scripts/manifests/tests/test_runner_dind_gc_tightening.py` (+59/−6). The body's "three files" heading undercounts; the fourth is the CI assertion, and the file list has four. 2. **No protected surfaces touched.** No requests/limits change, no image change (both container digests identical before and after), no storageClass or volumeClaimTemplates change, no RBAC, no NetworkPolicy, no `.forgejo/**`, no Terraform, no DB. `configmap.yaml` is untouched, as the body states. In the StatefulSet the only spec edits are `replicas: 8 -> 9` and the affinity block. 3. **Affinity shape is the safe one.** I read the whole file at the head, not just the diff: a single `nodeSelectorTerm` carries `In [prod-wk-02, prod-wk-05, prod-wk-06]` and `NotIn [prod-wk-01]` as two `matchExpressions` at the same level, so they AND. `topologySpreadConstraints` is unchanged (`maxSkew: 1`, `ScheduleAnyway`). A `NotIn` in its own term would indeed have widened the set, and the in-file comments say so. 4. **Headroom arithmetic.** The ephemeral leg — the one that decides 9 against 10 — recomputes exactly on every eligible node: at 14Gi per pod and the allocatables in the table, +1 pod is wk-02 26.0/92.4Gi = 28.1%, wk-05 76.1/87.8 = 86.7%, wk-06 70.0/87.8 = 79.8%; +2 is wk-05 102.6% (body says 102.7%, rounding — over 100% either way, i.e. unschedulable) and wk-06 95.7%. So every eligible node stays ≤86.7% after one pod and wk-05 is at its last. The memory columns recompute exactly too (48.0% / 61.8% / 73.6%). One derived column does not: "headroom left (mem)" of 12.5 / 20.4 / 5.7Gi is not allocatable−requests, which I compute as ~12.0 / 22.3 / 6.1Gi. The deviation is sub-GiB to ~1.9Gi and runs in both directions, and it crosses no bar — the smallest residual (~6Gi on wk-06) is still several times the Talos OOM trigger depth the body quotes. Worth recomputing if the body is reprinted; no claim that depends on it changes. 5. **Bounce disclosure is honest and specific.** The body states plainly that the pod-template change (plus `podManagementPolicy: Parallel`, not gated on readiness) replaces every replica in reverse ordinal order, kills in-flight jobs, and belongs in the runner gap; the commit message repeats it. Merge alone changes nothing until ArgoCD syncs. It is not understated anywhere. 6. **dind-gc follows the replica count.** At this head: CronJob `RUNNER_COUNT: "9"` == script default `${RUNNER_COUNT:-9}` == `spec.replicas: 9`. The ordinal loop is `while [ "$ordinal" -lt "$RUNNER_COUNT" ]` (it was already count-driven — the diff documents that rather than changing it), `main()` now exits 2 on a non-numeric count before any sweep, and the assertion derives the width from `spec.replicas` instead of a literal. The old module-level `RUNNER_COUNT = "8"` is gone, and nothing in `scripts/manifests/tests/` (three files there) imports or references it, so no dangling NameError. Both new `--self-check` cases genuinely fail their mutation: CronJob env 8 against replicas 9, and a loop rewritten to `-lt 8`, which no longer references RUNNER_COUNT. 7. **CI on the exact head.** I read the commit-statuses API for `a290cc94d6efeb71de52860213a150b7c544c0c2`: combined state `success`, 11 of 11 latest contexts success — CI Review: runner-manifest-test, helm-validate, receiver-test, alert-receiver-test, record-sql-test, dispatcher-test, cockpit-test, cockpit-e2e; Lint & Validate: shellcheck, yaml-lint, terraform-validate. The `pending` rows visible per context are older than that context's success row. I did not run the suites inside this pod (no PyYAML here); the runner-manifest-test job at this head is the executed evidence, and the static reading above is separate. 8. **Class claim matches the diff.** Routine is right: one integer, one required-affinity expression pair, a GC count plus default plus input guard, and its CI assertion. The only material risk is the required-affinity narrowing, and it is disclosed with failure mode and levers (ordinal 8 can sit Pending on Longhorn's provisioned-limit check, pool stays at 8 slots, no memory/OOM/data event). The whole-pool restart is operational and is placed in the gap. Nothing here is the dangerous class (no authz, secrets, data surface, or Terraform). Approving at this head. No seats and no merge from me. model: omniroute/deepseek-flash session: IR-20261111-runners10-seatB-20261112T0226Z

pi-landing merged commit d5f84aacca into live 2026-11-12 03:45:41 +00:00

pi-landing referenced this pull request from a commit 2026-11-12 03:45:42 +00:00

Merge pull request 'IR-20261111: chore(forgejo-runner): replicas 8 -> 10 (restore CI slots after capacity 1)' (#949) from plane/IR-20261111-runners10 into live

pi-landing referenced this pull request 2026-11-12 04:25:08 +00:00

IR-20261112: chore(forgejo-runner): replicas 9 -> 8 (ordinal-8 volume fits no node under Longhorn 110%) #954

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

operator-review

Requires human operator review before merge — dangerous class (RBAC, secrets, network policy, Terraform) or explicitly gated. Never auto-merge.

No labels

operator-review

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Homelab/infra-core!949

No description provided.