IR-20261112: chore(forgejo-runner): replicas 9 -> 8 (ordinal-8 volume fits no node under Longhorn 110%) #954
No reviewers
Labels
No labels
operator-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Homelab/infra-core!954
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "plane/IR-20261112-runners8"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Returns the forgejo-runner StatefulSet to 8 replicas (was 9). The dind-gc fix
is kept, the affinity is untouched, and nothing lands from this PR — the rollout
is its own runner gap, below.
Why 8, not 9: the Longhorn 110% math
Ordinal 8's 30 Gi strict-local (
longhorn-runner) docker volume fits on NOrunner node under Longhorn's over-provisioning rule
scheduled + size <= (max - reserved) x 110%:So the 9th pod could not schedule at all: ordinal 8 sat Pending with volume
insufficient storagewhile ordinals 0–7 ran the old revision at capacity 2.#948 (capacity 2 → 1) and #949 (replicas 8 → 9, dind-gc follows) are already
landed on
live; this PR removes the 9th slot that no node can host.8 slots at capacity 1 is accepted until the pve RAM and disk arrive. Back at 8
replicas the capacity-1 revision can roll ordinals 0–7 cleanly, one slot each.
Follow-up recorded (not this PR): option (C) — shrink the runner docker volume to
~20 Gi via a PVC migration, which buys the 9th slot back on today's nodes.
The dind-gc fix is kept, and it follows the count
The GC sweep width is not a constant:
RUNNER_COUNTtracksstatefulset.yaml spec.replicas, so it moves 9 → 8 here in both places it exists.dind-gc-cronjob.yaml— envRUNNER_COUNT: "9"→"8"(comment updated:8 = the replica count on this branch (was 9)).dind-gc-script.yaml— defaultRUNNER_COUNT="${RUNNER_COUNT:-9}"→:-8.while [ "$ordinal" -lt "$RUNNER_COUNT" ]— never ahard-coded
0..N-1range, which is the HOMELAB-2481 leak class.CI pins all three together:
scripts/manifests/tests/test_runner_dind_gc_tightening.pyA5 reads
spec.replicasfrom the StatefulSet and asserts the CronJob env and thescript default both equal it. Run locally at this head:
That test file carries the PR's only non-manifest change, and it is a mechanical
consequence of the count moving: its
--self-checkbad case "CronJob RUNNER_COUNTleft behind the StatefulSet replicas" carried the literal
"8", which stoppedbeing a mismatch the moment replicas became 8 — the self-check then printed
NOT REJECTEDand failed CI while the assertion itself was fine. The stale value is nowderived from
spec.replicas(str(int(_runner_count()) - 1)), which is the sameconstant-bumped-alongside rot that module's docstring already warns about. No
assertion was weakened; the run above shows all 31 bad cases still rejected.
Affinity is unchanged
nodeAffinity(wk-02 / wk-05 / wk-06, wk-01 removed in #949) and thetopologySpreadConstraintsare byte-for-byte identical in this diff. Onlyspec.replicasand comments move. The 2026-11-11 headroom paragraph is kept ashistory with its number corrected (
the CEILING is 9 and not 10— unchanged), and anew dated 2026-11-12 block records the 110% math above as the reason for 8.
At rollout: ordinal 8's pod goes, its PVCs stay as orphans
Scaling a StatefulSet down never deletes PVCs, so
docker-data-forgejo-runner-8andrunner-data-forgejo-runner-8are left behindas orphans — deliberately, not "cleaned up" by this change. They are for a
named cleanup later (an operator-gated PVC deletion, not part of this PR).
Rollout is NOT part of this PR
Ordering matters because the pool is the CI that would otherwise be rolling itself.
The rollout happens in a second runner gap: an idle check that includes Sendrast,
then START/END notices to a9 and 3c. Nothing is merged or rolled by the
builder that opened this PR.
Scope
Four files, one commit, no force-push:
core/manifests/apps/forgejo-runner/statefulset.yaml—spec.replicas: 9→8;comments corrected / the 2026-11-12 dated block added. Affinity untouched.
core/manifests/apps/forgejo-runner/dind-gc-cronjob.yaml—RUNNER_COUNTenv"9"→"8", two comments.core/manifests/apps/forgejo-runner/dind-gc-script.yaml—RUNNER_COUNTdefault9→8, two comments.scripts/manifests/tests/test_runner_dind_gc_tightening.py— the self-check badcase derives its stale value instead of hard-coding
8(above).Pushed as
pi-landing(repo activity feedact_user= pi-landing on the commit);opened through the mint.sh staged-header route with
GET /userassertedpi-landingin-process before the POST.
APPROVE — ld-reviewer — head
3e05d8a67cDiff reviewed against base live
8f490930554f40d8adf1d0f2a89af614eb9acfd8; PR head confirmed3e05d8a6…via the PR object. Diff is exactly 4 files (3 manifests + 1 test), one commit.replicas 9 -> 8 —
statefulset.yaml:- replicas: 9/+ replicas: 8(raw head file line 125 isspec.replicas: 8). ✔dind-gc fix kept and following the count —
dind-gc-cronjob.yaml: envRUNNER_COUNTvalue: "9"->value: "8".dind-gc-script.yaml: defaultRUNNER_COUNT="${RUNNER_COUNT:-9}"->:-8. Ordinal loop is still RUNNER_COUNT-driven and untouched (while [ "$ordinal" -lt "$RUNNER_COUNT" ]; do, head line 285); grepped the script for hard-coded ranges — no0..N, noseq 0, no literalfor i in 0 1 2 …; the only0..7mention is the comment explaining why a fixed range is forbidden (HOMELAB-2481 class). Comment edits are prose only (9 -> 8 runner arithmetic, and the "LAST ordinal / N-1 of N" rewording). ✔Affinity/spread byte-identical — the complete
statefulset.yamlbase->head diff contains only the two comment blocks and the singlereplicas:line;nodeAffinityandtopologySpreadConstraintsdo not appear in the diff at all (the block tail hashes identical in base and head). ✔Only other change is the test self-check — the whole-file base->head diff of
scripts/manifests/tests/test_runner_dind_gc_tightening.pyis one hunk:mutate_env(..., "RUNNER_COUNT", "8")becomesstale_count = str(int(_runner_count()) - 1)+mutate_env(..., stale_count), plus explanatory comment and an f-string case name._runner_count()really readsstatefulset.yaml spec.replicas(validates positive int, returnsstr(replicas)), so at 8 replicas the bad case now mutates to"7"— still a genuine mismatch, so the case is not weakened back into a tautology. No assertion removed or loosened: A5's three checks are intact and unchanged from base —env.get("RUNNER_COUNT") != replicas(line 283),defaults.get("RUNNER_COUNT") != replicas(line 290), and there.search(r"RUNNER_COUNT", _ordinal_loop_block(...))check that rejects a hard-coded ordinal range (line 295). No case was added or dropped, so the self-check still enumerates the same set. (Tests not executed here by rule — this is a by-inspection check; the hunk is mechanical.) ✔PR body math — body carries the Longhorn over-provisioning rule
scheduled + size <= (max - reserved) x 110%with the per-node table (wk-02 458+30=488 vs 438.9; wk-05 351+30=381 vs 350.9; wk-06 325+30=355 vs 333.3, all "no"), explains ordinal 8 sat Pending oninsufficient storage, and states verbatim "8 slots at capacity 1 is accepted until the pve RAM and disk arrive", with the same math mirrored in the new dated 2026-11-12 comment block in the StatefulSet and in the commit message. ✔Trailer — head commit
3e05d8a67c5888ea1a6e40210143aea00f819661message's last non-empty trailer line isLab-Run: IR-20261112-runners8. ✔No blocking findings: scope is minimal, the orphaned
docker-data-forgejo-runner-8/runner-data-forgejo-runner-8PVCs are explicitly declared as deliberate leftovers for a named cleanup, and the rollout is correctly deferred out of this PR.model: omniroute/deepseek-flash
session: IR-20261112-runners8-seatB-20261112T0435Z
APPROVE — ld-reviewer — head
3e05d8a67cReviewed as a diff against base
live(8f49093055), 4 files, +44/−16. I also diffed the base and head blobs of all three manifests directly to confirm the published diff is complete (no hidden hunks) and byte-compared the scheduler block.replicas 9 → 8 —
core/manifests/apps/forgejo-runner/statefulset.yaml: the only non-comment hunk is- replicas: 9/+ replicas: 8(head line 125), with the surroundingserviceName/podManagementPolicy: Parallel/updateStrategycontext unchanged.dind-gc fix kept and count-derived —
dind-gc-cronjob.yaml: envRUNNER_COUNT- value: "9"/+ value: "8"(head line 145).dind-gc-script.yaml: default- RUNNER_COUNT="${RUNNER_COUNT:-9}"/+ RUNNER_COUNT="${RUNNER_COUNT:-8}"(head line 93). The loop is untouched and stillwhile [ "$ordinal" -lt "$RUNNER_COUNT" ]; do(head line 285) — no hard-coded0..N-1anywhere in the script; thecase "$RUNNER_COUNT"non-negative-integer guard is also unchanged. Comment edits only restate the same invariant.Affinity / spread untouched —
sed-range comparison of the scheduler block (base 122–175 vs head 143–196,nodeAffinity→requiredDuringSchedulingIgnoredDuringExecution→nodeSelectorTerms→matchExpressions… throughtopologySpreadConstraints/whenUnsatisfiable: ScheduleAnyway/labelSelector) is byte-identical:sha256 eb1d03e82b02ceb57fcc388bbe3a165e68fc01d3eb6aabeb569d0b480ec0ea40on both sides. The only statefulset hunks are the header comments (2026-11-11 paragraph reworded to keep its history, "the CEILING is 9 and not 10", a new dated 2026-11-12 110% block, and the rollout-scope note) and the one replicas line.Test file is the only other change, and it is not weakened —
scripts/manifests/tests/test_runner_dind_gc_tightening.pyhas exactly one hunk:mutate_env(..., "RUNNER_COUNT", "8")becomesstale_count = str(int(_runner_count()) - 1)with the label interpolating it. Checked at head:_runner_count()(line 99) readsspec.replicasfromstatefulset.yamland fails loudly on a non-positive-integer, so at head the derived stale value is"7"against a live"8"— the bad case is still a genuine mismatch and still routed toassert_tier1_safety_contract(line 672). No assertion removed, none loosened, no othercases.appendtouched; the siblingordinal loop hard-codes 0..7case that this value must not collide with is intact and now correctly targets"8".PR body math — the body carries the Longhorn over-provisioning table (
scheduled + size <= (max − reserved) × 110%; wk-02 488 > 438.9, wk-05 381 > 350.9, wk-06 355 > 333.3) and states verbatim that 8 slots at capacity 1 is accepted until the pve RAM and disk arrive, with option (C) ~20Gi recorded as the follow-up. The same math appears in the statefulset's dated comment, consistent with the body.Trailer — commit
3e05d8a67c5888ea1a6e40210143aea00f819661message ends, after the last body paragraph and a blank line, withLab-Run: IR-20261112-runners8as the final trailer. Head SHA in the PR object matches the SHA under review exactly.Non-blocking observation only (not a request for change): at exactly 8 replicas a literal
0..7loop would coincidentally be correct, which is why the rewritten script comment says "at any count above 8" — the A5 derivation test is what keeps that honest, and it does. The ordinal-8 PVC orphaning is disclosed and correctly left out of scope.model: omniroute/deepseek-flash
session: IR-20261112-runners8-seatA-20261112T0435Z