HOMELAB-2479: feat(forgejo-runner): CI reservation trim 51.8->33.1 GiB, slots kept #876

Merged

pi-landing merged 1 commit from plane/HOMELAB-2479-ci-trim into live 2026-11-06 13:17:10 +00:00

Conversation 1 Commits 1 Files changed 3 +21 -3

pi-landing commented 2026-11-06 12:47:45 +00:00

Collaborator

Copy link

HOMELAB-2479 — CI reservation trim: 51.79 GiB → 33.14 GiB, all 23 slots kept, 2 h timeout untouched

Plan: working-files/infra-stability/orch16/ci-reservation-plan.md (APPROVED), §2 table.

GO (overseer, through El Director)

implement the CI trim to 33.1 GiB total (the plan's 33.14 line), ALL 23 slots kept, the 2 h runner timeout untouched. ... Do NOT touch sbx-sendrast anything. Do NOT touch limits except where the plan says.

Measured before (live objects, not the repo)

kubectl get sts -n forgejo-runner -o json, .spec.template.spec.containers[*].resources.requests.memory, clock date -u = 2026-11-06T12:44:42Z:

object	container	requests.memory (before)
statefulset/forgejo-runner (×8)	runner	2118Mi
	dind	1045Mi
statefulset/android-ci (×2)	runner	6Gi
statefulset/android-emu (×2)	runner / dind	2700Mi / 600Mi
statefulset/android-release (×1)	runner	6Gi
statefulset/ld-ci (×1)	runner	2Gi
statefulset/ld-push (×1)	runner / dind	172Mi / 412Mi

Pod request totals before = 51.79 GiB namespace-wide (pod sums: 3163, 6144, 3300, 6144, 2048, 584 Mi × replicas).

The change — requests only, per pool

pool (ns forgejo-runner)	replicas × cap = slots	pod request before	pod request after	total before	total after
forgejo-runner ("k8s")	8 × 2 = 16	3163Mi (2118 runner + 1045 dind)	2048Mi (1003 runner + 1045 dind)	24.71 GiB	16.00 GiB
android-ci	2 × 1 = 2	6Gi	1Gi	12.00 GiB	2.00 GiB
android-emu	2 × 1 = 2	3300Mi	3300Mi (unchanged)	6.45 GiB	6.45 GiB
android-release	1 × 1 = 1	6Gi	6Gi (unchanged)	6.00 GiB	6.00 GiB
ld-ci	1 × 1 = 1	2Gi	2176Mi (the one raise)	2.00 GiB	2.13 GiB
ld-push	1 × 1 = 1	584Mi	584Mi (unchanged)	0.57 GiB	0.57 GiB
all lanes	23 slots			51.79 GiB	33.14 GiB (−18.65)

• Limits untouched everywhere, including forgejo-runner runner 2560Mi / dind 8Gi, android-ci 12Gi, ld-ci 6Gi.
• Slot count unchanged: 23 → 23. No replica change, no lane deleted, no PVC touched.
• 2 h timeout untouched: cm/forgejo-runner-config runner.timeout: 2h is not in this diff; android-emu's 2h + shutdown_timeout: 40m and 2700 s grace are untouched.
• Nothing in sbx-sendrast, nothing in any sandbox pool, no node config, no prod change (no image is published by this diff, so no pin bump).
• Predicted per-node reservation under today's placement: wk-05 36.86 → 22.44 GiB (honest envelope 33.28) · wk-06 14.93 → 10.70 GiB (envelope 15.16).

Risk, named (plan §2 — do not read this PR as "fixed")

• Today's 51.79 GiB is itself 3.35 GiB past what the measured node-level 7 d peaks back: the honest ceiling on wk-05 + wk-06 is 48.44 GiB (wk-05 alone is 3.58 GiB over its own envelope; the 18.53 GiB gap there is page cache / kernel / tmpfs that no requests value can remove).
• 33.14 GiB is still not the 29.07 design target. It is the smallest reservation that equals this namespace's measured worst simultaneous week (33.37 GiB, 2026-11-03T21:32:02Z). The last 4.07 GiB needs a per-project CI peak, not another cut.
• This is a step; the node-floor work continues separately. Reservation freed is 18.65 GiB of reservation, not of usage.
• Executed literally as the plan's table (pod-level 2048Mi for the k8s pool). Honest deviation note: plan §5 additionally spelled the k8s edit as "runner 2118Mi → 2048Mi, dind 1045Mi unchanged", which is 3093Mi/pod and does not reach 33.14 GiB (it would be 41.28 GiB). The table's and the GO's 33.14 GiB both require 2048Mi per pod, so the whole delta lands on the runner container (2118 → 1003Mi). Consequence to be aware of: that container's own 7 d peak is 1786.55Mi and the worst pod's cgroup peak is 2258.79Mi, so both the request and its scheduling guarantee now sit below measured peaks. Limits are unchanged, so this changes OOM/eviction ranking, not the ceiling. If the overseer prefers the 3093Mi shape, it is a one-line follow-up and the total becomes 41.28 GiB.

Apply gate (NOT applied by this PR's author)

Apply only with no Sendrast job running, on homelab-ba's word through the overseer; read back from the live pod spec afterwards (kubectl get pods -n forgejo-runner -o json → .spec.containers[*].resources.requests.memory), and expect pods to roll (android-emu's roll can take up to 45 min at 2700 s grace; android-release 90 s). Do not merge/apply while a job is in flight.

Validation run

• YAML parsed; recomputed totals from the rendered StatefulSets: android-ci 2048 + android-emu 6600 + android-release 6144 + ld-ci 2176 + ld-push 584 + forgejo-runner 16384 = 33936Mi = 33.14 GiB.
• kubectl apply --dry-run=server -f <file> -n forgejo-runner: statefulset.apps/forgejo-runner configured (server dry run), statefulset.apps/android-ci configured (server dry run), statefulset.apps/ld-ci configured (server dry run), statefulset.apps/ld-push configured (server dry run).
• No live object was written; no kubectl apply without --dry-run.

## HOMELAB-2479 — CI reservation trim: 51.79 GiB → **33.14 GiB**, all 23 slots kept, 2 h timeout untouched Plan: `working-files/infra-stability/orch16/ci-reservation-plan.md` (APPROVED), §2 table. ### GO (overseer, through El Director) > implement the CI trim to **33.1 GiB total** (the plan's 33.14 line), ALL 23 slots kept, the 2 h runner timeout untouched. ... Do NOT touch sbx-sendrast anything. Do NOT touch limits except where the plan says. ### Measured before (live objects, not the repo) `kubectl get sts -n forgejo-runner -o json`, `.spec.template.spec.containers[*].resources.requests.memory`, clock `date -u` = **2026-11-06T12:44:42Z**: | object | container | requests.memory (before) | |---|---|---| | `statefulset/forgejo-runner` (×8) | runner | **2118Mi** | | | dind | **1045Mi** | | `statefulset/android-ci` (×2) | runner | **6Gi** | | `statefulset/android-emu` (×2) | runner / dind | **2700Mi / 600Mi** | | `statefulset/android-release` (×1) | runner | **6Gi** | | `statefulset/ld-ci` (×1) | runner | **2Gi** | | `statefulset/ld-push` (×1) | runner / dind | **172Mi / 412Mi** | Pod request totals before = 51.79 GiB namespace-wide (pod sums: 3163, 6144, 3300, 6144, 2048, 584 Mi × replicas). ### The change — requests only, per pool | pool (ns `forgejo-runner`) | replicas × cap = slots | pod request before | **pod request after** | total before | **total after** | |---|---|---|---|---|---| | `forgejo-runner` ("k8s") | 8 × 2 = 16 | 3163Mi (2118 runner + 1045 dind) | **2048Mi (1003 runner + 1045 dind)** | 24.71 GiB | **16.00 GiB** | | `android-ci` | 2 × 1 = 2 | 6Gi | **1Gi** | 12.00 GiB | **2.00 GiB** | | `android-emu` | 2 × 1 = 2 | 3300Mi | **3300Mi (unchanged)** | 6.45 GiB | **6.45 GiB** | | `android-release` | 1 × 1 = 1 | 6Gi | **6Gi (unchanged)** | 6.00 GiB | **6.00 GiB** | | `ld-ci` | 1 × 1 = 1 | 2Gi | **2176Mi** (the one raise) | 2.00 GiB | **2.13 GiB** | | `ld-push` | 1 × 1 = 1 | 584Mi | **584Mi (unchanged)** | 0.57 GiB | **0.57 GiB** | | **all lanes** | **23 slots** | | | **51.79 GiB** | **33.14 GiB** (−18.65) | - **Limits untouched** everywhere, including `forgejo-runner` runner 2560Mi / dind 8Gi, `android-ci` 12Gi, `ld-ci` 6Gi. - **Slot count unchanged: 23 → 23.** No replica change, no lane deleted, no PVC touched. - **2 h timeout untouched:** `cm/forgejo-runner-config` `runner.timeout: 2h` is not in this diff; `android-emu`'s 2h + `shutdown_timeout: 40m` and 2700 s grace are untouched. - **Nothing in `sbx-sendrast`, nothing in any sandbox pool**, no node config, no `prod` change (no image is published by this diff, so no pin bump). - Predicted per-node reservation under today's placement: wk-05 36.86 → **22.44 GiB** (honest envelope 33.28) · wk-06 14.93 → **10.70 GiB** (envelope 15.16). ### Risk, named (plan §2 — do not read this PR as "fixed") - Today's **51.79 GiB is itself 3.35 GiB past** what the measured node-level 7 d peaks back: the honest ceiling on wk-05 + wk-06 is **48.44 GiB** (wk-05 alone is 3.58 GiB over its own envelope; the 18.53 GiB gap there is page cache / kernel / tmpfs that no `requests` value can remove). - **33.14 GiB is still not the 29.07 design target.** It is the smallest reservation that equals this namespace's measured worst simultaneous week (33.37 GiB, 2026-11-03T21:32:02Z). The last 4.07 GiB needs a per-project CI peak, **not** another cut. - This is a **step**; the node-floor work continues separately. Reservation freed is 18.65 GiB of *reservation*, not of *usage*. - Executed literally as the plan's table (pod-level 2048Mi for the k8s pool). Honest deviation note: plan §5 additionally spelled the k8s edit as "runner 2118Mi → 2048Mi, dind 1045Mi unchanged", which is 3093Mi/pod and does **not** reach 33.14 GiB (it would be 41.28 GiB). The table's and the GO's 33.14 GiB both require 2048Mi per **pod**, so the whole delta lands on the runner container (2118 → 1003Mi). Consequence to be aware of: that container's own 7 d peak is 1786.55Mi and the worst pod's cgroup peak is 2258.79Mi, so both the request and its scheduling guarantee now sit below measured peaks. Limits are unchanged, so this changes OOM/eviction *ranking*, not the ceiling. If the overseer prefers the 3093Mi shape, it is a one-line follow-up and the total becomes 41.28 GiB. ### Apply gate (NOT applied by this PR's author) **Apply only with no Sendrast job running, on homelab-ba's word through the overseer; read back from the live pod spec afterwards** (`kubectl get pods -n forgejo-runner -o json` → `.spec.containers[*].resources.requests.memory`), and expect pods to roll (android-emu's roll can take up to 45 min at 2700 s grace; android-release 90 s). Do not merge/apply while a job is in flight. ### Validation run - YAML parsed; recomputed totals from the rendered StatefulSets: `android-ci 2048 + android-emu 6600 + android-release 6144 + ld-ci 2176 + ld-push 584 + forgejo-runner 16384 = 33936Mi = 33.14 GiB`. - `kubectl apply --dry-run=server -f <file> -n forgejo-runner`: `statefulset.apps/forgejo-runner configured (server dry run)`, `statefulset.apps/android-ci configured (server dry run)`, `statefulset.apps/ld-ci configured (server dry run)`, `statefulset.apps/ld-push configured (server dry run)`. - No live object was written; no `kubectl apply` without `--dry-run`.

pi-landing added 1 commit 2026-11-06 12:47:45 +00:00

HOMELAB-2479: feat(forgejo-runner): CI reservation trim 51.8->33.1 GiB, slots kept dc4e2a54cd

pi-landing commented 2026-11-06 13:10:14 +00:00

Author

Collaborator

Copy link

Review — APPROVE

Head reviewed: dc4e2a54cdfa1879a6b436a565550661014ac7b0 (exact head verified from the API before review; base live 960410f5f74cee5eeeff893bb8550cad359bbc29). Model: omniroute/deepseek-flash. Session: 01a1b0f8-4445-76f3-8e2c-eccb8c7b163f.

Verdict

APPROVE. No blockers. 3 concerns, 2 suggestions — every one is a follow-up, disclosed in the PR body or inherited from the approved plan, not introduced by this diff.

1. Diff vs the GO — exact

3 files, 3 hunks, three changed values, all requests.memory:

file	container	before	after	GO
statefulset.yaml	runner	2118Mi	1003Mi	pod 3163 → 2048 Mi ✔
android-ci-statefulset.yaml	runner	6Gi	1Gi	6 → 1 Gi ✔
ld-pool-statefulsets.yaml (ld-ci)	runner	2Gi	2176Mi	2048 → 2176 Mi ✔

k8s is a pod-level move: 1003 + 1045 (dind) = 2048 Mi ✔. Limits untouched (runner 2560Mi/dind 8Gi, android-ci 12Gi, ld-ci 6Gi), verified by reading the files at the head sha, not just the hunk context. Unchanged lanes verified from the head tree, not merely absent from the diff: android-emu 2×(2700+600), android-release 1×6144, ld-push 1×(172+412). Nothing in sbx-sendrast or any sandbox namespace — no file outside core/manifests/apps/forgejo-runner/ is touched, and the recursive tree at the head sha shows no duplicate/overriding definition of these StatefulSets anywhere else in the repo.

2. Arithmetic — recomputed from the head files

forgejo-runner   8 × 2048 = 16384
android-ci       2 × 1024 =  2048
android-emu      2 × 3300 =  6600
android-release  1 × 6144 =  6144
ld-ci            1 × 2176 =  2176
ld-push          1 ×  584 =   584
                    total = 33936 Mi = 33.1406 GiB

33 936 Mi = 33.14 GiB — matches the brief's 33 936 Mi, the GO's 33.1 and the PR's figure. No drift in any changed line or in the total. Per-node: wk-05 = 22 984 Mi = 22.44 GiB ✔, wk-06 = 10 952 Mi = 10.70 GiB ✔ (both as stated). Slots 23 → 23: 16+2+2+1+1+1; replicas: untouched in all six objects, no capacity/configmap in the diff.

3. Scope safety — clean

No labels, selector, nodeAffinity/nodeSelector/toleration, replicas, capacity, volumeClaimTemplates, grace period, image or timeout change. The only non-comment lines that changed are the three memory: request values.

4. Apply gate — honest, and cross-checked

The body carries the required gate: apply only with no Sendrast job running, on homelab-ba's word through the overseer, read the live pod spec back after (.spec.containers[*].resources.requests.memory), do not apply with a job in flight. Independently confirmed (read-only kubectl get sts -n forgejo-runner, 13:09Z) that live still reads 3163 / 6144 / 3300 / 6144 / 2048 / 584 Mi — the change is genuinely not applied and the PR's "before" column is the live truth. CI on this exact head: 11/11 success, incl. runner-manifest-test and yaml-lint.

Concern 1 — the k8s runner request is now below its own measured peak

runner.requests.memory = 1003Mi against that container's measured 7 d peak 1786.55 Mi and the worst pod's cgroup peak 2258.79 Mi. Limits unchanged, so this moves eviction/OOM ranking, not the ceiling — the 8 pods become earlier eviction candidates on a node whose 7 d node-level peak was 52.31 GiB of 58.35 GiB allocatable. It is the direct arithmetic consequence of the GO's 33.1 GiB with 23 slots kept, and it is disclosed; not a defect of the diff. Needs a post-apply watch on kube_pod_container_status_restarts_total{namespace="forgejo-runner"} and node memory-pressure evictions on wk-05/wk-06 for 24 h.

Concern 2 — the approved plan contradicts itself (PR resolved it correctly)

Plan §2 says pod 3163 → 2048 Mi; plan §5 says runner 2118 → 2048 with dind unchanged = 3093 Mi/pod = 41.30 GiB total (the PR's narrative says 41.28; exact is 41.30, 42 296 Mi). The PR implemented §2/the GO and disclosed the deviation explicitly — the honest resolution. Please correct the plan doc so the audit trail does not carry two shapes.

Concern 3 — the "before" total does not reconcile with its own rows (inherited from plan §1)

25 304 + 12 288 + 6 600 + 6 144 + 2 048 + 584 = 52 968 Mi = 51.73 GiB, not 53 032 Mi / 51.79 GiB (64 Mi; the plan's note about 2 extra Succeeded dind-gc pods is the likely origin). The delta is therefore −18.59 GiB, not −18.65. The after figure is exact, so nothing in the change depends on it — narrative only.

Suggestion 1 — do not let android-ci stay at 1 Gi

Correct per the GO and nothing dispatches to it today, but leave an explicit step on HOMELAB-2479: raise it back to ≈12 Gi before any job is dispatched, or the first re-enabled android job runs a 12 GiB limit against a 1 GiB reservation on wk-05.

Suggestion 2 — only three StatefulSets roll

The body warns of android-emu's 45 min roll (2700 s grace) and android-release's 90 s; neither template is in this diff and neither is drifted from live, so neither rolls. Only forgejo-runner (8), android-ci (2), ld-ci (1) do — the "no job in flight" window is far shorter than the body implies. Worth correcting so the operator does not wait 45 minutes for a roll that will not happen.

Not verified by me

The 7 d peak figures are inherited from the PR body/plan (no Prometheus query run in this review); android-ci/ld-ci idleness is inherited from the plan's live-branch check; whether prod carries an overriding patch was not read (out of scope for this diff, and the apply read-back is the right control); CI job logs were not read, only the head's status.

## Review — APPROVE **Head reviewed: `dc4e2a54cdfa1879a6b436a565550661014ac7b0`** (exact head verified from the API before review; base `live` `960410f5f74cee5eeeff893bb8550cad359bbc29`). Model: `omniroute/deepseek-flash`. Session: `01a1b0f8-4445-76f3-8e2c-eccb8c7b163f`. ### Verdict **APPROVE. No blockers.** 3 concerns, 2 suggestions — every one is a follow-up, disclosed in the PR body or inherited from the approved plan, not introduced by this diff. ### 1. Diff vs the GO — exact 3 files, 3 hunks, **three changed values**, all `requests.memory`: | file | container | before | after | GO | |---|---|---|---|---| | `statefulset.yaml` | `runner` | `2118Mi` | **`1003Mi`** | pod 3163 → 2048 Mi ✔ | | `android-ci-statefulset.yaml` | `runner` | `6Gi` | **`1Gi`** | 6 → 1 Gi ✔ | | `ld-pool-statefulsets.yaml` (`ld-ci`) | `runner` | `2Gi` | **`2176Mi`** | 2048 → 2176 Mi ✔ | k8s is a **pod-level** move: 1003 + 1045 (dind) = 2048 Mi ✔. **Limits untouched** (runner `2560Mi`/dind `8Gi`, android-ci `12Gi`, ld-ci `6Gi`), verified by reading the files at the head sha, not just the hunk context. Unchanged lanes verified from the head tree, not merely absent from the diff: android-emu 2×(2700+600), android-release 1×6144, ld-push 1×(172+412). **Nothing in `sbx-sendrast` or any sandbox namespace** — no file outside `core/manifests/apps/forgejo-runner/` is touched, and the recursive tree at the head sha shows no duplicate/overriding definition of these StatefulSets anywhere else in the repo. ### 2. Arithmetic — recomputed from the head files ``` forgejo-runner 8 × 2048 = 16384 android-ci 2 × 1024 = 2048 android-emu 2 × 3300 = 6600 android-release 1 × 6144 = 6144 ld-ci 1 × 2176 = 2176 ld-push 1 × 584 = 584 total = 33936 Mi = 33.1406 GiB ``` **33 936 Mi = 33.14 GiB — matches the brief's 33 936 Mi, the GO's 33.1 and the PR's figure. No drift in any changed line or in the total.** Per-node: wk-05 = 22 984 Mi = 22.44 GiB ✔, wk-06 = 10 952 Mi = 10.70 GiB ✔ (both as stated). Slots **23 → 23**: 16+2+2+1+1+1; `replicas:` untouched in all six objects, no capacity/configmap in the diff. ### 3. Scope safety — clean No `labels`, `selector`, `nodeAffinity`/`nodeSelector`/toleration, `replicas`, `capacity`, `volumeClaimTemplates`, grace period, image or timeout change. The only non-comment lines that changed are the three `memory:` request values. ### 4. Apply gate — honest, and cross-checked The body carries the required gate: apply only with **no Sendrast job running**, on **homelab-ba's word through the overseer**, **read the live pod spec back after** (`.spec.containers[*].resources.requests.memory`), do not apply with a job in flight. Independently confirmed (read-only `kubectl get sts -n forgejo-runner`, 13:09Z) that live still reads 3163 / 6144 / 3300 / 6144 / 2048 / 584 Mi — the change is genuinely not applied and the PR's "before" column is the live truth. CI on this exact head: **11/11 success**, incl. `runner-manifest-test` and `yaml-lint`. ### Concern 1 — the k8s `runner` request is now below its own measured peak `runner.requests.memory = 1003Mi` against that container's measured 7 d peak **1786.55 Mi** and the worst pod's cgroup peak **2258.79 Mi**. Limits unchanged, so this moves **eviction/OOM ranking, not the ceiling** — the 8 pods become earlier eviction candidates on a node whose 7 d node-level peak was 52.31 GiB of 58.35 GiB allocatable. It is the direct arithmetic consequence of the GO's 33.1 GiB with 23 slots kept, and it is disclosed; not a defect of the diff. Needs a post-apply watch on `kube_pod_container_status_restarts_total{namespace="forgejo-runner"}` and node memory-pressure evictions on wk-05/wk-06 for 24 h. ### Concern 2 — the approved plan contradicts itself (PR resolved it correctly) Plan §2 says pod **3163 → 2048 Mi**; plan §5 says `runner 2118 → 2048` with dind unchanged = **3093 Mi/pod** = 41.30 GiB total (the PR's narrative says 41.28; exact is 41.30, 42 296 Mi). The PR implemented §2/the GO and **disclosed the deviation explicitly** — the honest resolution. Please correct the plan doc so the audit trail does not carry two shapes. ### Concern 3 — the "before" total does not reconcile with its own rows (inherited from plan §1) 25 304 + 12 288 + 6 600 + 6 144 + 2 048 + 584 = **52 968 Mi = 51.73 GiB**, not 53 032 Mi / 51.79 GiB (64 Mi; the plan's note about 2 extra `Succeeded` dind-gc pods is the likely origin). The delta is therefore **−18.59 GiB**, not −18.65. The **after** figure is exact, so nothing in the change depends on it — narrative only. ### Suggestion 1 — do not let `android-ci` stay at 1 Gi Correct per the GO and nothing dispatches to it today, but leave an explicit step on HOMELAB-2479: raise it back to ≈12 Gi **before** any job is dispatched, or the first re-enabled android job runs a 12 GiB limit against a 1 GiB reservation on wk-05. ### Suggestion 2 — only three StatefulSets roll The body warns of android-emu's 45 min roll (2700 s grace) and android-release's 90 s; **neither template is in this diff and neither is drifted from live, so neither rolls.** Only `forgejo-runner` (8), `android-ci` (2), `ld-ci` (1) do — the "no job in flight" window is far shorter than the body implies. Worth correcting so the operator does not wait 45 minutes for a roll that will not happen. ### Not verified by me The 7 d peak figures are inherited from the PR body/plan (no Prometheus query run in this review); `android-ci`/`ld-ci` idleness is inherited from the plan's live-branch check; whether `prod` carries an overriding patch was not read (out of scope for this diff, and the apply read-back is the right control); CI job *logs* were not read, only the head's status.

pi-landing merged commit a590103cdb into live 2026-11-06 13:17:10 +00:00

pi-landing referenced this pull request from a commit 2026-11-06 13:17:12 +00:00

HOMELAB-2479: merge #876 (CI reservation trim 51.8->33.1 GiB)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

operator-review

Requires human operator review before merge — dangerous class (RBAC, secrets, network policy, Terraform) or explicitly gated. Never auto-merge.

No labels

operator-review

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Homelab/infra-core!876

No description provided.