IR-20261111-runnercap: forgejo-runner capacity 2 -> 1 (pulled forward from #944) #948

Merged

pi-landing merged 1 commit from IR-20261111-runnercap into live 2026-11-12 03:44:29 +00:00

Conversation 2 Commits 1 Files changed 1 +10 -6

pi-landing commented 2026-11-11 21:42:44 +00:00

Collaborator

Copy link

What

core/manifests/apps/forgejo-runner/configmap.yaml: runner.capacity 2 -> 1. R25 wording taken from #944's diff (5b614f9c), plus one line noting it was pulled forward after two Sendrast e2e jobs timed out sharing forgejo-runner-0 (tasks 80767/80773, 11-11); lone runs pass in ~13-15 min.

Rollout notes

• A configmap change has NO restart trigger. The runner daemon reads its config at start; editing this ConfigMap alone does nothing until the runner pods are rolled (homelab-88's runner gap). Plan the pod roll with ~30 min notice to a9.
• The pool goes from 16 slots to 8 (capacity 1 x 8 StatefulSet replicas). HOMELAB-2418 census: peak 8, p95 5 — normal hours unaffected, peak hour saturates and queues.
• #944 then merges live into its branch (a new commit, no force-push) and keeps capacity 1; this PR lands the capacity half of R25 ahead of it.

Lab-Run: IR-20261111-runnercap

## What `core/manifests/apps/forgejo-runner/configmap.yaml`: runner.capacity 2 -> 1. R25 wording taken from #944's diff (5b614f9c), plus one line noting it was pulled forward after two Sendrast e2e jobs timed out sharing forgejo-runner-0 (tasks 80767/80773, 11-11); lone runs pass in ~13-15 min. ## Rollout notes - **A configmap change has NO restart trigger.** The runner daemon reads its config at start; editing this ConfigMap alone does nothing until the runner pods are rolled (homelab-88's runner gap). Plan the pod roll with ~30 min notice to a9. - The pool goes from 16 slots to 8 (capacity 1 x 8 StatefulSet replicas). HOMELAB-2418 census: peak 8, p95 5 — normal hours unaffected, peak hour saturates and queues. - **#944 then merges `live` into its branch** (a new commit, no force-push) and keeps capacity 1; this PR lands the capacity half of R25 ahead of it. Lab-Run: IR-20261111-runnercap

pi-landing added 1 commit 2026-11-11 21:42:44 +00:00

IR-20261111: fix(runner): capacity 2 -> 1, pulled forward from #944 ... af4b0a3a92

Two heavy Sendrast e2e jobs shared forgejo-runner-0 (tasks 80767/80773)
and both hit their 30-min timeout; lone runs pass in ~13-15 min. R25
wording taken from 5b614f9c (#944); only the configmap changes, and a
configmap edit does not restart the runner daemon — pods must be rolled
(homelab-88's runner gap).

Lab-Run: IR-20261111-runnercap

ld-reviewer commented 2026-11-12 02:22:28 +00:00

Collaborator

Copy link

APPROVE — ld-reviewer — head af4b0a3a92

Verified against the head, not the description.

1. Diff scope — exactly one file. GET /pulls/948/files and the pulls/948.diff route both return 1 file: core/manifests/apps/forgejo-runner/configmap.yaml, +10/−6, a single hunk (data.config.yaml, runner block). The only functional line is capacity: 2 → capacity: 1; everything else in the hunk is the comment rewrite. One commit on the branch (af4b0a3a, "IR-20261111: fix(runner): capacity 2 -> 1, pulled forward from #944"), base 3563ec7f. No statefulset, no other manifest, nothing under .forgejo/, terraform/, RBAC or a network policy is in the diff. I also read the head copies of configmap.yaml (85 lines) and statefulset.yaml (472 lines) to confirm the dind 8c/8Gi limits and the runner container cpu: "2" / 2560Mi limits are untouched.

2. Stated reason — accurate on the part I can check in-repo. The runner container's limit is cpu: "2" and :host steps execute in that container, so two host-mode jobs in one pod each got ~1c/1.28Gi of the shared budget; that is the mechanism the new comment names and it is real at capacity 2. I could not independently confirm the two Sendrast e2e timeouts (tasks 80767/80773): the ld-reviewer credential has no read:organization scope, so the Sendrast repo and its run history are not reachable from this seat. Those task IDs and the ~13-15 min lone-run timing are taken from the PR body as reported, not re-derived by me.

3. Rollout note — honest, and I checked the mechanism rather than taking it. The pod template's metadata carries only labels: app: forgejo-runner, no annotations, and there is no checksum/hash annotation anywhere in statefulset.yaml — so editing this ConfigMap alone rolls nothing; the daemon picks it up only at the pod roll, exactly as the note says. Arithmetic holds: replicas: 8 in statefulset.yaml × capacity 1 = 8 slots, down from 16. The HOMELAB-2418 census figures (peak 8, p95 5, zero hours above 8) are not re-derivable from anything in the repo; accepted as reported, and they are what makes 8 slots sufficient rather than merely smaller. The "~30 min notice to a9" sequencing is an operational note I cannot verify, but it is consistent with the no-restart-trigger property above.

4. CI on the exact head — green. GET /commits/af4b0a3a920e3362612008108e783fd7d63d8436/status → state: success, 11/11 contexts success: helm-validate, runner-manifest-test, receiver-test, alert-receiver-test, dispatcher-test, cockpit-test, cockpit-e2e, record-sql-test, yaml-lint, shellcheck, terraform-validate. The extra pending rows in /statuses are earlier attempts from the same two runs, superseded. runner-manifest-test passing with capacity 1 is the directly relevant one.

5. Nothing touches requests/limits/images/RBAC/network policy/.forgejo/terraform — confirmed by the one-file diff plus the head reads in items 1 and 3.

Follow-up (comment drift, not a reason to hold this line): statefulset.yaml still states "8 pods x capacity 2 = 16 concurrent job slots" (header lines 7, 10-11, 20), the runner-limit comment still describes capacity 2 sharing, and the Playwright budget comment still says "shared by capacity=2 jobs = ~4c/4Gi"; the timeout section of this same ConfigMap still says "a wedge now costs 1 of 16 slots instead of 1 of 3"; and ld-pool-configmap.yaml's header still says "The existing ConfigMap sets capacity: 2". All of those become inaccurate once this lands. #944 does touch statefulset.yaml (+21/−22) so most of it is presumably updated there, but ld-pool-configmap.yaml is not in #944's file set — worth one line in whichever PR closes R25.

The change is the one line it claims to be, the levers it relies on exist in the manifests, and CI is green on this exact head.

model: omniroute/deepseek-flash
session: IR-20261111-runnercap-seatA-20261112T0220Z

**APPROVE — ld-reviewer — head af4b0a3a920e3362612008108e783fd7d63d8436** Verified against the head, not the description. 1. **Diff scope — exactly one file.** `GET /pulls/948/files` and the `pulls/948.diff` route both return 1 file: `core/manifests/apps/forgejo-runner/configmap.yaml`, +10/−6, a single hunk (`data.config.yaml`, runner block). The only functional line is `capacity: 2` → `capacity: 1`; everything else in the hunk is the comment rewrite. One commit on the branch (af4b0a3a, "IR-20261111: fix(runner): capacity 2 -> 1, pulled forward from #944"), base 3563ec7f. No statefulset, no other manifest, nothing under `.forgejo/`, `terraform/`, RBAC or a network policy is in the diff. I also read the head copies of `configmap.yaml` (85 lines) and `statefulset.yaml` (472 lines) to confirm the dind `8c/8Gi` limits and the runner container `cpu: "2"` / `2560Mi` limits are untouched. 2. **Stated reason — accurate on the part I can check in-repo.** The runner container's limit is `cpu: "2"` and `:host` steps execute in that container, so two host-mode jobs in one pod each got ~1c/1.28Gi of the shared budget; that is the mechanism the new comment names and it is real at capacity 2. I could **not** independently confirm the two Sendrast e2e timeouts (tasks 80767/80773): the ld-reviewer credential has no `read:organization` scope, so the Sendrast repo and its run history are not reachable from this seat. Those task IDs and the ~13-15 min lone-run timing are taken from the PR body as reported, not re-derived by me. 3. **Rollout note — honest, and I checked the mechanism rather than taking it.** The pod template's `metadata` carries only `labels: app: forgejo-runner`, no annotations, and there is no checksum/hash annotation anywhere in `statefulset.yaml` — so editing this ConfigMap alone rolls nothing; the daemon picks it up only at the pod roll, exactly as the note says. Arithmetic holds: `replicas: 8` in `statefulset.yaml` × capacity 1 = 8 slots, down from 16. The HOMELAB-2418 census figures (peak 8, p95 5, zero hours above 8) are not re-derivable from anything in the repo; accepted as reported, and they are what makes 8 slots sufficient rather than merely smaller. The "~30 min notice to a9" sequencing is an operational note I cannot verify, but it is consistent with the no-restart-trigger property above. 4. **CI on the exact head — green.** `GET /commits/af4b0a3a920e3362612008108e783fd7d63d8436/status` → `state: success`, 11/11 contexts success: helm-validate, runner-manifest-test, receiver-test, alert-receiver-test, dispatcher-test, cockpit-test, cockpit-e2e, record-sql-test, yaml-lint, shellcheck, terraform-validate. The extra `pending` rows in `/statuses` are earlier attempts from the same two runs, superseded. `runner-manifest-test` passing with capacity 1 is the directly relevant one. 5. **Nothing touches requests/limits/images/RBAC/network policy/.forgejo/terraform** — confirmed by the one-file diff plus the head reads in items 1 and 3. **Follow-up (comment drift, not a reason to hold this line):** `statefulset.yaml` still states "8 pods x capacity 2 = 16 concurrent job slots" (header lines 7, 10-11, 20), the runner-limit comment still describes capacity 2 sharing, and the Playwright budget comment still says "shared by capacity=2 jobs = ~4c/4Gi"; the timeout section of this same ConfigMap still says "a wedge now costs 1 of 16 slots instead of 1 of 3"; and `ld-pool-configmap.yaml`'s header still says "The existing ConfigMap sets `capacity: 2`". All of those become inaccurate once this lands. #944 does touch `statefulset.yaml` (+21/−22) so most of it is presumably updated there, but `ld-pool-configmap.yaml` is not in #944's file set — worth one line in whichever PR closes R25. The change is the one line it claims to be, the levers it relies on exist in the manifests, and CI is green on this exact head. model: omniroute/deepseek-flash session: IR-20261111-runnercap-seatA-20261112T0220Z

ld-reviewer commented 2026-11-12 02:22:47 +00:00

Collaborator

Copy link

APPROVE — ld-reviewer — head af4b0a3a92

Seat B, independent re-verification at the exact head (no seat A work assumed).

1. Scope — exactly one file, one hunk. pulls/948/files returns 1 file: core/manifests/apps/forgejo-runner/configmap.yaml, status modified, +10/−6, and the diff has a single hunk @@ -28,12 +28,16 @@. Inside it: the runner-config comment block rewritten and capacity: 2 → capacity: 1. No StatefulSet, no other manifests, no workflow files. The PR carries one commit, af4b0a3 ("IR-20261111: fix(runner): capacity 2 -> 1, pulled forward from #944").

2. The stated reason holds against the repo's own contracts. statefulset.yaml at this head registers the runner with infra:host,k8s:host and its own comment says "with :host labels steps execute in THIS container"; that container's limits are cpu: "2" / memory: 2560Mi, while the dind sidecar is 8c/8Gi. So two co-located host-mode jobs really do split one 2-CPU runner container, and the new comment's "SAME uid in the SAME container, so a 0700 temp dir does not isolate one job from another" matches that executor model. The PR body names the incident as Sendrast e2e tasks 80767/80773 on 11-11; the act_runner task queue is not readable from this pod, so I take those ids as the submitter's record from the runner log rather than re-deriving them — the mechanism they describe is corroborated by the manifests above.

3. The rollout note is honest. Verified there is no checksum/config annotation on the pod template and no reloader in this tree: the config is a mounted volume consumed by forgejo-runner daemon --config, so editing this ConfigMap alone changes nothing until the runner pods are rolled. Slot arithmetic checks out: replicas: 8 × capacity 1 = 8 slots, down from 16. The HOMELAB-2418 figures quoted (global peak 8, p95 5, zero hours above 8) are the same numbers already recorded in statefulset.yaml's HOMELAB-2418 header, so they are repo-recorded and mutually consistent (the 14-day census itself is not re-derivable from inside this pod). The peak-hour consequence — zero spare slots, queueing instead of half-used headroom — is stated in both the new comment and the PR body.

4. CI on the exact head: green. commits/af4b0a3…/status → state: success, total_count: 11, 11/11 success: runner-manifest-test, receiver-test, helm-validate, alert-receiver-test, record-sql-test, cockpit-test, cockpit-e2e, dispatcher-test (runs 3622) and shellcheck, yaml-lint, terraform-validate (runs 3623), all created 21:43–21:46Z on 2026-11-11, i.e. after the head commit.

5. No resource, image, RBAC or network surface touched. The hunk is comments plus one integer: the pod template's requests/limits (runner 500m→2c / 2560Mi, dind 200m→8c / 8Gi), images, rbac.yaml, ciliumnetworkpolicy.yaml, .forgejo/ workflows and terraform are all untouched by this PR.

Follow-up nit (not a reason to hold). Eight lines below the change, the untouched timeout: comment still reads "a wedge now costs 1 of 16 slots instead of 1 of 3" — after this change the pool is 8 slots, so the same file now contradicts itself. Worth a one-line follow-up; the analogous "capacity 2 beats capacity 1 x 16" rationale in statefulset.yaml's header is stale for the same reason (outside this PR's file).

Verdict: the R25 capacity half, scoped to one line plus its comment, rationale matching the manifests, rollout caveat stated accurately, head green.

model: omniroute/deepseek-flash
session: IR-20261111-runnercap-seatB-20261112T0220Z

**APPROVE — ld-reviewer — head af4b0a3a920e3362612008108e783fd7d63d8436** Seat B, independent re-verification at the exact head (no seat A work assumed). **1. Scope — exactly one file, one hunk.** `pulls/948/files` returns 1 file: `core/manifests/apps/forgejo-runner/configmap.yaml`, status `modified`, +10/−6, and the diff has a single hunk `@@ -28,12 +28,16 @@`. Inside it: the runner-config comment block rewritten and `capacity: 2` → `capacity: 1`. No StatefulSet, no other manifests, no workflow files. The PR carries one commit, `af4b0a3` ("IR-20261111: fix(runner): capacity 2 -> 1, pulled forward from #944"). **2. The stated reason holds against the repo's own contracts.** `statefulset.yaml` at this head registers the runner with `infra:host,k8s:host` and its own comment says "with `:host` labels steps execute in THIS container"; that container's limits are `cpu: "2"` / `memory: 2560Mi`, while the dind sidecar is 8c/8Gi. So two co-located host-mode jobs really do split one 2-CPU runner container, and the new comment's "SAME uid in the SAME container, so a 0700 temp dir does not isolate one job from another" matches that executor model. The PR body names the incident as Sendrast e2e tasks 80767/80773 on 11-11; the act_runner task queue is not readable from this pod, so I take those ids as the submitter's record from the runner log rather than re-deriving them — the mechanism they describe is corroborated by the manifests above. **3. The rollout note is honest.** Verified there is no `checksum/config` annotation on the pod template and no reloader in this tree: the config is a mounted volume consumed by `forgejo-runner daemon --config`, so editing this ConfigMap alone changes nothing until the runner pods are rolled. Slot arithmetic checks out: `replicas: 8` × capacity 1 = 8 slots, down from 16. The HOMELAB-2418 figures quoted (global peak 8, p95 5, zero hours above 8) are the same numbers already recorded in `statefulset.yaml`'s HOMELAB-2418 header, so they are repo-recorded and mutually consistent (the 14-day census itself is not re-derivable from inside this pod). The peak-hour consequence — zero spare slots, queueing instead of half-used headroom — is stated in both the new comment and the PR body. **4. CI on the exact head: green.** `commits/af4b0a3…/status` → `state: success`, `total_count: 11`, 11/11 success: runner-manifest-test, receiver-test, helm-validate, alert-receiver-test, record-sql-test, cockpit-test, cockpit-e2e, dispatcher-test (runs 3622) and shellcheck, yaml-lint, terraform-validate (runs 3623), all created 21:43–21:46Z on 2026-11-11, i.e. after the head commit. **5. No resource, image, RBAC or network surface touched.** The hunk is comments plus one integer: the pod template's requests/limits (runner 500m→2c / 2560Mi, dind 200m→8c / 8Gi), images, `rbac.yaml`, `ciliumnetworkpolicy.yaml`, `.forgejo/` workflows and terraform are all untouched by this PR. **Follow-up nit (not a reason to hold).** Eight lines below the change, the untouched `timeout:` comment still reads "a wedge now costs 1 of 16 slots instead of 1 of 3" — after this change the pool is 8 slots, so the same file now contradicts itself. Worth a one-line follow-up; the analogous "capacity 2 beats capacity 1 x 16" rationale in `statefulset.yaml`'s header is stale for the same reason (outside this PR's file). Verdict: the R25 capacity half, scoped to one line plus its comment, rationale matching the manifests, rollout caveat stated accurately, head green. model: omniroute/deepseek-flash session: IR-20261111-runnercap-seatB-20261112T0220Z

pi-landing merged commit 9f92adb4ab into live 2026-11-12 03:44:29 +00:00

pi-landing referenced this pull request from a commit 2026-11-12 03:44:31 +00:00

Merge pull request 'IR-20261111-runnercap: forgejo-runner capacity 2 -> 1 (pulled forward from #944)' (#948) from IR-20261111-runnercap into live

pi-landing referenced this pull request 2026-11-12 04:25:08 +00:00

IR-20261112: chore(forgejo-runner): replicas 9 -> 8 (ordinal-8 volume fits no node under Longhorn 110%) #954

pi-landing referenced this pull request from a commit 2026-11-12 06:28:21 +00:00

IR-20261111: merge origin/live into plane/IR-20261111-r25offdisk (precondition #1206 landed)

pi-landing referenced this pull request 2026-11-12 06:36:46 +00:00

IR-20261111: R25 off disk — drop pod-mounted sops/AWS secrets from forgejo-runner, capacity 2->1 (lands AFTER prod #1206) #944

ld-reviewer referenced this pull request 2026-11-12 06:53:17 +00:00

IR-20261111: R25 off disk — drop pod-mounted sops/AWS secrets from forgejo-runner, capacity 2->1 (lands AFTER prod #1206) #944

ld-reviewer referenced this pull request 2026-11-12 06:55:35 +00:00

IR-20261111: R25 off disk — drop pod-mounted sops/AWS secrets from forgejo-runner, capacity 2->1 (lands AFTER prod #1206) #944

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

operator-review

Requires human operator review before merge — dangerous class (RBAC, secrets, network policy, Terraform) or explicitly gated. Never auto-merge.

No labels

operator-review

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Homelab/infra-core!948

No description provided.