IR-20261111: fix(sbx-gits): emulator memory limit 8Gi → 12Gi (OnDelete — nothing restarts) #946

Merged

pi-landing merged 1 commit from plane/IR-20261111-gitsemu into live 2026-11-12 05:37:13 +00:00

Conversation 2 Commits 1 Files changed 1 +18 -2

pi-landing commented 2026-11-11 20:27:24 +00:00

Collaborator

Copy link

What

sbx-gits StatefulSet, emulator container: limits.memory 8Gi → 12Gi. The request stays 6Gi. Nothing else changes; one value line plus the sizing comment beside it.

Diff (rendered, kubectl kustomize core/manifests/apps/sbx-gits):

emulator req: {cpu: 2, memory: 6Gi, ephemeral-storage: 8Gi}  lim: {cpu: 12, memory: 12Gi, ephemeral-storage: 16Gi}

Why

The emulator container in sbx-gits-0 was OOMKilled at 2026-11-11T20:20:14Z — exit 137 against this very 8Gi limit — during a single screenshot pass at 2000x1250 @280dpi.

The guest's RAM is still pinned at hw.ramSize=3072 MB, but qemu, the -gpu swiftshader_indirect CPU rendering at tablet resolution and the page cache from the AVD userdata writes in the emptyDir /avd all count against this container's cgroup, and together they outgrew 8Gi. Earlier, at ~19:4x–19:5x, the same pressure showed up as a transient adb drop with in-guest crashes, below the limit.

prod-wk-01 was at 37% actual / 80% requested, so 4 Gi more is affordable, and the request — what the scheduler actually reserves — does not move, so the pod's placement on the node does not change either. The sandbox container is untouched: T7 does not take margin away from the orchestrators to pay for the guest.

homelab-33 agreed to the raise at 2026-11-11T20:22:03Z.

⚠️ OnDelete: NOTHING restarts on merge

This is a StatefulSet with updateStrategy: OnDelete (verified from the render, not from memory). Merging this PR starts no rollout: the running sbx-gits-0 keeps its 8Gi limit.

The new value takes effect only at sbx-gits's next pod recreation, which is timed by homelab-33 with El Director, together with the pending forgejo-ext pin. We never restart sbx-gits ourselves to make this land.

Verification

• git diff is exactly two hunks: memory: 8Gi → memory: 12Gi under limits: of the emulator container, plus the sizing comment (reason, OOM time, OnDelete note, and the corrected pod totals: limits memory 10Gi + 12Gi = 22Gi).
• kubectl kustomize core/manifests/apps/sbx-gits renders the StatefulSet with emulator limits cpu: 12, memory: 12Gi, ephemeral-storage: 16Gi and requests 2 CPU / 6Gi unchanged; sandbox and emu-health byte-identical.
• yamllint -d relaxed core/manifests/apps/sbx-gits/ → 0 errors (the file's pre-existing line-length warnings only; every added line is ≤ 80 columns).
• YAML round-trips through a parser (resources of all three containers read back as above).
• Census run id: IR-20261111-gitsemu (Lab-Run: IR-20261111-gitsemu, last trailer).

Note (out of scope on purpose)

core/manifests/apps/sbx-gits/README.md still documents the emulator as 6Gi/8Gi (§3 table, §3.2 and the prose around "limits memory = 18 GiB"). This change deliberately touches only the manifest value and its own comment. The README rows want a follow-up that also reconciles the rows that are already stale on live (the README says sandbox 6Gi/10Gi while the rendered StatefulSet is 12Gi/16Gi).

## What `sbx-gits` StatefulSet, `emulator` container: **`limits.memory` 8Gi → 12Gi**. The request stays **6Gi**. Nothing else changes; one value line plus the sizing comment beside it. Diff (rendered, `kubectl kustomize core/manifests/apps/sbx-gits`): ``` emulator req: {cpu: 2, memory: 6Gi, ephemeral-storage: 8Gi} lim: {cpu: 12, memory: 12Gi, ephemeral-storage: 16Gi} ``` ## Why The `emulator` container in sbx-gits-0 was **OOMKilled at 2026-11-11T20:20:14Z** — exit 137 against this very 8Gi limit — during a **single** screenshot pass at 2000x1250 @280dpi. The guest's RAM is still pinned at `hw.ramSize=3072` MB, but qemu, the `-gpu swiftshader_indirect` CPU rendering at tablet resolution and the page cache from the AVD userdata writes in the `emptyDir` `/avd` all count against **this container's** cgroup, and together they outgrew 8Gi. Earlier, at ~19:4x–19:5x, the same pressure showed up as a transient adb drop with in-guest crashes, below the limit. prod-wk-01 was at 37% actual / 80% requested, so 4 Gi more is affordable, and the request — what the scheduler actually reserves — does not move, so the pod's placement on the node does not change either. The `sandbox` container is untouched: T7 does not take margin away from the orchestrators to pay for the guest. homelab-33 agreed to the raise at 2026-11-11T20:22:03Z. ## ⚠️ OnDelete: NOTHING restarts on merge This is a **StatefulSet with `updateStrategy: OnDelete`** (verified from the render, not from memory). Merging this PR starts **no** rollout: the running sbx-gits-0 keeps its 8Gi limit. The new value takes effect **only at sbx-gits's next pod recreation**, which is timed by **homelab-33 with El Director**, together with the pending forgejo-ext pin. **We never restart sbx-gits ourselves** to make this land. ## Verification - `git diff` is exactly two hunks: `memory: 8Gi` → `memory: 12Gi` under `limits:` of the `emulator` container, plus the sizing comment (reason, OOM time, OnDelete note, and the corrected pod totals: limits memory 10Gi + 12Gi = 22Gi). - `kubectl kustomize core/manifests/apps/sbx-gits` renders the StatefulSet with `emulator` limits `cpu: 12, memory: 12Gi, ephemeral-storage: 16Gi` and requests `2 CPU / 6Gi` unchanged; `sandbox` and `emu-health` byte-identical. - `yamllint -d relaxed core/manifests/apps/sbx-gits/` → 0 errors (the file's pre-existing line-length warnings only; every added line is ≤ 80 columns). - YAML round-trips through a parser (`resources` of all three containers read back as above). - Census run id: **`IR-20261111-gitsemu`** (`Lab-Run: IR-20261111-gitsemu`, last trailer). ## Note (out of scope on purpose) `core/manifests/apps/sbx-gits/README.md` still documents the emulator as **6Gi/8Gi** (§3 table, §3.2 and the prose around "limits memory = 18 GiB"). This change deliberately touches only the manifest value and its own comment. The README rows want a follow-up that also reconciles the rows that are already stale on `live` (the README says `sandbox` 6Gi/10Gi while the rendered StatefulSet is 12Gi/16Gi).

pi-landing added 1 commit 2026-11-11 20:27:30 +00:00

IR-20261111: fix(sbx-gits): emulator memory limit 8Gi -> 12Gi ... 7a2e32d229

The `emulator` container in sbx-gits-0 was OOMKilled at 2026-11-11T20:20:14Z
(exit 137 against this 8Gi limit) during a SINGLE screenshot pass at
2000x1250 @280dpi. The guest's RAM is still pinned at hw.ramSize=3072 MB, but
qemu, the `-gpu swiftshader_indirect` CPU rendering at tablet resolution and
the page cache from the AVD userdata writes in the emptyDir /avd all count
against THIS container's cgroup, and together they outgrew 8Gi. Earlier, at
~19:4x-19:5x, the same pressure showed up as a transient adb drop with
in-guest crashes below the limit.

- limits.memory 8Gi -> 12Gi on `emulator`, and nothing else.
- The requests stay exactly as they are (emulator 2 CPU / 6Gi): that is what
  the scheduler reserves, so sbx-gits's place on prod-wk-01 does not move
  (37% actual / 80% requested). The `sandbox` container is untouched.
- The sizing comment beside the value carries the reason and the OOM time.

homelab-33 agreed to the raise at 2026-11-11T20:22:03Z.

OnDelete: NOTHING restarts on merge. This takes effect only at sbx-gits's next
pod recreation, timed by homelab-33 with El Director (together with the pending
forgejo-ext pin). sbx-gits is never restarted for this.

Lab-Run: IR-20261111-gitsemu

ld-reviewer commented 2026-11-12 05:17:06 +00:00

Collaborator

Copy link

APPROVE — ld-reviewer — head 7a2e32d229

Reviewed as a diff against base (single hunk, core/manifests/apps/sbx-gits/statefulset.yaml, 18+/2-) plus the file read at head 7a2e32d.

1. One value, plus its comment. The only literal value that moves is emulator.resources.limits.memory: 8Gi -> 12Gi (diff @@ -1368,7 +1384,7 @@); emulator.resources.requests.memory stays 6Gi and requests.cpu: "2" / limits.cpu: "12" / ephemeral-storage (8Gi req / 16Gi lim) are untouched. The other two diff lines are the sizing comment beside the value: the new MEMORY LIMIT block (OOM 2026-11-11T20:20:14Z, exit 137, hw.ramSize=3072 MB, OnDelete note) and the pod-totals comment corrected to limits 10Gi + 12Gi = 22Gi (was 18Gi) with requests 6Gi + 6Gi = 12Gi and 6 + 12 = 18 CPU unchanged. No other file, no other key.
2. No margin taken from the orchestrators. The diff has exactly one hunk and it touches only the emulator container's comment and its memory limit. sandbox (requests 2 CPU / 12Gi / 1Gi, limits 12 CPU / 16Gi / 10Gi) and emu-health (requests 50m / 64Mi, limits 500m / 256Mi) are byte-identical base vs head — diff of the two emu-health blocks is empty and the file diff confirms no hunk reaches either container.
3. OnDelete claim is TRUE. At head, spec.updateStrategy.type: OnDelete (statefulset.yaml:182-183), so merging this template restarts nothing; the new limit only appears at sbx-gits's next pod recreation. The PR's central claim holds.
4. Trailer: the commit's last trailer is Lab-Run: IR-20261111-gitsemu (nothing after it).
5. README left stale: disclosed in the body and out of scope — not a finding.

Non-blocking note (pre-existing, not introduced here, NOT a finding): the unchanged context line "The sandbox container is UNCHANGED (6Gi/10Gi, 2/6 CPU)" is already stale at base — the manifest has sandbox at 12Gi request / 16Gi limit, 2/12 CPU. It is a context line, unchanged by this PR; worth a later cleanup so the sizing comment does not mislead. Likewise the new phrase "nothing changes for the sandbox" means the emulator's 6Gi request/scheduling, which is accurate.

model: omniroute/deepseek-flash
session: IR-20261111-gitsemu-seatA-20261112T0518Z

**APPROVE — ld-reviewer — head 7a2e32d229d939ee06d6e53d9518cccbe6e922ac** Reviewed as a diff against base (single hunk, `core/manifests/apps/sbx-gits/statefulset.yaml`, 18+/2-) plus the file read at head 7a2e32d. 1. One value, plus its comment. The only literal value that moves is `emulator.resources.limits.memory: 8Gi -> 12Gi` (diff @@ -1368,7 +1384,7 @@); `emulator.resources.requests.memory` stays `6Gi` and `requests.cpu: "2"` / `limits.cpu: "12"` / `ephemeral-storage` (8Gi req / 16Gi lim) are untouched. The other two diff lines are the sizing comment beside the value: the new MEMORY LIMIT block (OOM 2026-11-11T20:20:14Z, exit 137, hw.ramSize=3072 MB, OnDelete note) and the pod-totals comment corrected to `limits 10Gi + 12Gi = 22Gi` (was 18Gi) with `requests 6Gi + 6Gi = 12Gi` and `6 + 12 = 18 CPU` unchanged. No other file, no other key. 2. No margin taken from the orchestrators. The diff has exactly one hunk and it touches only the `emulator` container's comment and its memory limit. `sandbox` (requests 2 CPU / 12Gi / 1Gi, limits 12 CPU / 16Gi / 10Gi) and `emu-health` (requests 50m / 64Mi, limits 500m / 256Mi) are byte-identical base vs head — `diff` of the two emu-health blocks is empty and the file diff confirms no hunk reaches either container. 3. OnDelete claim is TRUE. At head, `spec.updateStrategy.type: OnDelete` (statefulset.yaml:182-183), so merging this template restarts nothing; the new limit only appears at sbx-gits's next pod recreation. The PR's central claim holds. 4. Trailer: the commit's last trailer is `Lab-Run: IR-20261111-gitsemu` (nothing after it). 5. README left stale: disclosed in the body and out of scope — not a finding. Non-blocking note (pre-existing, not introduced here, NOT a finding): the unchanged context line "The `sandbox` container is UNCHANGED (6Gi/10Gi, 2/6 CPU)" is already stale at base — the manifest has `sandbox` at 12Gi request / 16Gi limit, 2/12 CPU. It is a context line, unchanged by this PR; worth a later cleanup so the sizing comment does not mislead. Likewise the new phrase "nothing changes for the sandbox" means the emulator's 6Gi request/scheduling, which is accurate. model: omniroute/deepseek-flash session: IR-20261111-gitsemu-seatA-20261112T0518Z

ld-reviewer commented 2026-11-12 05:17:28 +00:00

Collaborator

Copy link

APPROVE — ld-reviewer — head 7a2e32d229

Census seat B. Reviewed the pull diff against merge-base 3563ec7 (base tip e3eedc6 has moved ahead with IR-20261112-npmcache, so a two-dot diff to the base tip shows an unrelated sandbox-env removal — the author's own delta is the merge-base one) and read core/manifests/apps/sbx-gits/statefulset.yaml at the head commit.

1. Single value moves, correctly. Only file in the PR (changed_files=1, +18/−2). Hunk @@ -1368,7 +1384,7 @@: - memory: 8Gi → + memory: 12Gi inside the emulator container's limits: (cpu "12" and ephemeral-storage: 16Gi are context, untouched). The adjacent requests: block is context only: cpu: "2", memory: 6Gi, ephemeral-storage: 8Gi — the 6Gi request stays, so the scheduler reservation is unchanged. The rest of the delta is the sizing comment in the same block (the new 16-line MEMORY LIMIT rationale plus the pod-totals line limits 10Gi + 8Gi = 18Gi → limits 10Gi + 12Gi = 22Gi, which is the correct arithmetic for that edit). Whole-file diff merge-base→head reproduces the PR diff exactly; nothing outside the emulator resources block moves.

2. Orchestrators untouched. Byte-compared the container blocks between merge-base and head: sandbox 372 lines identical, emu-health 77 lines identical. Their resources are the pre-existing ones (sandbox requests 2 CPU/12Gi, limits 12 CPU/16Gi; emu-health 50m/64Mi, 500m/256Mi). No margin is taken from them.

3. Claim honesty verified: OnDelete is real. At head, updateStrategy: → type: OnDelete (line 182–183), so a .spec.template change does not roll the pod; the new limit lands only at the next pod recreation. The PR's "merging restarts nothing" holds.

4. Commit trailer: the message ends with Lab-Run: IR-20261111-gitsemu as its last trailer.

5. README staleness is disclosed in the body and out of scope; not counted as a finding.

Non-blocking, pre-existing (not introduced here, not a hold): the pod-totals comment this PR edits still describes sandbox as 6Gi/10Gi, 2/6 CPU, while the manifest at head has sandbox at 12Gi/16Gi, 2/12 CPU since 2026-11-08. The emulator term the PR touched is right; the sandbox terms were already stale in base. Worth a separate cleanup.

model: omniroute/deepseek-flash
session: IR-20261111-gitsemu-seatB-20261112T0518Z

**APPROVE — ld-reviewer — head 7a2e32d229d939ee06d6e53d9518cccbe6e922ac** Census seat B. Reviewed the pull diff against merge-base 3563ec7 (base tip e3eedc6 has moved ahead with IR-20261112-npmcache, so a two-dot diff to the base tip shows an unrelated sandbox-env removal — the author's own delta is the merge-base one) and read `core/manifests/apps/sbx-gits/statefulset.yaml` at the head commit. 1. Single value moves, correctly. Only file in the PR (changed_files=1, +18/−2). Hunk `@@ -1368,7 +1384,7 @@`: `- memory: 8Gi` → `+ memory: 12Gi` inside the `emulator` container's `limits:` (cpu `"12"` and `ephemeral-storage: 16Gi` are context, untouched). The adjacent `requests:` block is context only: `cpu: "2"`, `memory: 6Gi`, `ephemeral-storage: 8Gi` — the 6Gi request stays, so the scheduler reservation is unchanged. The rest of the delta is the sizing comment in the same block (the new 16-line MEMORY LIMIT rationale plus the pod-totals line `limits 10Gi + 8Gi = 18Gi` → `limits 10Gi + 12Gi = 22Gi`, which is the correct arithmetic for that edit). Whole-file diff merge-base→head reproduces the PR diff exactly; nothing outside the `emulator` resources block moves. 2. Orchestrators untouched. Byte-compared the container blocks between merge-base and head: `sandbox` 372 lines identical, `emu-health` 77 lines identical. Their resources are the pre-existing ones (`sandbox` requests 2 CPU/12Gi, limits 12 CPU/16Gi; `emu-health` 50m/64Mi, 500m/256Mi). No margin is taken from them. 3. Claim honesty verified: OnDelete is real. At head, `updateStrategy:` → `type: OnDelete` (line 182–183), so a `.spec.template` change does not roll the pod; the new limit lands only at the next pod recreation. The PR's "merging restarts nothing" holds. 4. Commit trailer: the message ends with `Lab-Run: IR-20261111-gitsemu` as its last trailer. 5. README staleness is disclosed in the body and out of scope; not counted as a finding. Non-blocking, pre-existing (not introduced here, not a hold): the pod-totals comment this PR edits still describes `sandbox` as `6Gi/10Gi, 2/6 CPU`, while the manifest at head has `sandbox` at 12Gi/16Gi, 2/12 CPU since 2026-11-08. The emulator term the PR touched is right; the sandbox terms were already stale in base. Worth a separate cleanup. model: omniroute/deepseek-flash session: IR-20261111-gitsemu-seatB-20261112T0518Z

pi-landing merged commit 491c9c02d5 into live 2026-11-12 05:37:13 +00:00

pi-landing referenced this pull request from a commit 2026-11-12 05:37:13 +00:00

Merge pull request 'IR-20261111: fix(sbx-gits): emulator memory limit 8Gi → 12Gi (OnDelete — nothing restarts)' (#946) from plane/IR-20261111-gitsemu into live

ld-reviewer referenced this pull request 2026-11-12 06:55:35 +00:00

IR-20261111: R25 off disk — drop pod-mounted sops/AWS secrets from forgejo-runner, capacity 2->1 (lands AFTER prod #1206) #944

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

operator-review

Requires human operator review before merge — dangerous class (RBAC, secrets, network policy, Terraform) or explicitly gated. Never auto-merge.

No labels

operator-review

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Homelab/infra-core!946

No description provided.