HOMELAB-2479: sbx-gits sandbox memory 13Gi limit / 12Gi request and CPU limit 12 #851

Merged

pi-landing merged 2 commits from plane/HOMELAB-2479-sbx-resources into live 2026-11-05 18:12:43 +00:00

Conversation 1 Commits 2 Files changed 1 +3 -3

pi-landing commented 2026-11-05 18:00:02 +00:00

Collaborator

Copy link

Why

pod/sbx-gits-0, container sandbox: .status.containerStatuses[name=sandbox].lastState.terminated.reason = OOMKilled at 17:53:52Z at a 10Gi memory limit. Aaron's order, quoted: "We need to give them big resources. Avoid sandboxes crashing."

This PR raises only that container: resources.requests.memory 6Gi -> 12Gi, resources.limits.memory 10Gi -> 13Gi, resources.limits.cpu 6 -> 12. Three lines.

Why 13 and not 16

At the 17:53Z kill the sandbox container's working set was 9.47 GiB and the emulator's was 6.02 GiB, while prod-wk-01's node_memory_MemAvailable_bytes was 3.9 GiB of 27.4 GiB. That puts the node's practical ceiling for this pod at ~13.4 GiB, so the limit is set to 13Gi. At 16Gi the sandbox could push wk-01 into Talos's OOMController, which kills the unlimited BestEffort Longhorn pods that hold the OmniRoute gateway's and Forgejo's replicas — a cluster-wide cascade. A contained sandbox kill is the lesser harm, and that trade is deliberate: the sandbox is allowed to die at its own limit rather than take the gateway's replicas with it.

Deliberately excluded (measured, resolved in a follow-up)

• An emptyDir tmp sizeLimit of 30Gi cannot be placed on wk-01: node_filesystem_avail_bytes{instance="10.10.10.20:9100",mountpoint="/var",device="/dev/vda5"} = 28.88 GiB free at 17:56:41Z, and with a ~10 % nodefs eviction floor a single pod can use at most ~18.6 GiB of emptyDir before node-pressure eviction starts evicting other pods.
• Separately, tmp 30Gi + avd 16Gi + herdr-sock 64Mi = 46.06 GiB of emptyDir ceilings against a 34 GiB container ephemeral-storage limit, so the pod would be evicted on ephemeral-storage mid-build and the avd limit would be unreachable.

Neither is changed here.

Merge behaviour

The StatefulSet is OnDelete, so merging does not restart anything; the values take effect only when the pod is next created, which the orchestrator does deliberately.

Untouched: the emulator container, the avd emptyDir, work, herdr-sock, probes, images, replicas, and every other object in the repo.

## Why `pod/sbx-gits-0`, container `sandbox`: `.status.containerStatuses[name=sandbox].lastState.terminated.reason = OOMKilled` at **17:53:52Z** at a **10Gi** memory limit. Aaron's order, quoted: *"We need to give them big resources. Avoid sandboxes crashing."* This PR raises only that container: `resources.requests.memory` 6Gi -> 12Gi, `resources.limits.memory` 10Gi -> 13Gi, `resources.limits.cpu` 6 -> 12. Three lines. ## Why 13 and not 16 At the 17:53Z kill the sandbox container's working set was **9.47 GiB** and the emulator's was **6.02 GiB**, while prod-wk-01's `node_memory_MemAvailable_bytes` was **3.9 GiB of 27.4 GiB**. That puts the node's practical ceiling for this pod at **~13.4 GiB**, so the limit is set to **13Gi**. At 16Gi the sandbox could push wk-01 into Talos's `OOMController`, which kills the unlimited BestEffort Longhorn pods that hold the OmniRoute gateway's and Forgejo's replicas — a cluster-wide cascade. A contained sandbox kill is the lesser harm, and that trade is deliberate: the sandbox is allowed to die at its own limit rather than take the gateway's replicas with it. ## Deliberately excluded (measured, resolved in a follow-up) - An `emptyDir` `tmp` `sizeLimit` of **30Gi cannot be placed on wk-01**: `node_filesystem_avail_bytes{instance="10.10.10.20:9100",mountpoint="/var",device="/dev/vda5"}` = **28.88 GiB free at 17:56:41Z**, and with a ~10 % nodefs eviction floor a single pod can use at most **~18.6 GiB** of emptyDir before node-pressure eviction starts evicting other pods. - Separately, `tmp` 30Gi + `avd` 16Gi + `herdr-sock` 64Mi = **46.06 GiB of emptyDir ceilings against a 34 GiB container ephemeral-storage limit**, so the pod would be evicted on ephemeral-storage mid-build and the `avd` limit would be unreachable. Neither is changed here. ## Merge behaviour The StatefulSet is `OnDelete`, so merging does **not** restart anything; the values take effect only when the pod is next created, which the orchestrator does deliberately. Untouched: the `emulator` container, the `avd` emptyDir, `work`, `herdr-sock`, probes, images, replicas, and every other object in the repo.

pi-landing added 1 commit 2026-11-05 18:00:02 +00:00

HOMELAB-2479: fix(sbx-gits): sandbox memory 16Gi/12Gi and CPU limit 12 0510398c4a

pi-landing added 1 commit 2026-11-05 18:02:51 +00:00

HOMELAB-2479: fix(sbx-gits): sandbox memory limit 13Gi (node ceiling ~13.4GiB) 61147ada71

pi-landing changed title from HOMELAB-2479: sbx-gits sandbox memory 16Gi/12Gi and CPU limit 12 to HOMELAB-2479: sbx-gits sandbox memory 13Gi limit / 12Gi request and CPU limit 12 2026-11-05 18:05:14 +00:00

ld-reviewer commented 2026-11-05 18:07:01 +00:00

Collaborator

Copy link

Verdict of record — ld-reviewer — head 61147ada71: APPROVE

model: omniroute/deepseek-flash
session: orch12-review-851-ld-reviewer

What I verified (object + field)

1. Diff is exactly three values. 1 file, 1 hunk, 3 additions / 3 deletions, all in core/manifests/apps/sbx-gits/statefulset.yaml at spec...containers[name=sandbox].resources: requests.memory 6Gi -> 12Gi, limits.cpu "6" -> "12", limits.memory 10Gi -> 13Gi. Nothing else moved in the hunk or the file. ephemeral-storage request 1Gi / limit 10Gi unchanged in both containers.

2. The request fits prod-wk-01. node/prod-wk-01.status.allocatable.memory = 28217332Ki (27556Mi). Before: allocation.memory.requests = 18934Mi (68%). Adding 6Gi (6144Mi): 25078Mi = 91.0% of allocatable, ~2478Mi (~2.4GiB) of request headroom left. It admits.

3. 13Gi is supported by the body's arithmetic; the ARITHMETIC IS NOT A BLOCKER. Ceiling = 9.47GiB sandbox working set + 3.9GiB node_memory_MemAvailable_bytes = 13.37GiB, and 13Gi sits under it with the 1Gi request->limit cushion fitting beneath the same ceiling (12Gi request < 13.37GiB). 16Gi would not have fit; 13 does. Live state confirms the trigger: pod/sbx-gits-0.status.containerStatuses[name=sandbox].lastState.terminated.reason = OOMKilled, finishedAt = 2026-11-05T17:53:52Z, current limits still 10Gi/cpu6 (OnDelete, as the body says).

4. Nothing else regresses. Probes, both image digests (sbx-fleet:6fa150a@sha256:0ee5e97e…, android-emulator:230756a@sha256:66430940…), replicas: 1, the emulator container, avd (sizeLimit 16Gi), work (4Gi), herdr-sock (64Mi) are byte-identical.

5. The body states the trade honestly — a contained sandbox kill at its own cgroup limit rather than a Talos OOMController cascade onto the BestEffort Longhorn pods. No overclaim found; the excluded tmp/ephemeral work is disclosed and deferred.

Concerns (non-blocking)

• C1 — the PR title contradicts the diff. Title says "memory 16Gi/12Gi and CPU limit 12"; the diff and body are 13Gi. A later GitOps audit reads the title. Correct it before merge or in the merge message.
• C2 — containment margin is ~0.4GiB, not a guarantee. At 13Gi the sandbox can reach within ~0.37GiB of the stated 13.37GiB ceiling, and containers[name=emulator] still has headroom (requests 6Gi / limits 8Gi memory, working set 6.02GiB) — if both grow together the node can still reach pressure and the kernel OOM killer can still prefer the BestEffort Longhorn pods, which is the outcome this PR exists to avoid. Also, memory requests land at 91% of allocatable, leaving ~2.4GiB to admit anything else on prod-wk-01. Separately, the body's excluded-work rationale refers to "a 34 GiB container ephemeral-storage limit" that does not exist in this manifest (sandbox 10Gi, emulator 16Gi); it is an intended figure and should be worded as such in the follow-up. None of this blocks: 13Gi is measured-supported, 16Gi is not.

APPROVE at 61147ada7177e84a55715fd6e242055358e4aabb. Not merged by me — landing is a separate worker's job.

**Verdict of record — ld-reviewer — head 61147ada7177e84a55715fd6e242055358e4aabb: APPROVE** model: omniroute/deepseek-flash session: orch12-review-851-ld-reviewer ## What I verified (object + field) 1. **Diff is exactly three values.** 1 file, 1 hunk, 3 additions / 3 deletions, all in `core/manifests/apps/sbx-gits/statefulset.yaml` at `spec...containers[name=sandbox].resources`: `requests.memory` 6Gi -> 12Gi, `limits.cpu` "6" -> "12", `limits.memory` 10Gi -> 13Gi. Nothing else moved in the hunk or the file. `ephemeral-storage` request 1Gi / limit 10Gi unchanged in both containers. 2. **The request fits prod-wk-01.** `node/prod-wk-01.status.allocatable.memory` = 28217332Ki (27556Mi). Before: `allocation.memory.requests` = 18934Mi (68%). Adding 6Gi (6144Mi): 25078Mi = **91.0% of allocatable**, ~2478Mi (~2.4GiB) of request headroom left. It admits. 3. **13Gi is supported by the body's arithmetic; the ARITHMETIC IS NOT A BLOCKER.** Ceiling = 9.47GiB sandbox working set + 3.9GiB `node_memory_MemAvailable_bytes` = 13.37GiB, and 13Gi sits under it with the 1Gi request->limit cushion fitting beneath the same ceiling (12Gi request < 13.37GiB). 16Gi would not have fit; 13 does. Live state confirms the trigger: `pod/sbx-gits-0.status.containerStatuses[name=sandbox].lastState.terminated.reason = OOMKilled`, `finishedAt = 2026-11-05T17:53:52Z`, current limits still 10Gi/cpu6 (OnDelete, as the body says). 4. **Nothing else regresses.** Probes, both image digests (`sbx-fleet:6fa150a@sha256:0ee5e97e…`, `android-emulator:230756a@sha256:66430940…`), `replicas: 1`, the `emulator` container, `avd` (sizeLimit 16Gi), `work` (4Gi), `herdr-sock` (64Mi) are byte-identical. 5. **The body states the trade honestly** — a contained sandbox kill at its own cgroup limit rather than a Talos `OOMController` cascade onto the BestEffort Longhorn pods. No overclaim found; the excluded tmp/ephemeral work is disclosed and deferred. ## Concerns (non-blocking) - **C1 — the PR title contradicts the diff.** Title says "memory 16Gi/12Gi and CPU limit 12"; the diff and body are **13Gi**. A later GitOps audit reads the title. Correct it before merge or in the merge message. - **C2 — containment margin is ~0.4GiB, not a guarantee.** At 13Gi the sandbox can reach within ~0.37GiB of the stated 13.37GiB ceiling, and `containers[name=emulator]` still has headroom (requests 6Gi / limits 8Gi memory, working set 6.02GiB) — if both grow together the node can still reach pressure and the kernel OOM killer can still prefer the BestEffort Longhorn pods, which is the outcome this PR exists to avoid. Also, memory requests land at 91% of allocatable, leaving ~2.4GiB to admit anything else on prod-wk-01. Separately, the body's excluded-work rationale refers to "a 34 GiB container ephemeral-storage limit" that does not exist in this manifest (sandbox 10Gi, emulator 16Gi); it is an intended figure and should be worded as such in the follow-up. None of this blocks: 13Gi is measured-supported, 16Gi is not. **APPROVE** at `61147ada7177e84a55715fd6e242055358e4aabb`. Not merged by me — landing is a separate worker's job.

pi-landing merged commit d35deab13f into live 2026-11-05 18:12:43 +00:00

pi-landing referenced this pull request from a commit 2026-11-05 18:12:45 +00:00

Merge pull request 'HOMELAB-2479: sbx-gits sandbox memory 13Gi limit / 12Gi request and CPU limit 12' (#851) from plane/HOMELAB-2479-sbx-resources into live

ld-reviewer referenced this pull request 2026-11-05 18:23:56 +00:00

lab-director: sandbox gate for the cockpit dev route #850

aaron referenced this pull request 2026-11-05 18:43:36 +00:00

HOMELAB-2682: fix(sbx-gits): seat transcripts move to the persistent volume (TMPDIR) — PR B of the evidence fixes #854

aaron referenced this pull request 2026-11-05 19:39:18 +00:00

HOMELAB-2682: fix(sbx-gits): the ledger says which class ran and keeps each run's results (PR A of the evidence fixes) #853

aaron referenced this pull request 2026-11-05 19:39:18 +00:00

HOMELAB-2682: fix(sbx-gits): seat transcripts move to the persistent volume (TMPDIR) — PR B of the evidence fixes #854

ld-reviewer referenced this pull request 2026-11-05 20:00:02 +00:00

HOMELAB-2682: fix(sbx-gits): the ledger says which class ran and keeps each run's results (PR A of the evidence fixes) #853

ld-reviewer referenced this pull request 2026-11-05 20:11:04 +00:00

HOMELAB-2682: fix(sbx-gits): the ledger says which class ran and keeps each run's results (PR A of the evidence fixes) #853

pi-landing referenced this pull request 2026-11-05 21:58:54 +00:00

HOMELAB-2682: fix(sbx-gits): proof check anchored to the smoke committed at the head #859

aaron referenced this pull request from a commit 2026-11-06 00:51:15 +00:00

HOMELAB-2682: fix(sbx-gits): the verdict says WHICH suite passed — smoke vs other suites

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

operator-review

Requires human operator review before merge — dangerous class (RBAC, secrets, network policy, Terraform) or explicitly gated. Never auto-merge.

No labels

operator-review

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Homelab/infra-core!851

No description provided.