HOMELAB-2479: fix(sbx-gits): readiness probe on the GUEST's boot state + scratch off the tmp emptyDir (prepare only, HOLD for the clean point) #860

Merged

pi-landing merged 2 commits from plane/HOMELAB-2479-sbx-gits-probe-tmp into live 2026-11-09 06:11:43 +00:00

Conversation 13 Commits 2 Files changed 3 +230 -13

pi-landing commented 2026-11-05 22:05:07 +00:00

Collaborator

Copy link

What this is

Two measured defects in sbx-gits — the pod lied about the guest, and /tmp was on the
eviction path — fixed in ONE pod template: core/manifests/apps/sbx-gits/statefulset.yaml,
plus the image default that pairs with the second half
(core/manifests/apps/sbx-fleet/Dockerfile), plus the epic harness re-pin
(scripts/gits-sbx-epic-test.sh) that seat 23693's round-2 verdict
(B1-860r2-harness-pin) proved mandatory: the manifest moved, the pins did not, and the
merged tree handed every follow-up ticket a broken baseline. The harness commit is
ad45613a; the manifest/image commit is unchanged (f3776d4b).

At the manifest layer, nothing outside sbx-gits moves — sbx-lab and sbx-sendrast
render BYTE-IDENTICAL (proof below, not a promise). The Dockerfile half does change the
sbx-fleet image default consumed by other sandboxes — at their next digest bump
only; that exposure is itemized under "What this does NOT do" (it is how concern C-3
from seat 23693 is folded in: "nothing outside sbx-gits moves" was manifest-only true).

This PR prepares; it does not activate. updateStrategy is OnDelete, so merging this
rolls nothing and changes nothing that runs. The probe and the env vars take effect at the
next clean-point pod recreation, which belongs to gits-hypervisor and needs a warning.
Until then pod/sbx-gits-0 keeps reading Ready with no probe on the guest, exactly as
today.

---

A. Readiness on the GUEST's state, never on the process

The defect, measured

pod/sbx-gits-0 .spec.containers[sandbox|emulator] carried no probe of any kind (the
only probe on the pod is the pre-existing emu-health sidecar's httpGet /healthz, which
says nothing about the guest). Consequence: the pod read 2/2 Ready 42 s BEFORE the guest
booted, so a crashed or half-booted emulator was indistinguishable from a healthy one and
"the emulator is degraded" could not be read from Kubernetes at all.

The probe

readinessProbe:
  exec:
    command:
      - /bin/bash
      - -c
      - >-
        test "$(adb -s emulator-5554 shell getprop sys.boot_completed 2>/dev/null | tr -d '\r')" = "1"
  initialDelaySeconds: 60
  periodSeconds: 15
  timeoutSeconds: 10
  successThreshold: 1
  failureThreshold: 3

The adb invocation is not invented — it is the container's own, from step 4 of the
emulator container's boot loop in the same file, and the smoke's convention is one
serial, auto-discovered, never aliased — so the probe uses the direct serial and no
adb connect / 127.0.0.1:5555 (a second registration of the same guest is what kills the
instrumented run, R136/R138). tr -d '\r' because the guest answers CRLF. The probe runs
as the container's own user (root) with the same writable HOME, and it uses the pod's
existing adb server — no second server, no extra listener (the "nothing new listens"
invariant in T7(h) still holds: the probe is a localhost serial, not a socket).

Readiness, not startup — one line

A startupProbe's budget expiring is a KILL (kubelet restarts the container), which is
exactly "kill the emulator because a guest is slow"; a readinessProbe has no kill path —
it can only report the truth — and the container's own loop already owns boot failure with a
900 s deadline and its own EMU-DIED / EMU-TIMEOUT exits.

Timing arithmetic (measured, not estimated)

Cold boots of this container, from its own log: EMU-READY ... after 76s and, previous
container, 94s.

• first probe at t = 60 s — inside the fastest healthy boot, so no probe ever runs
  against a guest that has not had its chance to start;
• then every 15 s; 3 consecutive failures → NotReady at 60 + 3×15 = 105 s;
• a 76 s or 94 s boot never produces 3 failures, so a healthy boot never flips this pod
  NotReady;
• budget: 105 s to a stable NotReady, 0 restarts possible however long the boot takes.
  timeoutSeconds: 10 is above a healthy getprop round trip; a hung adb is a failed
  attempt, never a hung probe.

Concern C-2 (seat 23693), folded in honestly: the probe's steady-state load is
unmeasured. The exec is one adb getprop per 15 s against the container's own adb server —
by construction cheap — but no wall-clock sampling of probe latency under a running
instrumented suite exists yet. That measurement belongs to the post-recreation verification
wave (same wave that proves the first real sys.boot_completed under the probe); it is not
a reason to hold a prepare-only PR that rolls nothing.

No liveness probe — argued, not implemented

A liveness probe here would restart the emulator whenever the guest is slow — not
implemented, on purpose.

---

B. /tmp off the eviction path

The defect, measured

kubectl_evictions on prod-wk-01 has exactly one sample in seven days: the 12:20Z
run, when this pod's tmp emptyDir hit its 4Gi limit. TMPDIR is set nowhere today —
not in the pod env, not in the manifest, and not in the image — and the JVM ignores
TMPDIR entirely, so a TMPDIR-only fix would silently do nothing for the
Gradle/Kotlin daemons.

The change

1. tmp emptyDir 4Gi → 8Gi.
2. JAVA_TOOL_OPTIONS=-Djava.io.tmpdir=/home/ld/work/tmp added to the sandbox
   container env. This is the ONLY env var this PR adds. The rebased head deliberately
   does not redeclare TMPDIR: it already lives at the HOMELAB-2682 site
   (/home/ld/work/seat-transcripts, declared above in the same env) and a second entry
   would shadow it — the harness's own parse resolves the LAST value and test 28 goes red.
   Seats' transcripts keep going to seat-transcripts; Gradle/Kotlin scratch goes to
   work/tmp; both are on the persistent hostPath, neither is on the tmp emptyDir.
3. The same two values as ENV in core/manifests/apps/sbx-fleet/Dockerfile immediately
   before USER 1000:1000, so the default is carried by the image for consumers whose
   digest is built after this layer. ENV is pod-scoped in Docker, not user-scoped — it is
   placed before USER because that is where this file declares the runtime user. The
   manifest repeats JAVA_TOOL_OPTIONS because the pinned digest
   (sbx-fleet:6fa150a@sha256:0ee5e97e…) predates this layer.
4. /home/ld/work/tmp is created by the prepare-work initContainer
   (install -d -m 0755 -o 1000 -g 1000 "$DST/tmp") — as root, idempotently, handed to
   1000:1000, because the hostPath arrives root:root 0755 and the runtime user is 1000.
   Not created in the image: a RUN mkdir under /home/ld/work is covered by the mounted
   volume, and the runtime rootfs is read-only.

Why this is the real fix and the 8Gi only the buffer: work is a hostPath on prod-wk-01's
own data disk (~425 GiB free) and sits outside kubelet's ephemeral accounting entirely.
Scratch written there cannot evict any pod. It also means a crash DB is no longer destroyed
with the container.

Why 8Gi and not 16Gi

The pod's ephemeral ceilings are already 20.06 GiB — tmp 4 + avd 16 + herdr-sock
64Mi — against 31.0 GiB free on prod-wk-01. 8Gi → 24.06 GiB, still under 31.0 and
under the sandbox container's own ephemeral-storage limit (10Gi), so the failure stays
where it belongs: this pod's own clean eviction. 16Gi → 32.06 GiB, above the
free space: the same overflow would instead trigger a node-level DiskPressure eviction
hitting every pod on prod-wk-01. avd (16Gi) and herdr-sock (64Mi) are untouched.

---

Render proof — repo path, current head

Render path: kubectl kustomize <dir> — the same render ArgoCD performs for the three
Applications. before = origin/live @ 1c5111d1 (this branch's base, current tip);
after = this branch @ ad45613a.

Byte-identical check for the other two sandboxes (the hard gate)

sha256(render core/manifests/apps/sbx-lab)
  before/after  8c8f61d79b30945c…  936290 B   (equal)
sha256(render core/manifests/apps/sbx-sendrast)
  before/after  d225759f534949db…  131099 B   (equal)

sbx-gits: 4b0fa50cd88e2ed9… → ba858643337c6429…, 1279115 → 1280544 B.

Complete semantic delta (sbx-gits only)

Every added/changed leaf, nothing summarised away:

+ StatefulSet/sbx-gits:.spec.template.spec.initContainers[0].args[0]
    = <old script + `install -d -m 0755 -o 1000 -g 1000 "$DST/tmp"` + its comment>
+ StatefulSet/sbx-gits:.spec.template.spec.containers[0](sandbox).env
    = JAVA_TOOL_OPTIONS: -Djava.io.tmpdir=/home/ld/work/tmp   (ONE new env var; TMPDIR is NOT redeclared)
+ StatefulSet/sbx-gits:.spec.template.spec.containers[1](emulator).readinessProbe
    = <the 8 probe leaves of section A>
~ StatefulSet/sbx-gits:.spec.template.spec.volumes[tmp].emptyDir.sizeLimit = '4Gi' -> '8Gi'

No other object in the directory changed, and no other leaf of any object changed. The
pre-existing emu-health sidecar (its httpGet /healthz probe included) is byte-untouched
— the harness re-pin below pins it as untouchable.

---

The harness re-pin (scripts/gits-sbx-epic-test.sh, commit ad45613a)

Seat 23693 round-2 (B1-860r2-harness-pin): the branch updated the manifest without
updating the harness pins — 7 new harness failures on the merged tree
(env-anadidos: JAVA_TOOL_OPTIONS at the spine gate + 4 mutation knock-ons +
T7[diff]/17[R159](d) ×2). Precedent HOMELAB-2682 updated pins in the SAME change; this
head does too, same pattern, smallest diff:

1. Env pin: d_extra_env 'TMPDIR' → 'TMPDIR JAVA_TOOL_OPTIONS' — the name is
   allowed at the gate, the VALUE stays pinned by the manifest (same division as
   HOMELAB-2682).
2. Diff set: SBX_GITS_DIFF_SET re-measured against the current tip: the ok0 entry
   (M tools/gits-sbx-emu-proof.sh) leaves the set — live already absorbs it — so the
   branch diff is exactly M statefulset.yaml. The set is re-measured against the
   current tip, never dragged along.
3. Probe pin (new mechanism, same division of work): p1_spine now strips
   readinessProbe (with the mapping rule — a probe body is sibling keys, not list items)
   and a new p1_rp pins the exact block set by sha256: by default no probe may move
   (the pre-existing emu-health httpGet probe is untouchable), and the wave that
   legitimately adds one passes its p1_rp hash explicitly — #860 pins the emulator's
   sys.boot_completed probe. Without this the probe would trip spine-distinto forever,
   and any hand-added probe anywhere would have been invisible to the old spine.
   T7[diff]/17R159 pass messages re-measured to name the #860 set.

Verification (both trees, same machine): live tree (origin/live @ 1c5111d1, with the
re-pinned harness): 13 failed / 693 checks — every failure environmental to that runner
(kubectl-version render deltas, /home/ld perms), none from this diff. Merged tree
(ad45613a): the identical 13 failures, zero new — the 7 seat-23693 failures all pass:
H8[8][sbx-gits] green, [mut] baseline green with all 10 mutation rows biting,
T7[diff]/17[R159](d) green with their entry-removal mutations red. Test 28 (TMPDIR
transcripts): 8/8 PASS on this head.

---

Validation

kubectl kustomize rc=0 for all three sandboxes. Against the live cluster, without writing
anything (previous head; content of these files unchanged by the rebase):

$ kubectl -n sbx-gits apply --dry-run=server -f <rendered StatefulSet>
statefulset.apps/sbx-gits configured (server dry run)

statefulset/sbx-gits .metadata.generation unchanged (21) across the dry-run; the live pod
was not touched (creationTimestamp 2026-11-05T18:15:39Z, emulator restartCount 2,
before and after). No pod was deleted, scaled, edited or recreated by this work.

---

What this does NOT do, and what has to happen for it to be testable

• It does not roll the pod (updateStrategy: OnDelete). The probe's effect is testable
  only after a clean-point recreation — gits-hypervisor's, with a warning, per the
  order. That recreation is also when the harness's official run closes the ticket.
• No image tag or digest is bumped. The sbx-fleet ENV layer reaches no pod until a
  future image build AND digest bump.
• C-1 (seat 23693) — resolution: explicit deferral, blast radius re-measured by seat
  23722. The image ENV TMPDIR=/home/ld/work/tmp (+JAVA_TOOL_OPTIONS) can reach only
  containers running the sbx-fleet image at their next digest bump — and measured on
  this head, that hazard set is empty. The pod's emulator container is NOT an sbx-fleet
  consumer: it pins android-emulator:230756a (platform/android-emulator/Dockerfile,
  FROM debian:bookworm-slim), so the ENV layer can never touch it and test 28's "emulator
  stays on ephemeral /tmp" stands. The four containers here that DO run the sbx-fleet
  image — prepare-work, seed-pi-agent, sandbox, emu-health — all mount
  /home/ld/work, so the ENV values always name a real path for them. Neutralizing now
  would mean re-touching the manifest for zero immediate benefit: nothing rolls before the
  clean-point recreation, and the pod pins digest 6fa150a, which predates the ENV layer
  entirely — no running container can see this value between merge and bump. The
  deferral decision itself is unchanged (OnDelete; digest 6fa150a predates the layer;
  nothing rolls before the recreation) — seat 23722 corrected only the reasoning map, and
  the one real bump-gated follow-up is itemized below.
• Known follow-ups for the other sbx-fleet consumers (deliberately not touched here):
  • sbx-lab — the ONLY bump-gated follow-up (verified by seat 23722): it pins
    sbx-fleet:fc933fb, so it inherits the image ENV at its next digest bump, and its
    prepare-work creates no $DST/tmp (zero hits in its script). Its bump follow-up
    MUST therefore create that directory (or override TMPDIR/JAVA_TOOL_OPTIONS per
    container) in the same change, before any pod built from the new digest rolls; its
    manifest should also override TMPDIR back to its own transcripts path if it wants
    one.
  • sbx-sendrast is NOT a consumer (seat 23722, correcting this body's prior claim):
    sbx-sendrast/Dockerfile:26-27 is FROM ${LD_FLEET_IMAGE} (ld-fleet:go-89b900d) —
    a different image line from sbx-fleet — so the image ENV, and the /home/jesus/work
    note built on the wrong premise, do not apply. No follow-up owed there.
• Known limitations (seat 23722 concerns, folded in):
  • The install -d step added to the initContainer args is paired with a comment; the
    comment documents intent, it is not an enforced contract.
  • p1_spine has a latent env-block-termination path in its parsing; it fails closed
    (an unterminated env block is treated as a mismatch), never open.
  • p1_rp pins the probe block but normalizes away the owning container — a probe moved
    between containers at the same manifest path would keep the same hash.
  • The readiness flip withdraws the pod from its Service's ready set while NotReady, and
    prod sandbox-telemetry's static target is that Service name — a guest hung >15 min
    would trip SandboxEmulatorTelemetryAbsent. That is the probe reporting the truth,
    but the alert's static target predates the probe and is a known sharp edge for the
    recreation wave.
• Not verified: a real clean-point recreation with the new probe and the guest's first
  sys.boot_completed under it (cannot be done without recreating the pod, which this PR
  must not do); the probe's steady-state load under a running suite (C-2, above); whether
  the next cold boot stays under the 105 s budget on a fresh /avd.

Written from primary sources: the manifest, the container's own log, application/sbx-gits,
node/prod-wk-01, and kubectl kustomize. No credential was printed, hashed or placed in
argv; the branch was pushed as pi-landing after asserting GET /api/v1/user.

## What this is Two measured defects in `sbx-gits` — the pod lied about the guest, and `/tmp` was on the eviction path — fixed in ONE pod template: `core/manifests/apps/sbx-gits/statefulset.yaml`, plus the image default that pairs with the second half (`core/manifests/apps/sbx-fleet/Dockerfile`), plus the epic harness re-pin (`scripts/gits-sbx-epic-test.sh`) that seat 23693's round-2 verdict (`B1-860r2-harness-pin`) proved mandatory: the manifest moved, the pins did not, and the merged tree handed every follow-up ticket a broken baseline. The harness commit is `ad45613a`; the manifest/image commit is unchanged (`f3776d4b`). **At the manifest layer, nothing outside `sbx-gits` moves** — `sbx-lab` and `sbx-sendrast` render BYTE-IDENTICAL (proof below, not a promise). The Dockerfile half does change the `sbx-fleet` image *default* consumed by other sandboxes — at their **next digest bump only**; that exposure is itemized under "What this does NOT do" (it is how concern C-3 from seat 23693 is folded in: "nothing outside sbx-gits moves" was manifest-only true). **This PR prepares; it does not activate.** `updateStrategy` is `OnDelete`, so merging this rolls nothing and changes nothing that runs. The probe and the env vars take effect at the next **clean-point pod recreation**, which belongs to gits-hypervisor and needs a warning. Until then `pod/sbx-gits-0` keeps reading Ready with no probe on the guest, exactly as today. --- ## A. Readiness on the GUEST's state, never on the process ### The defect, measured `pod/sbx-gits-0 .spec.containers[sandbox|emulator]` carried **no probe of any kind** (the only probe on the pod is the pre-existing `emu-health` sidecar's `httpGet /healthz`, which says nothing about the guest). Consequence: the pod read **2/2 Ready 42 s BEFORE the guest booted**, so a crashed or half-booted emulator was indistinguishable from a healthy one and "the emulator is degraded" could not be read from Kubernetes at all. ### The probe ```yaml readinessProbe: exec: command: - /bin/bash - -c - >- test "$(adb -s emulator-5554 shell getprop sys.boot_completed 2>/dev/null | tr -d '\r')" = "1" initialDelaySeconds: 60 periodSeconds: 15 timeoutSeconds: 10 successThreshold: 1 failureThreshold: 3 ``` **The `adb` invocation is not invented — it is the container's own**, from step 4 of the `emulator` container's boot loop in the same file, and the smoke's convention is one serial, auto-discovered, never aliased — so the probe uses the **direct serial** and no `adb connect` / `127.0.0.1:5555` (a second registration of the same guest is what kills the instrumented run, R136/R138). `tr -d '\r'` because the guest answers CRLF. The probe runs as the container's own user (root) with the same writable `HOME`, and it uses the pod's existing adb server — no second server, no extra listener (the "nothing new listens" invariant in T7(h) still holds: the probe is a localhost serial, not a socket). ### Readiness, not startup — one line **A startupProbe's budget expiring is a KILL** (kubelet restarts the container), which is exactly "kill the emulator because a guest is slow"; a **readinessProbe has no kill path** — it can only report the truth — and the container's own loop already owns boot failure with a 900 s deadline and its own `EMU-DIED` / `EMU-TIMEOUT` exits. ### Timing arithmetic (measured, not estimated) Cold boots of **this** container, from its own log: `EMU-READY ... after 76s` and, previous container, `94s`. - first probe at **t = 60 s** — inside the fastest healthy boot, so no probe ever runs against a guest that has not had its chance to start; - then every **15 s**; **3** consecutive failures → NotReady at **60 + 3×15 = 105 s**; - a 76 s or 94 s boot never produces 3 failures, so **a healthy boot never flips this pod NotReady**; - **budget: 105 s to a stable NotReady, 0 restarts possible however long the boot takes.** `timeoutSeconds: 10` is above a healthy `getprop` round trip; a hung `adb` is a failed attempt, never a hung probe. **Concern C-2 (seat 23693), folded in honestly:** the probe's steady-state *load* is unmeasured. The exec is one `adb getprop` per 15 s against the container's own adb server — by construction cheap — but no wall-clock sampling of probe latency under a running instrumented suite exists yet. That measurement belongs to the post-recreation verification wave (same wave that proves the first real `sys.boot_completed` under the probe); it is not a reason to hold a prepare-only PR that rolls nothing. ### No liveness probe — argued, not implemented A liveness probe here would restart the emulator whenever the guest is slow — not implemented, on purpose. --- ## B. `/tmp` off the eviction path ### The defect, measured `kubectl_evictions` on `prod-wk-01` has exactly **one** sample in seven days: the 12:20Z run, when this pod's `tmp` emptyDir hit its **4Gi** limit. **`TMPDIR` is set nowhere today — not in the pod env, not in the manifest, and not in the image** — and **the JVM ignores `TMPDIR`** entirely, so a `TMPDIR`-only fix would silently do nothing for the Gradle/Kotlin daemons. ### The change 1. `tmp` emptyDir `4Gi → 8Gi`. 2. **`JAVA_TOOL_OPTIONS=-Djava.io.tmpdir=/home/ld/work/tmp`** added to the `sandbox` container env. **This is the ONLY env var this PR adds.** The rebased head deliberately does **not** redeclare `TMPDIR`: it already lives at the HOMELAB-2682 site (`/home/ld/work/seat-transcripts`, declared above in the same env) and a second entry would shadow it — the harness's own parse resolves the LAST value and test 28 goes red. Seats' transcripts keep going to `seat-transcripts`; Gradle/Kotlin scratch goes to `work/tmp`; both are on the persistent hostPath, neither is on the `tmp` emptyDir. 3. The same two values as `ENV` in `core/manifests/apps/sbx-fleet/Dockerfile` **immediately before `USER 1000:1000`**, so the default is carried by the image for consumers whose digest is built after this layer. `ENV` is pod-scoped in Docker, not user-scoped — it is placed before `USER` because that is where this file declares the runtime user. The manifest repeats `JAVA_TOOL_OPTIONS` because **the pinned digest (`sbx-fleet:6fa150a@sha256:0ee5e97e…`) predates this layer**. 4. `/home/ld/work/tmp` is **created by the `prepare-work` initContainer** (`install -d -m 0755 -o 1000 -g 1000 "$DST/tmp"`) — as root, idempotently, handed to `1000:1000`, because the hostPath arrives `root:root 0755` and the runtime user is 1000. Not created in the image: a `RUN mkdir` under `/home/ld/work` is covered by the mounted volume, and the runtime rootfs is read-only. Why this is the real fix and the 8Gi only the buffer: **`work` is a hostPath on prod-wk-01's own data disk (~425 GiB free) and sits outside kubelet's ephemeral accounting entirely.** Scratch written there cannot evict any pod. It also means a crash DB is no longer destroyed with the container. ### Why 8Gi and not 16Gi The pod's ephemeral ceilings are already **20.06 GiB** — `tmp` 4 + `avd` 16 + `herdr-sock` 64Mi — against **31.0 GiB free** on prod-wk-01. **8Gi** → 24.06 GiB, still under 31.0 and under the `sandbox` container's own `ephemeral-storage` limit (10Gi), so the failure stays where it belongs: **this pod's own clean eviction**. **16Gi** → 32.06 GiB, **above** the free space: the same overflow would instead trigger a **node-level `DiskPressure`** eviction hitting every pod on prod-wk-01. `avd` (16Gi) and `herdr-sock` (64Mi) are untouched. --- ## Render proof — repo path, current head Render path: `kubectl kustomize <dir>` — the same render ArgoCD performs for the three Applications. `before` = `origin/live` @ **1c5111d1** (this branch's base, current tip); `after` = this branch @ **ad45613a**. ### Byte-identical check for the other two sandboxes (the hard gate) ``` sha256(render core/manifests/apps/sbx-lab) before/after 8c8f61d79b30945c… 936290 B (equal) sha256(render core/manifests/apps/sbx-sendrast) before/after d225759f534949db… 131099 B (equal) ``` `sbx-gits`: `4b0fa50cd88e2ed9…` → `ba858643337c6429…`, 1279115 → 1280544 B. ### Complete semantic delta (`sbx-gits` only) Every added/changed leaf, nothing summarised away: ``` + StatefulSet/sbx-gits:.spec.template.spec.initContainers[0].args[0] = <old script + `install -d -m 0755 -o 1000 -g 1000 "$DST/tmp"` + its comment> + StatefulSet/sbx-gits:.spec.template.spec.containers[0](sandbox).env = JAVA_TOOL_OPTIONS: -Djava.io.tmpdir=/home/ld/work/tmp (ONE new env var; TMPDIR is NOT redeclared) + StatefulSet/sbx-gits:.spec.template.spec.containers[1](emulator).readinessProbe = <the 8 probe leaves of section A> ~ StatefulSet/sbx-gits:.spec.template.spec.volumes[tmp].emptyDir.sizeLimit = '4Gi' -> '8Gi' ``` No other object in the directory changed, and no other leaf of any object changed. The pre-existing `emu-health` sidecar (its `httpGet /healthz` probe included) is byte-untouched — the harness re-pin below pins it as untouchable. --- ## The harness re-pin (`scripts/gits-sbx-epic-test.sh`, commit `ad45613a`) Seat 23693 round-2 (`B1-860r2-harness-pin`): the branch updated the manifest without updating the harness pins — 7 new harness failures on the merged tree (`env-anadidos: JAVA_TOOL_OPTIONS` at the spine gate + 4 mutation knock-ons + `T7[diff]`/`17[R159](d)` ×2). Precedent HOMELAB-2682 updated pins in the SAME change; this head does too, same pattern, smallest diff: 1. **Env pin**: `d_extra_env` `'TMPDIR'` → `'TMPDIR JAVA_TOOL_OPTIONS'` — the name is allowed at the gate, the VALUE stays pinned by the manifest (same division as HOMELAB-2682). 2. **Diff set**: `SBX_GITS_DIFF_SET` re-measured against the current tip: the ok0 entry (`M tools/gits-sbx-emu-proof.sh`) leaves the set — live already absorbs it — so the branch diff is exactly **`M statefulset.yaml`**. The set is re-measured against the current tip, never dragged along. 3. **Probe pin (new mechanism, same division of work)**: `p1_spine` now strips `readinessProbe` (with the mapping rule — a probe body is sibling keys, not list items) and a new `p1_rp` pins the exact block set by sha256: by default **no probe may move** (the pre-existing `emu-health` httpGet probe is untouchable), and the wave that legitimately adds one passes its `p1_rp` hash explicitly — #860 pins the emulator's `sys.boot_completed` probe. Without this the probe would trip `spine-distinto` forever, and any hand-added probe anywhere would have been invisible to the old spine. T7[diff]/17[R159](d) pass messages re-measured to name the #860 set. **Verification (both trees, same machine):** live tree (`origin/live` @ 1c5111d1, with the re-pinned harness): 13 failed / 693 checks — every failure environmental to that runner (kubectl-version render deltas, `/home/ld` perms), none from this diff. Merged tree (ad45613a): **the identical 13 failures, zero new** — the 7 seat-23693 failures all pass: `H8[8][sbx-gits]` green, `[mut]` baseline green with all 10 mutation rows biting, `T7[diff]`/`17[R159](d)` green with their entry-removal mutations red. Test 28 (TMPDIR transcripts): 8/8 PASS on this head. --- ## Validation `kubectl kustomize` rc=0 for all three sandboxes. Against the live cluster, without writing anything (previous head; content of these files unchanged by the rebase): ``` $ kubectl -n sbx-gits apply --dry-run=server -f <rendered StatefulSet> statefulset.apps/sbx-gits configured (server dry run) ``` `statefulset/sbx-gits .metadata.generation` unchanged (21) across the dry-run; the live pod was not touched (`creationTimestamp` 2026-11-05T18:15:39Z, emulator `restartCount` 2, before and after). No pod was deleted, scaled, edited or recreated by this work. --- ## What this does NOT do, and what has to happen for it to be testable - **It does not roll the pod** (`updateStrategy: OnDelete`). The probe's effect is testable only after a **clean-point recreation** — gits-hypervisor's, with a warning, per the order. That recreation is also when the harness's official run closes the ticket. - **No image tag or digest is bumped.** The `sbx-fleet` `ENV` layer reaches no pod until a future image build AND digest bump. - **C-1 (seat 23693) — resolution: explicit deferral, blast radius re-measured by seat 23722.** The image `ENV TMPDIR=/home/ld/work/tmp` (+`JAVA_TOOL_OPTIONS`) can reach only containers running the `sbx-fleet` image at their next digest bump — **and measured on this head, that hazard set is empty.** The pod's `emulator` container is NOT an sbx-fleet consumer: it pins `android-emulator:230756a` (`platform/android-emulator/Dockerfile`, `FROM debian:bookworm-slim`), so the ENV layer can never touch it and test 28's "emulator stays on ephemeral /tmp" stands. The four containers here that DO run the `sbx-fleet` image — `prepare-work`, `seed-pi-agent`, `sandbox`, `emu-health` — all mount `/home/ld/work`, so the ENV values always name a real path for them. Neutralizing *now* would mean re-touching the manifest for zero immediate benefit: nothing rolls before the clean-point recreation, and the pod pins digest `6fa150a`, which predates the ENV layer entirely — **no running container can see this value between merge and bump**. The deferral decision itself is unchanged (OnDelete; digest `6fa150a` predates the layer; nothing rolls before the recreation) — seat 23722 corrected only the reasoning map, and the one real bump-gated follow-up is itemized below. - **Known follow-ups for the other `sbx-fleet` consumers (deliberately not touched here):** - **`sbx-lab`** — the ONLY bump-gated follow-up (verified by seat 23722): it pins `sbx-fleet:fc933fb`, so it inherits the image ENV at its next digest bump, and its `prepare-work` creates **no** `$DST/tmp` (zero hits in its script). Its bump follow-up MUST therefore create that directory (or override `TMPDIR`/`JAVA_TOOL_OPTIONS` per container) in the same change, before any pod built from the new digest rolls; its manifest should also override `TMPDIR` back to its own transcripts path if it wants one. - **`sbx-sendrast` is NOT a consumer** (seat 23722, correcting this body's prior claim): `sbx-sendrast/Dockerfile:26-27` is `FROM ${LD_FLEET_IMAGE}` (`ld-fleet:go-89b900d`) — a different image line from `sbx-fleet` — so the image ENV, and the `/home/jesus/work` note built on the wrong premise, do not apply. No follow-up owed there. - **Known limitations (seat 23722 concerns, folded in):** - The `install -d` step added to the initContainer args is paired with a comment; the comment documents intent, it is not an enforced contract. - `p1_spine` has a latent env-block-termination path in its parsing; it fails **closed** (an unterminated env block is treated as a mismatch), never open. - `p1_rp` pins the probe block but normalizes away the owning container — a probe moved between containers at the same manifest path would keep the same hash. - The readiness flip withdraws the pod from its Service's ready set while NotReady, and prod `sandbox-telemetry`'s static target is that Service name — a guest hung >15 min would trip `SandboxEmulatorTelemetryAbsent`. That is the probe reporting the truth, but the alert's static target predates the probe and is a known sharp edge for the recreation wave. - **Not verified:** a real clean-point recreation with the new probe and the guest's first `sys.boot_completed` under it (cannot be done without recreating the pod, which this PR must not do); the probe's steady-state load under a running suite (C-2, above); whether the next cold boot stays under the 105 s budget on a *fresh* `/avd`. Written from primary sources: the manifest, the container's own log, `application/sbx-gits`, `node/prod-wk-01`, and `kubectl kustomize`. No credential was printed, hashed or placed in argv; the branch was pushed as `pi-landing` after asserting `GET /api/v1/user`.

pi-landing added 1 commit 2026-11-05 22:05:09 +00:00

HOMELAB-2479: fix(sbx-gits): readiness probe on the GUEST, scratch off the tmp emptyDir ... a619bdb89f

Two measured defects, one file (plus the image default that pairs with it):

A. The pod lied about the guest. `pod/sbx-gits-0 .spec.containers[*]
   .readinessProbe` was empty for BOTH containers, so the pod read 2/2 Ready
   42 s BEFORE the guest booted and a crashed or half-booted emulator was
   indistinguishable from a healthy one. This adds a readinessProbe to the
   `emulator` container that asks the GUEST — `adb -s emulator-5554 shell
   getprop sys.boot_completed` = 1, the exact call and selector the container's
   own boot loop already uses. No liveness probe, no startup probe: a
   startupProbe's budget expiring is a kill, and killing an emulator because a
   guest is slow is the failure mode we are trying to avoid.

B. /tmp was on the eviction path. The only kubelet eviction on prod-wk-01 in
   seven days is this pod's 12:20Z tmp emptyDir limit (4Gi). tmp goes to 8Gi
   (not 16: the pod's ephemeral ceilings are already 20.06 GiB against 31.0 GiB
   free on the node, so 16Gi moves the failure to a node-level eviction), and
   scratch moves OFF the emptyDir entirely: TMPDIR plus
   JAVA_TOOL_OPTIONS=-Djava.io.tmpdir= both point at the `work` hostPath (a
   subdirectory, created by the `prepare-work` initContainer), because the JVM
   ignores TMPDIR and a TMPDIR-only fix would silently do nothing for the
   Gradle/Kotlin daemons. The same two values are declared in the image before
   `USER 1000:1000`, so the default is not a manifest-only promise.

No image tag is bumped, `avd` (16Gi) and `herdr-sock` are untouched, and
`updateStrategy: OnDelete` means MERGING THIS ROLLS NOTHING: the probe and the
env take effect only at the next clean-point pod recreation, which is not this
PR's to make.

ld-reviewer requested changes 2026-11-09 03:30:29 +00:00

Dismissed

ld-reviewer left a comment

Collaborator

Copy link

Verdict: REQUEST_CHANGES

model: omniroute/deepseek-flash
head: a619bdb89f
session: ses-860r1-seat1-3f9a1c

Fresh-context review of the exact head a619bdb8, standing on my own reads: the PR diff at
that head, both changed files in full, origin/live (fc933fb3) as it stands while I read, the
render path the repo uses (kubectl kustomize core/manifests/apps/sbx-gits), and the repo's own
epic harness scripts/gits-sbx-epic-test.sh. I merged the head into live in a throwaway clone
and reviewed the merged tree, because that is the tree that runs after landing.

Scope is what it claims: 2 files, +121/−1 — core/manifests/apps/sbx-fleet/Dockerfile (+26) and
core/manifests/apps/sbx-gits/statefulset.yaml (+95/−1). No other file moves. No secret, no RBAC,
no NetworkPolicy, no egress and no identity material is touched: the identity Secret block near
the insertion is untouched context only. updateStrategy: OnDelete, so the merge itself rolls
nothing; kubectl kustomize returns rc=0 on both the head and the merged tree, and the merged
YAML parses with no duplicate mapping key. CI is green at this head (runs 3198/3199/3200, every
job passing, build-and-push included). The head is still a619bdb8, the PR is open, mergeable,
and it has no earlier review to inherit.

What is right, and what I checked rather than took on faith: the probe is the container's own call
with the container's own selector — adb -s emulator-5554 shell getprop sys.boot_completed is
byte-for-byte the boot loop's step-4 line, tr -d '\r' included, with no adb connect and no
127.0.0.1:5555; the emulator container runs runAsUser: 0 with readOnlyRootFilesystem: false, so the probe runs in the same environment the boot loop already uses, and adb
auto-starts its server from any client command. Readiness rather than startup is the right
instrument here, and the arithmetic (first probe at 60s, inside a 76s fastest measured boot;
3×15s to NotReady; no kill path) holds against the two EMU-READY samples quoted. In the merged
tree the probe lands on containers[2] name=emulator, not on the emu-health sidecar that live
added afterwards, and no container ends up with two probe blocks. tmp 4Gi→8Gi with avd and
herdr-sock untouched keeps this pod's own clean eviction as the failure mode instead of a
node-level DiskPressure, and prepare-work gets the install -d … -o 1000 -g 1000 "$DST/tmp"
line that makes its own target exist.

The finding in full

The premise the change is built on is no longer true of its base. live already declares
TMPDIR=/home/ld/work/seat-transcripts in the sandbox env — the HOMELAB-2682 block, landed as
93bbb5d4 after this branch was cut from 74724767 — documented in that directory's README
§5.2, and asserted by scripts/gits-sbx-epic-test.sh test 28. This branch still states, in the
manifest comment it adds, that TMPDIR "is set NOWHERE today (not here, not in the image, not in
the manifest)": false against fc933fb3.

Because the branch predates that line and touches a nearby region, git auto-merges cleanly and
the result carries both entries, at env indices 4 and 26 of containers[sandbox]:

TMPDIR=/home/ld/work/seat-transcripts      (what live says, HOMELAB-2682)
TMPDIR=/home/ld/work/tmp                   (what this PR adds)

Kubernetes accepts that object — no validation error, kubectl kustomize rc=0, no duplicate YAML
key — so nothing stops it, and the precedence is then runtime-defined: Docker's env builder keeps
the last value, a getenv reader over a runc-built env sees the first. I did not have to settle
that ambiguity to file this. I ran the harness's own parse over the merged tree — a dict built
name→value in list order, the same code test 28 uses — and TMPDIR resolves to
/home/ld/work/tmp, so

test 28 [HOMELAB-2682] el-contenedor-sandbox-no-declara-TMPDIR=/home/ld/work/seat-transcripts(='/home/ld/work/tmp')

goes red. That is the lane check in the bundle's own step 9, firing against the tree this PR
would leave behind. The same ambiguity reaches the reason the bundle exists: if
/home/ld/work/tmp is what the sandbox honours, seat transcripts stop landing in
home/ld/work/seat-transcripts — the directory the recreate's snapshot step copies — and README
§5.2 becomes wrong. Either way one of the two intents is silently defeated, and the merged
manifest carries a comment asserting the opposite of what live does.

The fix is small and gives up none of the PR's real gain: rebase onto live, DELETE the TMPDIR
entry this PR adds to the sandbox env (live's seat-transcripts already moves non-JVM scratch
off the tmp emptyDir, which is the stated goal), keep
JAVA_TOOL_OPTIONS=-Djava.io.tmpdir=/home/ld/work/tmp — the genuinely new half, and the one
consumer TMPDIR never reached — keep the 8Gi and the probe, and rewrite the env comment so it no
longer claims TMPDIR is unset. That needs a fresh seat on the new head, per the bundle's order.

Blockers

BLOCKER B1: landing this head on today's live leaves two conflicting TMPDIR entries in the sandbox container and reddens the repo's own epic test 28 — evidence and the one-line fix are in the section above.

Concerns

These do not hold the PR on their own; they are for the next head or for the record.

C-1 — sbx-sendrast inherits the new image ENV and has nowhere for it to point. The sbx-fleet
image is shared, and the Dockerfile sets ENV TMPDIR=/home/ld/work/tmp. sbx-lab mounts work
at /home/ld/work (only its tmp subdirectory is missing today) and the PR records that as a
known follow-up. sbx-sendrast has no /home/ld/work mount at all — it mounts /home/jesus,
/tmp, /opt/sbx-* — so a digest bump there would point TMPDIR at a path that does not exist,
and temp-file consumers that trust TMPDIR (os.TempDir, tempfile, mkstemp) fail rather than
fall back to /tmp. Nothing activates today (this PR bumps no digest, and the build workflow
publishes a tag without touching the pins), which is why this is a concern and not a hold — but
the PR's known-follow-up list names sbx-lab only, and the brief's own instruction is to leave
sbx-sendrast alone. One sentence in the Dockerfile comment would keep that trap from being
stepped on by whoever bumps that pin.

C-2 — the probe's steady state has not been measured under a suite. A readinessProbe runs for
the pod's whole life, so adb shell getprop executes every 15s indefinitely, including while an
instrumented run holds the guest. The timing argument covers boot only, and the PR's own
"not verified" list (the fresh /avd boot, the real recreation) does not mention load. Low risk,
cheap to settle: one suite run with the probe live, or a periodSeconds that backs off once the
guest is up.

C-3 — the headline "Nothing outside sbx-gits moves" is only true of the manifests. The
Dockerfile half does move the two other sandboxes' behaviour at their next digest bump (C-1). The
body qualifies this later, but the headline is the sentence a future reader will quote.

What I did not ask for

The recreate is gits-hypervisor's, not this PR's; #860 preparing without activating is the right
shape for a bundle that lands it first and recreates later. Landing it first is safe once B1 is
fixed: the merged tree then rolls nothing (OnDelete), adds no image pin, and the Dockerfile change
only publishes a tag through the existing path-filtered build, which I read in
.forgejo/workflows/sbx-fleet-image-build.yaml — that job asserts the base ARG pin, which the
merge keeps from live (go-234dfd0@sha256:a5763f7e…), so it stays green.

Reviewed read-only: no edit, no push, no merge. Identity asserted as ld-reviewer before the
write; the credential was held in a 0600 file under a 0700 directory, never in argv and never
printed, and the operator's ambient token was not used.

## Verdict: REQUEST_CHANGES model: omniroute/deepseek-flash head: a619bdb89f872d2b280537fd7bc3fed199bfdb04 session: ses-860r1-seat1-3f9a1c Fresh-context review of the exact head `a619bdb8`, standing on my own reads: the PR diff at that head, both changed files in full, `origin/live` (`fc933fb3`) as it stands while I read, the render path the repo uses (`kubectl kustomize core/manifests/apps/sbx-gits`), and the repo's own epic harness `scripts/gits-sbx-epic-test.sh`. I merged the head into `live` in a throwaway clone and reviewed the **merged** tree, because that is the tree that runs after landing. Scope is what it claims: 2 files, +121/−1 — `core/manifests/apps/sbx-fleet/Dockerfile` (+26) and `core/manifests/apps/sbx-gits/statefulset.yaml` (+95/−1). No other file moves. No secret, no RBAC, no NetworkPolicy, no egress and no identity material is touched: the identity Secret block near the insertion is untouched context only. `updateStrategy: OnDelete`, so the merge itself rolls nothing; `kubectl kustomize` returns rc=0 on both the head and the merged tree, and the merged YAML parses with no duplicate mapping key. CI is green at this head (runs 3198/3199/3200, every job passing, `build-and-push` included). The head is still `a619bdb8`, the PR is open, mergeable, and it has no earlier review to inherit. What is right, and what I checked rather than took on faith: the probe is the container's own call with the container's own selector — `adb -s emulator-5554 shell getprop sys.boot_completed` is byte-for-byte the boot loop's step-4 line, `tr -d '\r'` included, with no `adb connect` and no `127.0.0.1:5555`; the `emulator` container runs `runAsUser: 0` with `readOnlyRootFilesystem: false`, so the probe runs in the same environment the boot loop already uses, and `adb` auto-starts its server from any client command. Readiness rather than startup is the right instrument here, and the arithmetic (first probe at 60s, inside a 76s fastest measured boot; 3×15s to NotReady; no kill path) holds against the two `EMU-READY` samples quoted. In the merged tree the probe lands on `containers[2] name=emulator`, not on the `emu-health` sidecar that live added afterwards, and no container ends up with two probe blocks. `tmp` 4Gi→8Gi with `avd` and `herdr-sock` untouched keeps this pod's own clean eviction as the failure mode instead of a node-level `DiskPressure`, and `prepare-work` gets the `install -d … -o 1000 -g 1000 "$DST/tmp"` line that makes its own target exist. ## The finding in full The premise the change is built on is no longer true of its base. `live` already declares `TMPDIR=/home/ld/work/seat-transcripts` in the `sandbox` env — the HOMELAB-2682 block, landed as `93bbb5d4` after this branch was cut from `74724767` — documented in that directory's README §5.2, and asserted by `scripts/gits-sbx-epic-test.sh` test 28. This branch still states, in the manifest comment it adds, that `TMPDIR` "is set NOWHERE today (not here, not in the image, not in the manifest)": false against `fc933fb3`. Because the branch predates that line and touches a nearby region, git auto-merges cleanly and the result carries both entries, at env indices 4 and 26 of `containers[sandbox]`: TMPDIR=/home/ld/work/seat-transcripts (what live says, HOMELAB-2682) TMPDIR=/home/ld/work/tmp (what this PR adds) Kubernetes accepts that object — no validation error, `kubectl kustomize` rc=0, no duplicate YAML key — so nothing stops it, and the precedence is then runtime-defined: Docker's env builder keeps the last value, a `getenv` reader over a runc-built env sees the first. I did not have to settle that ambiguity to file this. I ran the harness's own parse over the merged tree — a dict built name→value in list order, the same code test 28 uses — and `TMPDIR` resolves to `/home/ld/work/tmp`, so test 28 [HOMELAB-2682] el-contenedor-sandbox-no-declara-TMPDIR=/home/ld/work/seat-transcripts(='/home/ld/work/tmp') goes red. That is the lane check in the bundle's own step 9, firing against the tree this PR would leave behind. The same ambiguity reaches the reason the bundle exists: if `/home/ld/work/tmp` is what the sandbox honours, seat transcripts stop landing in `home/ld/work/seat-transcripts` — the directory the recreate's snapshot step copies — and README §5.2 becomes wrong. Either way one of the two intents is silently defeated, and the merged manifest carries a comment asserting the opposite of what `live` does. The fix is small and gives up none of the PR's real gain: rebase onto `live`, DELETE the `TMPDIR` entry this PR adds to the `sandbox` env (live's `seat-transcripts` already moves non-JVM scratch off the `tmp` emptyDir, which is the stated goal), keep `JAVA_TOOL_OPTIONS=-Djava.io.tmpdir=/home/ld/work/tmp` — the genuinely new half, and the one consumer `TMPDIR` never reached — keep the 8Gi and the probe, and rewrite the env comment so it no longer claims `TMPDIR` is unset. That needs a fresh seat on the new head, per the bundle's order. ## Blockers BLOCKER B1: landing this head on today's live leaves two conflicting TMPDIR entries in the sandbox container and reddens the repo's own epic test 28 — evidence and the one-line fix are in the section above. ## Concerns These do not hold the PR on their own; they are for the next head or for the record. C-1 — `sbx-sendrast` inherits the new image `ENV` and has nowhere for it to point. The `sbx-fleet` image is shared, and the Dockerfile sets `ENV TMPDIR=/home/ld/work/tmp`. `sbx-lab` mounts `work` at `/home/ld/work` (only its `tmp` subdirectory is missing today) and the PR records that as a known follow-up. `sbx-sendrast` has no `/home/ld/work` mount at all — it mounts `/home/jesus`, `/tmp`, `/opt/sbx-*` — so a digest bump there would point `TMPDIR` at a path that does not exist, and temp-file consumers that trust `TMPDIR` (`os.TempDir`, `tempfile`, `mkstemp`) fail rather than fall back to `/tmp`. Nothing activates today (this PR bumps no digest, and the build workflow publishes a tag without touching the pins), which is why this is a concern and not a hold — but the PR's known-follow-up list names `sbx-lab` only, and the brief's own instruction is to leave `sbx-sendrast` alone. One sentence in the Dockerfile comment would keep that trap from being stepped on by whoever bumps that pin. C-2 — the probe's steady state has not been measured under a suite. A `readinessProbe` runs for the pod's whole life, so `adb shell getprop` executes every 15s indefinitely, including while an instrumented run holds the guest. The timing argument covers boot only, and the PR's own "not verified" list (the fresh `/avd` boot, the real recreation) does not mention load. Low risk, cheap to settle: one suite run with the probe live, or a `periodSeconds` that backs off once the guest is up. C-3 — the headline "Nothing outside `sbx-gits` moves" is only true of the manifests. The Dockerfile half does move the two other sandboxes' behaviour at their next digest bump (C-1). The body qualifies this later, but the headline is the sentence a future reader will quote. ## What I did not ask for The recreate is gits-hypervisor's, not this PR's; `#860` preparing without activating is the right shape for a bundle that lands it first and recreates later. Landing it first is safe once B1 is fixed: the merged tree then rolls nothing (OnDelete), adds no image pin, and the Dockerfile change only publishes a tag through the existing path-filtered build, which I read in `.forgejo/workflows/sbx-fleet-image-build.yaml` — that job asserts the base ARG pin, which the merge keeps from `live` (`go-234dfd0@sha256:a5763f7e…`), so it stays green. Reviewed read-only: no edit, no push, no merge. Identity asserted as `ld-reviewer` before the write; the credential was held in a 0600 file under a 0700 directory, never in argv and never printed, and the operator's ambient token was not used.

pi-landing force-pushed plane/HOMELAB-2479-sbx-gits-probe-tmp from a619bdb89f to f3776d4b31 2026-11-09 03:42:11 +00:00 Compare

pi-landing commented 2026-11-09 03:43:29 +00:00

Author

Collaborator

Copy link

Fix worker update — blocker B1 (seat 1947) resolved on new head f3776d4b.

Branch was 4 days stale: live (1c5111d1, carrying fc933fb3/HOMELAB-2682) already declares TMPDIR=/home/ld/work/seat-transcripts in the sandbox env, so the PR's own TMPDIR=/home/ld/work/tmp entry would have left TWO TMPDIR entries after auto-merge — the harness's own parse resolves the last one and epic test 28 (the step-9 lane check) would go red, and seat transcripts would stop landing where the recreate's snapshot copies them.

Changed on the rebased head: (1) rebased onto origin/live (1c5111d1); (2) dropped the manifest TMPDIR entry entirely — live's seat-transcripts value already does the job and is untouched; (3) kept JAVA_TOOL_OPTIONS=-Djava.io.tmpdir=/home/ld/work/tmp (the genuinely new half — the JVM ignores TMPDIR), the tmp emptyDir 4Gi→8Gi, the prepare-work $DST/tmp creation, and the guest-boot readiness probe; (4) rewrote the stale "TMPDIR is set NOWHERE" comment (also in the sbx-fleet Dockerfile) — it now states TMPDIR lives at the HOMELAB-2682 site and why a second entry must not exist; (5) verified on the rebased tree: the harness's own test-28 parse resolves sandbox TMPDIR -> /home/ld/work/seat-transcripts, emulator -> empty, test 28 standalone PASS; kubectl kustomize rc=0 for sbx-gits, sbx-lab, sbx-sendrast.

Seat 1947's non-blocking concerns, folded into the follow-up list:

• sbx-sendrast inherits image ENV TMPDIR=/home/ld/work/tmp on a future digest bump but has no /home/ld/work mount — its bump prep must name sbx-sendrast alongside sbx-lab before any sbx-fleet rollout reaches it.
• The probe's steady-state load (adb shell getprop every 15 s for pod life) is unmeasured — on the "not verified" list until observed on a recreated pod.
• "Nothing outside sbx-gits moves" is true of the manifests only; the image default reaches sbx-lab/sbx-sendrast on their next digest bump (by design, flagged above).

Not merging — landing waits for a fresh review seat on this exact head (f3776d4b).

**Fix worker update — blocker B1 (seat 1947) resolved on new head `f3776d4b`.** Branch was 4 days stale: live (`1c5111d1`, carrying `fc933fb3`/HOMELAB-2682) already declares `TMPDIR=/home/ld/work/seat-transcripts` in the sandbox env, so the PR's own `TMPDIR=/home/ld/work/tmp` entry would have left TWO TMPDIR entries after auto-merge — the harness's own parse resolves the last one and epic test 28 (the step-9 lane check) would go red, and seat transcripts would stop landing where the recreate's snapshot copies them. Changed on the rebased head: (1) rebased onto `origin/live` (`1c5111d1`); (2) **dropped the manifest `TMPDIR` entry entirely** — live's `seat-transcripts` value already does the job and is untouched; (3) **kept** `JAVA_TOOL_OPTIONS=-Djava.io.tmpdir=/home/ld/work/tmp` (the genuinely new half — the JVM ignores TMPDIR), the `tmp` emptyDir 4Gi→8Gi, the `prepare-work` `$DST/tmp` creation, and the guest-boot readiness probe; (4) rewrote the stale "TMPDIR is set NOWHERE" comment (also in the `sbx-fleet` Dockerfile) — it now states TMPDIR lives at the HOMELAB-2682 site and why a second entry must not exist; (5) verified on the rebased tree: the harness's own test-28 parse resolves sandbox `TMPDIR` -> `/home/ld/work/seat-transcripts`, emulator -> empty, test 28 standalone PASS; `kubectl kustomize` rc=0 for `sbx-gits`, `sbx-lab`, `sbx-sendrast`. **Seat 1947's non-blocking concerns, folded into the follow-up list:** - `sbx-sendrast` inherits image `ENV TMPDIR=/home/ld/work/tmp` on a future digest bump but has **no `/home/ld/work` mount** — its bump prep must name `sbx-sendrast` alongside `sbx-lab` before any `sbx-fleet` rollout reaches it. - The probe's steady-state load (`adb shell getprop` every 15 s for pod life) is **unmeasured** — on the "not verified" list until observed on a recreated pod. - "Nothing outside `sbx-gits` moves" is true of the **manifests** only; the image default reaches `sbx-lab`/`sbx-sendrast` on their next digest bump (by design, flagged above). Not merging — landing waits for a fresh review seat on this exact head (`f3776d4b`).

ld-reviewer commented 2026-11-09 04:05:22 +00:00

Collaborator

Copy link

Verdict: REQUEST_CHANGES

commit: f3776d4b31
model: omniroute/deepseek-flash
session: 01a1be68-4216-777b-86ff-4bae620aac82

Fresh-context review of the exact head f3776d4b31b1285854c06974feb6f71d9c6ae48e (base
1c5111d13d288e785492ad7874bce485e0bae6dc = the current origin/live tip, confirmed: the head
is a descendant of live, so the merged tree is the head's tree — I still merged head into base in
a throwaway worktree and both trees hash to b63dec94b7211935980c09f5efc6135796f5084d, i.e. the
rebase is clean and the merge is a fast-forward in effect). I did not read the prior seat's review
before forming a verdict; sources are the diff at this head, both changed files in the merged tree,
a PyYAML parse of the merged manifest, origin/live, and the repo's own harness
scripts/gits-sbx-epic-test.sh run on both trees.

The prior blocker B1 (seat #1947) — CLOSED, verified on the merged tree

• Exactly one TMPDIR in the merged manifest's pod spec: containers[name=sandbox].env
  → /home/ld/work/seat-transcripts (live's HOMELAB-2682 value, untouched). The PR's own
  /home/ld/work/tmp manifest entry is gone. emu-health and emulator declare no
  TMPDIR, as test 28 requires.
• The harness's own parse (t2682t_env, last-wins dict) resolves TMPDIR to
  seat-transcripts, emulator to empty, and the full test-28 block PASSES on the merged
  tree — including its java.io.tmpdir textual guard, which this PR satisfies because the new
  comment block names java.io.tmpdir.
• Retained as the PR claims: JAVA_TOOL_OPTIONS=-Djava.io.tmpdir=/home/ld/work/tmp on sandbox,
  tmp emptyDir 4Gi → 8Gi, the prepare-work install -d … "$DST/tmp" line, and the
  readinessProbe on the guest's sys.boot_completed.

The new blocker — the merged tree turns the epic's own gate red in 7 places

I ran scripts/gits-sbx-epic-test.sh on the merged tree and on origin/live (both with
GITS_SBX_ROOT set to the tree under test, same machine, same env):

tree	result
origin/live 1c5111d1 (baseline)	FAIL, 11 of 692
merged head f3776d4b	FAIL, 18 of 694

The baseline is already red for pre-existing reasons (image-digest pins, memory 6Gi/10Gi,
/home/ld unwritable for the H11 seed, loader-skills=0 — all unrelated to this diff), so I
judge the delta, not the absolute exit code. The delta is exactly 7 new failures, none
disappearing, every one traceable to this diff:

FAIL  H8[8][sbx-gits] render vs origin/live (spine-distinto env-anadidos: JAVA_TOOL_OPTIONS)
FAIL  H8[8][sbx-gits][mut] la linea base del head ya sale ROJA (spine-distinto env-anadidos: JAVA_TOOL_OPTIONS)
FAIL  H8[8][sbx-gits][mut] sandbox: variable de entorno no permitida -> esperaba 'env-anadidos: MUT_ENV', salio ' spine-distinto env-anadidos: JAVA_TOOL_OPTIONS MUT_ENV'
FAIL  H8[8][sbx-gits][mut] emulator: variable de entorno no permitida -> same
FAIL  H8[8][sbx-gits][mut] prepare-work: variable de entorno no permitida -> same
FAIL  T7[diff] sbx-gits NO debe cambiar contra origin/live salvo el conjunto esperado (cambios: M statefulset.yaml)
FAIL  17[R159] (d) T7[diff] sbx-gits NO debe cambiar contra origin/live fuera del conjunto esperado (cambios: M statefulset.yaml)

Two root causes, both mechanical:

1. H8's spine gate forbids any added env leaf against live (p1_added_bad … want_env →
   env-anadidos: JAVA_TOOL_OPTIONS). Five of the seven rows are this one red plus its knock-on:
   the mutation baseline is already red, so the three "env var no permitida" mutations can no
   longer fail for the right reason. The gate is not cosmetically red — its negative proof stops
   proving anything until the allowlist names the new variable.
2. T7[diff] pins the expected change-set of sbx-gits to SBX_GITS_DIFF_SET, which today
   is the HOMELAB-2682 set (M tools/gits-sbx-emu-proof.sh). M statefulset.yaml therefore
   fails it in two independent sections.

This is not the harness being unreasonable, and it is not a reason to weaken it: the repo's own
precedent is that the PR that changes the manifest updates the pin in the same change —
HOMELAB-2682 added its TMPDIR and carried both the d_extra_env='TMPDIR' allowlist line and
its rewritten expected diff-set. This PR changes the pod template's env, the probe, an
initContainer script and a volume cap, and touches scripts/gits-sbx-epic-test.sh not at all
(0 files of the harness in the diff).

Why it blocks rather than nags, given the bundle order that lands this FIRST: the epic's
coherence command is the one thing the orchestrator runs after every wave. Landing this first
hands every follow-up ticket a baseline that is red for reasons introduced here, and the next
worker either debugs someone else's env leaf or starts ignoring the harness — the failure mode the
gate exists to prevent. The fix is small and belongs here: extend the spine's want_env allowlist
with JAVA_TOOL_OPTIONS (and, if the intent is to keep it, say so in the message the gate prints)
and rewrite SBX_GITS_DIFF_SET to this change's real set, the way HOMELAB-2682 did.

What holds up on the rest (spot-checked, not taken on faith)

• Probe placement: in the merged tree the readinessProbe lands on containers[name=emulator]
  (containers[2] of sandbox, emu-health, emulator), not on live's later emu-health
  sidecar; no container carries two probe blocks. The exec is the boot loop's own
  adb -s emulator-5554 shell getprop sys.boot_completed … | tr -d '\r', no adb connect, no
  127.0.0.1:5555. Readiness-not-startup (no kill path) and the 60 s + 3×15 s = 105 s arithmetic
  against the two measured EMU-READY samples (76 s, 94 s) are internally consistent and I found
  no arithmetic error.
• No secret, RBAC, NetworkPolicy, egress or identity material is added or moved; the identity
  env block near the insertion is untouched context only, and the diff carries no credential
  value.
• Renders clean: kubectl kustomize rc=0 for the merged tree, the merged YAML parses with
  PyYAML with no duplicate mapping key, updateStrategy: OnDelete (the merge rolls nothing, the
  8Gi is a cap not a request), avd and herdr-sock untouched.
• Dockerfile path-filter: the ENV layer only reaches a pod after a future image build and a
  digest bump; this PR publishes a tag without moving any pin — correct and correctly stated.

Concerns (non-blocking, for the fix worker and the follow-up list)

• C-1 (new, mine): the image ENV TMPDIR=/home/ld/work/tmp also lands on the emulator
  container on the next sbx-fleet digest bump, and I verified that container has no
  /home/ld/work mount (work-mounted=False; only sandbox and emu-health mount it). So the
  bump would point adb's temp files at a path that does not exist in that container — and it
  contradicts test 28's stance that the emulator's ephemeral /tmp is the correct one. The
  Dockerfile comment argues the ENV is a default "for any consumer of this image"; the consumer
  list did not include the one container that lives in the same pod as the change. Scope it or
  mount it, before the bump.
• The fix worker folded the prior seat's three concerns into a PR comment (sbx-sendrast
  inheriting the image ENV with no /home/ld/work mount; steady-state adb getprop load every
  15 s unmeasured; "nothing outside sbx-gits moves" true of manifests only). They are mentioned,
  but they live in the thread, not the body — and the body is now stale against its own head:
  it still advertises TMPDIR=/home/ld/work/tmp "in the sandbox container env" and
  env indices [25]/[26] in the flattened leaf diff. On this head the manifest adds no TMPDIR
  at all. Please reconcile the body, because the next reader will look for the entry that was
  deliberately removed.
• Unverified and honestly flagged by the PR: a real clean-point recreation under the new probe, and
  a cold boot on a fresh /avd against the 105 s budget. I could not check those without the
  recreation this PR must not perform.

Blockers

BLOCKER B1-860r2-harness-pin: the merged tree runs scripts/gits-sbx-epic-test.sh with 7 new
failures versus origin/live (H8[8][sbx-gits] spine env-anadidos: JAVA_TOOL_OPTIONS and its
five knock-ons, plus T7[diff] twice on M statefulset.yaml), and the PR carries no update to
SBX_GITS_DIFF_SET or the spine allowlist. Extend both in this change, the way HOMELAB-2682 did
for TMPDIR, and re-run the harness on the merged tree: the delta must be zero new failures.
Everything the prior seat blocked on (B1, the duplicated TMPDIR) is resolved and verified;
test 28 is green on the merged tree.

## Verdict: REQUEST_CHANGES commit: f3776d4b31b1285854c06974feb6f71d9c6ae48e model: omniroute/deepseek-flash session: 01a1be68-4216-777b-86ff-4bae620aac82 Fresh-context review of the exact head `f3776d4b31b1285854c06974feb6f71d9c6ae48e` (base `1c5111d13d288e785492ad7874bce485e0bae6dc` = the current `origin/live` tip, confirmed: the head is a descendant of live, so the merged tree is the head's tree — I still merged head into base in a throwaway worktree and both trees hash to `b63dec94b7211935980c09f5efc6135796f5084d`, i.e. the rebase is clean and the merge is a fast-forward in effect). I did not read the prior seat's review before forming a verdict; sources are the diff at this head, both changed files in the merged tree, a PyYAML parse of the merged manifest, `origin/live`, and the repo's own harness `scripts/gits-sbx-epic-test.sh` run on **both** trees. ### The prior blocker B1 (seat #1947) — CLOSED, verified on the merged tree - Exactly **one** `TMPDIR` in the merged manifest's pod spec: `containers[name=sandbox].env` → `/home/ld/work/seat-transcripts` (live's HOMELAB-2682 value, untouched). The PR's own `/home/ld/work/tmp` manifest entry is **gone**. `emu-health` and `emulator` declare no `TMPDIR`, as test 28 requires. - The harness's own parse (`t2682t_env`, last-wins dict) resolves `TMPDIR` to `seat-transcripts`, `emulator` to empty, and the full test-28 block **PASSES** on the merged tree — including its `java.io.tmpdir` textual guard, which this PR satisfies because the new comment block names `java.io.tmpdir`. - Retained as the PR claims: `JAVA_TOOL_OPTIONS=-Djava.io.tmpdir=/home/ld/work/tmp` on `sandbox`, `tmp` `emptyDir` `4Gi → 8Gi`, the `prepare-work` `install -d … "$DST/tmp"` line, and the `readinessProbe` on the guest's `sys.boot_completed`. ### The new blocker — the merged tree turns the epic's own gate red in 7 places I ran `scripts/gits-sbx-epic-test.sh` on the merged tree and on `origin/live` (both with `GITS_SBX_ROOT` set to the tree under test, same machine, same env): | tree | result | |---|---| | `origin/live` `1c5111d1` (baseline) | FAIL, 11 of 692 | | merged head `f3776d4b` | FAIL, **18** of 694 | The baseline is **already** red for pre-existing reasons (image-digest pins, memory 6Gi/10Gi, `/home/ld` unwritable for the H11 seed, `loader-skills=0` — all unrelated to this diff), so I judge the delta, not the absolute exit code. The delta is exactly **7 new failures, none disappearing, every one traceable to this diff**: ``` FAIL H8[8][sbx-gits] render vs origin/live (spine-distinto env-anadidos: JAVA_TOOL_OPTIONS) FAIL H8[8][sbx-gits][mut] la linea base del head ya sale ROJA (spine-distinto env-anadidos: JAVA_TOOL_OPTIONS) FAIL H8[8][sbx-gits][mut] sandbox: variable de entorno no permitida -> esperaba 'env-anadidos: MUT_ENV', salio ' spine-distinto env-anadidos: JAVA_TOOL_OPTIONS MUT_ENV' FAIL H8[8][sbx-gits][mut] emulator: variable de entorno no permitida -> same FAIL H8[8][sbx-gits][mut] prepare-work: variable de entorno no permitida -> same FAIL T7[diff] sbx-gits NO debe cambiar contra origin/live salvo el conjunto esperado (cambios: M statefulset.yaml) FAIL 17[R159] (d) T7[diff] sbx-gits NO debe cambiar contra origin/live fuera del conjunto esperado (cambios: M statefulset.yaml) ``` Two root causes, both mechanical: 1. **H8's spine gate forbids any added env leaf against live** (`p1_added_bad … want_env` → `env-anadidos: JAVA_TOOL_OPTIONS`). Five of the seven rows are this one red plus its knock-on: the mutation baseline is already red, so the three "env var no permitida" mutations can no longer fail *for the right reason*. The gate is not cosmetically red — its negative proof stops proving anything until the allowlist names the new variable. 2. **`T7[diff]` pins the expected change-set of `sbx-gits` to `SBX_GITS_DIFF_SET`**, which today is the HOMELAB-2682 set (`M tools/gits-sbx-emu-proof.sh`). `M statefulset.yaml` therefore fails it in two independent sections. This is not the harness being unreasonable, and it is not a reason to weaken it: the repo's own precedent is that **the PR that changes the manifest updates the pin in the same change** — HOMELAB-2682 added its `TMPDIR` and carried both the `d_extra_env='TMPDIR'` allowlist line and its rewritten expected diff-set. This PR changes the pod template's env, the probe, an initContainer script and a volume cap, and touches `scripts/gits-sbx-epic-test.sh` not at all (`0` files of the harness in the diff). Why it blocks rather than nags, given the bundle order that lands this **FIRST**: the epic's coherence command is the one thing the orchestrator runs after every wave. Landing this first hands every follow-up ticket a baseline that is red for reasons introduced here, and the next worker either debugs someone else's env leaf or starts ignoring the harness — the failure mode the gate exists to prevent. The fix is small and belongs here: extend the spine's `want_env` allowlist with `JAVA_TOOL_OPTIONS` (and, if the intent is to keep it, say so in the message the gate prints) and rewrite `SBX_GITS_DIFF_SET` to this change's real set, the way HOMELAB-2682 did. ### What holds up on the rest (spot-checked, not taken on faith) - **Probe placement**: in the merged tree the `readinessProbe` lands on `containers[name=emulator]` (`containers[2]` of `sandbox`, `emu-health`, `emulator`), not on live's later `emu-health` sidecar; no container carries two probe blocks. The exec is the boot loop's own `adb -s emulator-5554 shell getprop sys.boot_completed … | tr -d '\r'`, no `adb connect`, no `127.0.0.1:5555`. Readiness-not-startup (no kill path) and the 60 s + 3×15 s = 105 s arithmetic against the two measured `EMU-READY` samples (76 s, 94 s) are internally consistent and I found no arithmetic error. - **No secret, RBAC, NetworkPolicy, egress or identity material** is added or moved; the identity env block near the insertion is untouched context only, and the diff carries no credential value. - **Renders clean**: `kubectl kustomize` rc=0 for the merged tree, the merged YAML parses with PyYAML with no duplicate mapping key, `updateStrategy: OnDelete` (the merge rolls nothing, the 8Gi is a cap not a request), `avd` and `herdr-sock` untouched. - **Dockerfile path-filter**: the `ENV` layer only reaches a pod after a future image build and a digest bump; this PR publishes a tag without moving any pin — correct and correctly stated. ### Concerns (non-blocking, for the fix worker and the follow-up list) - **C-1 (new, mine): the image `ENV TMPDIR=/home/ld/work/tmp` also lands on the `emulator` container** on the next `sbx-fleet` digest bump, and I verified that container has **no `/home/ld/work` mount** (`work-mounted=False`; only `sandbox` and `emu-health` mount it). So the bump would point `adb`'s temp files at a path that does not exist in that container — and it contradicts test 28's stance that the emulator's ephemeral `/tmp` is the correct one. The Dockerfile comment argues the `ENV` is a default "for any consumer of this image"; the consumer list did not include the one container that lives in the same pod as the change. Scope it or mount it, before the bump. - The fix worker folded the prior seat's three concerns into a PR **comment** (sbx-sendrast inheriting the image `ENV` with no `/home/ld/work` mount; steady-state `adb getprop` load every 15 s unmeasured; "nothing outside `sbx-gits` moves" true of manifests only). They are mentioned, but they live in the thread, not the body — and the body is now **stale against its own head**: it still advertises `TMPDIR=/home/ld/work/tmp` "in the `sandbox` container env" and env indices `[25]/[26]` in the flattened leaf diff. On this head the manifest adds no `TMPDIR` at all. Please reconcile the body, because the next reader will look for the entry that was deliberately removed. - Unverified and honestly flagged by the PR: a real clean-point recreation under the new probe, and a cold boot on a fresh `/avd` against the 105 s budget. I could not check those without the recreation this PR must not perform. ### Blockers BLOCKER B1-860r2-harness-pin: the merged tree runs `scripts/gits-sbx-epic-test.sh` with 7 new failures versus `origin/live` (`H8[8][sbx-gits]` spine `env-anadidos: JAVA_TOOL_OPTIONS` and its five knock-ons, plus `T7[diff]` twice on `M statefulset.yaml`), and the PR carries no update to `SBX_GITS_DIFF_SET` or the spine allowlist. Extend both in this change, the way HOMELAB-2682 did for `TMPDIR`, and re-run the harness on the merged tree: the delta must be zero new failures. Everything the prior seat blocked on (B1, the duplicated `TMPDIR`) is **resolved and verified**; test 28 is green on the merged tree.

pi-landing referenced this pull request from a commit 2026-11-09 04:41:54 +00:00

HOMELAB-2479: test(sbx-gits): re-pin the epic harness for the #860 head

pi-landing added 1 commit 2026-11-09 04:41:54 +00:00

HOMELAB-2479: test(sbx-gits): re-pin the epic harness for the #860 head ... ad45613a77

Seat 23693 round-2 verdict: the branch updates the manifest without updating
the harness pins, handing every follow-up ticket a broken baseline. Same
change as HOMELAB-2682's precedent, re-measured against origin/live = 1c5111d1:

- p1_lab_sts_bad env pin: d_extra_env 'TMPDIR' -> 'TMPDIR JAVA_TOOL_OPTIONS'
  (the JVM ignores TMPDIR; the Gradle/Kotlin scratch needs its own variable
  toward the work hostPath). Name allowed here, value pinned by the manifest.
- SBX_GITS_DIFF_SET: the ok0 entry (M tools/gits-sbx-emu-proof.sh) leaves the
  set — live already absorbs it, so the branch diff is exactly
  M statefulset.yaml. The set is re-measured against the current tip, never
  dragged along.
- readinessProbe: p1_spine strips it (mapping-rule, not the list rule — the
  probe body is sibling keys) and p1_rp pins the exact block set by sha256:
  by default no probe may move (emu-health's httpGet is untouchable); the
  wave that legitimately adds one passes its p1_rp hash via extra_rp
  (#860: the emulator's sys.boot_completed probe). Without this the added
  probe would trip spine-distinto forever.
- T7[diff] / 17[R159](d) pass messages re-measured to name the #860 set.

ld-reviewer commented 2026-11-09 04:57:43 +00:00

Collaborator

Copy link

commit: ad45613a77
model: omniroute/deepseek-flash
session: 01a1be9e-7ab1-777b-86ff-4bc71d8ab160

Verdict: REQUEST_CHANGES

Fresh seat, no prior-round context inherited. I reviewed the current head independently: the diff itself, the re-pinned harness, and my own re-run of the harness on a merged tree. The code is verified clean — the single blocker is in the PR body's consumer map, which is a durable instruction a follow-up worker will act on. Body-only fix, no code change, so the next round is a read, not a re-run.

Blockers

BLOCKER B1-860r3-consumer-map: the C-1 paragraph's blast-radius map is factually wrong in two ways, and it is the record that gates the deferred digest-bump work ("the digest-bump follow-up MUST land, in the same change, (i) … (ii) …").

• sbx-gits' emulator container is not an sbx-fleet consumer. The body says "the image ENV TMPDIR=/home/ld/work/tmp (+JAVA_TOOL_OPTIONS) would reach the emulator container at the next sbx-fleet digest bump — which mounts no /home/ld/work". The emulator container pins platform/android-emulator:230756a@sha256:66430940… (core/manifests/apps/sbx-gits/statefulset.yaml:1204), built by .forgejo/workflows/emu-images.yaml from platform/android-emulator/Dockerfile, whose base is FROM debian:bookworm-slim@sha256:734a72b3… (line 19) — no sbx-fleet layer anywhere in its lineage. The sbx-fleet containers in this StatefulSet are exactly prepare-work, seed-pi-agent, sandbox, emu-health, and all four mount /home/ld/work. So the C-1 hazard as written does not exist, and "contradicting test 28's 'emulator stays on ephemeral /tmp'" describes a contradiction the ENV layer cannot produce.
• sbx-sendrast is not an sbx-fleet consumer either. The body says it "inherits the image ENV at its next bump" and that "its bump must override both variables per container". core/manifests/apps/sbx-sendrast/Dockerfile:26-27 is ARG LD_FLEET_IMAGE=…/ld-target/ld-fleet:go-89b900d / FROM ${LD_FLEET_IMAGE} — it derives from ld-fleet directly, not from sbx-fleet, so this layer never reaches it, and /home/jesus/work in that manifest is irrelevant to this ENV.
• The real exposure is understated to one consumer and mis-scoped. The only genuine sbx-fleet consumer outside sbx-gits is sbx-lab (sbx-fleet:fc933fb@sha256:be81ce63…), and its work preparation creates no $DST/tmp — I grepped its statefulset.yaml for DST/tmp and /home/ld/work/tmp: zero hits. At its next digest bump it would inherit TMPDIR=/home/ld/work/tmp pointing at a directory nothing creates. The body mentions this ("its prepare-work must create tmp") but as one of three items inside a wrong list, which is how a correct item gets skipped.

Required correction (prose only, in this order):

1. Delete the emulator-inheritance claim and the test-28 contradiction built on it; state instead that sbx-gits' emulator runs android-emulator (FROM debian:bookworm-slim), so no sbx-fleet ENV can reach it, and every sbx-fleet container in sbx-gits mounts /home/ld/work and gets its tmp from the prepare-work line this PR adds — i.e. sbx-gits is fully self-consistent at bump.
2. Delete the sbx-sendrast item (not a consumer; /home/jesus/work is a different image lineage).
3. Keep, and promote to the only bump-gated follow-up: sbx-lab inherits the ENV at its next digest bump and must land install -d -m 0755 -o 1000 -g 1000 "$DST/tmp" (or a per-container override) in that same change, since its prep creates nothing there today.

Why this is a blocker rather than a concern: claim honesty is a review dimension here, and this paragraph is not a description — it is a MUST-do instruction for a future worker that names a non-consumer and the wrong image. Merged as-is it would produce a phantom emulator override and an unnecessary sbx-sendrast edit while the one real consumer stays a bullet in a wrong list. The deferral decision is safe (see below); the reasoning recorded for it is not.

What I verified clean (do not re-litigate)

• Diff scope: git diff --name-status 1c5111d1..ad45613a = M core/manifests/apps/sbx-gits/statefulset.yaml, M core/manifests/apps/sbx-fleet/Dockerfile, M scripts/gits-sbx-epic-test.sh. SBX_GITS_DIFF_SET is exactly M\tstatefulset.yaml — re-measured correctly; the sandbox directory changes in exactly that one file.
• Render byte-identity (sbx-lab, sbx-sendrast): confirmed by the harness on my own run — PASS 20[guard] el render de sbx-sendrast es BYTE-IDENTICO … sha256 d225759f534949db6… (matches the body), PASS for sbx-lab, and git diff against origin/live touches no file in sbx-sendrast.
• p1_rp pin: I recomputed it independently from kubectl kustomize core/manifests/apps/sbx-gits | sts_doc | p1_rp on this head and got 0ce65f6ed11b3b45f5cb56f2f58f76a40ba5404b2adb0a8e1c07dfd2b37bc602, exactly the constant the harness carries. It covers both blocks (the pre-existing emu-health httpGet and the new emulator exec probe), normalized to the block body.
• The new mechanism is sound and minimal, and it does grandfather emu-health correctly: with extra_rp empty (sbx-lab) the rule is rp(now) == rp(ref), so an existing probe that does not move is fine and any second probe is red; with extra_rp set the wave must publish the hash of its whole block set, so a change is visible in the harness diff — the same division of work as env: (names allowed, values pinned by the manifest). It fails closed in every shape I could construct.
• d_extra_env='TMPDIR JAVA_TOOL_OPTIONS' parses right: want_env="SEED_FORGEJO_EXT TMPDIR JAVA_TOOL_OPTIONS ", and p1_added_bad iterates name-by-name against p1_allowed word matching. H8[8][sbx-gits][mut] punto de partida … (con env permitido 'TMPDIR JAVA_TOOL_OPTIONS') is green on my run, and the three env-anadidos: MUT_ENV rows still bite.
• Comments do not pollute the spine: kubectl kustomize strips them (grep -c 'SCRATCH OUT OF THE EVICTION PATH' on the render = 0), so p1_spine's byte comparison is not broken by the large comment blocks this PR adds. Verified, not assumed.
• The deferral itself is safe: nothing rolls (updateStrategy: OnDelete), all four sbx-fleet containers in sbx-gits pin 6fa150a, which predates the new ENV layer, and the only non-sbx-gits sbx-fleet consumer is sbx-lab at an explicit future bump. The JAVA_TOOL_OPTIONS-only manifest env (no second TMPDIR) is right: /home/ld/work/tmp is created by the new prepare-work line as root then handed to 1000:1000, and the last-value-wins parse in test 28 stays green.

My harness numbers (merged tree, this head, my runner)

Scratch worktree, base merged into this head (1c5111d1 is an ancestor, so it is the merged tree): bash scripts/gits-sbx-epic-test.sh → exit 1, 683 PASS / 11 FAIL / 6 SKIP, 694 checks, 8m20s. Baseline, same runner, live tree at 1c5111d1 with this head's re-pinned harness → 12 FAIL, still in progress at section ~19 when I stopped it. Set comparison over the sections it had completed:

• Every failure in my merged run is present in the live run and none references this diff: stale manifest[2] memoria 6Gi/10Gi, stale manifest[3] imagenes … sha256:7e0513ca (live pins 6fa150a), manifest[3b], T7 emulador resource sums, T7+R99[10b], H8[8][mut] sandbox: memoria 13Gi -> 20Gi (the mutation's baseline value no longer exists in the render), probe uses an EMPTY pre-existing --dir, and the three 11[H11] rows that die on mkdir: cannot create directory '/home/ld': Permission denied.
• The only rows that differ are the live tree's two extra reds, H8[8][sbx-gits] render vs origin/live (readinessProbe-no-clavada) and its [mut] baseline row — which is the new pin doing its job on a tree that does not carry the probe. Zero new failures attributable to this diff, matching the fix worker's claim in shape; my raw counts are 11/694 here versus their 13/693, i.e. runner-relative, not set-relative.
• Named checks green on my run: H8[8][sbx-gits][mut] punto de partida (9 of 10 mutation rows biting; the 10th is the pre-existing stale-13Gi row), T7[diff] … M statefulset.yaml plus its red mutation, 17[R159] (d) plus its red mutation, and the 28[HOMELAB-2682] TMPDIR rows.

Concerns (non-blocking, do not need a round)

• C-a prepare-work comment ("Idempotent: a pre-existing $DST/tmp keeps its owner and its contents"): GNU install -d re-applies -m/-o/-g to an existing directory. Contents are kept; owner and mode are forced to 1000:1000 0755. The forced value is the intended one, so this is a comment that overstates, not a defect — but the harness's own culture is that comments are claims.
• C-b p1_spine strips readinessProbe: on a fresh marker line and inside the rskip terminator, but not in the "line that ENDED an env/volumeMounts/args block" re-evaluation. If a renderer ever emits readinessProbe: directly after one of those blocks, the probe leaks into the spine. It fails closed (spine-distinto, red), and kustomize sorts container keys so env: is never immediately followed by readinessProbe: — but given this function's history of layout assumptions, it deserves the same re-evaluation treatment or a two-line warning comment.
• C-c p1_rp normalizes the base indentation away, so the pin does not record which container a probe belongs to: two containers carrying byte-identical probe blocks could be swapped without moving the hash. Harmless here (the two blocks differ); worth knowing before it is relied on for an identical-probe case.
• C-d Side effect of the readiness flip, for the verification wave rather than this diff: the Service is headless with publishNotReadyAddresses unset, so sbx-gits-0.sbx-gits.sbx-gits.svc.cluster.local stops being published while the probe fails — and that exact name is the static target of prod's sandbox-telemetry ScrapeConfig. SandboxEmulatorTelemetryAbsent is for: 15m, so a normal boot cannot trip it, but a guest hung past ~15 min will now also fire "exporter absent". Arguably the right signal; it does change what that alert means, and nothing in the body says so.

Anything r2 broke

No. The probe strip, the p1_rp pin, the two-name env set, the re-measured diff set and the re-worded pass messages are consistent with each other and with the harness's existing conventions; T7[diff], 17[R159](d), the H8[8] table and the 20[guard] renders all behave as intended on a tree that carries this head, and red as intended on a tree that does not.

— ld-reviewer (fresh seat, DeepSeek), read-only: I edited nothing, pushed nothing, merged nothing. No token was placed in argv or printed.

commit: ad45613a77f76b1dda501a6f15f5dd48ec612721 model: omniroute/deepseek-flash session: 01a1be9e-7ab1-777b-86ff-4bc71d8ab160 ## Verdict: REQUEST_CHANGES Fresh seat, no prior-round context inherited. I reviewed the current head independently: the diff itself, the re-pinned harness, and my own re-run of the harness on a merged tree. **The code is verified clean — the single blocker is in the PR body's consumer map, which is a durable instruction a follow-up worker will act on.** Body-only fix, no code change, so the next round is a read, not a re-run. ## Blockers BLOCKER B1-860r3-consumer-map: the C-1 paragraph's blast-radius map is factually wrong in two ways, and it is the record that gates the deferred digest-bump work ("the digest-bump follow-up MUST land, in the same change, (i) … (ii) …"). - **`sbx-gits`' `emulator` container is not an `sbx-fleet` consumer.** The body says "the image `ENV TMPDIR=/home/ld/work/tmp` (+`JAVA_TOOL_OPTIONS`) would reach the **`emulator`** container at the next `sbx-fleet` digest bump — which mounts **no** `/home/ld/work`". The emulator container pins `platform/android-emulator:230756a@sha256:66430940…` (`core/manifests/apps/sbx-gits/statefulset.yaml:1204`), built by `.forgejo/workflows/emu-images.yaml` from `platform/android-emulator/Dockerfile`, whose base is `FROM debian:bookworm-slim@sha256:734a72b3…` (line 19) — no `sbx-fleet` layer anywhere in its lineage. The `sbx-fleet` containers in this StatefulSet are exactly `prepare-work`, `seed-pi-agent`, `sandbox`, `emu-health`, and all four mount `/home/ld/work`. So the C-1 hazard as written does not exist, and "contradicting test 28's 'emulator stays on ephemeral /tmp'" describes a contradiction the ENV layer cannot produce. - **`sbx-sendrast` is not an `sbx-fleet` consumer either.** The body says it "inherits the image ENV at its next bump" and that "its bump must override both variables per container". `core/manifests/apps/sbx-sendrast/Dockerfile:26-27` is `ARG LD_FLEET_IMAGE=…/ld-target/ld-fleet:go-89b900d` / `FROM ${LD_FLEET_IMAGE}` — it derives from `ld-fleet` directly, not from `sbx-fleet`, so this layer never reaches it, and `/home/jesus/work` in that manifest is irrelevant to this ENV. - **The real exposure is understated to one consumer and mis-scoped.** The only genuine `sbx-fleet` consumer outside `sbx-gits` is `sbx-lab` (`sbx-fleet:fc933fb@sha256:be81ce63…`), and its work preparation creates no `$DST/tmp` — I grepped its `statefulset.yaml` for `DST/tmp` and `/home/ld/work/tmp`: zero hits. At its next digest bump it would inherit `TMPDIR=/home/ld/work/tmp` pointing at a directory nothing creates. The body mentions this ("its `prepare-work` must create `tmp`") but as one of three items inside a wrong list, which is how a correct item gets skipped. Required correction (prose only, in this order): 1. Delete the emulator-inheritance claim and the test-28 contradiction built on it; state instead that `sbx-gits`' emulator runs `android-emulator` (FROM `debian:bookworm-slim`), so no `sbx-fleet` `ENV` can reach it, and every `sbx-fleet` container in `sbx-gits` mounts `/home/ld/work` and gets its `tmp` from the `prepare-work` line this PR adds — i.e. `sbx-gits` is fully self-consistent at bump. 2. Delete the `sbx-sendrast` item (not a consumer; `/home/jesus/work` is a different image lineage). 3. Keep, and promote to the only bump-gated follow-up: `sbx-lab` inherits the `ENV` at its next digest bump and must land `install -d -m 0755 -o 1000 -g 1000 "$DST/tmp"` (or a per-container override) **in that same change**, since its prep creates nothing there today. Why this is a blocker rather than a concern: claim honesty is a review dimension here, and this paragraph is not a description — it is a MUST-do instruction for a future worker that names a non-consumer and the wrong image. Merged as-is it would produce a phantom emulator override and an unnecessary `sbx-sendrast` edit while the one real consumer stays a bullet in a wrong list. The deferral *decision* is safe (see below); the *reasoning* recorded for it is not. ## What I verified clean (do not re-litigate) - **Diff scope**: `git diff --name-status 1c5111d1..ad45613a` = `M core/manifests/apps/sbx-gits/statefulset.yaml`, `M core/manifests/apps/sbx-fleet/Dockerfile`, `M scripts/gits-sbx-epic-test.sh`. `SBX_GITS_DIFF_SET` is exactly `M\tstatefulset.yaml` — re-measured correctly; the sandbox directory changes in exactly that one file. - **Render byte-identity** (`sbx-lab`, `sbx-sendrast`): confirmed by the harness on my own run — `PASS 20[guard] el render de sbx-sendrast es BYTE-IDENTICO … sha256 d225759f534949db6…` (matches the body), `PASS` for `sbx-lab`, and `git diff` against `origin/live` touches no file in `sbx-sendrast`. - **`p1_rp` pin**: I recomputed it independently from `kubectl kustomize core/manifests/apps/sbx-gits | sts_doc | p1_rp` on this head and got `0ce65f6ed11b3b45f5cb56f2f58f76a40ba5404b2adb0a8e1c07dfd2b37bc602`, exactly the constant the harness carries. It covers both blocks (the pre-existing `emu-health` `httpGet` and the new `emulator` exec probe), normalized to the block body. - **The new mechanism is sound and minimal**, and it does grandfather `emu-health` correctly: with `extra_rp` empty (`sbx-lab`) the rule is `rp(now) == rp(ref)`, so an existing probe that does not move is fine and any second probe is red; with `extra_rp` set the wave must publish the hash of its whole block set, so a change is visible in the harness diff — the same division of work as `env:` (names allowed, values pinned by the manifest). It fails closed in every shape I could construct. - **`d_extra_env='TMPDIR JAVA_TOOL_OPTIONS'`** parses right: `want_env="SEED_FORGEJO_EXT TMPDIR JAVA_TOOL_OPTIONS "`, and `p1_added_bad` iterates name-by-name against `p1_allowed` word matching. `H8[8][sbx-gits][mut] punto de partida … (con env permitido 'TMPDIR JAVA_TOOL_OPTIONS')` is green on my run, and the three `env-anadidos: MUT_ENV` rows still bite. - **Comments do not pollute the spine**: `kubectl kustomize` strips them (`grep -c 'SCRATCH OUT OF THE EVICTION PATH'` on the render = 0), so `p1_spine`'s byte comparison is not broken by the large comment blocks this PR adds. Verified, not assumed. - **The deferral itself is safe**: nothing rolls (`updateStrategy: OnDelete`), all four `sbx-fleet` containers in `sbx-gits` pin `6fa150a`, which predates the new `ENV` layer, and the only non-`sbx-gits` `sbx-fleet` consumer is `sbx-lab` at an explicit future bump. The `JAVA_TOOL_OPTIONS`-only manifest env (no second `TMPDIR`) is right: `/home/ld/work/tmp` is created by the new `prepare-work` line as root then handed to `1000:1000`, and the last-value-wins parse in test 28 stays green. ## My harness numbers (merged tree, this head, my runner) Scratch worktree, base merged into this head (`1c5111d1` is an ancestor, so it is the merged tree): `bash scripts/gits-sbx-epic-test.sh` → **exit 1, 683 PASS / 11 FAIL / 6 SKIP, 694 checks, 8m20s**. Baseline, same runner, live tree at `1c5111d1` with this head's re-pinned harness → **12 FAIL, still in progress at section ~19 when I stopped it**. Set comparison over the sections it had completed: - Every failure in my merged run is present in the live run and none references this diff: stale `manifest[2] memoria 6Gi/10Gi`, stale `manifest[3] imagenes … sha256:7e0513ca` (live pins `6fa150a`), `manifest[3b]`, `T7 emulador` resource sums, `T7+R99[10b]`, `H8[8][mut] sandbox: memoria 13Gi -> 20Gi` (the mutation's baseline value no longer exists in the render), `probe uses an EMPTY pre-existing --dir`, and the three `11[H11]` rows that die on `mkdir: cannot create directory '/home/ld': Permission denied`. - The only rows that differ are the live tree's two extra reds, `H8[8][sbx-gits] render vs origin/live (readinessProbe-no-clavada)` and its `[mut]` baseline row — which is the new pin doing its job on a tree that does not carry the probe. **Zero new failures attributable to this diff**, matching the fix worker's claim in shape; my raw counts are 11/694 here versus their 13/693, i.e. runner-relative, not set-relative. - Named checks green on my run: `H8[8][sbx-gits][mut] punto de partida` (9 of 10 mutation rows biting; the 10th is the pre-existing stale-13Gi row), `T7[diff] … M statefulset.yaml` plus its red mutation, `17[R159] (d)` plus its red mutation, and the `28[HOMELAB-2682]` TMPDIR rows. ## Concerns (non-blocking, do not need a round) - **C-a** `prepare-work` comment ("Idempotent: a pre-existing `$DST/tmp` keeps its owner and its contents"): GNU `install -d` re-applies `-m/-o/-g` to an existing directory. Contents are kept; owner and mode are forced to `1000:1000 0755`. The forced value is the intended one, so this is a comment that overstates, not a defect — but the harness's own culture is that comments are claims. - **C-b** `p1_spine` strips `readinessProbe:` on a fresh marker line and inside the `rskip` terminator, but not in the "line that ENDED an env/volumeMounts/args block" re-evaluation. If a renderer ever emits `readinessProbe:` directly after one of those blocks, the probe leaks into the spine. It fails closed (`spine-distinto`, red), and kustomize sorts container keys so `env:` is never immediately followed by `readinessProbe:` — but given this function's history of layout assumptions, it deserves the same re-evaluation treatment or a two-line warning comment. - **C-c** `p1_rp` normalizes the base indentation away, so the pin does not record *which container* a probe belongs to: two containers carrying byte-identical probe blocks could be swapped without moving the hash. Harmless here (the two blocks differ); worth knowing before it is relied on for an identical-probe case. - **C-d** Side effect of the readiness flip, for the verification wave rather than this diff: the Service is headless with `publishNotReadyAddresses` unset, so `sbx-gits-0.sbx-gits.sbx-gits.svc.cluster.local` stops being published while the probe fails — and that exact name is the static target of prod's `sandbox-telemetry` ScrapeConfig. `SandboxEmulatorTelemetryAbsent` is `for: 15m`, so a normal boot cannot trip it, but a guest hung past ~15 min will now also fire "exporter absent". Arguably the right signal; it does change what that alert means, and nothing in the body says so. ## Anything r2 broke No. The probe strip, the `p1_rp` pin, the two-name env set, the re-measured diff set and the re-worded pass messages are consistent with each other and with the harness's existing conventions; `T7[diff]`, `17[R159](d)`, the `H8[8]` table and the `20[guard]` renders all behave as intended on a tree that carries this head, and red as intended on a tree that does not. — ld-reviewer (fresh seat, DeepSeek), read-only: I edited nothing, pushed nothing, merged nothing. No token was placed in argv or printed.

ld-reviewer commented 2026-11-09 05:14:22 +00:00

Collaborator

Copy link

commit: ad45613a77
model: omniroute/deepseek-flash
session: 01a1beb3-c6bb-777b-86ff-4bd2a85d0fe5

Verdict: APPROVED

Blockers

None.

Confirmation seat for the body-only fix to BLOCKER B1-860r3-consumer-map (seat 23722). Fresh
context: I did not inherit that seat's summary as a premise. Every claim in the corrected C-1
paragraph was re-derived from the tree at this exact head (git archive ad45613a), the base
(1c5111d1), the render path (kubectl kustomize), and the repo's own harness. The prior seat's
code/harness verification is therefore not taken on faith either where I could cheaply re-measure
it (I re-measured the render gate and the p1_rp pin).

Head identity — nothing moved behind the body edit

• Branch tip == PR head == ad45613a77…, base 1c5111d1… (current origin/live tip). Two
  commits since base: f3776d4b (manifest + Dockerfile) and ad45613a (harness re-pin);
  3 files, +230/−13, and the manifest/image commit is still f3776d4b. The code is byte-identical
  to the state seats 23693/23722 verified; only the PR body changed — the defect class this round
  is the one that could be fixed without a new commit.
• CI green at this exact head: runs 3402/3403/3404, every job pass (build-and-push,
  yaml-lint, shellcheck, cockpit-test, cockpit-e2e, dispatcher-test, record-sql-test,
  runner-manifest-test, …).

The corrected consumer map — each claim, verified

1. emulator is NOT an sbx-fleet consumer → TRUE. At head, sbx-gits/statefulset.yaml
   pins harbor…/platform/sbx-fleet:6fa150a@sha256:0ee5e97e… on exactly prepare-work,
   seed-pi-agent, sandbox, emu-health, and harbor…/platform/android-emulator:230756a@sha256:66430940…
   on emulator. platform/android-emulator/Dockerfile:19 is
   FROM debian:bookworm-slim@sha256:734a72b3… — no sbx-fleet layer anywhere in that lineage, so
   no sbx-fleet ENV can reach the emulator and the "test 28's emulator-stays-on-ephemeral-/tmp
   stands" statement is now sound. The old claim (emulator inherits TMPDIR=/home/ld/work/tmp and
   mounts no work) is gone from the body.
2. All four sbx-fleet containers mount /home/ld/work → TRUE (prepare-work rw,
   seed-pi-agent rw, sandbox rw, emu-health read-only). Wording is accurate; the
   read-only qualifier is a separate, non-blocking refinement — concern C-2.
3. sbx-sendrast is NOT a consumer, no follow-up owed → TRUE.
   core/manifests/apps/sbx-sendrast/Dockerfile:26-27 is
   ARG LD_FLEET_IMAGE=…/ld-target/ld-fleet:go-89b900d / FROM ${LD_FLEET_IMAGE} — a different
   image line, no sbx-fleet layer. Repo-wide, the only objects referencing platform/sbx-fleet
   images are sbx-gits and sbx-lab (the rest are comments/docs/harness pins), no Dockerfile
   derives FROM it, and neither the sendrast nor the lab-director checkout references it.
4. sbx-lab is the ONE bump-gated follow-up → TRUE in substance. sbx-lab/statefulset.yaml
   pins sbx-fleet:fc933fb@sha256:be81ce63… on its two sbx-fleet containers, declares no
   TMPDIR/JAVA_TOOL_OPTIONS of its own, and its init script creates nothing at
   /home/ld/work/tmp (zero hits) — so at its next digest bump the image ENV would point at a
   directory nothing creates. Two residual wording defects inside that bullet are recorded as
   concern C-1; they do not change which change must happen, where, or when, which is
   what the blocker was about.

Deferral decision — intact, and now correctly reasoned

• updateStrategy: {type: OnDelete} is present in the template at head, so merge rolls nothing.
• The pinned digest sbx-fleet:6fa150a predates the ENV layer (the layer is added in
  sbx-fleet/Dockerfile immediately before USER 1000:1000, and no image was rebuilt in this
  branch: no tag/digest bump anywhere in the diff), so no running container can observe the new
  values between merge and bump.
• prepare-work runs as root and creates $DST/tmp with DST=/home/ld/work → /home/ld/work/tmp
  exists for every sbx-gits sbx-fleet container before they run. The manifest repeating
  JAVA_TOOL_OPTIONS (and NOT re-declaring TMPDIR, which lives at the HOMELAB-2682
  /home/ld/work/seat-transcripts site) matches the tree: one TMPDIR in the pod spec, one new
  env var (JAVA_TOOL_OPTIONS), tmp emptyDir 4Gi → 8Gi, avd 16Gi and herdr-sock 64Mi
  untouched.

Render gate and harness pins — independently re-measured

kubectl kustomize at base 1c5111d1 vs head ad45613a, this machine:

sbx-lab      8c8f61d79b30945b…  ==  8c8f61d79b30945b…   IDENTICAL
sbx-sendrast d225759f534949db…  ==  d225759f534949db…   IDENTICAL
sbx-gits     4b0fa50cd88e2ed9…  ->  ba858643337c6429…   CHANGED

Those are the exact digests the body quotes, so the byte-identical hard gate reproduces. I
recomputed p1_rp over the rendered head with the harness's own normalization (dedent the probe
body, drop blank lines, document order) and got
0ce65f6ed11b3b45f5cb56f2f58f76a40ba5404b2adb0a8e1c07dfd2b37bc602, which is byte-equal to the
pin at scripts/gits-sbx-epic-test.sh:3667 — the pin matches the render it claims to pin, and
SBX_GITS_DIFF_SET is M\tstatefulset.yaml, i.e. the branch diff set the body states.

Known limitations — folded in, and they are the right ones

The four seat-23722 concerns are present and correctly stated: install -d is documented by a
comment, not enforced; p1_spine's env-block termination path is described as failing closed
(I re-read the awk: a strip that runs long removes lines from the spine, which makes the spine
differ from origin/live and turns the gate red, never green); p1_rp normalizes the owning
container away (confirmed in the code — it prints readinessProbe: plus the dedented body, no
container identity, so a probe moved between containers at the same manifest path keeps its hash);
and the readiness-DNS / SandboxEmulatorTelemetryAbsent sharp edge for the recreation wave is
spelled out instead of hidden. C-2 (probe steady-state load unmeasured) and the explicit
"not verified" list are carried, not dropped.

Concerns (non-blocking, concerns may merge)

C-1 — sbx-lab has no prepare-work; and $DST there is not /home/ld/work. The corrected
bullet says "its prepare-work creates no $DST/tmp (zero hits in its script)". sbx-lab's
only initContainer is seed-pi-agent, and that script sets DST=/home/ld/work/pi-agent
(sbx-lab/statefulset.yaml:159,181) — prepare-work with DST=/home/ld/work is the sbx-gits
container. So the sentence is true of sbx-gits and imports another template's names into
sbx-lab. Operational consequence to carry into the follow-up ticket: in sbx-lab the directory
to create is the volume root /home/ld/work/tmp (the path the image ENV names), not
$DST/tmp, which would create /home/ld/work/pi-agent/tmp; and sbx-lab's init runs as the
image user (no root securityContext), so the literal
install -d -m 0755 -o 1000 -g 1000 recipe copied from sbx-gits will not work there — the
follow-up needs a root step (or a pre-created directory / fsGroup) plus the per-container
override. Not blocking: the actionable sentence in the body names the directory by reference to the
ENV value it just quoted, and that follow-up is a separate, separately reviewed change.

C-2 — emu-health mounts /home/ld/work read-only, so "always name a real path" is true but
not "a usable scratch path". At a future sbx-gits digest bump, emu-health (a sbx-fleet
container) inherits TMPDIR=/home/ld/work/tmp through a readOnly: true mount; node's
os.tmpdir() honours TMPDIR, so any temp-file use in the exporter would fail rather than fall
back to its /tmp. Bump-gated and today inert (6fa150a predates the layer), so it is not a
merger risk — but the same-change bump checklist should cover emu-health explicitly (mount
work rw, or neutralise TMPDIR for that container), not only sbx-lab.

Both concerns are wording/completeness items in the deferral record. They belong in the
bump-follow-up ticket; neither makes this prepare-only, OnDelete, no-image-bump change wrong,
and the blocker seat 23722 filed — a consumer map that would have produced a phantom emulator
override, an unnecessary sbx-sendrast edit, and the one real consumer as a bullet in a wrong
list — is resolved on this head.

Read-only: no file, branch, or PR state was edited; this comment is the only write, posted as
ld-reviewer after GET /api/v1/user returned that login.

commit: ad45613a77f76b1dda501a6f15f5dd48ec612721 model: omniroute/deepseek-flash session: 01a1beb3-c6bb-777b-86ff-4bd2a85d0fe5 ## Verdict: APPROVED ## Blockers None. Confirmation seat for the body-only fix to `BLOCKER B1-860r3-consumer-map` (seat 23722). Fresh context: I did not inherit that seat's summary as a premise. Every claim in the corrected C-1 paragraph was re-derived from the tree at this exact head (`git archive ad45613a`), the base (`1c5111d1`), the render path (`kubectl kustomize`), and the repo's own harness. The prior seat's code/harness verification is therefore not taken on faith either where I could cheaply re-measure it (I re-measured the render gate and the `p1_rp` pin). ### Head identity — nothing moved behind the body edit - Branch tip == PR head == `ad45613a77…`, base `1c5111d1…` (current `origin/live` tip). Two commits since base: `f3776d4b` (manifest + Dockerfile) and `ad45613a` (harness re-pin); 3 files, +230/−13, and the manifest/image commit is still `f3776d4b`. The code is byte-identical to the state seats 23693/23722 verified; only the PR body changed — the defect class this round is the one that could be fixed without a new commit. - CI green at this exact head: runs 3402/3403/3404, every job `pass` (`build-and-push`, `yaml-lint`, `shellcheck`, `cockpit-test`, `cockpit-e2e`, `dispatcher-test`, `record-sql-test`, `runner-manifest-test`, …). ### The corrected consumer map — each claim, verified 1. **`emulator` is NOT an `sbx-fleet` consumer → TRUE.** At head, `sbx-gits/statefulset.yaml` pins `harbor…/platform/sbx-fleet:6fa150a@sha256:0ee5e97e…` on exactly `prepare-work`, `seed-pi-agent`, `sandbox`, `emu-health`, and `harbor…/platform/android-emulator:230756a@sha256:66430940…` on `emulator`. `platform/android-emulator/Dockerfile:19` is `FROM debian:bookworm-slim@sha256:734a72b3…` — no `sbx-fleet` layer anywhere in that lineage, so no `sbx-fleet` `ENV` can reach the emulator and the "test 28's emulator-stays-on-ephemeral-/tmp stands" statement is now sound. The old claim (`emulator` inherits `TMPDIR=/home/ld/work/tmp` and mounts no `work`) is gone from the body. 2. **All four `sbx-fleet` containers mount `/home/ld/work` → TRUE** (`prepare-work` rw, `seed-pi-agent` rw, `sandbox` rw, `emu-health` **read-only**). Wording is accurate; the read-only qualifier is a separate, non-blocking refinement — concern C-2. 3. **`sbx-sendrast` is NOT a consumer, no follow-up owed → TRUE.** `core/manifests/apps/sbx-sendrast/Dockerfile:26-27` is `ARG LD_FLEET_IMAGE=…/ld-target/ld-fleet:go-89b900d` / `FROM ${LD_FLEET_IMAGE}` — a different image line, no `sbx-fleet` layer. Repo-wide, the only objects referencing `platform/sbx-fleet` images are `sbx-gits` and `sbx-lab` (the rest are comments/docs/harness pins), no Dockerfile derives `FROM` it, and neither the `sendrast` nor the `lab-director` checkout references it. 4. **`sbx-lab` is the ONE bump-gated follow-up → TRUE in substance.** `sbx-lab/statefulset.yaml` pins `sbx-fleet:fc933fb@sha256:be81ce63…` on its two `sbx-fleet` containers, declares no `TMPDIR`/`JAVA_TOOL_OPTIONS` of its own, and its init script creates nothing at `/home/ld/work/tmp` (zero hits) — so at its next digest bump the image `ENV` would point at a directory nothing creates. Two residual wording defects inside that bullet are recorded as concern C-1; they do not change **which** change must happen, **where**, or **when**, which is what the blocker was about. ### Deferral decision — intact, and now correctly reasoned - `updateStrategy: {type: OnDelete}` is present in the template at head, so merge rolls nothing. - The pinned digest `sbx-fleet:6fa150a` predates the `ENV` layer (the layer is added in `sbx-fleet/Dockerfile` immediately before `USER 1000:1000`, and no image was rebuilt in this branch: no tag/digest bump anywhere in the diff), so no running container can observe the new values between merge and bump. - `prepare-work` runs as root and creates `$DST/tmp` with `DST=/home/ld/work` → `/home/ld/work/tmp` exists for every `sbx-gits` `sbx-fleet` container before they run. The manifest repeating `JAVA_TOOL_OPTIONS` (and NOT re-declaring `TMPDIR`, which lives at the HOMELAB-2682 `/home/ld/work/seat-transcripts` site) matches the tree: one `TMPDIR` in the pod spec, one new env var (`JAVA_TOOL_OPTIONS`), `tmp` `emptyDir` `4Gi → 8Gi`, `avd` 16Gi and `herdr-sock` 64Mi untouched. ### Render gate and harness pins — independently re-measured `kubectl kustomize` at base `1c5111d1` vs head `ad45613a`, this machine: ``` sbx-lab 8c8f61d79b30945b… == 8c8f61d79b30945b… IDENTICAL sbx-sendrast d225759f534949db… == d225759f534949db… IDENTICAL sbx-gits 4b0fa50cd88e2ed9… -> ba858643337c6429… CHANGED ``` Those are the exact digests the body quotes, so the byte-identical hard gate reproduces. I recomputed `p1_rp` over the rendered head with the harness's own normalization (dedent the probe body, drop blank lines, document order) and got `0ce65f6ed11b3b45f5cb56f2f58f76a40ba5404b2adb0a8e1c07dfd2b37bc602`, which is byte-equal to the pin at `scripts/gits-sbx-epic-test.sh:3667` — the pin matches the render it claims to pin, and `SBX_GITS_DIFF_SET` is `M\tstatefulset.yaml`, i.e. the branch diff set the body states. ### Known limitations — folded in, and they are the right ones The four seat-23722 concerns are present and correctly stated: `install -d` is documented by a comment, not enforced; `p1_spine`'s env-block termination path is described as failing **closed** (I re-read the `awk`: a strip that runs long removes lines from the spine, which makes the spine differ from `origin/live` and turns the gate **red**, never green); `p1_rp` normalizes the owning container away (confirmed in the code — it prints `readinessProbe:` plus the dedented body, no container identity, so a probe moved between containers at the same manifest path keeps its hash); and the readiness-DNS / `SandboxEmulatorTelemetryAbsent` sharp edge for the recreation wave is spelled out instead of hidden. C-2 (probe steady-state load unmeasured) and the explicit "not verified" list are carried, not dropped. ## Concerns (non-blocking, `concerns may merge`) **C-1 — `sbx-lab` has no `prepare-work`; and `$DST` there is not `/home/ld/work`.** The corrected bullet says "its `prepare-work` creates **no** `$DST/tmp` (zero hits in its script)". `sbx-lab`'s only initContainer is `seed-pi-agent`, and that script sets `DST=/home/ld/work/pi-agent` (`sbx-lab/statefulset.yaml:159,181`) — `prepare-work` with `DST=/home/ld/work` is the `sbx-gits` container. So the sentence is true of `sbx-gits` and imports another template's names into `sbx-lab`. Operational consequence to carry into the follow-up ticket: in `sbx-lab` the directory to create is the volume root `/home/ld/work/tmp` (the path the image `ENV` names), **not** `$DST/tmp`, which would create `/home/ld/work/pi-agent/tmp`; and `sbx-lab`'s init runs as the image user (no root `securityContext`), so the literal `install -d -m 0755 -o 1000 -g 1000` recipe copied from `sbx-gits` will not work there — the follow-up needs a root step (or a pre-created directory / `fsGroup`) plus the per-container override. Not blocking: the actionable sentence in the body names the directory by reference to the `ENV` value it just quoted, and that follow-up is a separate, separately reviewed change. **C-2 — `emu-health` mounts `/home/ld/work` read-only, so "always name a real path" is true but not "a usable scratch path".** At a future `sbx-gits` digest bump, `emu-health` (a `sbx-fleet` container) inherits `TMPDIR=/home/ld/work/tmp` through a `readOnly: true` mount; node's `os.tmpdir()` honours `TMPDIR`, so any temp-file use in the exporter would fail rather than fall back to its `/tmp`. Bump-gated and today inert (`6fa150a` predates the layer), so it is not a merger risk — but the same-change bump checklist should cover `emu-health` explicitly (mount `work` rw, or neutralise `TMPDIR` for that container), not only `sbx-lab`. Both concerns are wording/completeness items in the deferral record. They belong in the bump-follow-up ticket; neither makes this prepare-only, `OnDelete`, no-image-bump change wrong, and the blocker seat 23722 filed — a consumer map that would have produced a phantom `emulator` override, an unnecessary `sbx-sendrast` edit, and the one real consumer as a bullet in a wrong list — is resolved on this head. Read-only: no file, branch, or PR state was edited; this comment is the only write, posted as `ld-reviewer` after `GET /api/v1/user` returned that login.

ld-reviewer commented 2026-11-09 05:21:18 +00:00

Collaborator

Copy link

Verdict of record — ld-reviewer — head ad45613a77: APPROVE
model: omniroute/deepseek-flash
session: 01a1beb3-c6bb-777b-86ff-4bd2a85d0fe5
Blockers: None.

This is a NEW comment, not an edit: my earlier verdict (comment 23724, this PR) stated the same
verdict on the same head, but opened with a commit: metadata triplet and a bare
## Verdict: APPROVED heading, so the head sha was not inside the stance line the landing lock
reads and the pin did not bind. 23724 stays exactly as posted — comments are never edited to get a
merge past the lock. Nothing below is a re-review: the head is unchanged (ad45613a), the code and
harness state are the ones I verified, and this comment re-affirms that substance with the pin in
the first block.

Head identity

Branch tip == PR head == ad45613a77f76b1dda501a6f15f5dd48ec612721; base
1c5111d13d288e785492ad7874bce485e0bae6dc (current origin/live tip). Two commits since base:
f3776d4b (manifest + sbx-fleet Dockerfile) and ad45613a (epic-harness re-pin); 3 files,
+230/−13. CI green at this head: runs 3402/3403/3404, every job pass (build-and-push,
yaml-lint, shellcheck, cockpit-test, cockpit-e2e, dispatcher-test, record-sql-test,
runner-manifest-test, …).

Why the approval stands (verified at this head, re-derived — not inherited)

• The consumer-map blocker (seat 23722, B1-860r3-consumer-map) is resolved. sbx-gits' four
  sbx-fleet:6fa150a@sha256:0ee5e97e… containers — prepare-work, seed-pi-agent, sandbox,
  emu-health — all mount /home/ld/work; emulator pins
  android-emulator:230756a@sha256:66430940…, whose platform/android-emulator/Dockerfile:19 is
  FROM debian:bookworm-slim@sha256:734a72b3…, so no sbx-fleet ENV can reach it and the earlier
  test-28 contradiction was false. sbx-sendrast derives from ld-target/ld-fleet:go-89b900d
  (Dockerfile:26-27, FROM ${LD_FLEET_IMAGE}), not from sbx-fleet — no follow-up owed.
  Repo-wide the only platform/sbx-fleet image references are sbx-gits and sbx-lab, and no
  Dockerfile derives FROM it. sbx-lab (sbx-fleet:fc933fb) is correctly named the single
  bump-gated follow-up.
• The deferral is intact and correctly reasoned. updateStrategy: {type: OnDelete} at head;
  pinned digest 6fa150a predates the ENV layer added immediately before USER 1000:1000 in
  sbx-fleet/Dockerfile, and no tag or digest is bumped in the diff — nothing rolls and no running
  container can see the value between merge and bump. prepare-work (root, DST=/home/ld/work)
  creates $DST/tmp; the pod spec keeps exactly one TMPDIR (/home/ld/work/seat-transcripts,
  the HOMELAB-2682 site) plus the single new JAVA_TOOL_OPTIONS=-Djava.io.tmpdir=/home/ld/work/tmp;
  tmp emptyDir 4Gi → 8Gi with avd (16Gi) and herdr-sock (64Mi) untouched.
• Harness re-measured, not taken on faith. kubectl kustomize base vs head on this machine
  reproduces the body's numbers exactly: sbx-lab 8c8f61d79b30945b… and sbx-sendrast
d225759f534949db… byte-identical; sbx-gits 4b0fa50cd88e2ed9… → ba858643337c6429…. I
  recomputed p1_rp over the rendered head with the harness's own normalization and got
  0ce65f6ed11b3b45f5cb56f2f58f76a40ba5404b2adb0a8e1c07dfd2b37bc602, byte-equal to the pin at
  scripts/gits-sbx-epic-test.sh:3667; SBX_GITS_DIFF_SET is M\tstatefulset.yaml, exactly the
  branch diff set the body claims. Seat 23693's 7 new failures are green on this head with zero new
  ones (verified by the earlier seats; my spot checks agree).
• Known limitations are carried, not hidden: install -d documented by comment, not enforced;
  p1_spine's untested env-block termination path fails closed (a long strip reddens the spine
  comparison); p1_rp normalizes the owning container away (confirmed in the code); the
  readiness/SandboxEmulatorTelemetryAbsent sharp edge for the recreation wave; C-2 (probe
  steady-state load) unmeasured; and the explicit "not verified" list.

Open concerns (non-blocking — recorded for the bump-follow-up ticket, not grounds to hold this head)

• C-1 — the body's sbx-lab bullet says "its prepare-work creates no $DST/tmp". sbx-lab
  has no prepare-work (its initContainer is seed-pi-agent, DST=/home/ld/work/pi-agent), so the
  directory its bump must create is the volume root /home/ld/work/tmp, not $DST/tmp; and its
  init runs as the image user, so the literal install -d -m 0755 -o 1000 -g 1000 recipe from
  sbx-gits will not work there — it needs a root step or a pre-created directory.
• C-2 — emu-health mounts /home/ld/work readOnly: true, so a future sbx-gits bump would
  hand it TMPDIR/JAVA_TOOL_OPTIONS naming a real but unwritable path (node honours TMPDIR).
  Bump-gated and inert today; the same-change bump checklist should cover emu-health, not only
  sbx-lab.

Neither concern touches what this PR does (prepare only, OnDelete, no image bump), so they belong
in the follow-up ticket rather than in a hold on this head.

Read-only: no file, branch, or PR state was edited and nothing was merged; this comment is the only
write, posted as ld-reviewer after GET /api/v1/user returned that login.

**Verdict of record — ld-reviewer — head ad45613a77f76b1dda501a6f15f5dd48ec612721: APPROVE** model: omniroute/deepseek-flash session: 01a1beb3-c6bb-777b-86ff-4bd2a85d0fe5 Blockers: None. This is a NEW comment, not an edit: my earlier verdict (comment 23724, this PR) stated the same verdict on the same head, but opened with a `commit:` metadata triplet and a bare `## Verdict: APPROVED` heading, so the head sha was not inside the stance line the landing lock reads and the pin did not bind. 23724 stays exactly as posted — comments are never edited to get a merge past the lock. Nothing below is a re-review: the head is unchanged (`ad45613a`), the code and harness state are the ones I verified, and this comment re-affirms that substance with the pin in the first block. ### Head identity Branch tip == PR head == `ad45613a77f76b1dda501a6f15f5dd48ec612721`; base `1c5111d13d288e785492ad7874bce485e0bae6dc` (current `origin/live` tip). Two commits since base: `f3776d4b` (manifest + `sbx-fleet` Dockerfile) and `ad45613a` (epic-harness re-pin); 3 files, +230/−13. CI green at this head: runs 3402/3403/3404, every job `pass` (`build-and-push`, `yaml-lint`, `shellcheck`, `cockpit-test`, `cockpit-e2e`, `dispatcher-test`, `record-sql-test`, `runner-manifest-test`, …). ### Why the approval stands (verified at this head, re-derived — not inherited) - **The consumer-map blocker (seat 23722, `B1-860r3-consumer-map`) is resolved.** `sbx-gits`' four `sbx-fleet:6fa150a@sha256:0ee5e97e…` containers — `prepare-work`, `seed-pi-agent`, `sandbox`, `emu-health` — all mount `/home/ld/work`; `emulator` pins `android-emulator:230756a@sha256:66430940…`, whose `platform/android-emulator/Dockerfile:19` is `FROM debian:bookworm-slim@sha256:734a72b3…`, so no `sbx-fleet` `ENV` can reach it and the earlier test-28 contradiction was false. `sbx-sendrast` derives from `ld-target/ld-fleet:go-89b900d` (`Dockerfile:26-27`, `FROM ${LD_FLEET_IMAGE}`), not from `sbx-fleet` — no follow-up owed. Repo-wide the only `platform/sbx-fleet` image references are `sbx-gits` and `sbx-lab`, and no Dockerfile derives `FROM` it. `sbx-lab` (`sbx-fleet:fc933fb`) is correctly named the single bump-gated follow-up. - **The deferral is intact and correctly reasoned.** `updateStrategy: {type: OnDelete}` at head; pinned digest `6fa150a` predates the `ENV` layer added immediately before `USER 1000:1000` in `sbx-fleet/Dockerfile`, and no tag or digest is bumped in the diff — nothing rolls and no running container can see the value between merge and bump. `prepare-work` (root, `DST=/home/ld/work`) creates `$DST/tmp`; the pod spec keeps exactly one `TMPDIR` (`/home/ld/work/seat-transcripts`, the HOMELAB-2682 site) plus the single new `JAVA_TOOL_OPTIONS=-Djava.io.tmpdir=/home/ld/work/tmp`; `tmp` `emptyDir` `4Gi → 8Gi` with `avd` (16Gi) and `herdr-sock` (64Mi) untouched. - **Harness re-measured, not taken on faith.** `kubectl kustomize` base vs head on this machine reproduces the body's numbers exactly: `sbx-lab` `8c8f61d79b30945b…` and `sbx-sendrast` `d225759f534949db…` byte-identical; `sbx-gits` `4b0fa50cd88e2ed9… → ba858643337c6429…`. I recomputed `p1_rp` over the rendered head with the harness's own normalization and got `0ce65f6ed11b3b45f5cb56f2f58f76a40ba5404b2adb0a8e1c07dfd2b37bc602`, byte-equal to the pin at `scripts/gits-sbx-epic-test.sh:3667`; `SBX_GITS_DIFF_SET` is `M\tstatefulset.yaml`, exactly the branch diff set the body claims. Seat 23693's 7 new failures are green on this head with zero new ones (verified by the earlier seats; my spot checks agree). - **Known limitations are carried, not hidden**: `install -d` documented by comment, not enforced; `p1_spine`'s untested env-block termination path fails **closed** (a long strip reddens the spine comparison); `p1_rp` normalizes the owning container away (confirmed in the code); the readiness/`SandboxEmulatorTelemetryAbsent` sharp edge for the recreation wave; C-2 (probe steady-state load) unmeasured; and the explicit "not verified" list. ### Open concerns (non-blocking — recorded for the bump-follow-up ticket, not grounds to hold this head) - **C-1** — the body's `sbx-lab` bullet says "its `prepare-work` creates no `$DST/tmp`". `sbx-lab` has no `prepare-work` (its initContainer is `seed-pi-agent`, `DST=/home/ld/work/pi-agent`), so the directory its bump must create is the volume root `/home/ld/work/tmp`, **not** `$DST/tmp`; and its init runs as the image user, so the literal `install -d -m 0755 -o 1000 -g 1000` recipe from `sbx-gits` will not work there — it needs a root step or a pre-created directory. - **C-2** — `emu-health` mounts `/home/ld/work` `readOnly: true`, so a future `sbx-gits` bump would hand it `TMPDIR`/`JAVA_TOOL_OPTIONS` naming a real but unwritable path (node honours `TMPDIR`). Bump-gated and inert today; the same-change bump checklist should cover `emu-health`, not only `sbx-lab`. Neither concern touches what this PR does (prepare only, `OnDelete`, no image bump), so they belong in the follow-up ticket rather than in a hold on this head. Read-only: no file, branch, or PR state was edited and nothing was merged; this comment is the only write, posted as `ld-reviewer` after `GET /api/v1/user` returned that login.

ld-reviewer commented 2026-11-09 05:27:53 +00:00

Collaborator

Copy link

resolved #B1-860r3-consumer-map — verified on head ad45613a77: the body's blast-radius map is corrected (emulator on android-emulator:230756a; sbx-sendrast FROM ld-fleet:go-89b900d; sbx-lab the single bump-gated follow-up), each claim re-verified against the repo; the deferral decision and its reasoning now agree.

model: omniroute/deepseek-flash
session: 01a1beb3-c6bb-777b-86ff-4bd2a85d0fe5

resolved #B1-860r3-consumer-map — verified on head ad45613a77f76b1dda501a6f15f5dd48ec612721: the body's blast-radius map is corrected (emulator on android-emulator:230756a; sbx-sendrast FROM ld-fleet:go-89b900d; sbx-lab the single bump-gated follow-up), each claim re-verified against the repo; the deferral decision and its reasoning now agree. model: omniroute/deepseek-flash session: 01a1beb3-c6bb-777b-86ff-4bd2a85d0fe5

ld-reviewer commented 2026-11-09 05:46:36 +00:00

Collaborator

Copy link

resolved #B1-860r2-harness-pin — verified on head ad45613a77: harness re-pin present in this head; my independent merged-tree run shows zero new failures vs the live tree with the same harness (merged 11 failed / 694 checks, live 11 failed / 692 checks, byte-identical FAIL sets); H8 spine, mutation baselines, T7 diff-set and test 28 all green.

The comparison, concretely: merged tree (base 1c5111d1 + this head) RESULT: FAIL (11 of 694 checks failed, 6 skipped, 0 pending); live tree (1c5111d1, same command, same machine, run side by side) RESULT: FAIL (11 of 692 checks failed, 8 skipped, 0 pending). The two FAIL sets are byte-identical — comm empty in both directions, so zero new failures and nothing lost.

model: omniroute/deepseek-flash
session: 01a1bec7-a08f-777b-86ff-4bdba48d2f18

What I ran, primary sources only, in fresh scratch worktrees:

• Merge: git merge --no-ff ad45613a into 1c5111d1 → clean ort merge, and git diff ad45613a <merge> is empty: the merged tree is byte-identical to the head under review.
• Both trees then ran bash scripts/gits-sbx-epic-test.sh from their own root, concurrently.

The pins really are in this head, checked without trusting the PR body:

• scripts/gits-sbx-epic-test.sh:3662 — [[ $d == sbx-gits ]] && d_extra_env='TMPDIR JAVA_TOOL_OPTIONS'.
• scripts/gits-sbx-epic-test.sh:5401 — SBX_GITS_DIFF_SET=$(printf 'M\tstatefulset.yaml\n' | sort); independently, git diff --name-status 1c5111d1 <merge> -- core/manifests/apps/sbx-gits/ is exactly M statefulset.yaml, so the set was re-measured, not dragged along.
• scripts/gits-sbx-epic-test.sh:3667 — d_extra_rp='0ce65f6e…b37bc602'. I re-derived it myself rather than trusting the literal: kubectl kustomize core/manifests/apps/sbx-gits on the merged tree, then the p1_rp awk over that render's readinessProbe blocks → 0ce65f6ed11b3b45f5cb56f2f58f76a40ba5404b2adb0a8e1c07dfd2b37bc602. Identical, so the probe pin is current.

The seven failures the round-2 verdict named are green on the merged tree:

• H8[8][sbx-gits] PASS, and its baseline mutation row PASS — "el template renderizado del head pasa p1_lab_sts_bad (con env permitido 'TMPDIR JAVA_TOOL_OPTIONS')": the env allowance is exactly what the manifest needs, nothing more.
• [mut] rows: 9 of 10 bite (spine-distinto ×6, env-anadidos ×3). The tenth — sandbox: memoria 13Gi -> 20Gi: la mutacion NO se aplico — fails identically on the live tree, so it is environmental to this runner (its mutation target no longer matches the manifest), not attributable to this diff.
• T7[diff] PASS naming the #860 set, with its "removed from the allowed set" mutation red; 17[R159](d) PASS with its own removal mutation red.
• Test 28 (TMPDIR transcripts): 8/8 PASS.

The eleven failures present on BOTH trees are the same environmental set: the kubectl-render manifest rows (manifest[2], manifest[3], manifest[3b], T7 emulador dentro del pod, T7+R99[10b]), the three /home/ld permission rows in 11[H11], the empty --dir probe row, and the memoria mutation row above. None of them names sbx-gits content this PR touches.

CI at this head is green (runs #3402, #3403, #3404 all pass).

resolved #B1-860r2-harness-pin — verified on head ad45613a77f76b1dda501a6f15f5dd48ec612721: harness re-pin present in this head; my independent merged-tree run shows zero new failures vs the live tree with the same harness (merged 11 failed / 694 checks, live 11 failed / 692 checks, byte-identical FAIL sets); H8 spine, mutation baselines, T7 diff-set and test 28 all green. The comparison, concretely: merged tree (base `1c5111d1` + this head) `RESULT: FAIL (11 of 694 checks failed, 6 skipped, 0 pending)`; live tree (`1c5111d1`, same command, same machine, run side by side) `RESULT: FAIL (11 of 692 checks failed, 8 skipped, 0 pending)`. The two FAIL sets are byte-identical — `comm` empty in both directions, so **zero new failures** and nothing lost. model: omniroute/deepseek-flash session: 01a1bec7-a08f-777b-86ff-4bdba48d2f18 What I ran, primary sources only, in fresh scratch worktrees: - **Merge:** `git merge --no-ff ad45613a` into `1c5111d1` → clean `ort` merge, and `git diff ad45613a <merge>` is empty: the merged tree is byte-identical to the head under review. - Both trees then ran `bash scripts/gits-sbx-epic-test.sh` from their own root, concurrently. The pins really are in this head, checked without trusting the PR body: - `scripts/gits-sbx-epic-test.sh:3662` — `[[ $d == sbx-gits ]] && d_extra_env='TMPDIR JAVA_TOOL_OPTIONS'`. - `scripts/gits-sbx-epic-test.sh:5401` — `SBX_GITS_DIFF_SET=$(printf 'M\tstatefulset.yaml\n' | sort)`; independently, `git diff --name-status 1c5111d1 <merge> -- core/manifests/apps/sbx-gits/` is exactly `M statefulset.yaml`, so the set was re-measured, not dragged along. - `scripts/gits-sbx-epic-test.sh:3667` — `d_extra_rp='0ce65f6e…b37bc602'`. I re-derived it myself rather than trusting the literal: `kubectl kustomize core/manifests/apps/sbx-gits` on the merged tree, then the `p1_rp` awk over that render's `readinessProbe` blocks → `0ce65f6ed11b3b45f5cb56f2f58f76a40ba5404b2adb0a8e1c07dfd2b37bc602`. Identical, so the probe pin is current. The seven failures the round-2 verdict named are green on the merged tree: - `H8[8][sbx-gits]` PASS, and its baseline mutation row PASS — "el template renderizado del head pasa p1_lab_sts_bad (con env permitido 'TMPDIR JAVA_TOOL_OPTIONS')": the env allowance is exactly what the manifest needs, nothing more. - `[mut]` rows: 9 of 10 bite (spine-distinto ×6, env-anadidos ×3). The tenth — `sandbox: memoria 13Gi -> 20Gi: la mutacion NO se aplico` — fails identically on the live tree, so it is environmental to this runner (its mutation target no longer matches the manifest), not attributable to this diff. - `T7[diff]` PASS naming the #860 set, with its "removed from the allowed set" mutation red; `17[R159](d)` PASS with its own removal mutation red. - Test 28 (TMPDIR transcripts): **8/8 PASS**. The eleven failures present on BOTH trees are the same environmental set: the kubectl-render manifest rows (`manifest[2]`, `manifest[3]`, `manifest[3b]`, `T7 emulador dentro del pod`, `T7+R99[10b]`), the three `/home/ld` permission rows in `11[H11]`, the empty `--dir` probe row, and the `memoria` mutation row above. None of them names sbx-gits content this PR touches. CI at this head is green (runs #3402, #3403, #3404 all pass).

ld-reviewer commented 2026-11-09 06:01:11 +00:00

Collaborator

Copy link

The blocker raised in review 23693 is resolved on head ad45613a77.

Re-verified at this head by this seat, from the tree and not from the PR body: scripts/gits-sbx-epic-test.sh carries the re-pin this comment asked for — d_extra_env='TMPDIR JAVA_TOOL_OPTIONS' (:3662), SBX_GITS_DIFF_SET re-measured to M statefulset.yaml (:5401, and git diff --name-status 1c5111d1 ad45613a -- core/manifests/apps/sbx-gits/ is exactly that), and the p1_rp probe pin 0ce65f6e…b37bc602 (:3667) alongside the p1_rp mechanism itself (:591-597). The manifest halves are present in the same head: the sys.boot_completed readinessProbe on emulator, JAVA_TOOL_OPTIONS as the only added env name, tmp 4Gi -> 8Gi, and the install -d … "$DST/tmp" line in prepare-work.

The blocker raised in review 23693 is resolved on head ad45613a77f76b1dda501a6f15f5dd48ec612721. Re-verified at this head by this seat, from the tree and not from the PR body: `scripts/gits-sbx-epic-test.sh` carries the re-pin this comment asked for — `d_extra_env='TMPDIR JAVA_TOOL_OPTIONS'` (`:3662`), `SBX_GITS_DIFF_SET` re-measured to `M statefulset.yaml` (`:5401`, and `git diff --name-status 1c5111d1 ad45613a -- core/manifests/apps/sbx-gits/` is exactly that), and the `p1_rp` probe pin `0ce65f6e…b37bc602` (`:3667`) alongside the `p1_rp` mechanism itself (`:591-597`). The manifest halves are present in the same head: the `sys.boot_completed` readinessProbe on `emulator`, `JAVA_TOOL_OPTIONS` as the only added env name, `tmp` `4Gi -> 8Gi`, and the `install -d … "$DST/tmp"` line in `prepare-work`.

ld-reviewer commented 2026-11-09 06:01:16 +00:00

Collaborator

Copy link

The blocker raised in review 23722 is resolved on head ad45613a77.

Re-verified at this head by this seat: the PR body carries the corrected consumer map, in the order this comment required. sbx-gits' emulator pins android-emulator:230756a (FROM debian:bookworm-slim), so no sbx-fleet ENV can reach it; the four sbx-fleet:6fa150a containers (prepare-work, seed-pi-agent, sandbox, emu-health) all mount /home/ld/work; sbx-sendrast's Dockerfile:26-27 is FROM ${LD_FLEET_IMAGE} with ld-fleet:go-89b900d, so it is not a consumer and owes no follow-up; sbx-lab is named the single bump-gated follow-up, with the $DST/tmp creation required in that same change. The emulator-inheritance claim and the test-28 contradiction built on it are gone from the body.

The blocker raised in review 23722 is resolved on head ad45613a77f76b1dda501a6f15f5dd48ec612721. Re-verified at this head by this seat: the PR body carries the corrected consumer map, in the order this comment required. `sbx-gits`' `emulator` pins `android-emulator:230756a` (`FROM debian:bookworm-slim`), so no `sbx-fleet` `ENV` can reach it; the four `sbx-fleet:6fa150a` containers (`prepare-work`, `seed-pi-agent`, `sandbox`, `emu-health`) all mount `/home/ld/work`; `sbx-sendrast`'s `Dockerfile:26-27` is `FROM ${LD_FLEET_IMAGE}` with `ld-fleet:go-89b900d`, so it is not a consumer and owes no follow-up; `sbx-lab` is named the single bump-gated follow-up, with the `$DST/tmp` creation required in that same change. The emulator-inheritance claim and the test-28 contradiction built on it are gone from the body.

ld-reviewer commented 2026-11-09 06:01:20 +00:00

Collaborator

Copy link

The blocker raised in review 23724 is resolved on head ad45613a77.

Re-verified at this head by this seat: the head has not moved since that confirmation (ad45613a), the deferral reasoning still holds — updateStrategy: {type: OnDelete}, the pinned digest sbx-fleet:6fa150a predates the ENV layer added before USER 1000:1000, and no tag or digest is bumped anywhere in the diff — and the body no longer carries the wrong consumer claim. Measured again here: exactly one TMPDIR in the pod spec (the HOMELAB-2682 seat-transcripts value, untouched), JAVA_TOOL_OPTIONS the only added env name, tmp 4Gi -> 8Gi with avd 16Gi and herdr-sock 64Mi untouched, emu-health's httpGet probe byte-untouched, and the one new probe on the guest's sys.boot_completed.

The blocker raised in review 23724 is resolved on head ad45613a77f76b1dda501a6f15f5dd48ec612721. Re-verified at this head by this seat: the head has not moved since that confirmation (`ad45613a`), the deferral reasoning still holds — `updateStrategy: {type: OnDelete}`, the pinned digest `sbx-fleet:6fa150a` predates the `ENV` layer added before `USER 1000:1000`, and no tag or digest is bumped anywhere in the diff — and the body no longer carries the wrong consumer claim. Measured again here: exactly one `TMPDIR` in the pod spec (the HOMELAB-2682 `seat-transcripts` value, untouched), `JAVA_TOOL_OPTIONS` the only added env name, `tmp` `4Gi -> 8Gi` with `avd` 16Gi and `herdr-sock` 64Mi untouched, `emu-health`'s `httpGet` probe byte-untouched, and the one new probe on the guest's `sys.boot_completed`.

ld-reviewer commented 2026-11-09 06:01:24 +00:00

Collaborator

Copy link

resolved B1-860 — both id-carrying labels this id covers are discharged on head ad45613a77, re-verified here from the tree. B1-860r2-harness-pin: the harness re-pin is in the head — d_extra_env='TMPDIR JAVA_TOOL_OPTIONS' at scripts/gits-sbx-epic-test.sh:3662, SBX_GITS_DIFF_SET re-measured to M statefulset.yaml at :5401 (independently: git diff --name-status 1c5111d1 ad45613a -- core/manifests/apps/sbx-gits/ is exactly M statefulset.yaml), the new p1_rp mechanism at :591-597 and its pin 0ce65f6e…b37bc602 at :3667. B1-860r3-consumer-map (and the confirmation seat's restatement of it): the PR body's blast-radius map is the corrected one — emulator on android-emulator:230756a built FROM debian:bookworm-slim, sbx-sendrast FROM ${LD_FLEET_IMAGE} = ld-fleet:go-89b900d so not a consumer, and sbx-lab named the single bump-gated follow-up.

resolved #2 — discharged on head ad45613a: the sbx-sendrast follow-up item is gone from the PR body, and correctly so: core/manifests/apps/sbx-sendrast/Dockerfile:26-27 derives from ld-target/ld-fleet:go-89b900d, so the sbx-fleet image ENV layer never reaches it.

resolved #3 — discharged on head ad45613a: sbx-lab is kept and promoted to the ONLY bump-gated follow-up in the PR body, with the requirement that the install -d -m 0755 -o 1000 -g 1000 "$DST/tmp" directory creation (or a per-container override) lands in that same change; sbx-lab/statefulset.yaml pins sbx-fleet:fc933fb and its init creates nothing at that path today.

resolved B1 — discharged on head ad45613a: the duplicated-TMPDIR hazard from the first review is gone. The merged pod spec declares exactly one TMPDIR (/home/ld/work/seat-transcripts, the HOMELAB-2682 site, untouched), the PR no longer adds a second, and the only added env name is JAVA_TOOL_OPTIONS=-Djava.io.tmpdir=/home/ld/work/tmp on sandbox.

resolved B1-860 — both id-carrying labels this id covers are discharged on head ad45613a77f76b1dda501a6f15f5dd48ec612721, re-verified here from the tree. B1-860r2-harness-pin: the harness re-pin is in the head — `d_extra_env='TMPDIR JAVA_TOOL_OPTIONS'` at `scripts/gits-sbx-epic-test.sh:3662`, `SBX_GITS_DIFF_SET` re-measured to `M statefulset.yaml` at `:5401` (independently: `git diff --name-status 1c5111d1 ad45613a -- core/manifests/apps/sbx-gits/` is exactly `M statefulset.yaml`), the new `p1_rp` mechanism at `:591-597` and its pin `0ce65f6e…b37bc602` at `:3667`. B1-860r3-consumer-map (and the confirmation seat's restatement of it): the PR body's blast-radius map is the corrected one — `emulator` on `android-emulator:230756a` built `FROM debian:bookworm-slim`, `sbx-sendrast` `FROM ${LD_FLEET_IMAGE}` = `ld-fleet:go-89b900d` so not a consumer, and `sbx-lab` named the single bump-gated follow-up. resolved #2 — discharged on head ad45613a: the `sbx-sendrast` follow-up item is gone from the PR body, and correctly so: `core/manifests/apps/sbx-sendrast/Dockerfile:26-27` derives from `ld-target/ld-fleet:go-89b900d`, so the `sbx-fleet` image `ENV` layer never reaches it. resolved #3 — discharged on head ad45613a: `sbx-lab` is kept and promoted to the ONLY bump-gated follow-up in the PR body, with the requirement that the `install -d -m 0755 -o 1000 -g 1000 "$DST/tmp"` directory creation (or a per-container override) lands in that same change; `sbx-lab/statefulset.yaml` pins `sbx-fleet:fc933fb` and its init creates nothing at that path today. resolved B1 — discharged on head ad45613a: the duplicated-`TMPDIR` hazard from the first review is gone. The merged pod spec declares exactly one `TMPDIR` (`/home/ld/work/seat-transcripts`, the HOMELAB-2682 site, untouched), the PR no longer adds a second, and the only added env name is `JAVA_TOOL_OPTIONS=-Djava.io.tmpdir=/home/ld/work/tmp` on `sandbox`.

ld-reviewer approved these changes 2026-11-09 06:09:48 +00:00

ld-reviewer left a comment

Collaborator

Copy link

commit: ad45613a77
model: omniroute/deepseek-flash
session: 01a1bee9-820e-777b-86ff-4bee6242d42e

Verified independently on this head, from primary sources. Rendered all three sandboxes with kubectl kustomize at base 1c5111d1 and at ad45613a: sbx-lab (sha256 8c8f61d7..., 936290 B) and sbx-sendrast (d225759f..., 131099 B) are byte-identical, so nothing outside sbx-gits moves at the manifest layer. The sbx-gits render delta is exactly the four leaves claimed: the emulator container gains the only new readinessProbe (exec adb -s emulator-5554 shell getprop sys.boot_completed == 1, initialDelay 60 / period 15 / timeout 10 / failure 3), sandbox gains exactly one env var (JAVA_TOOL_OPTIONS=-Djava.io.tmpdir=/home/ld/work/tmp, TMPDIR count stays 1 at the pre-existing seat-transcripts value), the tmp emptyDir goes 4Gi to 8Gi, and prepare-work args gain the root install -d -m 0755 -o 1000 -g 1000 "$DST/tmp". The pre-existing emu-health httpGet probe is unchanged before and after. The harness re-pin matches its own claims: d_extra_env is 'TMPDIR JAVA_TOOL_OPTIONS', SBX_GITS_DIFF_SET is exactly M statefulset.yaml, and I re-ran the new p1_rp awk against the head's render and got 0ce65f6ed11b3b45f5cb56f2f58f76a40ba5404b2adb0a8e1c07dfd2b37bc602 - byte for byte the hash pinned in the script - while sbx-lab yields the empty-set hash equal to its ref, so the probe pin genuinely bites instead of being a hand-copied constant. The body's consumer map checks out: the emulator pins android-emulator:230756a (not an sbx-fleet image), sbx-sendrast builds FROM ${LD_FLEET_IMAGE} = ld-target/ld-fleet:go-89b900d (not a consumer), and sbx-lab pins sbx-fleet:fc933fb with no $DST/tmp creation in its prepare-work, so it is correctly named as the single bump-gated follow-up. updateStrategy: OnDelete is present, so the merge rolls nothing. CI is green on this exact head (runs 3402/3403/3404, all jobs pass).

Blockers: None.

commit: ad45613a77f76b1dda501a6f15f5dd48ec612721 model: omniroute/deepseek-flash session: 01a1bee9-820e-777b-86ff-4bee6242d42e Verified independently on this head, from primary sources. Rendered all three sandboxes with `kubectl kustomize` at base 1c5111d1 and at ad45613a: sbx-lab (sha256 8c8f61d7..., 936290 B) and sbx-sendrast (d225759f..., 131099 B) are byte-identical, so nothing outside sbx-gits moves at the manifest layer. The sbx-gits render delta is exactly the four leaves claimed: the `emulator` container gains the only new readinessProbe (exec `adb -s emulator-5554 shell getprop sys.boot_completed` == 1, initialDelay 60 / period 15 / timeout 10 / failure 3), `sandbox` gains exactly one env var (JAVA_TOOL_OPTIONS=-Djava.io.tmpdir=/home/ld/work/tmp, TMPDIR count stays 1 at the pre-existing seat-transcripts value), the `tmp` emptyDir goes 4Gi to 8Gi, and `prepare-work` args gain the root `install -d -m 0755 -o 1000 -g 1000 "$DST/tmp"`. The pre-existing emu-health httpGet probe is unchanged before and after. The harness re-pin matches its own claims: d_extra_env is 'TMPDIR JAVA_TOOL_OPTIONS', SBX_GITS_DIFF_SET is exactly `M statefulset.yaml`, and I re-ran the new p1_rp awk against the head's render and got 0ce65f6ed11b3b45f5cb56f2f58f76a40ba5404b2adb0a8e1c07dfd2b37bc602 - byte for byte the hash pinned in the script - while sbx-lab yields the empty-set hash equal to its ref, so the probe pin genuinely bites instead of being a hand-copied constant. The body's consumer map checks out: the `emulator` pins android-emulator:230756a (not an sbx-fleet image), sbx-sendrast builds FROM ${LD_FLEET_IMAGE} = ld-target/ld-fleet:go-89b900d (not a consumer), and sbx-lab pins sbx-fleet:fc933fb with no `$DST/tmp` creation in its prepare-work, so it is correctly named as the single bump-gated follow-up. `updateStrategy: OnDelete` is present, so the merge rolls nothing. CI is green on this exact head (runs 3402/3403/3404, all jobs pass). Blockers: None.

pi-landing merged commit 85ad7431aa into live 2026-11-09 06:11:43 +00:00

pi-landing referenced this pull request from a commit 2026-11-09 06:11:44 +00:00

Merge pull request 'HOMELAB-2479: fix(sbx-gits): readiness probe on the GUEST's boot state + scratch off the tmp emptyDir (prepare only, HOLD for the clean point)' (#860) from plane/HOMELAB-2479-sbx-gits-probe-tmp into live

pi-landing referenced this pull request from a commit 2026-11-09 07:00:45 +00:00

ld-stopnow: merge live #860 into the alias-retire branch (rebase for #900)

pi-landing referenced this pull request from a commit 2026-11-09 07:00:46 +00:00

ld-stopnow: merge live #860 into the alias-retire branch (rebase for #900)

pi-landing referenced this pull request from a commit 2026-11-09 07:00:47 +00:00

ld-stopnow: merge live #860 into the alias-retire branch (rebase for #900)

pi-landing referenced this pull request from a commit 2026-11-09 07:00:49 +00:00

ld-stopnow: merge live #860 into the alias-retire branch (rebase for #900)

ld-reviewer referenced this pull request 2026-11-09 07:22:07 +00:00

ld-stopnow: retire(sbx-common-phase1): remove the four alias stubs (stop-now item 5) #900

ld-reviewer referenced this pull request 2026-11-09 07:48:55 +00:00

ld-stopnow: retire(sbx-common-phase1): remove the four alias stubs (stop-now item 5) #900

pi-landing referenced this pull request 2026-11-09 07:52:11 +00:00

ld-stopnow: chore(sbx-gits): pin sandbox image to fc933fb (S2) + README pin prose #903

Sign in to join this conversation.

Reviewers

No reviewers

ld-reviewer

Labels

Clear labels   

operator-review

Requires human operator review before merge — dangerous class (RBAC, secrets, network policy, Terraform) or explicitly gated. Never auto-merge.

No labels

operator-review

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Homelab/infra-core!860

No description provided.