HOMELAB-1665: feat(agent-sandbox): pi sandbox image → Harbor #519

Merged

aaron merged 4 commits from plane/HOMELAB-1665-pi-image into live 2026-09-29 04:39:26 +00:00

Conversation 3 Commits 4 Files changed 4 +438

aaron commented 2026-09-29 04:01:48 +00:00

Owner

Copy link

What

The pi sandbox image the Lab Director fleet's agents run in, built and published by Forgejo CI. Ticket: HOMELAB-1665 (LD0 epic / LD0.2).

New files only:

Path	Role
agent-sandbox/pi-image/Dockerfile	node 24 LTS (digest-pinned) + pi 0.87.1 + git/openssh-client/ripgrep/fd; non-root (uid/gid 1000); ENTRYPOINT ["pi"]
agent-sandbox/pi-image/README.md	pins, tag/promotion rules, volume + user contract for LD0.3, verification recipe, open questions
agent-sandbox/pi-image/examples/pi-pod.example.yaml	reference only, not wired — makes the volume/user contract concrete (LD0.3 owns the real Sandbox)
.forgejo/workflows/pi-image-build.yaml	build → smoke → size gate → push to Harbor

Why

LD0.2's deliverable is a reproducible pi image in the registry the cluster can pull from, so LD0.3 can stand up a real Sandbox without a node-local image. Builds have to happen in CI: the agent user has no docker socket, and contract §12 §4.3 rules in-sandbox builds out ("builds-stay-in-CI").

Design choices worth a reviewer's eye:

• Base pinned by digest (node:24-bookworm-slim@sha256:0e0ff40c…, node 24.21.0). Re-derived with crane digest node:24-bookworm-slim.
• ripgrep + fd-find pre-baked. pi's grep/find tools resolve binaries from PATH first and only download their own copies when absent (dist/utils/tools-manager.js); in a default-deny sandbox that download path is unreachable, so first run would otherwise have no grep/find. Debian ships fd as fdfind (pi accepts that name); the image also symlinks /usr/local/bin/fd.
• npm install -g --ignore-scripts — pi's own containerization guidance (docs/containerization.md).
• Non-root by reusing the base image's node user (uid/gid 1000) rather than minting a second uid-1000 account.
• 0.87.1 / latest only move from live. Any branch push (including this one) publishes only its immutable <short-sha> tag. A feature branch or PR can never republish the alias the fleet pins to — that keeps "merges never deploy" intact while still letting a PR prove the image builds.
• Smoke test asserts the AC line-wise: pi --version must equal the pin (grep -qxE), so a wrapped warning or update notice can't pass as a version match. Also asserts git/rg/fd/node/npm on PATH, uid 1000, and that /workspace + /home/node/.pi/agent exist and are writable.
• Size gate runs inline via scripts/container-image-size-gate.sh (1 GB policy) rather than the reusable image-size-gate.yaml workflow — the reusable one runs in a separate job whose runner may be a different pod, so the locally-built image wouldn't be inspectable.

Naming deviation (already ruled on, documented here as required)

The ticket says agent/pi:<pin>. There is no Harbor project named agent, and the fleet tooling cannot create Harbor projects, so the image publishes to the existing lab-director project as repository pi:

harbor.aaron.reynoza.org/lab-director/pi

That keeps it in the same project family as the historical Lab Director images (lab-director/sandbox-base, lab-director/worker, …). Recorded on HOMELAB-1665 and in the README.

Verify plan

Re-derived evidence, not pasted logs:

1. CI on this branch: build succeeds, both smoke steps pass, size gate passes, <short-sha> tag lands in Harbor (run + digest linked on the ticket).
2. crane manifest harbor.aaron.reynoza.org/lab-director/pi:<sha> and crane config … → User=node, Entrypoint=[pi], PI_CODING_AGENT_DIR present.
3. Post-merge (push to live): the same run publishes 0.87.1 + latest; crane pull harbor.aaron.reynoza.org/lab-director/pi:0.87.1 succeeds. The lab-director project allows anonymous pull (Harbor audit log shows anonymous pulls), so a worker node pulls it through the existing Harbor mirror with no imagePullSecret.
4. LD0.3 (not this PR): a Sandbox based on this image attaches a working pi TUI on PVC-backed /workspace + agent-state.

Auth convention

Uses the repo's existing HARBOR_USERNAME / HARBOR_PASSWORD Actions secrets — the same pair runner-image-build.yaml and docs-build.yaml already use to push to Harbor. No new secrets, no new credentials. (Cross-checked: Harbor's audit log shows those workflows pushing as admin, and the only push-capable robot in the registry is robot$ci-push with project * scope, so lab-director is in scope either way.)

Out of scope / open questions (in the README)

• PVCs and the Sandbox spec are LD0.3 — this PR only fixes the paths the image expects (/workspace, /home/node/.pi/agent).
• The agent-sandbox router defaults to an HTTP service on 8888, and the Go/Python SDKs assume it. This image is a TUI with no HTTP endpoint, so router/SDK-mediated access is a separate decision; kubectl attach works today.
• PI_SKIP_VERSION_CHECK deliberately left unset (ships pi's default behaviour) — flagged as a ruling to make when the Cilium egress policy lands.

Class

C (standard pipeline) — builder does not merge; a fresh-context landing worker owns review/merge.

## What The pi sandbox image the Lab Director fleet's agents run in, built and published by Forgejo CI. Ticket: **HOMELAB-1665** (LD0 epic / LD0.2). New files only: | Path | Role | |---|---| | `agent-sandbox/pi-image/Dockerfile` | node 24 LTS (digest-pinned) + pi 0.87.1 + git/openssh-client/ripgrep/fd; non-root (uid/gid 1000); `ENTRYPOINT ["pi"]` | | `agent-sandbox/pi-image/README.md` | pins, tag/promotion rules, volume + user contract for LD0.3, verification recipe, open questions | | `agent-sandbox/pi-image/examples/pi-pod.example.yaml` | **reference only, not wired** — makes the volume/user contract concrete (LD0.3 owns the real Sandbox) | | `.forgejo/workflows/pi-image-build.yaml` | build → smoke → size gate → push to Harbor | ## Why LD0.2's deliverable is a reproducible pi image in the registry the cluster can pull from, so LD0.3 can stand up a real Sandbox without a node-local image. Builds have to happen in CI: the agent user has no docker socket, and contract §12 §4.3 rules in-sandbox builds out ("builds-stay-in-CI"). Design choices worth a reviewer's eye: - **Base pinned by digest** (`node:24-bookworm-slim@sha256:0e0ff40c…`, node 24.21.0). Re-derived with `crane digest node:24-bookworm-slim`. - **`ripgrep` + `fd-find` pre-baked.** pi's `grep`/`find` tools resolve binaries from `PATH` first and only download their own copies when absent (`dist/utils/tools-manager.js`); in a default-deny sandbox that download path is unreachable, so first run would otherwise have no grep/find. Debian ships `fd` as `fdfind` (pi accepts that name); the image also symlinks `/usr/local/bin/fd`. - **`npm install -g --ignore-scripts`** — pi's own containerization guidance (`docs/containerization.md`). - **Non-root by reusing the base image's `node` user** (uid/gid 1000) rather than minting a second uid-1000 account. - **`0.87.1` / `latest` only move from `live`.** Any branch push (including this one) publishes only its immutable `<short-sha>` tag. A feature branch or PR can never republish the alias the fleet pins to — that keeps "merges never deploy" intact while still letting a PR prove the image builds. - **Smoke test asserts the AC line-wise**: `pi --version` must equal the pin (`grep -qxE`), so a wrapped warning or update notice can't pass as a version match. Also asserts `git`/`rg`/`fd`/`node`/`npm` on PATH, uid 1000, and that `/workspace` + `/home/node/.pi/agent` exist and are writable. - **Size gate** runs inline via `scripts/container-image-size-gate.sh` (1 GB policy) rather than the reusable `image-size-gate.yaml` workflow — the reusable one runs in a *separate job* whose runner may be a different pod, so the locally-built image wouldn't be inspectable. ## Naming deviation (already ruled on, documented here as required) The ticket says `agent/pi:<pin>`. There is **no Harbor project named `agent`**, and the fleet tooling cannot create Harbor projects, so the image publishes to the existing `lab-director` project as repository `pi`: ``` harbor.aaron.reynoza.org/lab-director/pi ``` That keeps it in the same project family as the historical Lab Director images (`lab-director/sandbox-base`, `lab-director/worker`, …). Recorded on HOMELAB-1665 and in the README. ## Verify plan Re-derived evidence, not pasted logs: 1. CI on this branch: build succeeds, both smoke steps pass, size gate passes, `<short-sha>` tag lands in Harbor (run + digest linked on the ticket). 2. `crane manifest harbor.aaron.reynoza.org/lab-director/pi:<sha>` and `crane config …` → `User=node`, `Entrypoint=[pi]`, `PI_CODING_AGENT_DIR` present. 3. Post-merge (push to `live`): the same run publishes `0.87.1` + `latest`; `crane pull harbor.aaron.reynoza.org/lab-director/pi:0.87.1` succeeds. The `lab-director` project allows anonymous pull (Harbor audit log shows anonymous pulls), so a worker node pulls it through the existing Harbor mirror with no imagePullSecret. 4. LD0.3 (not this PR): a Sandbox based on this image attaches a working pi TUI on PVC-backed `/workspace` + agent-state. ## Auth convention Uses the repo's existing `HARBOR_USERNAME` / `HARBOR_PASSWORD` Actions secrets — the same pair `runner-image-build.yaml` and `docs-build.yaml` already use to push to Harbor. No new secrets, no new credentials. (Cross-checked: Harbor's audit log shows those workflows pushing as `admin`, and the only push-capable robot in the registry is `robot$ci-push` with project `*` scope, so `lab-director` is in scope either way.) ## Out of scope / open questions (in the README) - PVCs and the Sandbox spec are **LD0.3** — this PR only fixes the paths the image expects (`/workspace`, `/home/node/.pi/agent`). - The agent-sandbox router defaults to an HTTP service on `8888`, and the Go/Python SDKs assume it. This image is a TUI with no HTTP endpoint, so router/SDK-mediated access is a separate decision; `kubectl attach` works today. - `PI_SKIP_VERSION_CHECK` deliberately left unset (ships pi's default behaviour) — flagged as a ruling to make when the Cilium egress policy lands. ## Class C (standard pipeline) — builder does not merge; a fresh-context landing worker owns review/merge.

aaron added 1 commit 2026-09-29 04:01:48 +00:00

HOMELAB-1665: feat(agent-sandbox): pi sandbox image -> Harbor ... 3d31e9b1d9

The container the Lab Director fleet's agents run in: pi plus the toolchain
builders/reviewers actually shell out to, built and published by Forgejo CI
(the agent user has no local docker socket, and contract §12 §4.3 keeps image
builds in CI).

- Dockerfile: node 24 LTS (digest-pinned) + pi 0.87.1 + git, openssh-client,
  ripgrep, fd-find; non-root (base image's uid/gid 1000 `node`); ENTRYPOINT pi
- ripgrep/fd are pre-baked deliberately: pi resolves them from PATH and only
  downloads its own copies when absent, which a default-deny sandbox cannot do
- volume contract documented for LD0.3: /workspace + /home/node/.pi/agent
  (PI_CODING_AGENT_DIR), both created empty, both owned by uid/gid 1000
- workflow: build + smoke (pi --version == the pin, toolchain present, non-root,
  mount points writable) + the homelab 1 GB size gate; pushes to Harbor using
  the repo's existing HARBOR_USERNAME/HARBOR_PASSWORD convention
- promotion: `0.87.1` / `latest` only move from `live`; any branch publishes
  only its immutable <short-sha> tag, so unmerged work can never republish the
  alias the fleet pins

Naming deviation from the ticket: it says `agent/pi:<pin>`, but no Harbor
project named `agent` exists and the fleet tooling cannot create Harbor
projects, so the image publishes to the existing `lab-director` project as
`lab-director/pi` -> harbor.aaron.reynoza.org/lab-director/pi. Recorded on
HOMELAB-1665 and in agent-sandbox/pi-image/README.md.

examples/pi-pod.example.yaml is reference-only (no ArgoCD app, no CI
validation); LD0.3 owns the real Sandbox.

aaron added 1 commit 2026-09-29 04:05:51 +00:00

HOMELAB-1665: fix(agent-sandbox): read the pushed digest from the registry ... fbb372f932

Run 1466 (push to the branch) built, smoke-tested, size-gated and pushed the
image successfully, then failed at the digest-report step:

  template parsing error: template: :1:2: executing "" at
  <index .RepoDigests 0>: error calling index: reflect: slice index out of range

`docker inspect .RepoDigests` comes back empty on this dind for a pushed tag,
so read the digest back from Harbor with crane instead (crane ships in the
runner image and reuses the docker login). Push + digest are now one step so
the registry call happens while the login is live.

Evidence from run 1466, for the record: pi --version -> 0.87.1, git 2.39.5,
ripgrep 13.0.0, fdfind 8.6.0, node v24.21.0, npm 11.19.0, uid/gid 1000,
/workspace + /home/node/.pi/agent writable, image 0.70 GB < 1 GB policy,
"Login Succeeded", push digest sha256:a478908e767de53740f01605d126532aed6074321f62559d3bd5f620d82ad5bc
(artifact 3d31e9b in harbor.aaron.reynoza.org/lab-director/pi).

aaron commented 2026-09-29 04:11:27 +00:00

Author

Owner

Copy link

CI evidence on head fbb372f (re-derived, not pasted):

Run	Workflow	Result
1470	pi-image-build.yaml (push to branch)	success — build, both smoke steps, size gate, Harbor push, digest
1473	pi-image-build.yaml (PR)	success — build + smoke + size gate, no push
1471	ci.yaml (PR)	success — helm-validate, receiver-test
1472	lint.yaml (PR)	success — yaml-lint, shellcheck, terraform-validate

Key log lines from run 1470:

Ref: refs/heads/plane/HOMELAB-1665-pi-image
pi --version -> 0.87.1
git version 2.39.5 / ripgrep 13.0.0 / fdfind 8.6.0 / v24.21.0 / 11.19.0
test 1000 = 1000  (uid and gid)
Size: 0.70 GB (747349924 bytes)  Limit: 1 GB  →  Image size OK.
Login Succeeded
Pushed: harbor.aaron.reynoza.org/lab-director/pi@sha256:feef409b41ae80b3e80b27e5ec7a1ba6e501038135a973a1db00aec42625efe3

Independent re-derivation from this machine:

crane pull harbor.aaron.reynoza.org/lab-director/pi:3d31e9b   # succeeds anonymously, 767 MB unpacked
crane config harbor.aaron.reynoza.org/lab-director/pi:3d31e9b
  User=node  Entrypoint=[pi]  WorkingDir=/workspace
  PI_CODING_AGENT_DIR=/home/node/.pi/agent  label version=0.87.1  linux/amd64

The pulled config digest b00ff8818449… equals the image id the CI build reported (Successfully built b00ff8818449), so the registry artifact is the artifact CI built.

Promotion gate is holding on purpose: Harbor holds exactly two artifacts (3d31e9b, fbb372f) — 0.87.1/latest are not published from a feature branch. They publish on the live push after merge.

One CI failure occurred on the first push (run 1466) and is fixed in the second commit: the digest step used docker inspect .RepoDigests, which is empty on this dind for a pushed tag (slice index out of range) after a successful push. It now reads the digest from the registry with crane. Kept as a separate commit so the failure and its evidence stay in the history.

**CI evidence on head `fbb372f`** (re-derived, not pasted): | Run | Workflow | Result | |---|---|---| | 1470 | `pi-image-build.yaml` (push to branch) | success — build, both smoke steps, size gate, Harbor push, digest | | 1473 | `pi-image-build.yaml` (PR) | success — build + smoke + size gate, no push | | 1471 | `ci.yaml` (PR) | success — helm-validate, receiver-test | | 1472 | `lint.yaml` (PR) | success — yaml-lint, shellcheck, terraform-validate | Key log lines from run 1470: ``` Ref: refs/heads/plane/HOMELAB-1665-pi-image pi --version -> 0.87.1 git version 2.39.5 / ripgrep 13.0.0 / fdfind 8.6.0 / v24.21.0 / 11.19.0 test 1000 = 1000 (uid and gid) Size: 0.70 GB (747349924 bytes) Limit: 1 GB → Image size OK. Login Succeeded Pushed: harbor.aaron.reynoza.org/lab-director/pi@sha256:feef409b41ae80b3e80b27e5ec7a1ba6e501038135a973a1db00aec42625efe3 ``` Independent re-derivation from this machine: ``` crane pull harbor.aaron.reynoza.org/lab-director/pi:3d31e9b # succeeds anonymously, 767 MB unpacked crane config harbor.aaron.reynoza.org/lab-director/pi:3d31e9b User=node Entrypoint=[pi] WorkingDir=/workspace PI_CODING_AGENT_DIR=/home/node/.pi/agent label version=0.87.1 linux/amd64 ``` The pulled config digest `b00ff8818449…` equals the image id the CI build reported (`Successfully built b00ff8818449`), so the registry artifact is the artifact CI built. **Promotion gate is holding on purpose**: Harbor holds exactly two artifacts (`3d31e9b`, `fbb372f`) — `0.87.1`/`latest` are not published from a feature branch. They publish on the `live` push after merge. One CI failure occurred on the first push (run 1466) and is fixed in the second commit: the digest step used `docker inspect .RepoDigests`, which is empty on this dind for a pushed tag (`slice index out of range`) *after* a successful push. It now reads the digest from the registry with `crane`. Kept as a separate commit so the failure and its evidence stay in the history.

aaron added 1 commit 2026-09-29 04:29:14 +00:00

HOMELAB-1665: fix(agent-sandbox): don't clobber the shared runner docker config; guard the pi pin 4a77d0380d

aaron commented 2026-09-29 04:33:21 +00:00

Author

Owner

Copy link

Landing report — HOMELAB-1665 (LD0.2) · NOT MERGED (escalated)

Head reviewed: fbb372f → fixed to 4a77d03. CI on 4a77d03: green (all four required contexts + both Build Pi Sandbox Image legs).

Independent review ran in two fresh contexts (pi -p --no-session subprocesses, separate from the implementer) with a brief built only from this diff, the ticket AC and the frozen constraints. Verdicts were APPROVE_WITH_CONCERNS (correctness/quality/conventions) and REQUEST_CHANGES (security/provenance) — 1 blocker, 10 concerns, 7 suggestions. Everything below was re-derived from primary sources; nothing is taken from the PR description.

1 blocker — found, fixed, and verified fixed

docker logout destroyed the pod-shared Harbor credential. The k8s label is :host mode (statefulset.yaml: DESIRED_LABELS="infra:host,k8s:host"), so steps run inside the runner container as runner with HOME=/home/runner and DOCKER_CONFIG unset. /home/runner/.docker is a pod-shared emptyDir seeded once per pod from the runner-harbor-robot secret, and the runner daemon needs it to pre-pull private job images (that is exactly what HOMELAB-1634 added it for). docker logout deleted the entry pod-wide.

Forensics (the mtimes match CI pushes to the second):

pod	/home/runner/.docker/config.json mtime	auths
runner-4	2026-09-29 04:03:37 (== push run for 3d31e9b)	{} (16 bytes)
runner-7	2026-09-29 04:07:57 (== push run for fbb372f)	{} (16 bytes)

The seed secret is 202 bytes with auths → harbor.aaron.reynoza.org, so the empty file is not the seed — it is the logout's residue.

Fixed in 4a77d03: export DOCKER_CONFIG="$(mktemp -d)" before docker login, logout removed, digest readback unchanged. Verified behaviourally, not just by reading the diff: after the post-fix push run, all 8 pods' config mtimes are byte-identical to the pre-run snapshot (including runner-4/runner-7) — the workflow no longer writes the shared file at all.

The same footgun pre-dates this PR in runner-image-build.yaml:36 and docs-build.yaml:40 and has already stripped the credential from all 8 pods (HOMELAB-1634's fix is currently ineffective). Filed separately as HOMELAB-1735 — out of this PR's territory, not caused by it.

Other changes in 4a77d03 (both reviewer concerns, cheap and in-territory)

• The documented pi bump flow was a silent no-op. The workflow always passes --build-arg PI_VERSION from its own env:, which overrides the Dockerfile's ARG, so a Dockerfile-only bump (what the README told you to do) would build the old pi and stay green. Added a guard step that fails the build when the two values drift.
• README stated the wrong digest mechanism — it quoted docker inspect --format='{{index .RepoDigests 0}}', the exact command fbb372f removed because it returns nothing on this dind. Corrected, with the bump flow.

Re-derived evidence for the AC

• crane digest node:24-bookworm-slim == the digest pinned in the Dockerfile (sha256:0e0ff40c…) — the base pin is real and derivable.
• The published artifact's config, read from Harbor: User=node, Entrypoint=["pi"], WorkingDir=/workspace, PI_CODING_AGENT_DIR=/home/node/.pi/agent, label version=0.87.1. Exported the image filesystem and confirmed etc/passwd → node:x:1000:1000, /workspace and /home/node/.pi/agent owned 1000/1000, usr/local/bin/pi → the pinned @earendil-works/pi-coding-agent/dist/bundle/cli.js, rg/fdfind present, and no docker/kubectl CLI (contract §12 "builds-stay-in-CI" honoured).
• pi --version on 0.87.1 prints exactly 0.87.1, so the workflow's grep -qxE line-wise assertion is real and not over-fitted.
• Only HARBOR_USERNAME/HARBOR_PASSWORD are referenced — no new secrets. Exactly 4 files added, 0 modified, 0 deleted; nothing outside the declared territory.
• Anonymous pull (the ticket's Verify line): lab-director is public: true; with an empty DOCKER_CONFIG, crane digest harbor.aaron.reynoza.org/lab-director/pi:4a77d03 succeeds anonymously — the path a worker node uses. The node-side pull itself lands with LD0.3's first Sandbox.
• Promotion gate holds: pre-merge Harbor has only 3d31e9b, fbb372f, 4a77d03; 0.87.1/latest are NOT_FOUND, as designed.

Why this is not merged — operator ruling needed (secrets scope)

The review surfaced a credential-scope question beyond the documented additive scope, which per landing rules stops the convoy rather than merging:

The trigger has no branches: filter, so HARBOR_USERNAME/HARBOR_PASSWORD (which push as admin per the Harbor audit log) are now reachable from a push to any branch, where both peers deliberately restrict the same credential to push: branches: [live]. The robot's actual project scope is not documented anywhere in the repo, and the same credential publishes platform/forgejo-runner — the image that runs with the SOPS age key and AWS credentials mounted. "Only live promotes" is enforced by workflow source text, not by the credential. The README also states that guarantee more strongly than the mechanism delivers.

Two options:

1. Merge as-is, with the trigger narrowed in a follow-up (or accept the widened exposure as a deliberate choice and document the credential's scope).
2. Narrow the credential's reach before merge — e.g. keep pre-merge runs credential-free (build + smoke + size gate; publish only from live), or mint a robot scoped to lab-director/pi. The cost is losing the pre-merge <short-sha> evidence tag.

Everything else is green and the blocker is closed, so this is a one-line decision.

Concerns/suggestions posted, not blocking (carried forward for LD0.3)

• README:34 calls <short-sha> "immutable evidence" — it is neither immutable (no retention/immutability control) nor reproducible: 4a77d03 rebuilt the npm layer on a different pod, so its config/layers differ from fbb372f for identical source.
• The 0.87.1/latest promotion is gated on a shell string compare and never asserted afterwards — invisible until a Sandbox runs an old pi.
• The example pod has no container-level hardening (capabilities.drop, allowPrivilegeEscalation: false) while being the template LD0.3 is told to build from; readOnlyRootFilesystem needs a writable path for pi's compile cache.
• README:22/README:108 assert "default-deny sandbox" behaviour that is not true today (the agent-fleet namespace and netpols do not exist yet), so those rationales are aspirational. Worth not letting LD0.3 inherit them as facts.
• The npm layer is version-pinned but not hash-pinned (--ignore-scripts is present, integrity for the top-level package is not asserted).
• No ruling yet on which credential goes into the per-worker /home/node/.pi/agent — the path of least resistance for LD0.3 would put a live operator key inside a container with shell + curl.

## Landing report — HOMELAB-1665 (LD0.2) · NOT MERGED (escalated) Head reviewed: `fbb372f` → fixed to **`4a77d03`**. CI on `4a77d03`: **green** (all four required contexts + both `Build Pi Sandbox Image` legs). Independent review ran in two fresh contexts (`pi -p --no-session` subprocesses, separate from the implementer) with a brief built only from this diff, the ticket AC and the frozen constraints. Verdicts were `APPROVE_WITH_CONCERNS` (correctness/quality/conventions) and `REQUEST_CHANGES` (security/provenance) — 1 blocker, 10 concerns, 7 suggestions. Everything below was re-derived from primary sources; nothing is taken from the PR description. ### 1 blocker — found, fixed, and verified fixed **`docker logout` destroyed the pod-shared Harbor credential.** The `k8s` label is `:host` mode (`statefulset.yaml`: `DESIRED_LABELS="infra:host,k8s:host"`), so steps run *inside the runner container* as `runner` with `HOME=/home/runner` and `DOCKER_CONFIG` unset. `/home/runner/.docker` is a pod-shared emptyDir seeded **once per pod** from the `runner-harbor-robot` secret, and the runner daemon needs it to pre-pull private job images (that is exactly what HOMELAB-1634 added it for). `docker logout` deleted the entry pod-wide. Forensics (the mtimes match CI pushes to the second): | pod | `/home/runner/.docker/config.json` mtime | auths | |---|---|---| | `runner-4` | `2026-09-29 04:03:37` (== push run for `3d31e9b`) | `{}` (16 bytes) | | `runner-7` | `2026-09-29 04:07:57` (== push run for `fbb372f`) | `{}` (16 bytes) | The seed secret is **202 bytes** with `auths → harbor.aaron.reynoza.org`, so the empty file is not the seed — it is the logout's residue. Fixed in `4a77d03`: `export DOCKER_CONFIG="$(mktemp -d)"` before `docker login`, logout removed, digest readback unchanged. **Verified behaviourally, not just by reading the diff:** after the post-fix push run, all 8 pods' config mtimes are byte-identical to the pre-run snapshot (including `runner-4`/`runner-7`) — the workflow no longer writes the shared file at all. The same footgun pre-dates this PR in `runner-image-build.yaml:36` and `docs-build.yaml:40` and has already stripped the credential from **all 8 pods** (HOMELAB-1634's fix is currently ineffective). Filed separately as **HOMELAB-1735** — out of this PR's territory, not caused by it. ### Other changes in `4a77d03` (both reviewer concerns, cheap and in-territory) - **The documented pi bump flow was a silent no-op.** The workflow always passes `--build-arg PI_VERSION` from its own `env:`, which overrides the Dockerfile's `ARG`, so a Dockerfile-only bump (what the README told you to do) would build the old pi and stay green. Added a guard step that fails the build when the two values drift. - **README stated the wrong digest mechanism** — it quoted `docker inspect --format='{{index .RepoDigests 0}}'`, the exact command `fbb372f` removed because it returns nothing on this dind. Corrected, with the bump flow. ### Re-derived evidence for the AC - `crane digest node:24-bookworm-slim` == the digest pinned in the Dockerfile (`sha256:0e0ff40c…`) — the base pin is real and derivable. - The published artifact's config, read from Harbor: `User=node`, `Entrypoint=["pi"]`, `WorkingDir=/workspace`, `PI_CODING_AGENT_DIR=/home/node/.pi/agent`, label `version=0.87.1`. Exported the image filesystem and confirmed `etc/passwd` → `node:x:1000:1000`, `/workspace` and `/home/node/.pi/agent` owned `1000/1000`, `usr/local/bin/pi` → the pinned `@earendil-works/pi-coding-agent/dist/bundle/cli.js`, `rg`/`fdfind` present, and **no** `docker`/`kubectl` CLI (contract §12 "builds-stay-in-CI" honoured). - `pi --version` on 0.87.1 prints exactly `0.87.1`, so the workflow's `grep -qxE` line-wise assertion is real and not over-fitted. - Only `HARBOR_USERNAME`/`HARBOR_PASSWORD` are referenced **— no new secrets.** Exactly 4 files added, 0 modified, 0 deleted; nothing outside the declared territory. - **Anonymous pull (the ticket's Verify line)**: `lab-director` is `public: true`; with an empty `DOCKER_CONFIG`, `crane digest harbor.aaron.reynoza.org/lab-director/pi:4a77d03` succeeds anonymously — the path a worker node uses. The node-side pull itself lands with LD0.3's first Sandbox. - Promotion gate holds: pre-merge Harbor has only `3d31e9b`, `fbb372f`, `4a77d03`; `0.87.1`/`latest` are `NOT_FOUND`, as designed. ### Why this is not merged — operator ruling needed (secrets scope) The review surfaced a **credential-scope** question beyond the documented additive scope, which per landing rules stops the convoy rather than merging: The trigger has no `branches:` filter, so `HARBOR_USERNAME`/`HARBOR_PASSWORD` (which push as `admin` per the Harbor audit log) are now reachable from a push to *any* branch, where both peers deliberately restrict the same credential to `push: branches: [live]`. The robot's actual project scope is not documented anywhere in the repo, and the same credential publishes `platform/forgejo-runner` — the image that runs with the SOPS age key and AWS credentials mounted. "Only `live` promotes" is enforced by workflow source text, not by the credential. The README also states that guarantee more strongly than the mechanism delivers. Two options: 1. **Merge as-is**, with the trigger narrowed in a follow-up (or accept the widened exposure as a deliberate choice and document the credential's scope). 2. **Narrow the credential's reach before merge** — e.g. keep pre-merge runs credential-free (build + smoke + size gate; publish only from `live`), or mint a robot scoped to `lab-director/pi`. The cost is losing the pre-merge `<short-sha>` evidence tag. Everything else is green and the blocker is closed, so this is a one-line decision. ### Concerns/suggestions posted, not blocking (carried forward for LD0.3) - `README:34` calls `<short-sha>` "immutable evidence" — it is neither immutable (no retention/immutability control) nor reproducible: `4a77d03` rebuilt the npm layer on a different pod, so its config/layers differ from `fbb372f` for identical source. - The `0.87.1`/`latest` promotion is gated on a shell string compare and never asserted afterwards — invisible until a Sandbox runs an old pi. - The example pod has no container-level hardening (`capabilities.drop`, `allowPrivilegeEscalation: false`) while being the template LD0.3 is told to build from; `readOnlyRootFilesystem` needs a writable path for pi's compile cache. - `README:22`/`README:108` assert "default-deny sandbox" behaviour that is not true today (the `agent-fleet` namespace and netpols do not exist yet), so those rationales are aspirational. Worth not letting LD0.3 inherit them as facts. - The npm layer is version-pinned but not hash-pinned (`--ignore-scripts` is present, integrity for the top-level package is not asserted). - No ruling yet on *which* credential goes into the per-worker `/home/node/.pi/agent` — the path of least resistance for LD0.3 would put a live operator key inside a container with shell + curl.

aaron added 1 commit 2026-09-29 04:35:42 +00:00

HOMELAB-1665: ci(pi-image): restrict push trigger to live (secrets scope) ... 52bd784073

HARBOR_USERNAME/HARBOR_PASSWORD are repo-level secrets that push as a
Harbor admin shared with platform/forgejo-runner publishing. Without a
branches filter the push trigger made them reachable from a push to any
branch. Restrict to live, matching runner-image-build.yaml and
docs-build.yaml.

Pre-merge <short-sha> evidence tags are sacrificed by design; the
dedicated-scoped-robot option is deferred to operator review.

PRs were already credential-free (the push step is gated on
github.event_name != 'pull_request') and remain so. No other behaviour
changes.

aaron merged commit c7a74c7d4a into live 2026-09-29 04:39:26 +00:00

aaron referenced this pull request from a commit 2026-09-29 04:39:27 +00:00

Merge pull request 'HOMELAB-1665: feat(agent-sandbox): pi sandbox image → Harbor' (#519) from plane/HOMELAB-1665-pi-image into live

aaron commented 2026-09-29 04:43:05 +00:00

Author

Owner

Copy link

Merged — HOMELAB-1665 (LD0.2)

Merge commit: c7a74c7d4a02d49bb77087ef63399b96f0811be8
Merged head: 52bd784073ec0a03a2553bb72c59db0cae5aa7f4 → live
Method: merge · no force-push · branch left intact

Trigger narrowing (the orchestrator ruling), applied pre-merge

52bd784 adds the only change requested by the ruling — branches: [live] on the push trigger:

on:
  workflow_dispatch: {}
  push:
    branches: [live]        # was: no branches filter
    paths: [...]
  pull_request:
    branches: [live]
    paths: [...]

This matches the peer convention exactly (runner-image-build.yaml, docs-build.yaml).

Credential reach before the fix: with no branches: filter, a push to any branch reached HARBOR_USERNAME/HARBOR_PASSWORD — repo-level secrets that push as admin and that also publish platform/forgejo-runner, the image that runs with the SOPS age key and AWS credentials mounted. "Only live promotes" was enforced by workflow source text, not by the credential. Now the credential is only reachable from a live push (or a deliberate workflow_dispatch by a write-access actor).

PRs remain credential-free — unchanged and confirmed in the file: the publish step is gated if: github.event_name != 'pull_request', so the PR leg runs build → two smoke steps → size gate and never logs in.

The 3 comments above the trigger that described the old behaviour were corrected in the same commit (they would otherwise have stated a promotion rule the trigger no longer implements). That is the only text change; no logic changed.

CI

• New head 52bd784 — all 6 contexts green: CI Review / helm-validate, CI Review / receiver-test, Lint & Validate / {yaml-lint, shellcheck, terraform-validate}, and Build Pi Sandbox Image / build-and-push (pull_request) (green, credential-free). All 4 required contexts on live present and passing.
• No push leg fired on the feature branch — correct, and the point of the change.
• forgejo_ci_status/ci_failures still return 500 on actions/tasks (HOMELAB-1736); status was read via commits/{sha}/status instead.

GitOps continuation — live publish verified

The merge to live fired the image workflow: run 1481, Build Pi Sandbox Image / build-and-push (push) — success in 1m41s
https://forgejo.aaron.reynoza.org/Homelab/infra-core/actions/runs/1481

Re-derived from the registry (not from the log), anonymously via crane to mirror the worker-node path:

crane digest harbor.aaron.reynoza.org/lab-director/pi:c7a74c7 → sha256:739c7b8307cb3b7985eff771e95ad4077fca7880eb60fba4b8d89cb343e5a9f3
crane digest harbor.aaron.reynoza.org/lab-director/pi:0.87.1 → sha256:739c7b8307cb3b7985eff771e95ad4077fca7880eb60fba4b8d89cb343e5a9f3
crane digest harbor.aaron.reynoza.org/lab-director/pi:latest → sha256:739c7b8307cb3b7985eff771e95ad4077fca7880eb60fba4b8d89cb343e5a9f3

0.87.1 and latest exist, are identical to the live head's sha tag, and were resolved with an empty DOCKER_CONFIG (anonymous) — the ticket's Verify line. Config of the promoted tag:

User=node  Entrypoint=["pi"]  WorkingDir=/workspace
Env: PI_CODING_AGENT_DIR=/home/node/.pi/agent  NODE_VERSION=24.21.0
label org.opencontainers.image.version=0.87.1   linux/amd64   layers 249 MiB compressed

The promotion gate now reads correctly in Harbor. Tag list after merge: 3d31e9b, fbb372f, 4a77d03, c7a74c7, 0.87.1, latest. Note 52bd784 is absent — direct evidence the narrowing took effect: the pre-merge evidence tag is gone by design, as the ruling accepted.

Blocker fix still holding on the live leg

The 4a77d03 docker-config fix (throwaway DOCKER_CONFIG, no docker logout) was re-checked behaviourally across this run, not just read from the diff. All 8 runner pods' /home/runner/.docker/config.json mtimes:

runner-0 2026-09-25 05:27:34   runner-4 2026-09-29 04:03:37
runner-1 2026-09-28 04:04:31   runner-5 2026-09-25 05:47:18
runner-2 2026-09-23 05:23:47   runner-6 2026-09-24 06:25:42
runner-3 2026-09-22 05:23:25   runner-7 2026-09-29 04:07:57

Run 1481 executed 04:39–04:41 and no pod's config moved — the workflow does not write the pod-shared Harbor credential at all. (The 04:03:37/04:07:57 stamps are the pre-fix 3d31e9b/fbb372f runs.)

Anomaly noted, NOT addressed here (out of scope)

All 8 pods' shared config is 16 bytes with zero auths — the HOMELAB-1735 damage in runner-image-build.yaml/docs-build.yaml is still live, so the runner daemon currently cannot pre-pull private job images. Pre-existing, not caused by this PR, and explicitly out of this PR's territory. Filed as HOMELAB-1735.

Deferred to operator review

The dedicated-scoped-robot option (mint a robot scoped to lab-director/pi instead of sharing the admin-pushing repo secret) is not implemented — deferred by the ruling. Until that lands, the credential's project scope remains undocumented in-repo and the guarantee rests on the live-only trigger.

Concerns/suggestions from the review round remain open for LD0.3 and were not re-opened or re-litigated here.

## Merged — HOMELAB-1665 (LD0.2) Merge commit: `c7a74c7d4a02d49bb77087ef63399b96f0811be8` Merged head: `52bd784073ec0a03a2553bb72c59db0cae5aa7f4` → `live` Method: merge · no force-push · branch left intact ### Trigger narrowing (the orchestrator ruling), applied pre-merge `52bd784` adds the **only** change requested by the ruling — `branches: [live]` on the `push` trigger: ```yaml on: workflow_dispatch: {} push: branches: [live] # was: no branches filter paths: [...] pull_request: branches: [live] paths: [...] ``` This matches the peer convention exactly (`runner-image-build.yaml`, `docs-build.yaml`). **Credential reach before the fix:** with no `branches:` filter, a push to *any* branch reached `HARBOR_USERNAME`/`HARBOR_PASSWORD` — repo-level secrets that push as `admin` and that also publish `platform/forgejo-runner`, the image that runs with the SOPS age key and AWS credentials mounted. "Only `live` promotes" was enforced by workflow source text, not by the credential. Now the credential is only reachable from a `live` push (or a deliberate `workflow_dispatch` by a write-access actor). **PRs remain credential-free** — unchanged and confirmed in the file: the publish step is gated `if: github.event_name != 'pull_request'`, so the PR leg runs build → two smoke steps → size gate and never logs in. The 3 comments above the trigger that described the old behaviour were corrected in the same commit (they would otherwise have stated a promotion rule the trigger no longer implements). That is the only text change; no logic changed. ### CI - **New head `52bd784`** — all 6 contexts green: `CI Review / helm-validate`, `CI Review / receiver-test`, `Lint & Validate / {yaml-lint, shellcheck, terraform-validate}`, and `Build Pi Sandbox Image / build-and-push (pull_request)` (green, credential-free). All 4 required contexts on `live` present and passing. - **No `push` leg fired on the feature branch** — correct, and the point of the change. - `forgejo_ci_status`/`ci_failures` still return 500 on `actions/tasks` (HOMELAB-1736); status was read via `commits/{sha}/status` instead. ### GitOps continuation — live publish verified The merge to `live` fired the image workflow: **run 1481, `Build Pi Sandbox Image / build-and-push (push)` — success in 1m41s** <https://forgejo.aaron.reynoza.org/Homelab/infra-core/actions/runs/1481> Re-derived from the registry (not from the log), anonymously via `crane` to mirror the worker-node path: ``` crane digest harbor.aaron.reynoza.org/lab-director/pi:c7a74c7 → sha256:739c7b8307cb3b7985eff771e95ad4077fca7880eb60fba4b8d89cb343e5a9f3 crane digest harbor.aaron.reynoza.org/lab-director/pi:0.87.1 → sha256:739c7b8307cb3b7985eff771e95ad4077fca7880eb60fba4b8d89cb343e5a9f3 crane digest harbor.aaron.reynoza.org/lab-director/pi:latest → sha256:739c7b8307cb3b7985eff771e95ad4077fca7880eb60fba4b8d89cb343e5a9f3 ``` `0.87.1` and `latest` exist, are identical to the live head's sha tag, and were resolved with an empty `DOCKER_CONFIG` (anonymous) — the ticket's Verify line. Config of the promoted tag: ``` User=node Entrypoint=["pi"] WorkingDir=/workspace Env: PI_CODING_AGENT_DIR=/home/node/.pi/agent NODE_VERSION=24.21.0 label org.opencontainers.image.version=0.87.1 linux/amd64 layers 249 MiB compressed ``` **The promotion gate now reads correctly in Harbor.** Tag list after merge: `3d31e9b`, `fbb372f`, `4a77d03`, `c7a74c7`, `0.87.1`, `latest`. Note **`52bd784` is absent** — direct evidence the narrowing took effect: the pre-merge evidence tag is gone by design, as the ruling accepted. ### Blocker fix still holding on the live leg The `4a77d03` docker-config fix (throwaway `DOCKER_CONFIG`, no `docker logout`) was re-checked behaviourally across this run, not just read from the diff. All 8 runner pods' `/home/runner/.docker/config.json` mtimes: ``` runner-0 2026-09-25 05:27:34 runner-4 2026-09-29 04:03:37 runner-1 2026-09-28 04:04:31 runner-5 2026-09-25 05:47:18 runner-2 2026-09-23 05:23:47 runner-6 2026-09-24 06:25:42 runner-3 2026-09-22 05:23:25 runner-7 2026-09-29 04:07:57 ``` Run 1481 executed 04:39–04:41 and **no pod's config moved** — the workflow does not write the pod-shared Harbor credential at all. (The `04:03:37`/`04:07:57` stamps are the pre-fix `3d31e9b`/`fbb372f` runs.) ### Anomaly noted, NOT addressed here (out of scope) All 8 pods' shared config is **16 bytes with zero `auths`** — the HOMELAB-1735 damage in `runner-image-build.yaml`/`docs-build.yaml` is still live, so the runner daemon currently cannot pre-pull private job images. Pre-existing, not caused by this PR, and explicitly out of this PR's territory. Filed as HOMELAB-1735. ### Deferred to operator review The dedicated-scoped-robot option (mint a robot scoped to `lab-director/pi` instead of sharing the admin-pushing repo secret) is **not** implemented — deferred by the ruling. Until that lands, the credential's project scope remains undocumented in-repo and the guarantee rests on the `live`-only trigger. Concerns/suggestions from the review round remain open for LD0.3 and were not re-opened or re-litigated here.

aaron referenced this pull request 2026-09-29 05:27:31 +00:00

HOMELAB-1666: feat(agent-sandbox): pinned v1.0.4 install + namespace + metrics #520

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

operator-review

Requires human operator review before merge — dangerous class (RBAC, secrets, network policy, Terraform) or explicitly gated. Never auto-merge.

No labels

operator-review

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Homelab/infra-core!519

No description provided.