HOMELAB-1666: feat(agent-sandbox): pinned v1.0.4 install + namespace + metrics #520
No reviewers
Labels
No labels
operator-review
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Homelab/infra-core!520
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "plane/HOMELAB-1666-agent-sandbox"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What
LD0.3 (Lab Director rebuild): the sandbox substrate — kubernetes-sigs/agent-sandbox — as a pinned, GitOps-installable module in infra-core. The
Sandboxinstance itself is in the sibling prod PR (Homelab/prod#…, same branch name).core/manifests/agent-sandbox/vendor/sandbox-with-extensions-v1.0.4.yamlvendor/SHA256SUMS:8cbe7f4c2524…57052c0)core/manifests/agent-sandbox/kustomization.yamlcore/manifests/agent-sandbox/README.mdcore/manifests/namespaces/agent-sandbox-system.yamlcore/manifests/monitoring/servicemonitors/agent-sandbox-controller.yamlcore/manifests/monitoring/rules/agent-sandbox-alerts.yamlrelease: kubernetes-prometheus-stackrule labelWhy vendored manifest, not the Helm chart
Upstream publishes no chart repo (
kubernetes-sigs.github.io/agent-sandbox/index.yaml→ 404, only 3 release assets), and names GitOps engines as the primary consumer of the pinned manifest. Vendoring means: the deployed artifact is verifiable byte-for-byte, re-derivable offline, and needs no external source at sync time (repo-wide convention: the only remote sources are Helm repos, everything else is Forgejo).Extensions (
sandboxtemplates/sandboxwarmpools/sandboxclaims+--extensions) are installed now rather than later, because the extension path is the one that gets a controller-generated default-deny NetworkPolicy — upstream does not default bareSandboxpods. RBAC delta vs core-only is tabulated in the module README (addsnetworkpolicieswrite + the three extension kinds +leases.delete).The three deltas (all in kustomization.yaml, all commented in place)
Namespaceobject removed — the namespace is declared once incore/manifests/namespaces/, so one object is not owned by two Applications, and the namespace/PSA inventory stays in one directory (longhorn-system,headscale,cilium-systemdo the same).argocd.argoproj.io/sync-wave: "-10"+sync-options: Prune=false. Upstream's own warning: deleting a CRD cascade-deletes every resource of that kind — forsandboxes.agents.x-k8s.iothat is every Sandbox pod and its PVCs. The annotation is the "prune: false on CRDs" requirement, scoped to the CRDs; the Application keepsprune: trueso upstream upgrades still clean stale RBAC/Deployment objects. Added as two JSON6902adds on individual annotation keys, so the CRDs'controller-genannotation survives and new CRDs are covered automatically.runAsNonRoot+seccompProfile: RuntimeDefault; containerallowPrivilegeEscalation: false,drop: [ALL],runAsNonRoot,runAsUser: 65532(the image's uid, verified withcrane config),seccompProfile; requests10m/64Mi, limits500m/128Mi. With these the controller pod satisfies PSArestricted.Deliberately not changed
automountServiceAccountTokenon the controller staystrue(it authenticates to the API server). The zero-keys requirement is on the Sandbox podTemplate — see the prod PR.readOnlyRootFilesystemon the controller: not set. Unverified for this static Go binary in this cluster; flipping it blind risks a crash-loop that looks like a controller bug. Documented + tracked on the ticket.agent-sandbox/pi-image/**(LD0.2) — touching it would trigger that directory's image build and it is another ticket's territory. Note: the LD0.2 example pod saysnamespace: agent-fleet # placeholder namespace — created by LD0.3; the real namespace isagent-sandbox-system(single new namespace, per the ticket's constraint). Worth a one-line comment fix in a later PR.Verify (all agent-doable, no cluster access)
registry.k8s.iopulls are already cached through Harbor at the node level (prod/talos/mirror-patch.yaml→v2/k8s-cache), same ascore/manifests/external-snapshotter.Review notes / risk
networkpoliciesRBAC). Not to be auto-merged; this PR is expected to stop at operator review.apps/omniroute.yaml).PI_SKIP_VERSION_CHECK, the in-cluster model endpoint, and escalatingagent-sandbox-systemfrom PSAbaselinetorestricted.LD0.3 of the Lab Director rebuild: the sandbox substrate as an ArgoCD app. - core/manifests/agent-sandbox/ — kubernetes-sigs/agent-sandbox v1.0.4 vendored verbatim from the pinned release asset (sha256 in vendor/SHA256SUMS), rendered through kustomize with three explicit deltas: upstream Namespace dropped (declared once in core/manifests/namespaces/), CRDs annotated sync-wave -10 + Prune=false (deleting a CRD cascade-deletes every Sandbox and its PVCs), controller pod hardened + resourced to upstream's own documented example (satisfies PSA restricted). - core/manifests/namespaces/agent-sandbox-system.yaml — namespace with PSA enforce baseline (+ audit/warn restricted), Prune=false. - core/manifests/monitoring/{servicemonitors,rules}/agent-sandbox-* — scrape target + upstream's starter alert, in the homelab's monitoring convention. The Sandbox instance itself lives in prod (manifests/agent-sandbox).Cross-link + evidence update (the body has a placeholder where this was not known yet):
agent-sandbox-namespacewave -1,agent-sandboxwave 10 multi-source) plus the firstSandbox(manifests/agent-sandbox/pi-sandbox.yaml). Merge this one first (or together): #794's Application points atcore/manifests/agent-sandboxonlive.CI on this branch (
abe8b7e) is green, 6/6:yaml-lint(the job that actually coverscore/manifests/),whitespace/shellcheck,terraform-validate,CI Review / helm-validate,CI Review / receiver-test, Atlantis0/0 projects applied.Note for whoever reviews the CI tab:
Homelab/infra-core's Forgejoactions/tasksAPI endpoint answers 500 for this repo, so the run list is not browsable from the API — commit status had to be read from/api/v1/repos/Homelab/infra-core/commits/abe8b7e/status. The web UI shows the same six contexts.Also added, beyond the files in this diff: an Outline runbook for the operator side — "agent-sandbox — pi Sandbox operations (HOMELAB-1666 / LD0.3)" in the Runbooks collection.
Not merged by design (dangerous class): stopping here for operator review.
Correction to my previous comment: the six green contexts are
Lint & Validate / yaml-lint,Lint & Validate / shellcheck,Lint & Validate / terraform-validate,CI Review / helm-validate,CI Review / receiver-test, and the Atlantis status ("0/0 projects applied"). I mistyped one of them as "whitespace/shellcheck".Landing-prep review package — HOMELAB-1666 (LD0.3) · DANGEROUS CLASS · operator ratifies
Nothing was merged, nothing was applied to the cluster, no agent decision to merge. This is the decision input: verdict, what was independently verified, exposure, options, checklist. You ratify.
Heads reviewed — infra-core
cadb8d7(wasabe8b7e;cadb8d7is docs-only, render byte-identical) · prodca921ff(was5d968e7; comments/README only,spec:deep-equal).CI on those heads: infra-core 5/5 green (yaml-lint, shellcheck, terraform-validate, helm-validate, receiver-test), prod 4/4 green (validate-manifests, Validate Manifests/validate, plan-cluster, plan-zitadel) — read via
commits/{sha}/status; the actions API still 500s (HOMELAB-1736).1. Verdict
APPROVE FOR RATIFICATION — merge as they now stand, infra-core first (Option A). No manifest change is needed for the AC. Two reviewers ran in fresh contexts (diffs + ticket AC + frozen constraints only, implementer summary withheld) and were then adjudicated against primary sources.
2. Independently re-derived (not taken from the PR text)
v1.0.4/sandbox-with-extensions.yaml→ sha2568cbe7f4c…57052c0,diffbyte-identical,sha256sum -c OK. v1.0.4 is the latest release (2026-09-24), 3 assets, no chart repo (index.yaml→ 404)kubectl kustomize core/manifests/agent-sandbox→ 11 objects, no Namespace, 4/4 CRDssync-wave:"-10"+sync-options: Prune=false(controller-gen annotation preserved), controller patch present (podrunAsNonRoot+seccomp, containerallowPrivilegeEscalation:false,drop:[ALL],runAsUser:65532, 10m/64Mi → 500m/128Mi)pi-sandbox.yamlspec:+metadata:deep-equalcrane digest registry.k8s.io/agent-sandbox/agent-sandbox-controller:v1.0.4=sha256:c241245d…f8b2020;crane config→User 65532(matches therunAsUserand the README)crane digest harbor.aaron.reynoza.org/lab-director/pi:0.87.1=sha256:739c7b83…343e5a9f3— byte-exact the digest pinned inpi-sandbox.yamland the digest LD0.2's landing reported for the live-promoted0.87.1/latest(#519). Config:User node,Entrypoint [pi],Workdir /workspace,PI_CODING_AGENT_DIR=/home/node/.pi/agent,/etc/passwdnode:x:1000:1000→ matches runAsUser/Group/fsGroup 1000 and both mountslab-directoris public: anonymous token carriespull, anonymous manifestHEAD→ 200. NoimagePullSecretneeded (a wrong answer here = permanent ImagePullBackOff)automountServiceAccountToken: falseexplicit on the Sandbox podTemplate; noenv/envFrom/extra volumes/hostPath/host*; no webhooks or admission config; CRDs are Namespaced-scope,v1beta1only, no conversion webhook, nopreserveUnknownFieldspi-sandbox.yamlwalked against the pinned v1.0.4sandboxes.agents.x-k8s.ioopenAPIV3Schema(v1beta1served+storage,podTemplaterequired) — all resolve; negative controls (unknown spec field, unknown container field, unknown volumeClaimTemplate field) all detected, so the walk is not vacuousinfra-applyremainsworkflow_dispatch3. Findings
reconcilePVCsstamps each PVC with a controller ownerReference to the Sandbox (SetControllerReference, also on adoption) and adds no Sandbox finalizer, so deleting the Sandbox garbage-collectsworkspace-pi+agent-state-pi. Upstream issue #1001 is the open request forpersistentVolumeClaimRetentionPolicy; the field does not exist in v1.0.4.controllers/sandbox_controller.go:1739,1773; e2evolumeclaimtemplate_test.goasserts controller ownerRef; upstream #1001cadb8d7+ca921ff(docs/comments + corrected PR body). Re-verified: comment-only diffs, render byte-identical,spec:deep-equal. Data is still recoverable: thelonghornclass isreclaimPolicy: Retain, so the volumes becomeReleased(detached) PVs — but the PVCs are gone, a re-created Sandbox gets new empty volumes, and recovery is manual.Prune=falsewas credited with protecting CRDs from Application deletion. Wrong mechanism (right outcome):Prune=falseonly governs the sync-prune path. App-deletion cascade runsshouldBeDeleted()=!kube.IsCRD && !Delete=false && !helm-keep, so ArgoCD never cascade-deletes CRDs (guard present since ≤ v2.4, verified at v2.13.2) — but the Sandbox is not protected:kubectl delete application agent-sandboxdeletes the controller SA/RBAC/Service/Deployment and the Sandbox → its PVCs.controller/appcontroller.go:1102; gitops-enginepkg/utils/kube/kube.goIsCRDDelete=false, notPrune=false→ Option B.registry.k8s.io/…-controller:v1.0.4, tag only; digest only documented).crane digestmatches READMEregistry.k8s.iois mirrored via Harbork8s-cache, andexternal-snapshotteralready pulls the same way in this repo. Follow-up if you want the invariant literal → Option C.monitoring-servicemonitors/monitoring-rulesapps, and the rule has anabsent(up{…})clause →AgentSandboxControllerMetricsTargetsDown(warning) fires ~5–6 min later and keeps firing until prod #794 lands.path/targetRevision: live; rule exprabsent()clause on the target existing (follow-up).argoproj.io/Applicationsince 1.8 (no health customization in this cluster's argocd values), so wave 10 can start before wave −1's namespace exists; it converges viaretry:5+selfHeal(with a briefCreateNamespace=falseerror). Inside the app the ordering is real: CRD −10 → controller 0 → Sandbox 10, with the Deployment's built-in health and ArgoCD's ensure-CRD-ready wait.grep argoproj.io_Application→ none in argocd valueshealth.luainargocd-cm.pods,persistentvolumeclaims,servicesin every namespace; extensions role adds pods patch/update,leasesdelete, fullnetworkpoliciesCRUD. Nopods/exec, nopods/attach, nosecrets, no impersonate/escalate/bind.ConfigMap/agent-sandbox-configis referencedoptional: trueand not shipped (harmless empty volume). Controller limits (500m/128Mi) are upstream's example, unmeasured. RBAC delta table overstates thepodsrow (the core role already had those verbs).readOnlyRootFilesystemdeliberately off (pi writes/tmp).4. Exposure bound
What this touches. 4 new cluster-scoped CRDs (
agents.x-k8s.io,extensions.agents.x-k8s.io); 2 ClusterRoles + 2 ClusterRoleBindings (cluster-wide write to pods/PVCs/services/networkpolicies); 1 new namespaceagent-sandbox-system(PSAenforce: baseline,audit/warn: restricted,Prune=false); 1 controller Deployment/SA/Service; 1Sandbox/pi→ Pod + 2 Longhorn PVCs (6Gi); 1 ServiceMonitor + 1 warning rule. No webhooks, no hooks, no Jobs, no Secrets, no CI apply path.Blast radius if wrong.
kubectl delete crd …: cascades to every Sandbox of that kind and, via the PVC ownerReferences, every Sandbox PVC.kubectl delete application agent-sandbox(it carries the ArgoCD resources finalizer): controller SA/RBAC/Service/Deployment + the Sandbox + its PVCs; CRDs and the Namespace survive.What is protected, by which mechanism. sync-prune of CRDs/Namespace →
Prune=false; CRDs under Application deletion → ArgoCD's!kube.IsCRDcascade exclusion (notPrune=false); Sandbox API credentials →automountServiceAccountToken: false(no Secrets anywhere); install integrity → sha256-pinned vendored asset + digest-pinned pi image; PVC data →longhornreclaimPolicy: Retain(volume survives asReleased); merge-order safety → infra-core is inert on its own.Rollback (operator-actionable).
Merge order. infra-core #520 first, then prod #794 — or back-to-back. Nothing in infra-core is applied until the prod Application points at it, so #520 alone is inert apart from the monitoring alert (F4). Do not merge prod first alone: its Application would reference a path that does not exist on infra-core
live(SyncError noise; nothing is applied).5. Options + recommendation
cadb8d7→ prodca921ff. Manifest set unchanged since review except the docs fix; the AC only needs a cluster check, and every destructive path left is explicit and now documented truthfully. Follow-ups (recorded below) do not gate the pilot.argocd.argoproj.io/sync-options: Delete=falseto the Sandbox (and namespace) first. Makes Application deletion non-destructive. Cost: a one-line annotation, CI re-run, and it changes semantics — a prunedpi-sandbox.yamlwould then leave the Sandbox running (arguably worse than "Git is the source of truth"). My read: not needed now; take it with the VAP follow-up if you want the guard.absent()clause, install upstream'sValidatingAdmissionPolicyforautomountServiceAccountToken=false. Each is a real improvement but none affects the AC, and each costs a re-render + CI cycle. My read: separate follow-up PRs.Recommendation: Option A. Then, in rough priority: (1) VAP for the automount invariant +
Delete=falsedecision; (2) alert gating; (3) controller image digest pin or amended constraint wording; (4) Application-health customization inargocd-cm; (5)readOnlyRootFilesystem+/tmpemptyDir experiment; (6) PSAbaseline → restrictedflip and the Cilium egress allowlist; (7) the RBAC-delta doc nit and the LD0.2 example-pod namespace comment.6. Operator-verify checklist (your AC)
kubectl -n argocd get applications agent-sandbox agent-sandbox-namespace→Synced+Healthy. Thenkubectl -n agent-sandbox-system get sandbox,pods,pvc,deploy,svcandkubectl -n agent-sandbox-system describe sandbox pi→Ready=True,Pod/piRunning 1/1, PVCsworkspace-pi(5Gi) andagent-state-pi(1Gi)Bound, 4 CRDsEstablished.kubectl -n argocd get application agent-sandbox -o jsonpath='{.status.operationState.syncResult.resources[*].name}'/ the sync panel: CRDs at wave −10 before the Deployment, and the Sandbox only after the Deployment was Healthy. Expect a brief namespace-race retry on a cold install (F5) — that is expected, not a failure.kubectl -n agent-sandbox-system attach -it pi→ working pi TUI. Detach with Ctrl-P Ctrl-Q and confirm the pod survives. If rendering is off,kubectl -n agent-sandbox-system exec -it pi -- env TERM=xterm-256color pi(the image sets noTERM).settings.json. Intended steady state: pi pointed at the in-cluster OmniRoute gateway (open question on the ticket).pwd=/workspace;ls -la /home/node/.pi/agent; write a file,kubectl -n agent-sandbox-system delete pod pi, confirm the file is still there.kubectl delete crd(cascades every Sandbox and PVC), do not delete theagent-sandboxApplication (it deletes the Sandbox and its PVCs), do not delete the namespace. To clear the Sandbox while keeping data: orphan the PVCs first (§4B).longhornvolumes survive asReleasedif you do delete the PVCs.7. What I could not verify (no cluster access)
Live
Applicationstatus / cold-install convergence;Sandbox/piReady, PodRunning, PVCsBound; the attach/TUI behaviour and detach; PSA admission actually admitting both pods; the Harbork8s-cachepull of the controller image at apply time; and the exact ArgoCD build in-cluster (chartargo-cd 7.7.10→ v2.13.x; the!kube.IsCRDcascade guard was verified at v2.13.2 and is present since at least v2.4). Everything else in this package was re-derived from primary sources on this machine.LANDED — infra-core #520 merged (operator-ratified Option A)
Merge method:
merge. Merge commit:bf90e6af41893b42a9aacc571a9a66338ae89b49at2026-09-29T06:26:38Z.livewasc7a74c7d4a02d49bb77087ef63399b96f0811be8before.Pre-merge re-verification (at merge time)
cadb8d7(plane/HOMELAB-1666-agent-sandbox) — exactly the reviewed head, no new commits.commits/cadb8d7/status(actions API still 500s, HOMELAB-1736): 5/5 success —yaml-lint,shellcheck,terraform-validate,CI Review / helm-validate,CI Review / receiver-test.live; reviewed diff is purely additive (0 deletion lines, 7 files, +9489).ca921ff, 4/4 green) and merged 9 s later (2e44e41,06:26:47Z), keeping the two inside the F4 alert window.Tree integrity (post-merge)
git diff c7a74c7d..bf90e6a(livebefore → after) is byte-identical to the reviewed PR diff (diff -qclean). The merge introduced nothing beyond the reviewed contents.GitOps continuation — converged, read-only verified
rootpicked up prodlive2e44e41(~3 min ArgoCD poll latency)agent-sandbox-namespacesync finished → Synced + Healthysync-wave: -10+Prune=falseagent-sandboxsync finished → Synced + Healthy, synced revisions["bf90e6af…","2e44e413…"]= exactly the two merged SHAsSandbox/piReady=True (DependenciesReady)Pod/pi1/1 Running,workspace-pi5Gi +agent-state-pi1Gi BoundMerge → fully healthy: 5 m 14 s (well inside the 20-minute budget).
Wave ordering held inside the app —
syncResult.resourcesorder: 4 CRDs → SA → 2 ClusterRoles → 2 ClusterRoleBindings → Service → Deployment → Sandbox.Deployment/agent-sandbox-controller1/1 onprod-wk-05atregistry.k8s.io/agent-sandbox/agent-sandbox-controller:v1.0.4 --leader-elect=true --extensions.F4 did not materialise. The
AgentSandboxControllerMetricsTargetsDownrule landed (PrometheusRule/agent-sandbox-alerts,release: kube-prometheus-stack) but never fired: the ServiceMonitor target is up (up{namespace="agent-sandbox-system",service="agent-sandbox-controller"} = 1) and prod merged 9 s after infra-core, soabsent(up{…})was never armed. No alert worth explaining to the operator.Read-only access note (no new credentials created)
The
argocdCLI is configured in~/.config/argocdbut its session token is expired — per the landing instruction I did not re-login. Health was read through the path the ratified package itself documents (kubectl -n argocd get applications …), whichspec/tools.mdsanctions ("kubectl — cluster operations (read freely)"). Zero mutating calls were made: onlyget/describe/logs.Anomaly observed (pre-existing, NOT caused by this change)
agent-sandbox-controllerrestarted once (exit 1,06:33:54Z). Logs show the cause:Error retrieving lease lock … context deadline exceededagainsthttps://10.96.0.1:443→ controller-runtime exits on leader-election loss by design. It is collateral, not a defect in these manifests: 9 unrelated components lost leases in the same 2-second window (06:33:52–54) —cilium-operator,kube-controller-manager×2,kube-scheduler×3,csi-attacher,csi-provisioner,csi-resizer,csi-snapshotter,sops-secrets-operator— andKubeAPIErrorBudgetBurnis firing at critical (5m/1h). Restart counters in the thousands (kube-scheduler 1255, sops-secrets-operator 3091) and the ArgoCD repo-server'sx253 over 20dliveness failures show the condition long predates this landing. This is the tracked incident HOMELAB-1427 (control-plane instability / etcd storage-latency stalls).Impact on this change: none lasting. The controller re-acquired the lease at 06:33:56 and has been stable since;
Sandbox/pistayedReady=True,Pod/pihas 0 restarts, both PVCs stayedBound.Also confirmed live: PVCs carry a controller
ownerReferenceto theSandbox(F1's fix is materially correct), and the namespace PSA labels areenforce: baseline/audit+warn: restrictedas declared.