HOMELAB-1683: feat(cockpit): live attach registry + go-live wiring (LD3.3) #588
No reviewers
Labels
No labels
operator-review
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Homelab/infra-core!588
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "plane/HOMELAB-1683-attach-live"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What
The live attach registry + go-live wiring for LD3.3 (HOMELAB-1683), off
epic/LD3@76625d2.lab-director/cockpit/**Go only — noweb/**, no manifests, no flags in the Deployment (those are the prod PR).#579shipped the attach bridge behindLD_ATTACH_ENABLEDwith a registry seam that refused everything, because the read-only RBAC it needs was not ratified. The package §4/§5 delta is operator-ratified (2026-10-02), so this slice builds the live implementation.The live registry (
attach/kube_registry.go, new)Three hops, every one the controller's answer rather than a name this process composed:
Group/version read out of the pinned CRDs (
core/manifests/agent-sandbox/vendor/sandbox-with-extensions-v1.0.4.yaml), not docs prose. Confirmed against the live cluster (read-only):Sandbox.status.selector=agents.x-k8s.io/sandbox-name-hash=…is exactly the label the pod carries;pihastty=true stdin=true;ld-claude-code-pool-*'sclaude-codecontainer has neither.hasTTYis evidence AND policy. browser/firmware/meet-ai ship the samepicontainer with a terminal, and the package §1 refuses those classes — so the conservative answer is reported, and the info read and the refusal cannot disagree.claude-codeis not in the harness table and its shipped template declares no stdin/tty, so it refuses (unknown) rather than silently inheriting a shell.Failure shapes are preserved: nothing attachable under an id →
ErrUnknownSandboxwith the registry'sWhy(unbound claim / no pod / torn down); aForbiddenis not an unknown sandbox — it survives as an error naming the grant, which the bridge renders asregistry_unwired(503), so nobody is taught to stop retrying a sandbox that is there.Wiring
The enabled path builds the executor and the live registry, once each, from one client configuration (
NewKubeClient, shared with the executor so the two cannot read different configs). The disabled path constructs neither, asserted as call counts —TestWireDisabledConstructsNoKubeClientuntouched and green, plus a newTestWireDisabledConstructsNoRegistry.LD_ATTACH_NAMESPACE(defaultagent-sandbox-system) added; the registry refuses an empty namespace rather than defaulting, because a wrong namespace looks exactly like a fleet with no sandboxes.Verification
Tests are envtest-free: the live registry is driven by fixture CRD objects through the
ClusterReaderseam (mapping, the per-class TTY table taken from the prod templates, ticket/identity sources and their order, every unknown-vs-cannot-look shape, ambiguity determinism, offline construction, and one end-to-end pass throughWire+Info). Against the live cluster (read-only, real kubeconfig, dev mode):/api/v1/attach/<a warm-pool Sandbox name>/info→ 404unknown_sandbox, audited — proving the live registry performs real API reads rather than reading a stub.Go-live gap, reported not hidden
No object in the cluster carries the Plane ticket today: the spawn arm passes no
labelstocreate_sandbox, solab-director/ticketis absent and every attach is refused assandbox_without_ticket(409, audited). The remedy is one line at the spawn call site (cmd/dispatcher/pipeline.go), dispatcher territory, deliberately not in this PR. Same forLD_HERDR_LABEL(recorded empty, never guessed). Documented inlab-director/cockpit/README.md.Refusals remain fail-closed and audited end to end; no new credentials, no writes, no watch.
Review 1/2 — safety & inertness lens
VERDICT: APPROVE-WITH-CONCERNS — 0 blockers.
Fresh context, head
07105c058f(post-dates neither review). The safety invariant holds: inert when the flag is off, read-only cluster surface, identity resolved from controller-owned objects, the 404-vs-503 split intact, and no sentinel/dispatcher file touched.Confirmed
wire.godisabled branch returns before readingopts.NewExecutor/opts.NewRegistry; only production call sites ofNewKubeExecutor/NewKubeRegistry/NewKubeClientare the seam defaults. CLI probe: flag off +LD_ATTACH_NAMESPACEset → exit 0,NO kube client constructed. Flag off +LD_ATTACH_KUBECONFIG→ exit 2 (FaultIncompleteAttach, pre-existing).TestWireDisabledConstructsNoRegistryis load-bearing for the seam (mutation callingopts.NewRegistryon the disabled branch → FAIL). It cannot catch a disabled branch calling the concreteNewKubeRegistrydirectly — hardening suggestion only.Get(sandboxClaimGVR),Get(sandboxGVR),List(podGVR);ClusterReaderis Get/List only; no Create/Update/Patch/Delete/Watch. Allremotecommand/NewSPDYExecutorhits are inkube.go(the Executor).identityOfreadsspec.podTemplate...env[LD_HERDR_LABEL],harnessContainerreads the template containers'tty/stdin,ticketOfreads claim/Sandbox metadata +spec.additionalPodMetadata;podForreturns only pod names — nopod.GetLabels()anywhere.UnknownSandboxError→ 404; Forbidden/other → 503liveRegistryReason; no-TTY class refused; empty namespace → construction refusal.lab-director/cockpit/**; no sentinel/dispatcher path.origin/epic/LD3(web parity guard intact).Concerns (no blockers)
bridge.go:56-58,64,66— the 503 sentence says the grant is "sandboxclaims + sandboxes + pods … ratified in §4/§5", but §4 names onlysandboxclaims+pods. Adjudicated below: the merged prod manifest PR #884 explicitly carriesagents.x-k8s.io/sandboxes get,listin its "RBAC scope, as written" table, so the grant went through operator-merged review; the residual is the §4 prose, not the code.cockpit-rbac.yaml/cockpit-ciliumnetworkpolicy.yaml. Adjudicated: both files exist on prodlive(PR #884 merged,40490c8).wire_test.gohardening: assert inertness on the concrete path too (construction counter).bridge.go:425-430— the live reason is attached to any non-unknown registry error (can misdirect).main.go:330—kube_client_constructed=%tprintsEnabled()(tautological).kube.go/kube_registry.go— "one client configuration" is two independentNewKubeClientresolutions that agree by construction.Full report:
.rpiv/artifacts/build/ld3-1683-golive-review-1-out.mdReview 2/2 — correctness lens
VERDICT: APPROVE-WITH-CONCERNS — 0 blockers.
Fresh context, head
07105c058f. The three-hop CRD mapping is correct against the actual install, the class→TTY table matches the shipped templates, no test is a tautology, andgo vet/gofmt/go test -raceare clean at the head.Confirmed
sandboxclaims.extensions.agents.x-k8s.io/v1beta1(Namespaced, printer column.status.sandbox.name),sandboxes.agents.x-k8s.io/v1beta1(Namespaced);status.selectoristype: string;spec.podTemplate.spec.containers[]carriestty/stdinbooleans andenv[].{name,value};spec.podTemplate.metadata.labelsand claimspec.additionalPodMetadatapresent. Independently re-checked against the live CRDs (v1beta1 / Namespaced).Forbiddenis not an unknown sandbox:Resolvebranches only onapierrors.IsNotFound; every other error is wrapped.--checkprobe).8d5eab2had rewritten the pinned sentence and CI run #1752 failedcockpit-test;07105c0(#1756) is all-jobs green.Class-map adjudication (explicit):
light/build-test→ TTY;browser/firmware/meet-ai→ refused by class per §1 (their templates DO ship the terminalpicontainer —sandbox-*-template.yaml:242-243/283-284);claude-code→ no table entry → honest refusal (its shipped template declares no stdin/tty, andspawnbinds only five classes). The table is exhaustive over the declared class vocabulary and correct. The compositeHasTTY = template-tty AND class-tableis the right choice:/infoand the refusal cannot disagree.Concerns (no blockers)
kube_registry.go:236-238+bridge.go:424-430— a malformed Sandbox (containers: [], schema-reachable, nominItems) is diagnosed as a missing RBAC grant (503 + apply-the-RBAC chips). Fail-closed and audited, but misdirects the operator; the code comment claims this bucket is only "a Forbidden, a timeout".kube_registry.gounstructured.Nested*errors silently discarded — a wrong-typedstatus.selector/status.sandbox.namereads as a normal lifecycle state (404 "not created yet"). Latent (CRD types bothstring).attach.go:208-210,219,230-232+bridge.go:447-450— the class-refusal prose says browser/firmware/meet-ai "have no TTY" while the substrate declares one in the same PR. The reason is policy, not absence. Behaviour unaffected; the sentence is pinned in a web fixture so it cannot be reworded in this territory.UnknownSandboxError.Whynever reaches the wire: it travels inRefusal.Detail, and the/infobody is theInfostruct which has no detail field — the five 404 cases collapse to one sentence. Visible only in the ledger.attach.go:168-171—Sandbox.HasTTY's doc (evidence only) contradictskube_registry.go:243-260(evidence AND policy).Full report:
.rpiv/artifacts/build/ld3-1683-golive-review-2-out.mdLanding adjudication — deviations ruled, concerns classified
Both independent reviews returned APPROVE-WITH-CONCERNS, 0 blockers at head
07105c058f. All findings are diagnostic-prose/doc-accuracy or enabled-mode hardening; none breaks flag-off inertness, the read-only registry, identity provenance, the refusal matrix, or scope.Deviation 1 — class→TTY table vs the templates (from manifest evidence). Ruled CORRECT. The composed rule is
HasTTY = template declares stdin+tty AND the ratified class table admits it. Verified against the shipped prod templates:light/build-testdeclare the terminalpicontainer;browser/firmware/meet-aialso ship the samepicontainer withstdin: true/tty: trueyet are refused by class under §1 — so the conservative answer is reported and the/inforead and the socket refusal cannot disagree.claude-codehas no table entry and its template declares no tty (sandbox-claude-code-template.yaml:335, singleclaude-codecontainer), andspawnbinds only five classes — the honest answer is an unknown-class refusal, not a silent shell. The §1 prose namingclaude-codeas a TTY class is the stale artifact, not the code; the builder flagged it.Deviation 2 — namespaced Role vs ClusterRole. Ruled CORRECT, same scope. §4 says "ClusterRole"; every kind is
scope: Namespacedin the pinned CRDs, the fleet is one namespace, and the merged prod manifest (#884) implements a namespaced Role + RoleBinding with exactly four rules and nothing wider (nowatch, nosandboxtemplates/sandboxwarmpools, no write, nopods/attach/pods/log/pods/portforward). The in-cluster precedent (agent-sandbox-mcp/rbac.yaml) is a namespaced Role for the same blast-radius reason. A ClusterRole would be strictly wider with nothing to show for it.Deviation 3 — sandboxes read not named in §4. Resolved. Review 1 flagged that the live registry GETs
agents.x-k8s.io/sandboxeswhile §4 names onlysandboxclaims+pods. The hop is genuinely required (claim.status.sandbox.name→ Sandbox →status.selector→ pod), read-only, and fail-closed; and the merged prod manifest PR #884 listsagents.x-k8s.io/sandboxes get,listexplicitly in its "RBAC scope, as written" table, so the grant passed operator-merged review. Residual: §4's prose. Likewise the 503 sentence's pointers tocockpit-rbac.yaml/cockpit-ciliumnetworkpolicy.yamlare accurate — both files are on prodlive.Deviation 4 — sentinel NOT armed. Confirmed out of scope, correctly. This PR does not touch the sentinel or the dispatcher: 13 files, all under
lab-director/cockpit/**, nocmd/dispatcher/**, no sentinel path. The S1 ruling stays parked as HOMELAB-1874.Concerns are posted above for the follow-up slices (live-error typing,
Whyon the wire, class-refusal prose,--checkcolumn). None blocks the flag-off merge.