LD3 EPIC: LD Cockpit — observer app (attach, telemetry panes, gates queue, image build) #589
No reviewers
Labels
No labels
operator-review
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Homelab/infra-core!589
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "epic/LD3"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What
Epic-close merge for LD3: LD Cockpit —
epic/LD3→live.Every slice on this branch was individually reviewed and landed through the pipeline. This PR is the integration-level gate: base-drift reconciliation, union-diff review, merge on green. It is a merge commit, not a squash — the slice history is preserved.
Slices contained
workflow_dispatchfor pre-live tags) — HOMELAB-1681Union diff vs
live203 files, +41151 / -0 — all additions:
lab-director/cockpit/**.forgejo/workflows/cockpit-image-build.yaml.forgejo/workflows/ci.yamlcockpit-test,cockpit-e2eappended)lab-director/.dockerignorelab-director/dispatcher/**— untouched (0 lines in the delta).No prod manifests (separate repo). No unrelated files.
Base drift
epic/LD3was 31 commits behindlive.liveadvanced.forgejo/workflows/ci.yaml(HOMELAB-1871helm-validatesteps at line ~36). Dry-run merge (git merge-tree+ temp worktree) is clean — the two ci.yaml hunks are additive in disjoint regions, so the union carries bothlive's chart-script/render steps and the epic's cockpit jobs. Base is merged into the head before merge.Integration review
Tracked in the comments below (fresh context, primary sources, integration lens).
The three seams LD3.2a is built on, all credentialless: - plane/: a three-method GET-only read seam (issues, issue, comments) with a live client that reuses the dispatcher's watch.HTTPClient for issues/states/labels/project and adds only the comment read (the gate model is a comment marker, and that client cannot read comments). State-group filtering is client-side, because the proxy ignores state__group; a partial scan is a named error, never a short queue. The write seam is a SEPARATE interface (Writer) whose production implementation in this slice is the named refusal, so a read token and a write token cannot arrive through one type. - gates/: ld.gate/v1 (options, recommendation, exposure{blast,recovery}, expires_at, gate_owner, nonce) and ld.gate-disposition/v1 as markers inside Plane comments, mirroring claims' record discipline: append-time and read-time validation, newest-wins with an ascending-nonce tiebreak, and a pure Resolve for the lifecycle (open / answered / expired). Refusals are typed and carry the policy line (P-GATE-001..006) they offend. - audit/: the ld.audit.sentinel/v1 entry shape with an interim log sink, so the entry an operator will eventually query in JetStream is the entry being emitted today. Verified separates attribution from evidence: a dev-mode header identity is named and marked unverified. Tests: the Plane wire format (httptest loopback, no Plane), the gate lifecycle (open->ratify, open->defer->re-gate, expiry, a disposition older than its gate, a foreign nonce), record validation, the store's enumeration and every decision refusal, and the ledger shape.Four routes on the existing ServeMux discipline, all operator-only: GET /api/v1/gates the queue + the write seam's posture POST /api/v1/gates/{ticket}/disposition one decision, written as a comment GET /api/v1/overview wiring census + counts census POST /api/v1/fleet/stop R8: 503 "sentinel not armed", always The queue answers 200 with an honest state in EVERY case, including unwired and a failed read: the state IS the answer, and a 500 would say "the cockpit is broken" when the truth is "the cockpit cannot see the queue yet, and here is why". A decision is different — it is refused with a status a client can branch on (503 unwired, 404 unknown ticket, 410 expired, 409 no-gate/foreign-nonce/ already-decided, 400 malformed, 502 write failed) plus the policy line that decided it, and the principal always comes from the session, never the body (a body-supplied principal is a 400, not an accepted forgery). The census keeps measured counts and leaves unmeasured ones ABSENT, never zero: a zero is a measurement, and an outage is not a measurement of zero. The read model's wiring census gains its own gates row, separate from plane, so an operator can see the queue reading while a decision still cannot be written. The STOP endpoint invokes nothing: it appends an ld.audit.sentinel/v1 entry (kind stop, outcome refused, P-SENT-001/P-SENT-002) before it answers, reports audit_recorded honestly when the sink refuses, and has no arm flag and no override. Config: PLANE_WORKSPACE (defaulted to this instance's workspace) is the one new key; a wrong workspace is a loud 404, never a silently different project.The shell's three panes become the real surfaces, and the 3.1a placeholder frame (Pane + its empty-state contract) is superseded by them — each pane now names its own missing dependencies in place, which is what the placeholder was standing in for. - Gates Queue (the centerpiece): rows with recommendation, exposure bound (blast + recovery), options, expiry countdown, owner and nonce, plus the ratify/defer/veto controls from the closed vocabulary. Class-E rows render ratify/defer; a class-E ticket with no gate record renders NO controls and the sentence explaining why. The unwired state says "gates unwired" and names LD3.1b; the write posture is stated before it is discovered via a 503. - Fleet Overview: the three card groups (homelab / fleet / updates), every count through StatCell — a hatched ? when the backend did not measure it, never a zero — the wiring census as named rows, the project switcher, the sandbox table skeleton whose only row is the honest unknown (hatched, never green), and the four-state vocabulary legend. - Director chat: text-first, with the transport seam wired to the refusing implementation, so a typed message is recorded as NOT SENT with the reason (LD4.3 / HOMELAB-1687) and no reply is invented. - Header STOP: always visible, one confirmation dialog naming fleet scope and the ledger entry, then the real fail-closed request. The 503 renders amber with the audit entry echoed back (schema, outcome, policy lines, principal, verified, request id). - Pane navigation: a tablist under the header. All three panes stay mounted (inactive ones hidden) so their state survives a glance and the shell's structure is stable across routes; the gates badge renders only a measured count. The emoji-block glyphs the design names (⏸ ✋ ⏹) have no font coverage in this stack — verified in a browser, they paint as tofu — so the derived-state marks are lucide strokes and the STOP mark is a drawn square. Intent preserved, tofu avoided.Adds the attach bridge behind LD_ATTACH_ENABLED (default false): the cockpit/** mechanism, the server wiring and the refusal matrix, with no manifests, no RBAC, no CNP, no dispatcher change and no web/**. attach/ (new): - Registry seam (read-only interface, recording fixture, and the refusing unwired registry that names the unratified RBAC) — the live CRD reads wait for the operator's ratification package §4/§5. - Executor seam + client-go impl: remotecommand.NewSPDYExecutor over pods/exec with Tty:true and a TerminalSizeQueue; construction reads a kubeconfig (LD_ATTACH_KUBECONFIG, empty = in-cluster) and contacts nothing. - Refusal matrix with a P-ATT-001..008 policy book, the class harness allowlist (spawn's own class constants; unknown class refuses), and the ld.audit.attach/v1 writer (audit.Entry gains attach kinds, outcomes and fields; Validate enforces the schema/kind/outcome pairing both ways and keeps sentinel entries free of attach fields). - Wire() is the one place a kube client can be constructed: disabled builds none (asserted by a construction counter), enabled builds one and refuses per attempt while the registry is unwired. server/: - GET /api/v1/attach/{sandbox}/info and WebSocket /ws/attach/{sandbox}, registered outside RequireOperator so that a refused handshake is still a ledger entry; identity resolution, refusals and the WS protocol all audit (open/close/refused), statusLogger gains the Hijack passthrough a websocket upgrade needs through the chain. - One attach_open is answered by exactly one attach_close (a failed dial writes the aborted close), and an attach never opens when its open entry cannot be written. config: LD_ATTACH_ENABLED (default false) + LD_ATTACH_KUBECONFIG; a kubeconfig configured while the flag is off is refused as ambiguous. main: attach declared in the read-model census, in --check's OK lines and in the boot log ("no kube client constructed"). Tests: 192 pass under -race (attach 32, audit 9, server 47, config 13 new or extended); full refusal matrix, WS protocol over a real socket (upgrade/resize/EOF both ways/abrupt disconnect/protocol error/slow consumer), ledger pairing and entry JSON shape.The deploy slice (LD3.1b) pins a DIGEST, so the digest has to exist before the manifests can name it — and the cockpit image workflow's only push trigger is `live`, which the epic has not reached. This adds the dispatch path that publishes an epic/LD3 build. ADDITIVE, by construction: - `workflow_dispatch` gains one optional `ref` input (default "" = build the ref the dispatch ran on, i.e. the branch picked in the UI). A new step in BOTH jobs checks out a non-empty `ref` with a shallow single-ref fetch and `--detach`, so preflight answers "is the cockpit on THIS ref?" for the ref that will actually be built. - `SHORT_SHA` now comes from `git rev-parse HEAD` instead of `${{ github.sha }}`. Same seven characters on the push/PR paths (checkout HEAD IS github.sha there); on the dispatch path it is the BUILT commit, so the tag, the smoke results and the reported digest all describe one commit. - The `latest` promotion decision reads the BUILT ref rather than the run's ref. With no input the two strings are identical, so a `push` to `live` still promotes and nothing else does; the case this closes is a dispatch from `live` whose `ref` input names an epic branch, which must never move `latest`. Unchanged: both existing triggers byte-for-byte (branches/paths), the `if: github.event_name != 'pull_request'` push gate, the scoped HARBOR_PUSH_USERNAME/PASSWORD robot (no admin credential, no new secret), the SHA-tag-only push, and the size gate. Verified: YAML parses (PyYAML); every `run:` block passes `bash -n`; yamllint finding count unchanged at the file's pre-existing level (67 line-length/style findings before and after, none of a new class).The frontend half of LD3.3 starts at the contract. Every shape in types/attach.ts was read off the Go it mirrors (attach.Info, attach.Server*Frame, attach.RefusalKind, server.attachRefusal) and then VERIFIED by marshalling those structs and diffing the JSON — which is how two things the Go doc comments get wrong were found: - attach.Term carries NO json tags, so the info payload's terminal object is {"Columns":120,"Rows":32} (PascalCase), NOT the cols/rows the comment above the type claims. normaliseTerm accepts both spellings so a future tag addition is a no-op rather than a silent 0x0 geometry. - class_without_tty is answered 200, not the refusal kind's own 409: the read succeeded, so a 4xx would tell the frontend its request was wrong. attachApi.info therefore does NOT throw on non-2xx. The backend answers a decidable refusal as HTTP so a browser's onerror and an operator's curl see the same JSON — but a browser cannot read a failed WebSocket handshake's status at all, which is why every 400/401/403/404/409/502/503 is a POSTURE the button renders rather than a transport failure it retries. xterm 6.0.0 and @xterm/addon-fit 0.11.0 are pinned exactly (both are lazily imported, so they stay out of the home screen's first-paint chunk). formatBytes joins the format helpers for the session status strip.The UI half: a per-row attach cell, a by-id launcher, an xterm terminal and the Sandbox Environment View as a modal (design Part 2 §4.1 screen 4 — "what is this agent doing, and let me type into it"). Three decisions worth a reviewer's attention: - The rows carry the affordance, and the LAUNCHER exists because the cockpit has no sandbox-list read at all: info is a lookup BY ID, and the read model's census names four sources, none of which enumerates sandboxes. Rows-only would have shipped an unreachable feature. The launcher is the same posture as every other seam here — declare what is missing, make what IS possible possible — and the empty row renders a disabled button WITH the reason rather than no button, because "the registry is not read" and "this sandbox cannot be attached" are different facts. - The attach column is SECOND, not last, and that was decided by measurement: at 1440px the Fleet Overview's middle column is ~520px while the table wants ~640px, so a trailing column put the operator's one action off the right edge on the screen it was built for. - The dialog draws NO terminal until a session has actually opened. A refusal renders the status alone; an empty xterm below it would read as a shell that has not printed yet, which is the dead pane R6 forbids. It is portalled to <body> (the LD3.2b containing-block lesson), and its height is the terminal's height — with no terminal it shrinks to its five lines of status instead of wasting the screen. One measurement worth recording, because nothing in review would have found it: the visually-hidden reason sentence is position:absolute with no positioned ancestor, so it escaped the table's overflow-x-auto scroll container and grew the DOCUMENT's scrollWidth to 575px at a 390px viewport — the exact sideways-scrolling page the 3.1a responsive net exists to catch, and it caught it. Anchoring the cell (position: relative) keeps the sentence inside the column the scroller clips.14 new Playwright specs (48 -> 62), against the real cockpit binary in dev mode serving the real frontend build. The split is forced by what this build can produce. LD_ATTACH_ENABLED is false by design (the RBAC/CNP/arming delta is the operator's ratification package), so the LIVE server answers 503 "attach disabled" on both transports and records every attempt — the disabled posture is therefore asserted completely end to end, including reading the response the page actually received to check `disabled` and `audit_recorded` are true. An attachable answer is impossible live, so the other two postures run with the info payload fulfilled in the browser, from the same fixtures the Go tests assert. The most valuable spec is the one that follows the real refusal through the frontend: a fulfilled "attachable" answer opens the dialog, the live server then refuses the WebSocket upgrade, and the dialog has to land on an honest end state — the disabled refusal, its policy line and "refused and recorded" — with NO terminal element in the DOM. It is the R6 rule ("never a dead black box") as an assertion rather than a promise, and it is also the path a real operator hits whenever the registry moves under them. The viewport guard from LD3.2b is parameterised rather than copied, so the attach dialog — a second portal-to-body surface, mounted from inside a pane nested under the sticky blurred header — gets the same measured containing-block assertions at 1440 and 390, plus a structural check that its parent really is <body>. Measured at the end of this branch: at 390px the open dialog sits 675..836 inside an 844px viewport with its backdrop at 0..844, and the document scrollWidth is 390 against a 390px clientWidth.The §5.2 vocabulary as a component pair, pure and unit-tested: - `StateGlyph` draws one derived state: the mark (lucide strokes — the design's ⏸ and ✋ live in the emoji/symbol blocks and neither Inter nor the mono stack carries them, so the literal glyphs painted tofu in a real browser), the word, and the tone/texture the vocabulary declares. - `DerivedStateRow` paints the ROW: §5.2's dim / amber / blue / hatch textures, the duration label ("parked 23m"), the gate a HELD row is held by (with a link when there is one), and the row's own sentence. Column-agnostic — the caller supplies its cells as `lead`/`trail` snippets — so the wave table, a future project wave list and a future firmware list share one visual language instead of three hand-rolled approximations of it. - `unknown` is the DEFAULT state for both components: a caller that forgets to derive a state renders the hatched idle one, never green. - `pending_rule` joins the vocabulary as the hardware-gated hole (§5.2's rule text, §6's ApexDirector hero card): a dashed mark, hatched, idle, never a tick. It is deliberately NOT in `DERIVED_STATE_ORDER` — §5.2's four registry-derivable states keep the legend unchanged. The never-healthy rule is enforced three ways: a vocabulary test over every state, a guard (`isHealthy`) that takes its table as an argument so the guard itself can be tested against a MUTATED green-unknown table, and the dim moved off the word onto the aria-hidden mark — an ancestor's opacity is invisible to the responsive spec's contrast read, so dimming the whole chip was a silent contrast regression hiding under a passing test. `StatCell`'s value region gains `data-value`, so a spec can assert the absence rule on the VALUE alone: an unmeasured cell's explanatory sentence may legitimately contain digits ("not 41/41"), the value may not.Integration review — LD3 epic union diff
Fresh context, primary sources only (the union diff
origin/live...origin/epic/LD3, the merged tree, the repo refs — not the implementer's summaries). Reviewed as ONE change set for integration defects; the individual slices were reviewed and landed separately.Reviewed tree:
b382e185591a45e9da7612376aaa81cc714bb907— byte-identical to the head of this PR after the base-drift merge, and to the pre-pushmerge-treedry run.VERDICT: 0 blockers, 1 concern, 1 suggestion
Scope exactness — PASS
Delta vs
live: 203 files, +41151 / −0. Exactly:lab-director/cockpit/**.forgejo/workflows/cockpit-image-build.yaml.forgejo/workflows/ci.yamllab-director/.dockerignore0 deletions,0 renames,0 mode changes. No prod manifests, no unrelated files.lab-director/dispatcher/**— untouched: tree blobbf48180b…identical tolive,go.mod9576f925…andgo.sum85113c7e…byte-identical tolive.git diff origin/live...origin/epic/LD3 -- lab-director/dispatcher/ | wc -l→0.ci.yaml — conflict-free union, both job sets present
livealso advancedci.yaml(HOMELAB-1871 extended the existinghelm-validatejob). The epic's hunk appendscockpit-test/cockpit-e2ein a disjoint region. The merged file parses and carries:helm-validate(with live'sTest chart helper scripts+Render optional chart features),receiver-test,dispatcher-test,alert-receiver-test,cockpit-test,cockpit-e2e.Additive in both directions: vs merge-base
+112/−0; vslive+112/−0; vs epic+26/−0.go.mod discipline — PASS
Cockpit direct requires all justified by imports (
go-oidc/v3→auth,gorilla/websocket→attach,oauth2,apimachinery,client-go, localdispatcherviareplace ⇒ ../dispatcher);go mod tidy -diffempty,go mod verifyok. Dispatcher module byte-identical tolive.Secrets — none
No
BEGIN *PRIVATE KEY, noscheme://user:pass@DSN, noghp_/glpat-/xox/AKIAshapes, no high-entropy literals beyond digests/URLs/test names.${{ secrets.X }}references and env-var names are not credentials; the onlysecret-named values are test fixtures.pushstays restricted tolive; the image push uses the scopedHARBOR_PUSH_*robot.Landed blocker-fixes — all present in the union
server/static.go:179(derivation),server/middleware.go:155-169(script-src 'self' …'sha256-…'), served atserver/server.go:212web/svelte.config.js:30-32(serviceWorker.register: true) + built shell registers/service-worker.js; e2e asserts SW activeaudit/audit.go:62(SchemaSentinel = sentinel.AuditSchemaVersion);Entry.Principal/Scopeare the sentinel package's own types; decode asserted ataudit/audit_test.go:210web/src/lib/honest-state.ts:204(glyphPaint— texture outranks tone), used atstate-glyph.svelte:78, asserted instate-glyph.test.ts+dashboard-pane.test.ts:119Integration coherence — PASS
Fixture↔wire parity guard resolves
../../../../plane/plane.goat the module root and asserts the Go constants (17 tests collected ≥ the guard's floor). SSE vocabularyhello|heartbeat|readmodelmatchessse/sse.go↔events.svelte.ts; API paths match the Go mux; SPA routes/,/login,/unauthorizedare served by the fallback while/api,/authare excluded. Image workflowCONTEXT=lab-director+DOCKERFILE=lab-director/cockpit/Dockerfile+lab-director/.dockerignoreagree with the Dockerfile'sCOPYpaths.Local evidence on the reviewed tree:
go vetclean,go test ./...ok, vitest 312 passed / 33 files,svelte-check0 errors, SPA build ok, Playwright e2e 76 passed against the real binary and real SPA. Live CSP probe on the served shell returned the hashed policy with no'unsafe-inline', hash independently reproduced from the served bytes.[CONCERN] cockpit-image-build.yaml paths filter omits the dispatcher module
.forgejo/workflows/cockpit-image-build.yaml:58-68— thepush/pull_requestpathsfilters list onlylab-director/cockpit/**,lab-director/.dockerignoreand the workflow file, but the cockpit image builds and links the dispatcher module (Dockerfile:82,89doCOPY dispatcher/go.mod dispatcher/go.sumandCOPY dispatcher/ /src/dispatcher/), and the cockpit importsdispatcher/config,dispatcher/events,dispatcher/sentinel,dispatcher/spawn,dispatcher/watch.Consequence: a dispatcher-only change merged to
livenever rebuilds or re-pushes the cockpit image, so the published cockpit image can carry a stale dispatcher — including thesentinelaudit schema the cockpit reconciles against. Low severity (the image stays self-consistent for its own commit; this is staleness, not a broken build), and this is a mergeable concern, not a blocker: the epic's own delta is correct and the missing trigger is a follow-up.[SUGGESTION] Go CI leg uses a floating toolchain
.forgejo/workflows/ci.yaml:106,144,204run floatinggolang:1.24while the image that ships this code pinsgolang:1.24.1-alpine@sha256:43c094ad…(lab-director/cockpit/Dockerfile:62). The Node steps in the same workflow deliberately pin the Dockerfile's exact digest, so the Go half is the inconsistent leg. Noted honestly:dispatcher-image-build.yaml's Go step has the same shape and predates this epic — a pre-existing repo convention, not a defect introduced here.Base drift reconciled: this head is
epic/LD3merged withorigin/live(5a1e50a), conflict-free per the dry run, and its tree hash is byte-identical to the tree reviewed above. Merging on green CI.Merged — LD3 epic landed on
livelive @ 5215bcb05e—Merge pull request 'LD3 EPIC: LD Cockpit — observer app (attach, telemetry panes, gates queue, image build)' (#589) from epic/LD3 into liveTrue merge commit, not a squash — spot-checked slice commits (
e9e7a68,009c76b,8d5eab2,74f6343,4348525,b0d4eea,65e357f,bb99fd9) are all ancestors oflive.git diff origin/live 5a1e50ais empty, i.e. the merged tree is exactly the tree reviewed above.Landing lock:
review verdict 2026-10-02T21:33:46Z post-dates head commit 2026-10-02T21:31:23Z.CI on the merged head — green
ci.yamlcockpit-e2esuccess 3m0s,cockpit-test2m34s,dispatcher-test2m10s,helm-validate8s,receiver-test5s,alert-receiver-test20scockpit-image-build.yamllint.yamlThe
telemetry.spec.ts:275failure seen on the pre-drift head (#1758) passed here — confirming it was flakiness (that spec has no readiness wait before itspage.evaluate), not a defect in this change set. It also passed 8/8 local runs and in the full 76-test local suite on the identical tree.GitOps continuation
Exactly one downstream workflow fired (matching the repo's path filters):
Build Lab Director Cockpit Image@5215bcb— pass (1m53s).lab-director/cockpit:5215bcb→sha256:ca76fd2e0f9c84c58f7d943530fbc5f25e3da877a751cdc3a851bd53b1e61b92latestmoved to the same digest (built fromrefs/heads/live, so the promotion rule applied). Previouslatest/65e357fwassha256:a248f426….dispatcher-image-build.yamlcorrectly did not fire —lab-director/dispatcher/**is untouched.manifests/lab-director/namespace.yaml). The LD3.1b deploy slice owns that; until it lands, pinsha256:ca76fd2e….Open follow-ups (non-blocking)
cockpit-image-build.yaml:58-68— addlab-director/dispatcher/**to thepathsfilters, or a dispatcher-only change onliveleaves the published cockpit image carrying a stale shared module (incl. thesentinelaudit schema).ci.yamlGo steps run floatinggolang:1.24while the shipped image pinsgolang:1.24.1-alpine@sha256:43c094ad…; the Node steps already pin.Full landing log:
.rpiv/artifacts/build/ld3-epic-pr-landing.md.