HOMELAB-1859: LD4 slot 6 — sandbox class egress + result-return path (the result gateway) #597
No reviewers
Labels
No labels
operator-review
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Homelab/infra-core!597
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "plane/HOMELAB-1859-gateway"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Implements the ratified design (
.rpiv/artifacts/epic-ld4/design-1859-sandbox-egress.md) and the orchestrator adjudication on HOMELAB-1859: the result gateway. No sandbox class gains Plane egress; the product returns through one authenticated hop, and the lane watcher ends a lane on the bundle instead of the 4h TTL. Prod manifests are a separate PR.What is built, against the brief's map
gateway/api.go—Submission{Verb, Comment, Drafts}, no project/state/target fieldgateway/api.goErrUnknownField(400), not ignored. Size caps. Verb allowlist + role charter (roundtable comments; architect comments+drafts; every other role writes nothing).BacklogStateis a constant, not a fieldgateway/nonce.go— HMAC over role/ticket/lane/expirygateway/nonce.gomin(claim lease deadline, now+15m). Deviation (documented in the file): the brief's shorthand isbase64url(HMAC(...)), which a verifier cannot recompute the claims from; the shipped token is<base64url(payload)>.<base64url(mac)>over the same fields. Constant-time compare,Bearer-onlygateway/server.go—POST /v1/submit+ healthgateway/server.gogateway/plane_writer.go—CreateWorkItem(state=Backlog)gateway/plane_writer.gobackloggroup). Deliberately notclaims.APIWriter: the dispatcher owns state/leases, the gateway owns products — stated in the READMEgateway/server.go(roleToken),cmd/ld-result-gateway/main.goTokenRefFor→ mounted Secret map. The token map is built FROM the registry, so adding a role's credential is a registry entry + a Secret keycmd/ld-result-gateway/main.gocmd/ld-result-gateway/+Dockerfile--check/--version, graceful shutdown,/healthz+/readyz. The binary rides in the dispatcher image (same HOMELAB-1688 precedent ascmd/routines: one module, one reviewed digest)spawn/spawn.go,spawn/evidence.go,spawn/errors.go.sandbox-init/submit-token, OUTSIDE the pinned package dir, thenchmod 600through the sandbox's own shell. First out-of-package driver file; the git token (§2.6.4's other.sandbox-init/file) is still unshipped and will use the same helperresultBundleWatcher(lane ends on the bundle, not the TTL)cmd/dispatcher/resultwatch.go, wired inapp.go.sandbox-state/agent/result.jsonwith the sandbox's owntest -f(a verified shape;download_filecannot distinguish absence from a broken transport, and the MCPfile_existsresponse shape is not verified against the pinned server — guessing it would be fabrication).teardown/*unchanged; the old TTL ending is the fallbackLaneOutcome.Contractfrom the compiled ticket section (OQ-7(ii))cmd/dispatcher/resultwatch.go(contractFromPackage)```ld-result-contractblock ofname: sha256:<hex>. Malformed → refused BEFORE the claim; absent → the honest empty contractidentity/roundtable entries +ActiveRolesInV1+ pinned testidentity/example-registry.yaml,model.go, testskv/app-credentials(OQ-2(a)); the architect's plane ref reconciled to the same object. Pinned counts moved (9 roles / 39 entries / 10 chat accounts)identity/surface_plane.goTokenRefForwas already the seam; this is its first real consumerVerify lines, mechanized
gateway/server_test.gonames every refusal and asserts zero Plane calls — no nonce, forged nonce, expired nonce, wrong-ticket assertion, unknown field (state/project/ticket/assignee), unchartered role, replay 409, unwired-ref 503.gateway/plane_writer_test.gopins the wire shapes (one-field comment body;name/description_html/stateonly, with the resolved Backlog UUID). The live probe half (sandbox → Plane000, sandbox → gateway200/409,cilium-dbg policy trace) runs at activation; the class rule is in the prod PR.cmd/dispatcher/resultwatch_test.godrives the whole loop over the fixtures through a real gateway onhttptestwith the real mint/verify path — the record lands, the bundle ends the lane,disposition=completed,evidence.verified=true, nono_result. The regression is in the same file: a lane that submits nothing still endsno_result+failedwith the nonce delivered, and a bundle declaring its own contract is refused.RoleForLanebeing closed so A–E class lanes get no nonce and keep exactly their current reach. Live landing awaits activation (step 6) — see below.go vet ./...andgo test -race ./...are green forlab-director/dispatcher.Deviations from the brief (all stated in-code)
<base64url(payload)>.<base64url(mac)>instead of the brief's MAC-only shorthand — a verifier needs the claims (gateway/nonce.goheader)./v1/submit/{ticket}) was added so a caller CAN state the ticket it believes it is writing to; the statement is checked against the nonce and never honored. The type itself still has no target field.product-receiptpresence is enforced by the lane watcher, not by the ticket contract: a runtime-produced artifact cannot be digest-pinned before the run, so teardown's declared-vs-observed math plus the watcher's ticket/role/lane/body_sha256checks carry it. (An empty contract is still honest, not a pass.)min(claim deadline, now+15m), as specified). A run that takes longer than that gets a 401 and its lane closesReasonNoResult— loud and named. Re-minting on the heartbeat is the follow-up; a longer-lived nonce would silently widen the boundary. Flagged for the reviewer.What activation still needs (steps 1 and 6, operator)
kv/app-credentials:plane_architect_api_token,plane_roundtable_api_token(step 1, the two Plane service users per README §As-built) andld_submit_hmac_key.live(the image workflow triggers onlive, not on this epic branch) and pin it in the gateway Deployment.prod/manifests/lab-director/dispatcher-config.yamland re-apply (step 6) — that is what turns the lanes on. Until then the whole path is inert: no lane resolves, no nonce is minted, and no A–E lane changes behaviour at all.intakehad never reached the mount the dispatcher reads) — the gateway mounts that same key.Related: HOMELAB-1685 (intake/architect drafting), HOMELAB-1686 (roundtable record), HOMELAB-1784/1786 (the CNP convention and the Gateway-VIP hole this design does not depend on).
The one authenticated hop by which a sandbox's product reaches Plane, plus the process that serves it. - gateway/api.go: Submission{Verb, Comment, Drafts} — deliberately NO project, state, assignee or target field (the ticket is the nonce's, a draft's state is the Backlog constant the watch never publishes). Strict JSON decoding refuses an unknown field at the boundary, so no request can smuggle a target or a state. Verb allowlist + role charter: the roundtable comments, the architect drafts, every other role writes nothing. Size caps. Receipt shape. - gateway/nonce.go: HMAC-SHA256 over role||ticket||lane||expiry, minted by the dispatcher (min(claim deadline, now+15m)), presented as "Authorization: Bearer <nonce>", verified statelessly in constant time. The token is <base64url(payload)>.<base64url(mac)> — the brief's shorthand writes the MAC alone, which a verifier cannot recompute the claims from; the file documents the deviation. Fingerprint-for-ledger, Bearer-only scheme. - gateway/plane_writer.go: the gateway's own narrow Plane client (Comment, CreateItem which resolves the project's Backlog state by name or group; no transition, no edit, no read on the hot path). Separate from claims.APIWriter on purpose: the dispatcher owns state/leases with one key, the gateway owns product writes with per-role keys. - gateway/server.go: the request path (nonce → target assertion → strict decode → shape → charter → replay ledger → per-role token → write → receipt). The replay ledger is a real single-use guarantee for ONE replica (reservation taken before the write, released on failure so an outage does not burn the nonce) and the documented TOCTOU beyond that (OQ-4). A path-form asserted ticket must equal the nonce's, so a nonce cannot be retargeted. - cmd/ld-result-gateway: env config (LD_SUBMIT_HMAC_KEY, PLANE_*, the registry-derived per-role LD_PLANE_TOKEN_<ROLE>), --check/--version, graceful shutdown, /healthz + /readyz. The token map is built FROM the registry's tokenRefs, so adding a role's credential is a registry entry plus a Secret key. - Dockerfile: the gateway binary rides in the dispatcher image (the same HOMELAB-1688 precedent as routines): one module, one reviewed digest. Tests (no sockets): every Verify-line-1 refusal is a named fixture test asserting ZERO Plane calls (no nonce, forged nonce, expired nonce, wrong-ticket assertion, unknown field, unchartered role), plus attribution, Backlog-only creation, replay 409, 503-with-the-missing-ref and nonce-release on outage.The result-return path: a lane now ends when its run's bundle lands, and Plane gets written as the ROLE through the gateway. - cmd/dispatcher/resultwatch.go: the resultBundleWatcher. It probes .sandbox-state/agent/result.json with the sandbox's own `test -f` through execute_command — a shape the contract verifies (a non-zero exit is DATA, not a tool error), unlike download_file (where absence and a broken transport look alike) and unlike the MCP file_exists tool, whose response shape is not verified against the pinned server. On presence it reads, strictly decodes into teardown.Result, and returns LaneOutcome{Result, Contract}: teardown is unchanged. On the sandbox TTL with nothing, the OLD ending stands — ReasonNoResultSource, a FAILED run. A bundle that declares its OWN contract is refused at the decoder boundary, and a gateway-wired lane whose bundle has no product-receipt (or a receipt from another ticket/role/lane) is refused. - contractFromPackage: the ResultContract comes from the compiled package's ticket-and-acceptance-criteria section (OQ-7(ii)) — a fenced ld-result-contract block of name: sha256:<hex> entries. A malformed declaration is a refusal BEFORE the claim; no declaration is the honest empty contract. - pipeline.go: the lane's dispatch nonce is minted after the claim (expiry = min(the claim's lease deadline, now+15m)) and handed to spawn; the watcher receives the lane's expectations (ticket UUID, role, whether a nonce was delivered, the ticket's contract) through the optional laneExpecter interface, so every existing watcher and fixture keeps working unchanged. An A-E class lane gets no nonce and keeps exactly the reach it had before. - gateway/lanes.go: the v1 lane→role binding (roundtable→roundtable, direction→architect). The compiled package's static role is a section-ordering input, NOT the attribution authority; an unknown lane is inert. - app.go / main.go: the default watcher becomes the bundle watcher; Options gains SubmitHMACKey (read from LD_SUBMIT_HMAC_KEY), so the whole product path is inert until the operator flips it. - spawn/fixture.go: the fixture now MODELS `test -f <path>` from its own file map (a fixture that answers "present" for every path would make every watcher test a lie); a scripted ExecResult still wins. Tests (Verify lines 2 and 3, mechanized through a REAL gateway over httptest, the real mint path and the real verifier): the roundtable record lands and the lane ends on the bundle with disposition=completed / evidence.verified=true and NO no_result; the architect drafts land in Backlog; a lane that submits nothing still ends reason=no_result + disposition=failed with the nonce delivered; a self-declared contract is refused; a bundle without product-receipt is refused; a class lane gets no nonce. Plus the extraction matrix and the malformed-contract refusal before the claim. Honest limit carried in the code: the nonce lives 15 minutes (min(claim deadline, now+15m) as designed), so a run longer than that gets a 401 and its lane closes ReasonNoResult — a loud, named failure. Re-minting on the heartbeat (or a per-lane nonce TTL) is the follow-up; minting a longer-lived nonce here would silently widen the boundary.Review verdict — APPROVE (no blocker findings), on head
7b276a7after an integration unionReviewed in a fresh context from primary sources (the diff, HOMELAB-1859's adjudication, the design brief, the ticket's Verify lines). Reviewer ≠ implementer.
Integration first (this PR's head moved)
origin/epic/LD4advanced after this branch was cut (#592, HOMELAB-1690's friction package — which also activated LD4.5b'son-call). Merging it produced conflicts in exactly five files, every one the SAME additive edit on both sides. Resolved as unions in7b276a7, no product code touched:identity/example-registry.yaml+model.go:activeRolesandActiveRolesInV1()carry bothon-callandroundtable(8 active roles), ordered on-call-then-roundtable so the file/Go order comparison still holds.identity/registry_test.go+surfaces_test.go: pinned totals to their true merged values — 10 roles described, 43 identity entries, 11 chat accounts (9 full four-surface sets +sendrast+ el-director's 3).Dockerfile: four binaries (dispatcher, routines, alertintake, ld-result-gateway); header +EXPOSEline union both sides' reasons. Note :8091 is shared withALERT_INTAKE_LISTENby number only — a mgmt-VM host unit vs a pod, no shared network path.go vet ./...,go test -race ./...(whole module) androutines -check(27 routines, 5 active / 22 paused) are green locally; CI is green on this head (dispatcher-test 3m05s).The security core — verified
.sandbox-init/submit-token, 0600 via the sandbox's ownchmod, outside the pin-verified package dir). Per-role Plane PATs live only in the gateway process (env from the registry'sTokenRefFor→ mounted Secret), andSpawnEvidencerecords the nonce's path + digest —String()deliberately omits even the digest. The dispatcher mounts onlyLD_SUBMIT_HMAC_KEY.role‖ticket‖lane‖expiry,hmac.Equalcompare,Bearer-only (every other scheme refused), key floor 16 bytes enforced at both mint and verify, expirymin(claim deadline, now+15m). The brief's MAC-only shorthand is unverifiable by construction, so the<b64(payload)>.<b64(mac)>deviation is correct and documented.reserve()takes the ledger slot inside one critical section before the write andrelease()s it only on failure, so two concurrent same-nonce submissions cannot both write;LedgerTTL(30m) >NonceTTL(15m), so there is no forget-then-replay window. A deliberate double-POST across N replicas remains the documented TOCTOU (the Deployment runs 1). Honest.Submissionhas no project/state/target/assignee field, unknown fields areErrUnknownField(400) rather than ignored,BacklogStateis a constant (resolved by name or thebackloggroup; a project without one is refused, never approximated), charter is closed (roundtablecomment;architectcomment+drafts; all else 403). The path-form asserted ticket is checked against the nonce and never honored. Every refusal is a named fixture test asserting zero Plane calls; wire shapes pinned (one-field comment body;name/description_html/stateonly).resultBundleWatcherends on the bundle;teardown/*is untouched (0 files in the diff) andReasonNoResultkeeps its meaning — the TTL fallback and the "nothing submitted →no_result+failed, nonce still delivered" regression live in the same test file. Gateway-wired lanes must declare aproduct-receiptbound to the lane's ticket/role/lane, and a self-declared contract is refused at the decoder.f6a944c) and the E-adj rule is prod #888's5761770, in an unmerged PR. Order holds.Verify lines
1 and 2 are fixture-mechanized as claimed (2 runs end-to-end through a real gateway on
httptest, real mint/verify); 3 is honestly fixture-only, with live landing deferred to activation — stated in both PR bodies, not overclaimed. Thetest -fprobe shape is the contract's (execute_commandrunscwd=/workspace), and its failure mode is fail-safe (probe error → keep waiting → TTLno_result, logged).Non-blocking concerns (mergeable; posted, not fixed here)
mountedTokenskeys the token map bytokenRefbut renders the env var from the role alone (LD_PLANE_TOKEN_<ROLE>). Two instances of one chartered role would therefore share one credential and cross-attribute. Unreachable today (both chartered roles are single-instance); worth closing before a chartered role gains an instance.draft_itemssubmission that fails after the Nth create releases the nonce, so a retry re-creates the earlier items. Impact is low (Backlog items are inert), but the duplication is silent.no_result.Config.ProjectIDis required but the write path usesHTTPPlane's own project id — redundant but harmless.Out of scope here: the one class egress rule is prod #888 (
lightwidened with exactly one rule → gateway:8091; Plane still absent). That PR is a dangerous-class, E-adj change and stays for operator review — do not auto-merge it.