HOMELAB-1682: feat(cockpit): LD3.2a — Gates Queue, census, fail-closed STOP (backend) #572
No reviewers
Labels
No labels
operator-review
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Homelab/infra-core!572
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "plane/HOMELAB-1682-gates-backend"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What
The backend half of LD3.2 (ticket HOMELAB-1682, slice "three panes"): the Gates
Queue data model and read backend, the Fleet Overview counts census, and the R8
STOP endpoint. Credentialless, class B/C — no frontend, no manifests, no CI
changes, nothing in
lab-director/dispatcher/**.New packages under
lab-director/cockpit/:plane/gates/ld.gate/v1+ld.gate-disposition/v1comment records, markers, lifecycle (Resolve), the store (enumeration +Decide), policy lines P-GATE-001..006, typed refusalsaudit/ld.audit.sentinel/v1entry shape + interim log sink (JetStream when LD1.5b lands)Extended:
readmodel/(census + agatessource row, P-COCKPIT-005/006),server/(four routes),config/(PLANE_WORKSPACE, defaulted),cmd/cockpit(wiring +
--checklines).GET /api/v1/gatesstate(configured/unwired/error), the rows, and the write seam's posturePOST /api/v1/gates/{ticket}/disposition{decision, reason[, gate_nonce]}→ oneld.gate-disposition/v1comment; 503 while the write credential is unwiredGET /api/v1/overviewPOST /api/v1/fleet/stopsentinel not armed+ anld.audit.sentinel/v1entry, recorded before the answerWhy
R5: no gate data model exists anywhere today (class-E is prose in a ticket
description; the scout found no
Gate/Exposuretype in the fleet). The modelis therefore a JSON comment marker with
claims' discipline (append-time andread-time validation, newest-wins with a nonce tiebreak), enumerated from a Plane
state scan +
events.ParseClass+ marker parse. R4: every unmeasured value isabsent rather than zero — an outage is not a measurement of zero — and every
unwired seam renders a reason and the keys that would wire it. R8: the stop is
fail-closed with no arm flag and no override; the operator decides when to arm it.
R10: reads and writes are separate credentials behind separate interfaces, so the
read token cannot reach the write call site.
Test plan
140 top-level tests / 212 including subtests, 0 failures, all packages
(incl. the pre-existing 3.1a suites) green under
-race. New:plane13,gates25,audit5, plus the readmodel census and server fleet suites (33 inserver). No Plane, no network: the wire format is driven throughhttptestloopback, everything else is fixture-driven with a fake clock.
Every refusal path is asserted: unwired (503 + reason), unknown ticket (404),
expired gate (410), no-gate / foreign nonce / already decided (409), malformed or
body-supplied principal (400), failed write (502 — never a 201). The lifecycle is
asserted as a pure function: open → ratify; open → defer → re-gate; expiry; a
disposition older than the gate; a retried decision is idempotent and writes no
second comment.
Dev-mode run + curl (credentialless image,
go run ./cmd/cockpit):GET /api/v1/gates→ 200{"state":"unwired", ... "count":0, "items":[], "writes":{"state":"unwired", ...}}with the policy linesGET /api/v1/overview→ 200, four source rows,census.state:"unwired"withno
total/class_e_open/gates_needing_decisionkeysPOST …/disposition→ 503 unwired with the reasonPOST /api/v1/fleet/stop→ 503 with the audit entry (kindstop, outcomerefused,verified:falsefor the dev identity,policies:[P-SENT-001, P-SENT-002]) and the matchingmsg="ledger entry"log lineSlice boundary
In: the model, the enumeration, the census, the four routes, the audit seam, the
tests. Out (deliberately): the SvelteKit panes and the STOP button (3.2b), the
Plane write credential (3.1b ratification package — until it lands a decision
is a 503, never a silent no-op), the sentinel arm (an operator decision), the
derived-state visuals (3.4).
Open items for the reviewer
gates.Recordcarries noticketfield (R5's list is exact); foreignness isenforced by the disposition's
gate_nonceinstead. If the reviewer wantsclaims-style copy detection, that is a one-field change.offer ratify/defer/veto; a class-E ticket with no gate record renders
decisions: []+ anext_actionsentence rather than buttons that would 409.3.2b should confirm that rendering.
a 200-ticket budget (exceeding it is an error, not a short queue). Scout §8.4
suggests a cached/timed scan; deferred as a later slice's call.
PLANE_WORKSPACEis new (defaulted tohomelab); 3.1b's deploy slice shouldknow about it.
The three seams LD3.2a is built on, all credentialless: - plane/: a three-method GET-only read seam (issues, issue, comments) with a live client that reuses the dispatcher's watch.HTTPClient for issues/states/labels/project and adds only the comment read (the gate model is a comment marker, and that client cannot read comments). State-group filtering is client-side, because the proxy ignores state__group; a partial scan is a named error, never a short queue. The write seam is a SEPARATE interface (Writer) whose production implementation in this slice is the named refusal, so a read token and a write token cannot arrive through one type. - gates/: ld.gate/v1 (options, recommendation, exposure{blast,recovery}, expires_at, gate_owner, nonce) and ld.gate-disposition/v1 as markers inside Plane comments, mirroring claims' record discipline: append-time and read-time validation, newest-wins with an ascending-nonce tiebreak, and a pure Resolve for the lifecycle (open / answered / expired). Refusals are typed and carry the policy line (P-GATE-001..006) they offend. - audit/: the ld.audit.sentinel/v1 entry shape with an interim log sink, so the entry an operator will eventually query in JetStream is the entry being emitted today. Verified separates attribution from evidence: a dev-mode header identity is named and marked unverified. Tests: the Plane wire format (httptest loopback, no Plane), the gate lifecycle (open->ratify, open->defer->re-gate, expiry, a disposition older than its gate, a foreign nonce), record validation, the store's enumeration and every decision refusal, and the ledger shape.Four routes on the existing ServeMux discipline, all operator-only: GET /api/v1/gates the queue + the write seam's posture POST /api/v1/gates/{ticket}/disposition one decision, written as a comment GET /api/v1/overview wiring census + counts census POST /api/v1/fleet/stop R8: 503 "sentinel not armed", always The queue answers 200 with an honest state in EVERY case, including unwired and a failed read: the state IS the answer, and a 500 would say "the cockpit is broken" when the truth is "the cockpit cannot see the queue yet, and here is why". A decision is different — it is refused with a status a client can branch on (503 unwired, 404 unknown ticket, 410 expired, 409 no-gate/foreign-nonce/ already-decided, 400 malformed, 502 write failed) plus the policy line that decided it, and the principal always comes from the session, never the body (a body-supplied principal is a 400, not an accepted forgery). The census keeps measured counts and leaves unmeasured ones ABSENT, never zero: a zero is a measurement, and an outage is not a measurement of zero. The read model's wiring census gains its own gates row, separate from plane, so an operator can see the queue reading while a decision still cannot be written. The STOP endpoint invokes nothing: it appends an ld.audit.sentinel/v1 entry (kind stop, outcome refused, P-SENT-001/P-SENT-002) before it answers, reports audit_recorded honestly when the sink refuses, and has no arm flag and no override. Config: PLANE_WORKSPACE (defaulted to this instance's workspace) is the one new key; a wrong workspace is a loud 404, never a silently different project.Landing worker — CI check on head
0e3f479(run #1683): RED — landing haltedThe builder could not confirm the final CI state (its polls 502'd). Verified via the Forgejo API + the raw job log. Combined commit status for
0e3f47922cb0492fd780472c76bbd280ef1976b1=failure(9 statuses; the earlier poll results were stale/absent, this one is authoritative).Root cause (deterministic, introduced by this PR)
lab-director/cockpit/web/e2e/shell.spec.ts:75—expect(getByTestId('source-row')).toHaveCount(3)→ Received 4. 0 elements at first paint, then 4 stably; the assertion fails on both the initial attempt and retry #1.
This PR adds a fourth census source row (
readmodel/readmodel.gosources()now returnsplane, gates, dispatcher, bus), but the PR touches nolab-director/cockpit/web/**file — the existing e2e spec still asserts exactly 3 rows. The web UI renders whatever the census returns, so the shell now shows 4. Minimal fix (for a fresh fix worker, not done here): update the spec to expect 4 and assert thegatesrow's posture alongsideplane/dispatcher/bus.Not a blocker (classified flaky)
shell.spec.ts:95"ships the PWA shell … service worker ACTIVE" failed once withExpected "activated" / Received "activating", then passed on retry — Playwright classifies it1 flaky, not1 failed. Timing-sensitive SW activation, unrelated to this backend diff. Suite summary: 1 failed, 1 flaky, 24 passed (49.6s).Disposition
No reviews were run and no merge was attempted: the landing contract stops on a red CI job on head. The fix is a code change, which belongs to a fresh fix worker (per the landing pipeline), not to me. Re-dispatch this landing worker once a new head is green.
Log evidence: raw job log fetched from
actions/runs/1683/jobs/5/attempt/1/logs; combined status fromapi/v1/repos/Homelab/infra-core/commits/0e3f479/status.Review verdict 1/2 — correctness & security (fresh context)
Reviewer: independent fresh process (
pi -p --no-session --model deepseek/deepseek-flash), no implementer context. Head reviewed:b3c1bec. Primary sources: head tree export oflab-director/cockpit/**,pr572.diff(30 files, baseepic/LD3), design brief R4/R5/R8/R10, PR description.VERDICT: BLOCKERS (2 blockers, 8 concerns, 3 nits). Adjudicated by the landing worker against the code: all three claimed facts below (sentinel schema field types,
Queue.Countshape, thefleetscope value) are confirmed.VERDICT: BLOCKERS
BLOCKER 1 — the audit entry stamped
ld.audit.sentinel/v1is not that schema; the "a serializer away, never a rewrite" claim is falseaudit/audit.go:105declaresPrincipal string \json:"principal"`and:110, while the schema it claims —audit/audit.go:44(SchemaSentinel = "ld.audit.sentinel/v1") — is defined by the dispatcher atdispatcher/sentinel/ledger.go:90asPrincipal Principaland:95asScope StopScope, i.e. JSON **objects** (sentinel/identity.go:38;sentinel/stop.go:26), with noUnmarshalJSON` shim.So the entry
server/stop.go:58-70emits —"principal":"dev:e2e-operator","scope":"fleet"— cannot be decoded intosentinel.AuditEntryat all, and fails the sentinel's ownValidate()twice over (Principal.Valid()→ kind"";StopScope.Validate()→ onlyall|ticket|classare declared and"fleet"is not among them; the value should be{"kind":"all"}).Why it is wrong:
audit/audit.go:8-12claims "the cockpit ships the same shape … a later durable sink is a serializer away, never a rewrite", and the PR description repeats "ld.audit.sentinel/v1entry shape". That is checkably false for two of the nine fields. Concrete consequence: when LD1.5b swaps the sink for the JetStream ledger (or a sentinel reader consumes the stream),Appendis refused by validation — the stop path's evidence stops being recorded, the exact failure P-SENT-002/R8 exist to prevent. A schema string asserted without conformance is worse than a distinct one.Fix: use the dispatcher's types (
Principal: sentinel.Principal{Kind: sentinel.PrincipalOperator, ID: <subject>},Scope: sentinel.AllWorkers()— the dispatcher module is already imported fordispatcher/events), or stop claiming the sentinel schema and name the cockpit's own until it conforms. Either wayEntry.Validate(audit/audit.go:129-161) should validateScopeagainst the vocabulary — today any string passes.BLOCKER 2 — an unwired or failed queue renders
"count":0, contradicting R4/P-COCKPIT-005 added by this same diffgates/store.go:131declaresCount int \json:"count"`(no pointer, noomitempty).UnwiredQueue(store.go:137-152) andErrorQueue(store.go:156-165) leave it at the zero value, soGET /api/v1/gatesreturns{"state":"unwired"|"error","count":0,"items":[]}— andserver/fleet_test.go:103*pins* it (if body.Count != 0 || len(body.Items) != 0`).Meanwhile this same diff adds
readmodel/readmodel.go:285P-COCKPIT-005: an unmeasured count is absent from the payload, never zero, README.md:131-132 ("An unmeasured count is absent from a payload, never0"), and the PR body's "R4: every unmeasured value is absent rather than zero". The identical quantity is handled the opposite way one package over:Census.GatesNeedingDecision *int \json:"gates_needing_decision,omitempty"`(readmodel/census.go:54), left unset with a reason when unwired (census.go:141-152`), with tests asserting the key is absent.Why it is wrong: exactly the harm R4 names — a client that reads
count(the field the pane renders as "N need you") shows "0 gates need you" during an outage, and a zero is indistinguishable from a measured empty queue. The honeststateonly helps a client that already knows to branch on it, and this payload is the contract 3.2b will code against.Fix:
Count *int \json:"count,omitempty"`(absent unlessstate == QueueConfigured);itemsdeserves the same treatment. Updateserver/fleet_test.go:103` to assert absence.CONCERN 1 —
Resolvedoes not use the documented newest-wins tie-break;EffectiveDispositionis dead codegates/resolve.go:70-73picks the answer as "first in comment-slice order wins on equalCreatedAt", whiledisposition.go:152-161documents and implements the other rule (newest, ties by ascending nonce).EffectiveDisposition/sortStableByNewesthave no production call site. Plane'screated_atis second-resolution in practice, andFixtureClient.Commentsreturns newest-first whileHTTPClient.Commentsreturns Plane's page order. Bounded impact (both dispositions still answer the gate) butDecide's idempotency comparison and the renderedReasoncan differ between readers, so a legitimate retry can 409 depending on order.CONCERN 2 — the tested ticket-reference form (human identifier) cannot resolve against live Plane
FixtureClient.Issuematches ID or Identifier (plane/fixture.go:163), so every test — and the PR's "unknown ticket (404)" case — exercises identifier refs. The live path cannot:plane/http.go:175-183→watch.HTTPClient.IssuebuildsGET /projects/{projectID}/issues/{issueID}/, Plane's by-UUID route. A decision on a real ticket named the way the queue exposes it (Item.Identifier) returnsRefusalUnknownTicket→ 404 "Plane has no such ticket in this project" — a refusal that is simply untrue. Fix: resolve identifiers in the seam or make the ref a documented UUID and remove identifier matching from the fixture so the divergence cannot hide.CONCERN 3 — idempotency is not concurrency-safe (TOCTOU)
Decidere-reads comments (store.go:319-322) then writes (store.go:372) with no lock/CAS. Two in-flight identical POSTs can both observeres.Answered == falseand both post. The claim holds for a sequential retry (tested) but not a concurrent one. Duplicate comment (noisy), not an unauthorized decision.CONCERN 4 — dev mode is the only thing in front of the new write route
RequireOperatorsynthesizes an operator identity for every request whenIssuer == ""; a manifest that omits all OIDC keys serves an unprotected operator surface with a WARN. Today no write can occur (plane.UnwiredWriter), so this is not a hole in this slice — it becomes one when 3.1b wires the real writer. No server-level test of the fleet routes in non-dev mode (everynewTestServerFleetpassesdev=true).CONCERN 5 —
FixtureClientimplements both seams, so the suite cannot detect an R10 violationplane/fixture.go:210-213asserts_ Clientand_ Writer; every test passes the same object as both. Production is genuinely split, so separation holds structurally today, but the tests would keep passing ifStorewrote through its read client.CONCERN 6 —
LogSink.Appendcannot report a dropped entry, soaudit_recorded:truecan be a lieaudit/audit.go:186-214validates then callslog.LogAttrs(...)(which returns nothing) and returnsnil; onlyMemorySinkcan actually fail. Mitigated by the body echoing the entry, but the flag means more than the sink can honour.CONCERN 7 — advertised vs accepted decision vocabulary diverge
decisionsForadvertises ratify/defer for class-E rows (store.go:440-445) butDecideacceptsvetofor any gate (store.go:289-296). R5 says the class-E restriction is encoded in UI, so server-side enforcement is absent by design, yet the queue'sdecisionsfield reads as a contract.CONCERN 8 — closed/expired gates vanish with no counter; a >200-candidate project takes the whole overview down
Openexcludes closed tickets at scan time and drops closed/answered/expired rows, so a gate on a ticket later marked Done, or an expired class-E gate, is absent from the queue,CountOpenGatesandCensus.GatesNeedingDecision— with no "N expired" signal anywhere. Alsostore.go:179-182errors past 200 candidates andPlaneCensus.Censuspropagates that as a hard error, so the entire Fleet Overview loses all ticket counts too.NIT 1 —
checkMarkerPayloadis unreachable from the encoders (json.MarshalHTML-escapes</>by default); harmless defence-in-depth, overstated comment.NIT 2 —
extractMarkersilently ignores a second marker in one comment (<!--ld-gate:A--><!--ld-gate:B-->reads as A, B dropped).NIT 3 — unused exported surface:
Record.HasOption,EffectiveDisposition,Decisions(),cite().Dimensions checked
record.go:214-217,disposition.go:89-91), parse validates after unmarshal (record.go:250-252,disposition.go:131-133); blank/duplicate/missing fields, bad version/kind/decision and unterminated/empty/bad-JSON markers all refuse; the two marker prefixes are not prefixes of each other. Findings: CONCERN 1, NIT 1, NIT 2.auth.IdentityFrom(ctx)(server/gates.go:107-108); the body struct has no principal field andDisallowUnknownFieldsturns a supplied one into 400; the route lives on the mux wrapped inauth.RequireOperator(server/server.go:156,:162). Findings: CONCERN 4, 5, 7.Clienthas three GET methods and no write; the only request helper setsX-API-Key; neitherAPIErrornorUnwiredErrornor thewatchconstructor errors contain the token;ScanIssues/Commentsfail loud on the page budget and on a cursor-less "more" page.events.ParseClass+ marker, budget is a hard error before any comment read, cross-page scan, corrupted marker and read failures fail the whole read,Decideorders refusals before network and re-reads current comments,CountOpenGatescallsOpen. Findings: CONCERN 2, 3, 8.audit_recorded:false,Verifiedfalse for a dev identity. Audit shape is BLOCKER 1; sink durability is CONCERN 6.*int+omitempty, absent keys asserted). BLOCKER 2 is the adjacentgates.Queuepayload contradicting the same policy line.Claims vs code
record.go:214-217,:250-252;disposition.go:89-91,:131-133server/gates.go:88-95,:107-108;server/server.go:156,:162store.go:333-352,:319-372resolve.go:80-87;store.go:355-357;server/gates.go:167-170store.go:329-332,:365-368server/stop.go(no config/flag read anywhere)readmodel/census.go— REFUTED forgates.Queue.count(BLOCKER 2)census.go:54,:141-152vsstore.go:131,:137-165;fleet_test.go:103;README.md:131-132ld.audit.sentinel/v1shapeaudit/audit.go:105,:110vsdispatcher/sentinel/ledger.go:90,:95;sentinel/stop.go:26plane/fixture.go:163vsplane/http.go:175-183Review verdict 2/2 — model adequacy vs R5, test adequacy, fix commit
b3c1bec(fresh context)Reviewer: independent fresh process (
pi -p --no-session --model deepseek/deepseek-flash), no implementer context, no handover of the other reviewer's findings. Head reviewed:b3c1bec. Primary sources: head tree export,pr572.diff, design brief R5 (plus R4/R8/R2/R11), PR description.VERDICT: BLOCKERS (1 blocker, 5 concerns, 5 nits). Adjudicated by the landing worker: the missing cross-principal idempotency case is confirmed absent — every
Decidecall in the suite usesoperator@example/operator/dev:e2e-operator; no test submits the same decision under a different principal.VERDICT: BLOCKERS
BLOCKER 1 — the principal leg of the idempotency key has no test at all
gates/store.go:333-352— the idempotent-repeat branch isif existing.Decision == req.Decision && strings.EqualFold(existing.Principal, req.Principal). The claim it implements is written down atREADME.md:161-163: "a retried decision is idempotent (same decision + principal + nonce = success, no second comment)" — the principal is load-bearing in the key, and P-GATE-003 makes attribution the point of the surface.The only idempotency tests are
TestDecideIsIdempotentForTheSamePrincipal(store_test.go:239-269) and the HTTP retry (server/fleet_test.go:228-243). Both re-submit as the same principal. The complement — same nonce, same decision, different principal — appears in no test (everyDecidecall usesoperator@example/operator/dev:e2e-operator; the only "other" identity isprincipal:"someone-else"inside a JSON body, refused as an unknown field, not as a second subject).Why it is a blocker: the
EqualFold(existing.Principal, …)comparison can be deleted and every test still passes — the same-decision different-principal case would then silently returnalready_appliedpointing at the first operator's comment, i.e. operator B's action attributed to operator A, with no failure anywhere. The code is, as far as it reads, correct; the coverage is not.Fix: one subtest that has operator A ratify
gate-1, then operator B submitratifyfor the same nonce, assertingIsRefusalKind(err, RefusalAlreadyDecided)andlen(fixture.Writes()) == 1.CONCERN 1 — an expired gate is closed but never reported as expired on any read surface
Openkeeps a row only forHasGate && Openor!HasGate && class E(store.go:199-208); an expired gate falls todefault: continue.Resolvecomputesres.Expired/res.ReasonbutitemFromGateis only called for open gates, so that reason never reaches the queue payload or the census. P-GATE-004 andREADME.md:167-168promise "an expired gate is reported as expired, never silently still open." The "never silently still open" half holds (410 + absent from queue); the "reported as expired" half has no observable surface. Not a blocker because R5 does not require expired gates to occupy the queue, but the policy line and README over-promise.CONCERN 2 — class-E "ratify/defer" is render-only; the API accepts
veto, and the README claims it is enforceddecisionsForreturns[ratify, defer]for class E (store.go:440-447) — correct per R5 ("class-E items render ratify/defer"). ButDecision.Valid()acceptsvetofor every gate, soPOST …/disposition {"decision":"veto"}on a class-E ticket is accepted and written. R5's parenthetical ("encoded in UI") means the model is not violated, butREADME.md:171-173says the class-E rule is "enforced where it cannot be bypassed" — false for the render rule, and nothing pins the vocabulary the API accepts for a class-E gate.CONCERN 3 — a stop attempt refused by authentication leaves no ledger entry
handleFleetStopaudits every request that reaches it, andfleet_test.go:562-604proves that for two dev-mode attempts. But the route is behindRequireOperator; in prod an unauthenticatedPOST /api/v1/fleet/stopis answered 403 by the middleware and never reaches the handler, so nold.audit.sentinel/v1entry is written. The audit package even definesKindStopIgnored/OutcomeIgnoredfor exactly this case and never emits them. Given P-SENT-002's own framing ("a refused stop that leaves no record is indistinguishable from an attack that never happened"), worth recording — though R8 only mandates the handler fail closed.CONCERN 4 — the queue row's
options[]is never assertedstore.go:401copiesrec.OptionsintoItem.Options, but no test asserts it (TestOpenKeepsClassEAndMarkedGatesasserts Exposure, GateNonce, Author, Class, RaisedAt, ExpiresAt, Decisions — not Options; the HTTP test asserts only GateNonce/Recommendation).Optionsis the R5 field the operator chooses between; a regression dropping it fromitemFromGatewould pass the whole suite.CONCERN 5 — P-GATE-005 ("one decision, one comment") is not guaranteed under concurrent replicas
store.go:333-365is a read-then-write with no conditional write; two replicas or a double-submit can both observe an unanswered gate and both post. The class doc (store.go:49-54) says re-reading at write time "makes two cockpit replicas safe" — true for a sequential retry (proven), not a concurrent pair. Plane comments offer no CAS; needs a documented single-writer assumption or a de-duplicating read-back.NIT 1 —
decisionsFor's comment describes a different rule than the code (it is only ever called for marker rows, and stripsvetofrom a marker gate on a class-E ticket); behaviour matches R5, the prose does not.NIT 2 —
README.md:146-148shows"gate_owner":""but the field isomitempty, so an unrouted gate omits it.NIT 3 — one malformed marker anywhere makes the whole queue unreadable (intended trade-off, tested) — a human comment quoting the marker grammar turns
GET /api/v1/gatesintostate:"error"for every operator. Worth a README blast-radius line.NIT 4 —
201 Createdfor an already-applied repeat; 200 would read more honestly for a no-op.NIT 5 — this brief's own pointer was wrong: the defer→re-gate rule is proven by
TestResolveLifecycle/"open then defer then a new gate reopens"(resolve_test.go:61-89), not byTestOpenKeepsClassEAndMarkedGates(which proves queue membership only).R5 field-by-field
options[]record.go:113; validation:154-170(required, ≥1, no blank, no case-insensitive dupe); rowstore.go:401; human line:226recommendationrecord.go:118, required:171; rowstore.go:402options, which the ruling permitsexposure.blastrecord.go:90, required:175-177; rowstore.go:403exposure.recoveryrecord.go:91, required:178-180expires_atrecord.go:124(omitzero),Expired():182; consumedresolve.go:82gate_ownerrecord.go:126-128(omitempty), optional; rowstore.go:404; human line rendersunroutedstore.go:329-332(a nonce the ticket does not carry is refused). Real code, but it only detects a copy when nonces are globally unique — nothing enforces or generates uniqueness, and fixtures reusegate-1/n-1across tickets. Load-bearing within one ticket (re-gate/idempotency); as cross-ticket copy detection it is an assumption, as the PR's open item #1 concedesv,kind,author,at,nonce(not in R5)record.go:101-131, required:132-181claims.ClaimRecord";nonceis load-bearing forgate_nonce"Never auto" is structurally true. The only caller of
DecideishandleDisposition(an authenticated POST); no ticker, hook or startup path decides anything.decisionsFormatches the ruling exactly (class-E → ratify/defer; marked non-class-E → ratify/defer/veto), with the enforcement caveat in CONCERN 2.Lifecycle coverage (excerpt)
Adequate: open→ratify; defer→re-gate (
TestResolveLifecycle/"open then defer then a new gate reopens"); expiry closes;Deciderefuses expired + HTTP 410; foreign nonce refused and not applied; a foreign disposition does not close a gate; no-gate refusal; malformed decision; missing reason; missing/forged principal; cross-page enumeration; partial-queue refusal; corrupted marker fails loud; read failure propagates; closed tickets excluded; unwired write seam (503 + reason); STOP 503-always for every handler attempt; census absence-not-zero.Not adequate: same decision, different principal →
already_decided(BLOCKER 1); expired gate reported as expired (CONCERN 1); audit entry for auth-refused stop attempts (CONCERN 3); queue row carriesoptions[](CONCERN 4). "No config/env path arms the STOP" is verified by inspection (config/load.gohas no arm key;server/stop.go:44-52is unconditional).Refusal plumbing is otherwise clean: eight distinct
RefusalKinds, everyrefuse(...)call site cites one P-GATE-00N line with an operator-legible sentence, andwriteDecisionErrormaps 503/404/410/409/400/502 exactly as documented with a behavior test per class. A failed write is 502, never 201.Fix commit
b3c1becgit show --stat→ exactly one file,lab-director/cockpit/web/e2e/shell.spec.ts(+20/−2); nothing else in the commit. The PWA service-worker spec is byte-identical to base — nothing was softened to paper over the earlier flaky activation. The contract it asserts holds against head:[data-source]for plane/gates/dispatcher/bus (matchessource-row.svelte:35andsources()returning exactly those four), so a renamed/dropped source fails naming the row;toHaveCount(4)catches an unclaimed fifth row;gatesrow'sdata-state="unwired"+ "enumerated from Plane" text matchesgatesSource'sStateUnwireddetail (readmodel.go:230-238) under the credentialless e2e harness.This commit is clean — it fixes a real hole (the old bare
toHaveCount(3)asserted a number, not identities) and is confined to the one spec file.Landing worker (re-dispatch) — CI verified green, reviews returned BLOCKERS, landing halted
Check-first (verified myself, not from a report):
b3c1beca3c35066969faed2d1732677862ae6152(the named-row e2e fix on top of0e3f479).b3c1bec= success, 8/8 contexts green, includingCI Review / cockpit-e2e(1m36s),cockpit-test(1m41s),dispatcher-test(2m6s), the cockpit imagepreflight/build-and-push,helm-validate,receiver-test,alert-receiver-test. The earlier redcockpit-e2eon0e3f479is resolved.lab-director/cockpit/web/e2e/shell.spec.ts, +20/−2) — reviewed and clean.Reviews: two fresh-context reviewers (separate processes, primary sources only, no implementer context) both returned BLOCKERS. Verdicts posted above as comments. Three blockers, adjudicated by me against the code:
audit.Entryclaims schemald.audit.sentinel/v1but emits"principal":"<string>"/"scope":"fleet"where the schema owner (dispatcher/sentinel) requiresPrincipal{kind,id,…}andStopScope{kind}with vocabularyall|ticket|class. The emitted entry cannot be decoded by that schema — the "a serializer away, never a rewrite" claim is false, and"fleet"is not a legal scope. AlsoEntry.Validatechecks no scope vocabulary.gates.Queue.Countis a non-pointerint, so an unwired/error queue renders"count":0— contradicting R4 and theP-COCKPIT-005line this same diff adds ("an unmeasured count is absent from the payload, never zero"), which the census half implements correctly with*int+omitempty.EqualFold(existing.Principal, req.Principal)) has no test: deleting it keeps the whole suite green while attributing operator B's decision to operator A.Not merged. The landing contract stops on open blocker findings. Next: a fresh fix worker addresses the three blockers (plus any concerns it chooses to fold in) on a new head; then re-dispatch a landing worker. No image push expected either way (
lab-director/cockpitis live-gated).Fix worker — all three blockers addressed on head
c11c725Fresh fix worker, no implementer context. Branch
plane/HOMELAB-1682-gates-backend, commitc11c7254fcff1a61c89b7fdcaf4d260a381ce431(8 files, all underlab-director/cockpit/; noweb/**and nodispatcher/**change). Working tree:go vet ./...clean,go build ./...clean,go test -race -count=1 ./...→ 8/8 packages ok, 141 PASS, 0 FAIL.B1 — the audit entry is now the schema it claims
audit.Entry.Principal/Scopewere flat strings; they are now the schema owner's own types —sentinel.Principalandsentinel.StopScope— andSchemaSentinelissentinel.AuditSchemaVersion(referenced, not re-typed).Entry.Validatevalidates both against the schema's vocabulary, so"scope":"fleet"is refused at append time instead of shipped (it was never a declared scope).The STOP endpoint builds the principal from the session: a verified session →
operator:<subject>; a dev-mode header identity →kind:"unknown"withverified:false(named, not granted authority it did not prove); an identity the cockpit cannot name →unknown/unattributed(still an entry, P-SENT-002). No cockpit-specific extension was needed, so nold.audit.cockpit/v1.New test
TestEntryDecodesAsTheSentinelSchemaItClaimsdecodes an emitted entry intosentinel.AuditEntry, runs the schema's ownValidate, round-trips a sentinel-authored entry back intoaudit.Entry, and asserts the wire shape is objects. Emitted entry (the exact response body of a dev-modePOST /api/v1/fleet/stop):B2 — an unmeasured count is absent, not zero
gates.Queue.Countis now*intwithomitempty.UnwiredQueue/ErrorQueueleave it nil, so the wire carries nocountkey; onlyOpensets it.CountOpenGatesreturns an error rather than a fabricated0if a configured queue ever lacks a count, and the doctor command printscount=unmeasured. Test pins it three ways: the pointer, the marshalled key set, and the HTTP body.Unwired response (exact,
GET /api/v1/gates,writes/policy_lineselided):A configured queue still carries
"count":2(asserted over the HTTP surface).B3 — the principal leg of the idempotency key is load-bearing in a test
New
TestDecideIsNotIdempotentAcrossPrincipals: operator A ratifiesgate-1; operator B replays the byte-identical request (same decision, same nonce) and is refusedalready_decidednaming A;fixture.Writes()stays at 1 and the written disposition is attributed to A; A's own retry is stillalready_appliedpointing at the same comment.Mutation proof: with
&& strings.EqualFold(strings.TrimSpace(existing.Principal), strings.TrimSpace(req.Principal))removed from the branch ingates/store.go:364,go test ./gates/fails —TestDecideIsNotIdempotentAcrossPrincipals(only it) atstore_test.go:320: "a different principal deciding an already-decided gate must be refused, not answered with A's success". Comparison restored; suite green again.Also updated:
README.md(route table + STOP section now state the count-absence and the sentinel entry shape), andcmd/cockpit/main.go(%d queue.Count→%s queue.CountText()— the one-line change the pointer type forces; it is the cockpit's own doctor output and now printsunmeasuredrather than a fabricated0).Not fixed here (recorded concerns, not blockers):
Resolvenewest-wins tie-break,EffectiveDispositiondead code, class-Evetoadvertised-then-accepted, expired gates with no counter, log sink cannot report a dropped entry. Next step: re-dispatch a landing worker for review on this head.Landing verdict (final) — delta review of
c11c725(fresh context, post-head)Landing worker, closing gate. Verdict posted after head
c11c7254fcff1a61c89b7fdcaf4d260a381ce431.Reviewer: one independent fresh process (
pi -p --no-session --model deepseek/deepseek-flash), no implementer context, not handed the prior verdicts. Primary sources:git show c11c725(8 files, alllab-director/cockpit/**),diff b3c1bec..c11c725, the schema ownerdispatcher/sentinel/ledger.go, design brief R4/R8. Read-only; the reviewer rango vet/go build/go test -raceand its mutation probes in a/tmpcopy — the worktree was never modified.VERDICT: CLEAN — 0 blockers, 3/3 prior blockers closed
B1 — CLOSED.
audit.Entry.Principal/Scopeare nowsentinel.Principal/sentinel.StopScope(audit/audit.go:123,131, omitempty dropped on scope),SchemaSentinel = sentinel.AuditSchemaVersion(:52), andValidatecalls the owner'sPrincipal.Valid()/Scope.Validate()(:171,174) against theoperator|role|system|unknown+all|ticket|classvocabularies. Independently re-probed the live STOP endpoint: the emittedauditobject decodes intosentinel.AuditEntryand passes the sentinel's ownValidate; the wire now carries"principal":{"kind":…}and no bare"scope":"string. No"scope":"fleet"literal remains; one emitter site only (server/stop.go:85). The README/PR claim is now literally true. (sentinel.Validatedoes not check theSchemastamp, so the test assertingdecoded.Schemaseparately is what makes the claim honest — it does.)B2 — CLOSED.
Count *int \json:"count,omitempty"`(gates/store.go:138);UnwiredQueue/ErrorQueuenever assign it, onlyOpendoes (:234); handlers return the queue unmodified (server/gates.go:70,75,78);CountOpenGateserrors rather than fabricating zero (:246). Wire-probed all three cases: configured-1 →"count":1, configured-0 →"count":0(a measured zero is still a statement — no over-correction), error read → **key absent**. Tests assert the marshalled key set, notCount != 0(server/fleet_test.go:107,116;gates/store_test.go:582,593`).B3 — CLOSED and load-bearing.
TestDecideIsNotIdempotentAcrossPrincipals(gates/store_test.go:294-360) drives the real store; B's exact replay is refusedalready_decidednaming A,len(Writes())==1, the stored disposition is A's, A's retry staysalready_applied. Mutation proofs run by the reviewer in a copy: deleting the principal leg fails only this test (at:320); inverting theEqualFoldfails this test and the same-principal one — so it is not a duplicate of the old coverage.Regression scan: every hunk accounted for; no test deleted, no
t.Skip/TODO added, test counts rose, two assertions replaced by equal-or-stronger ones. The rewrittenstopPrincipalalso fixes a latentverified:true-with-no-subject bug.readmodel/census.gounaffected; no integer consumer ofQueue.Countremains.Concerns (non-blocking, recorded not blocking): the doctor's "unmeasured" string is reachable only in tests (the error branch prints no count at all — no fabricated 0 either);
audit.Kind/Outcomeremain cockpit-local lookalike string types pinned by value, andaudit.Outcome.Valid()cannot expressbudget_exceeded/resume_failed, so a sentinel-authored entry with those outcomes would fail the cockpit's Validate (round-trip test only coversrefused);Entry.ValidateacceptsSchemaAttach(no attach emitter exists yet); README's "decodes every emitted entry" is broader than the test (the emission path was verified independently here); one cosmetic assertion shift inserver/fleet_test.godrops the second entry's principal kind.CI: verified green on
c11c725myself, not from the fix worker's report — 8/8 contexts: run #1698 (preflight,build-and-push) and run #1697 (cockpit-test,cockpit-e2e,dispatcher-test,helm-validate,receiver-test,alert-receiver-test), allsuccess.Disposition: MERGE (no blocker findings, CI green).