LD4 (HOMELAB-1713): The Loops — intake, El Director, routines, readiness, plan-freshness, meetings, alert audit → live #591
No reviewers
Labels
No labels
operator-review
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Homelab/infra-core!591
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "epic/LD4"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Epic integration PR for LD4 · The Loops (HOMELAB-1713). Operator-gated: merging rolls a new dispatcher image into the fleet backbone via ArgoCD — do not merge without the operator's review session.
What lands (all PRs reviewed by fresh-context landing workers, CI green)
What this merge triggers
Deliberately OFF / deferred (each named on its ticket)
Operator review queue attached to this merge
① §11 spool ruling (HOMELAB-1685) · ② Signal demotion gate (HOMELAB-1687) · ③ OpenBao policies create-only + token seeding (HOMELAB-1688) · ④ prod queue #868/#870/#871/#875/#876 in that order after the image · ⑤ meetings: create the Outline
Meetingscollection + RC ld-el-director account before arming.Known non-blockers
Atlantis orphaned statuses (HOMELAB-1861) may leave phantom pending contexts — CI is the required leg; the phantom is tracked infra-side. The alert-audit base lag (148 lines of monitoring rules behind live) means one page-eligible rule could not be annotated from this branch — a live merge-forward before the next census window closes it.
New package persona/: el-director.yaml (the canonical persona, CONTENT not config) plus a strict loader that refuses what the contract forbids: * an unknown key (KnownFields) or an out-of-vocabulary tone/behavior/tier value — a silently dropped setting behaves like one that never existed; * identity.account that is not the name identity.AccountNameFor derives, or that the committed registry does not bind on Plane and RocketChat (§8); * boundaries.never that is not EXACTLY the contract §2 set for the role, in that order — adding a prohibition is refused (a standing rule is a mechanism, a measurement or a deletion, §5) and removing one is refused. Persona.Render() emits the fixed block and PersonaSource wraps the role-lore corpus source so ONE sha256 pin covers persona + lore: a changed tone re-derives a different digest, which is the point of pinning it. The persona carries tone, never authority — enforcement lives where the Director is not (design §2.4), which is the structural answer to risk R-1.New package comms/: MessageClass (attention.push, nudge.r04, escalation.p1, fallback.notice, director.reply, direction-ack, refusal), Message, Health, the Ladder, the SignalClient seam and FixtureSignalClient, and the probe loop the arm runs. The ladder is where contract §7 ('Signal = outage-fallback + attention-push only') becomes code, with two anti-storm properties that are the reason it is a package and not three if-statements in an arm: * ONE fallback.notice per degradation episode (edge-latched, emitted by the first message that actually needs the fallback — a probe failure alone pages nobody); * a degraded ladder never touches RocketChat, so an outage costs one failed probe per interval rather than one timeout per message. escalation.p1 dual-posts unconditionally; the 24h soft nudge never does; the 72h ACT nudge does because the caller marks it, not because of its class. Deliver returns what actually happened (route, room, signal calls, notice, degraded) and reports a failed attention leg rather than claiming delivery. rccbridge stays the transport and holds the credential: comms is handed a DMClient (EnsureDM+Post) and a SignalClient, so 'the ladder can only post' is structural. Tests use the committed chat fixture: no socket is opened.R6 (the cheap-to-fix, easy-to-forget risk): ActiveRolesInV1 omits intake and the registry loader refuses an active role that is missing a surface. The role is activated as DATA — four bindings (plane, forgejo, rocketchat, herdr) with tokenRefs into OpenBao — exactly the way model.go documents ('adding a role is a registry edit, never a code change here'). Empty role lore is NOT part of LD4.1 (O6). The three count assertions that pin the artifact (described roles, entries, chat invite list) move with it.New package director/: the El Director's warranted machinery, fixture-first. * warrants: a closed source vocabulary, a per-class ref grammar, and a Resolver seam. Resolvability is CHECKED, not assumed — a record whose citation does not resolve is excluded from the active set with a fault naming file + line (the anti-hallucination boundary, AC-5). * memory: the git-backed record store (preference / lore / conversation-state), the maturity ladder (established needs two INDEPENDENT warrants — provenance, not count; canonical needs an operator-ruling on top), the 90-day TTL, contradiction = ESCALATE not coin-flip, and Compose() which renders only the active set. A quarantined or contradicted record is composed into no package. * veto: named gap + resolvable warrant, or refused with a teaching message and NOTHING written to Plane (AC-2). A stop veto releases the claim and leaves the rest to the TTL reaper (R-6: no sentinel scope). Recording one veto is exactly one ld.veto/v1 comment. * hand-off: exactly ONE Plane item, zero spawn/claim calls (AC-1), asserted from the fixture's whole call log; plus the OQ-6 direction intake that mints an inbox item quoting the operator's own words. * meeting: a complete meeting.completed convenes the roundtable once; a gap opens exactly ONE decision-queue item naming every missing artifact and convenes nothing (AC-6). * counters: veto_rate = N/(N+M) with refusals counted SEPARATELY, the overturn rate as the calibration input, and the OQ-5 band (0.05-0.35/quarter, with a proposal floor so a zero rate reads 'unmeasured' rather than 'in band') (AC-7). The package has no spawn, claim, merge, delete or transition surface at all — PlaneSink is exactly CreateItem + Comment — and boundary_test.go asserts that over the package's own syntax tree. The live Plane writer is HTTPPlaneSink (item create + comment, redacting String, stub-transport tests: no socket is opened anywhere in this package).Wires El Director v1 into the process as two NON-FATAL arms (the rcTopology posture: a chat outage is an observability outage, not a dispatch outage): * director the DM listener. Four intents and no fifth: a !decision marker in a DM maps to no ticket, so the R-7 rail replies with ONE escalation question (and names the item when the operator scoped it with ticket=<KEY>); a meeting-done DM becomes the manual meeting.completed trigger; an utterance taken as direction mints an inbox item and is acknowledged with its identifier (OQ-6); conversation is recorded, not answered (the model surface is not wired in v1, and a template reply would be worse than silence). * comms the ladder's health probe, one observation per interval, so 'is RocketChat up?' is answered once per probe instead of once per message. Config: a director block (personaPath, memoryRoot, userEnv/tokenEnv for the SECOND chat account, operator, signalFallback, fallbackProbeInterval, pollInterval), arms.director/arms.comms/arms.directorArchive, a Secrets pair for the el-director account, and the example document. The keys are named userEnv/tokenEnv rather than the design's dmUserEnv/dmTokenEnv because load.go's credential scan refuses a credential-shaped key that is not on its exact-match allowlist and dmTokenEnv is not. Two refusals are deliberate and loud, not stubs: * director.signalFallback: true is refused (FaultUnwiredArm). The fleet has exactly one Signal egress — the pi-signal-bridge systemd USER unit on the mgmt VM, reached over the pi-intercom socket — and a dispatcher pod cannot reach it. comms.UnwiredSignalClient fails with that reason instead of reporting pages that never arrive. * arms.directorArchive: true is refused for the same reason: the warrant archive reader does not exist yet, so the loader validates warrant SHAPES and says so (a nil Resolver). buildDirector fails loud on a persona that does not validate or a corpus that cannot be read, and REPORTS memory faults rather than dying on them: one record whose warrant does not resolve must not take the Director off the air. The veto stop is a claim release via the ordinary Plane writer (R-6) — no sentinel scope, and the arm cannot stop a sandbox it does not own. Fixture-driven arm tests use the committed persona, registry and chat fixture: no socket is opened.The LD4.9 mechanism as a pure, fixture-testable package: scope.go the plan of record: the epic description's <h2>Scope</h2> checklist, parsed and normalised (entities, tags, three citation shapes; a citation key must be >=3 chars so `R-25` cannot become a manufactured finding) diff.go the 4-class diff. D1 (planned, unticketed) and D2 (unplanned child) are identifier mismatches -> drift. D3 (stale tick) and D4 (uncited scope item) are advisory lines only. That asymmetry is the cost brake: prose can never dispatch the architect lane. pass.go one pass over one epic: read, diff, record on the epic, and on drift create exactly ONE queued dispatch issue (Todo + label plan-drift + `ld:plan-drift` marker and JSON payload). R-25 never drafts a ticket body. Receipts follow the R-25 design: a no-change pass returns a VISIBLE success-noop carrying the epic comment ref (the comment itself opens with "no change"); a pass that could not LOOK returns an EMPTY receipt which is `missed`, never success-noop. A failed disposition returns an error so the runner records `failed`, keeping the epic comment ref as evidence. Child enumeration is modelled as `Children(ctx, epicID)` rather than a query because the self-hosted proxy ignores `?parent=` (see the watch commit). 92.9% statement coverage; every test is fixture-driven and none opens a socket.The window is MEASURED at runtime, never assumed: retention here is size-bound (storage.tsdb.retention.size=30GiB) even though retention.time reads 15d, so the realised window was 10.00d. A census that assumed 30d would report every slow-cadence alert as never-firing, and a delete disposition built on that would be wrong. census.json therefore records the window, both retention flags, the evaluation interval, and queries[] — every PromQL string with its step and bounds, so a reviewer can re-derive every number by hand. Measured 2026-10-02 over 10.00d: repo-owned (the denominator) 63 rules, 27 page-eligible = 42.9% all live rules (annex) 210 rules, 66 page-eligible instances (64 names) fired and page-eligible 9 repo-owned The generator FAILS CLOSED below a 7-day window (--min-window-days): a short window manufactures false never-firing rules. Page-eligibility is a walk of the LIVE Alertmanager route tree (parsed from /api/v2/status config.original), never a hard-coded list; repo attribution comes from the PrometheusRule manifests. Two traps are fixed and regression-tested: summing series for %window (348% on a three-series rule) and classifying by substring on the rule file path (the mount directory contains 'kube-prometheus-stack'). Findings the census surfaced, in notes/2026-10-02-live-probe-findings.md: - 6 repo-owned page-eligible rules are DEAD: their metrics have no series at all - the route tree names 5 alertnames of which 4 have no live rule; one of those (EtcdHealthCritical) was the #2 source of notifications in the rail, i.e. Alertmanager re-notifying stale state for a rule that is not deployed - ControlPlaneStaticPodsCrashLooping was notified on 2026-09-30 with no ALERTS samples anywhere in the window: recorded as an open anomaly, not papered over Also: --record appends a sampled daily line per alertname (the honest path to the ticket's literal 30-day census, true from ~2026-11-01), and --severity-from manifests computes the PROJECTED page-eligible count for a dispositions change that is not deployed yet, using the same route walk.Standing rule, enforced: every alert rule ships with a disposition contract — ld-owner, ld-action, ld-frequency, ld-disposition (keep-page|keep-record), ld-silence-policy, ld-review-after. A rule without a consumer does not merge. Annotations, deliberately NOT labels: annotations do not participate in a series' identity, so adding them creates no new ALERTS series, churns no fingerprints and costs no cardinality. Labels would change the identity of every series — the cheapest possible way to corrupt the census in the previous commit. Two legs, because each alone has a hole: - diff-scoped refusal (ALERT-CONTRACT-001/002): rules added or modified by the PR must carry all six annotations with valid values - legacy ratchet (ALERT-CONTRACT-003/004): legacy-rules.txt lists the 32 rules that predate the contract; a listed rule that is MODIFIED without acquiring the contract is refused, and the file may only shrink (a new line is refused) Stated limits, in the README rather than hidden: the gate can be bypassed by the E-adj paths (Alertmanager route tree, defaultRules) which are protected-namespace and controlled by review class; it refuses the ABSENCE of a contract, not a dishonest one; and prod's 5 edge-probe rules carry the contract but are not yet CI-gated (follow-up).Separate commit from the tooling so a revert of the paging change is one revert. 27 repo-owned rules were page-eligible (severity=critical reaches pi-alert-receiver). 15 are demoted to severity=info with the ld-* contract, which moves the repo-owned page fraction 42.9% -> 19.1% (27 -> 12 of 63), the ticket's <20% target. Evidence per rule is in the ld-note annotation and in lab-director/alerts/dispositions.yaml; every demotion is one of: (a) DEAD RULES (6) — the metric has no series anywhere in the TSDB, so the rule can never fire: VeleroBackupFailure, VeleroBackupMissing, LonghornBackupTargetMissing, LonghornVolumeReadOnly, PVEDiskFull, PVENodeUnreachable. Demotion is a no-op SAFETY step, not a paging reduction; deletion is proposed for the follow-up PR (demote first, delete after one clean cycle). Said plainly so the delta is not oversold. (b) CHRONIC STATE (3) — paging on a condition rather than an event, and duplicating a page-eligible chart rule: PersistentVolumeCritical (83.1% of the window at >90% PVC fill), NodeDiskFull (66 episodes around a flat 90% line), ClusterMemoryHeadroomCritical (fires below 2 GiB cluster-available while the measured minimum was 107 GiB — the threshold is ~54x below reality, with per-node usage peaking at 74% against NodeMemoryPressure's 90% line). (c) FIRES CONSTANTLY, NEVER ACTED ON (5) — LonghornCSIAttacherLeaderLost (190 episodes: the threshold is literally '> 0 restarts'), PodCrashLooping (99 series, 44 notifications, for: 0m, duplicating the chart's damped KubePodCrashLooping), LonghornVolumeFaulted (47 volumes matched, consistent with detached/stale series), NodeMemoryOvercommit (55% observed against a 200% line — a capacity signal, not an incident), LokiIngesterFailing (compares a raw counter to 0), TempoSpanPipelineStalled (an 'and' between differently-sampled counters: 22 pending episodes, 0 firing). Deliberately NOT demoted, with reasons recorded in the annotations: - EdgeProbeDown (critical) — measured and found NOT to be a duplicate: it matches only the three self-hosted public services while the warning variant carries the other 25 edge hosts. Its flapping is a 'for:' problem, not a silencing problem. - ControlPlaneStaticPodsCrashLooping — a severity demotion CANNOT stop it paging: the Alertmanager route tree names it explicitly ahead of the severity=critical route. - CiliumEnvoyDown, NodeDown, PrometheusWALCorruption, LonghornDiskCritical, CNPGBackupStale — unique coverage, verified against the metric's observed range over the window (e.g. cilium-envoy up=1 and node-exporter up=1 for the whole window, Longhorn disk peaking at 70% against a 90% line). No deletion disposition is proposed: the window is 10.0d, so a slow-cadence rule would show zero evidence and a deletion would be window truncation, not proof. Every annotated rule's expression was verified byte-identical (whitespace normalised) to the running Prometheus revision, so the dispositions are authored against what the cluster actually evaluates.The trigger source, the disposition lane and the registry record: * `arms.boundary` (OFF by default) plus a `boundary:` block — interval, coldStartPolicy, pageSize/maxPages, overlap, emittedRetention. The cold-start policy has NO default and is REQUIRED when the arm is on: a scan that finds no watermark must be told whether the downtime was absorbed or lost, and the loader will not choose that for the operator (the watch's rule, restated for the second observer). * `templates.plan-drift` — the LD4.9 item kind, alongside LD4.1's two. It answers "which lane serves this item kind", not "how risky is this ticket", so it is a template key and not a class letter. * R-25 registers in the committed registry as `onEvent: ticket.done` + `scope: epic`, `cadencePeriod: week`, outputContract planeComment, status PAUSED with `pausedReason: dep:routine-executor` — the honest reason: the event-trigger seam through routines.Runner, a live ExecutionPlane and an Executor are not wired. `emitsWork` is deliberately unset (the entry is paused, so it creates nothing); when the executor lands, the drift dispatch issue must be declared there as an edge to whichever routine consumes `plan-drift` items — inventing an edge to R-12 (a calendar sweep) would be a fiction. * app.go wires the arm as a NON-FATAL runner and builds its OWN in-process watermark store (never the watch's). It reads the routines registry from ROUTINES_REGISTRY_PATH best-effort: a missing mount is a WARN and the sink logs `registry_loaded=false`, so "no mount" is never confused with "no routine wants this event". Three sibling-facing test adjustments, all growth-tolerant so the other 4.x registrants (R-24 next) do not have to touch them again: the registry census count is now a floor plus a named R-25 shape assertion; the corpus-parity rule now requires an entry outside R-01..R-22 to name its owning ticket; and `routines -check`'s expected total is derived from the registry instead of hardcoded. `routines -check` is green (23 routines, 3 active, R-25 paused on dep:routine-executor).Landing review blocker 2 (undeclared API fan-out): the adapter read one GET /raw/{path} per test file, so cost was linear in the repo's test count (207 files ~= 5 min against R-24's 900 s window). - File() is now served from the revision tarball, materialized by ONE /archive/{rev}.tar.gz request; the raw endpoint survives only as a bounded fallback (archive disabled, gitlink). - Runs are two bounded, filtered windows (the endpoint ignores limit without a filter): head_sha=<rev> and head_branch=<gate>&event=push. Both cached. - MaxRequests hard stop with a loud ForgejoRequestBudgetError. - BranchRunSource + probeContext.runs() merge the gate-branch window. - cmd/readiness reports per-target wall time + GET count on stderr.First slice of LD4.2, per the orchestrator's assignment: prove or disprove the brief's riskiest assumption (§5.3 / R-5) before anything is built on it. Verdict: DISPROVED on both halves. * prod/manifests/agent-sandbox/ declares six SandboxTemplates (ld-sandbox, ld-build-test, ld-browser, ld-firmware, ld-meet-ai, ld-claude-code) and sandbox-init/roles.json declares two roles (light, builder-core). There is no roundtable class and no planner class. * The union of every class's egress allowlist is DNS, Forgejo, OmniRoute, the NATS bus and (light only) OTel. Outline appears nowhere in cilium-network-policies.yaml, and the dispatcher's own policy omits it too. The unratified Gateway-VIP egress hole (HOMELAB-1786) is explicitly NOT a path this slice may build on. Consequence: the documented fallback applies. The meeting-record Plane item is the record's durable home in v0; a roundtable class and its Outline egress are an operator decision (new SandboxTemplate + role + CNP, all on AGENTS.md's dangerous list), recorded in the PR body and the ticket's verification comment.The routing half of LD4.2. A meeting has no ticket, so the meeting-record Plane item IS the ticket: marker ld:meeting-record (+ label roundtable for humans), routed by laneForIssue to a new template kind. * intake: MarkerMeetingRecord + LabelRoundtable, and MarkerLinePayload so director/ parses a marker payload the same way intake/ delimits one. * config: TemplateKindRoundtable (the third item-kind key) so the loader accepts it; the shipped binding is ld-sandbox, because there is no roundtable SandboxTemplate (see docs/ld4.2-roundtable-class-verdict.md). * laneForIssue: marker ld:meeting-record -> roundtable, checked before the LD4.1 markers. The MARKER routes, not the label - the same posture ReadUndigested takes; requiring the label too would let a missing label silently demote a meeting to ordinary ticket work. * config: arms.meetings (Bool, unset=OFF) + a meetings block (interval/state/pageSize/maxPages). meetings.state defaults to the watch's first dispatchableStates entry, so the arm's view of "open" and the watch's view of "dispatchable" cannot drift; the loader requires it when the arm is on.The image half of S1. The image does not exist yet, so this is what makes it buildable and publishable through the normal pipeline: * lab-director/whisper-asr/Dockerfile — multi-stage whisper.cpp v1.9.4 build (GGML_NATIVE=OFF: the CI runner and prod-wk-01 are different CPUs), with ggml-large-v3-turbo-q5_0 baked in and its sha256 VERIFIED at build time (the hash was read off the mgmt VM's own copy, so a drift fails the build rather than silently baking a different file). CPU only; no ffmpeg (the caller sends 16 kHz mono WAV, which is the contract); curl is present so the smoke test and an operator's in-pod check need no second image. * .forgejo/workflows/whisper-asr-image-build.yaml — PR: build + smoke test (no credentials); live: publish <short-sha> + the 0.1.0/latest aliases and print the digest. The smoke test exercises the real contract on a generated WAV. * lab-director/whisper-asr/README.md — the contract, the consumers it copies (meeting-media.py, voice-whisper.json), and what is deliberately absent. Also fixes a real defect this slice exposed: the FILE context author ignored a resolved lane, so a roundtable dispatch under context.author=files (prod's setting) would compile class "default" while the spawn arm was handed "roundtable" — and spawn's preflight refuses a package whose class disagrees with its spec. filesContextSource now implements PackageForLane, with the regression test at cmd/dispatcher/roundtable_pipeline_test.go:TestTheFileAuthorHonoursTheRoundtableLane.LD4.2's producer: the operator-mediated half of the recording-ingest convention. Proton Meet has no server-side recording API (E2E MLS), so the pipeline's input is an exported MP4 an operator drops. This component runs on the mgmt VM as a systemd USER unit — the pi-signal-bridge precedent (no npm deps, env file, loopback tunnel) — because the orchestrator's ingest-v1 ruling is exactly that shape, and because no workload in this cluster mounts the media NFS (the brief's in-cluster watcher would have meant inventing a storage contract to satisfy an ingest). inbox/<recording> --ffmpeg 16kHz mono--> POST whisper-asr:8171/inference --> Outline transcript doc (Context Block + #t=HH:MM:SS cues) --> ONE meeting-record Plane item (marker ld:meeting-record, label roundtable, Todo) carrying the meeting.completed payload The last step is the trigger the dispatcher's meetings arm sweeps. Design points that are load-bearing rather than incidental: * The wire is the SAME as meeting-media.py's: hand-assembled multipart, field "file" + response_format/temperature/language, 16 kHz mono pcm_s16le. A test asserts the exact bytes. * The Plane payload is the Go marker grammar: marker line + ONE JSON line whose field names are director.MeetingRecord's. A Node-rendered item is committed to director/testdata/ and PARSED BY THE GO READER in a test, so a drift on either side breaks a test instead of a roundtable. * MEETING_ATTENDEES is REQUIRED: events.MeetingCompleted.Gaps() treats an empty attendee list as a gap, so an ingest without the series' seats would produce a meeting that never convenes, and that failure would look like a broken roundtable rather than missing configuration. Diarization is deferred, so the seats are stated once by the operator, not guessed from the audio. * Evidence is never deleted: sources are moved (inbox -> done/failed) by an atomic rename claim, and a failure leaves the file plus a .error.txt beside it. * One writer per item class: this component creates meeting-record items only. Direction items are the roundtable's. Also adds whisper-asr-tunnel.service (the loopback port-forward that reaches the in-cluster ClusterIP without a new network surface). Evidence: node --test -> 21 tests, 0 sockets opened; go test -race ./... and go vet ./... green.The check over what the emission side writes. `rollup.go` is untouched (its sha256 pin is a promise an auditor re-derives, and LD4.4 owns it); this is a new file that READS RollupResult: * `SplitPhaseTotals` / `SplitPhases`: the per-sandbox census whose arithmetic is the ticket's verify line 2 — `Σ(build, review, verify) + untagged == total` exactly, integer tokens, no tolerance. `unphased` (the value the telemetry writes when nothing was declared) and any label outside the closed enum are UNTAGGED and NAMED, never a fourth bucket and never folded into one of the three. A blank-phase sample lands in untagged by construction (GroupSamples skips it deliberately). * `ValidatePhaseSplit` / `ValidatePhaseSplitWithDeclarations`: the enforcement point. A non-summing split is an episode-class mismatch (a folding bug); a bucket that DECLARED a phase and still carries untagged spend is a dispatcher defect; untagged spend on an undeclared/legacy window is reported with the instrument's own coverage. Both cite P-ROUT-012. * `CalibrationLine` / `PhaseCensusMarkdown` / `PhaseCensusDigest`: verify lines 3 and 4. The line quotes THIS fleet's measured validation share with n, the window and the untagged share, and the vendor datum appears only as an external, single-mission comparison ("not reproduced at this sample size" / "one comparison, not corroboration"). An empty window is a visible no-op, never a published 0 %. The table gives untagged its own column: unknown is not healthy. * `PhaseCensusExecutor`: the runner seam for R-29 (the R-06 ExecutorAdapter precedent) — names the series it emitted and refuses to claim success from a failed read. Registry: R-29 `phase-census`, ACTIVE (its mechanism ships in the same build), pinned to this file's sha256. It takes R-29, NOT the briefed R-27: the header has reserved R-27 for 4.11 since LD4.4 and `testdata/registry-4x-shapes.yaml` already carries R-27's maintainability-fitness shape (asserted by TestSchemaAccommodatesHosted468Shapes), so a second R-27 would break the registry's own "ids are stable forever" rule. R-28 is LD4.5b's on-call window; R-29 is the next free id. The census floor in registry_test.go is now derived from the corpus fixture instead of a hardcoded 24. policy.go/policy.md: P-ROUT-012 (both files change together — the doc is parsed back by TestPolicyBookMatchesDoc). Tests: phase_test.go (TestC6*) covers the identity, the untagged cases, the declared-phase defect, the mismatch episode, the calibration line, the rendered census row and the no-op receipt. `go test -race ./...` + `go vet ./...` green.Companion PR note: #595 (LD4.5d's Tempo
histogram_buckets+phasedimension) is reviewed CLEAN, minimal (exactly the two promised keys, no retention/query changes), and deliberately held for operator review (protected monitoring ns). Its base isepic/LD4, so its observable effect rides this PR's merge (prod pulls Tempo values from infra-corelive). Operator review of #591 should include #595. 1593's premise independently reproduced during review: subagent span p99 = 16.38s.The one authenticated hop by which a sandbox's product reaches Plane, plus the process that serves it. - gateway/api.go: Submission{Verb, Comment, Drafts} — deliberately NO project, state, assignee or target field (the ticket is the nonce's, a draft's state is the Backlog constant the watch never publishes). Strict JSON decoding refuses an unknown field at the boundary, so no request can smuggle a target or a state. Verb allowlist + role charter: the roundtable comments, the architect drafts, every other role writes nothing. Size caps. Receipt shape. - gateway/nonce.go: HMAC-SHA256 over role||ticket||lane||expiry, minted by the dispatcher (min(claim deadline, now+15m)), presented as "Authorization: Bearer <nonce>", verified statelessly in constant time. The token is <base64url(payload)>.<base64url(mac)> — the brief's shorthand writes the MAC alone, which a verifier cannot recompute the claims from; the file documents the deviation. Fingerprint-for-ledger, Bearer-only scheme. - gateway/plane_writer.go: the gateway's own narrow Plane client (Comment, CreateItem which resolves the project's Backlog state by name or group; no transition, no edit, no read on the hot path). Separate from claims.APIWriter on purpose: the dispatcher owns state/leases with one key, the gateway owns product writes with per-role keys. - gateway/server.go: the request path (nonce → target assertion → strict decode → shape → charter → replay ledger → per-role token → write → receipt). The replay ledger is a real single-use guarantee for ONE replica (reservation taken before the write, released on failure so an outage does not burn the nonce) and the documented TOCTOU beyond that (OQ-4). A path-form asserted ticket must equal the nonce's, so a nonce cannot be retargeted. - cmd/ld-result-gateway: env config (LD_SUBMIT_HMAC_KEY, PLANE_*, the registry-derived per-role LD_PLANE_TOKEN_<ROLE>), --check/--version, graceful shutdown, /healthz + /readyz. The token map is built FROM the registry's tokenRefs, so adding a role's credential is a registry entry plus a Secret key. - Dockerfile: the gateway binary rides in the dispatcher image (the same HOMELAB-1688 precedent as routines): one module, one reviewed digest. Tests (no sockets): every Verify-line-1 refusal is a named fixture test asserting ZERO Plane calls (no nonce, forged nonce, expired nonce, wrong-ticket assertion, unknown field, unchartered role), plus attribution, Backlog-only creation, replay 409, 503-with-the-missing-ref and nonce-release on outage.The result-return path: a lane now ends when its run's bundle lands, and Plane gets written as the ROLE through the gateway. - cmd/dispatcher/resultwatch.go: the resultBundleWatcher. It probes .sandbox-state/agent/result.json with the sandbox's own `test -f` through execute_command — a shape the contract verifies (a non-zero exit is DATA, not a tool error), unlike download_file (where absence and a broken transport look alike) and unlike the MCP file_exists tool, whose response shape is not verified against the pinned server. On presence it reads, strictly decodes into teardown.Result, and returns LaneOutcome{Result, Contract}: teardown is unchanged. On the sandbox TTL with nothing, the OLD ending stands — ReasonNoResultSource, a FAILED run. A bundle that declares its OWN contract is refused at the decoder boundary, and a gateway-wired lane whose bundle has no product-receipt (or a receipt from another ticket/role/lane) is refused. - contractFromPackage: the ResultContract comes from the compiled package's ticket-and-acceptance-criteria section (OQ-7(ii)) — a fenced ld-result-contract block of name: sha256:<hex> entries. A malformed declaration is a refusal BEFORE the claim; no declaration is the honest empty contract. - pipeline.go: the lane's dispatch nonce is minted after the claim (expiry = min(the claim's lease deadline, now+15m)) and handed to spawn; the watcher receives the lane's expectations (ticket UUID, role, whether a nonce was delivered, the ticket's contract) through the optional laneExpecter interface, so every existing watcher and fixture keeps working unchanged. An A-E class lane gets no nonce and keeps exactly the reach it had before. - gateway/lanes.go: the v1 lane→role binding (roundtable→roundtable, direction→architect). The compiled package's static role is a section-ordering input, NOT the attribution authority; an unknown lane is inert. - app.go / main.go: the default watcher becomes the bundle watcher; Options gains SubmitHMACKey (read from LD_SUBMIT_HMAC_KEY), so the whole product path is inert until the operator flips it. - spawn/fixture.go: the fixture now MODELS `test -f <path>` from its own file map (a fixture that answers "present" for every path would make every watcher test a lie); a scripted ExecResult still wins. Tests (Verify lines 2 and 3, mechanized through a REAL gateway over httptest, the real mint path and the real verifier): the roundtable record lands and the lane ends on the bundle with disposition=completed / evidence.verified=true and NO no_result; the architect drafts land in Backlog; a lane that submits nothing still ends reason=no_result + disposition=failed with the nonce delivered; a self-declared contract is refused; a bundle without product-receipt is refused; a class lane gets no nonce. Plus the extraction matrix and the malformed-contract refusal before the claim. Honest limit carried in the code: the nonce lives 15 minutes (min(claim deadline, now+15m) as designed), so a run longer than that gets a 401 and its lane closes ReasonNoResult — a loud, named failure. Re-minting on the heartbeat (or a per-lane nonce TTL) is the follow-up; minting a longer-lived nonce here would silently widen the boundary.LD4.7's pure half: the signal schema, the gauge burn arithmetic over probe_success, the priority ladder, the planner's ErrUncitedSignal seam, the digest hash and CitesDigest. - burn.go owns avg_over_time SEMANTICS as code (a 0/1 gauge is a MEAN, not a rate()), half-open windows, ErrNoSamples for an empty window (a coverage gap must never read as a quiet window), and budget_consumed_ratio. - priority.go: Propose takes the citation as a PARAMETER and refuses one that is not measured, so there is no code path that yields a priority without evidence. Retiring surfaces are forced to P4 with the retire ticket as the citation; an edge-observed fault caps at P3. - plan.go reuses 4.1's WindowKey/Plan shape and refuses the whole window on an unmeasured group (batch-wide, the 4.1 ruling). - digest.go renders the item body with signal_digest_hash=..., cites.go decides whether a draft quotes a hash the ledger actually holds (Verify line 3). - direction.Entry gains ONE additive omitempty DigestHash field (the transition 4.1's KindDigest comment reserves for 4.7; the 4.8 Run.Target pattern). - Policy as data: lab-director/signals/{slo-policy,priority-ladder}.yaml (target 0.99, retire surfaces flagged with SEND-546, fixture SLO flagged).CLASS: E-adj / DANGEROUS. `monitoring` is a protected namespace (AGENTS.md), so this commit must not auto-merge and is deliberately kept SEPARATE from the routines/ and pi-extensions/ legs of HOMELAB-1850 so the class boundary is visible on the board. Operator review required. 1. HOMELAB-1593 — histogram_buckets. The premise measured worse than stated: Tempo's default ladder tops out at 16.384s while subagent runs have a measured MEAN of ~1640s, so p99 was not coarse but CONSTANT (16.38s, one saturated bucket) and stall-outlier detection could not work, because the stall signature IS the top bucket. The ladder added here keeps the sub-second regime (tool spans run 2ms-64ms) and extends to 1h for agent spans — the ladder is GLOBAL to the metrics generator, not per span name, so dropping the small buckets would coarsen every `tool:*` span and both Pi dashboards. 2. HOMELAB-1850 — `phase` in `span_metrics.dimensions`, so the call/duration side of the Costs pane can be split by stage the way the token side already is. `phase` is a closed enum (build|review|verify|unphased; <= 4 values), so its cardinality cost is bounded by the existing dimension set. Not taken here, deliberately: the `sandbox_env` dimension (LD4.4's key). It multiplies with subagent.type x llm.model and the brief makes it a MEASURED decision (R4) — enable it once the series count after this change is known. Verified: `helm template grafana/tempo --version 1.24.2 -f <values>` renders both keys at `metrics_generator.processor.span_metrics.{dimensions,histogram_buckets}` (the chart passes `processor` through verbatim — chart 1.24.2 values lines 53-63), i.e. the Tempo 2.9 span-metrics schema, so the keys are the chart's own and not a guess. Rollout: infra-core `epic/LD4` → `live` → prod `apps/tempo.yaml` ($values reference, targetRevision: live) → ArgoCD sync. The metrics-generator restarts with the change and the new ladder starts fresh (a step, not a backfill); the default-ladder series age out of Prometheus' 10.03d retention.Review —
3514b82verdict: pass — integration review of the epic→live PR. Every constituent PR was independently reviewed by a fresh-context landing worker with an after-head verdict and merged CI-green (#567, #568, #569, #573, #575, #576, #577, #592, #594, #596, #597, #598, #599, #600 + pi-extensions #57/#60/#61/#63); the PR body inventories the stack, the OFF-by-default arm postures, and the activation sequence. Delta since the last constituent merge: three CI-leg fixes with diagnosed root causes (whisper static-link/exit-127, the alert-contract gate's merge-base bug, sandbox-init test pins), a terraform-init retry wrapper, and retrigger commits — all verified green on this head (CI run set on
3514b82: all jobs success). No blockers. Deployment risk posture: every new arm ships OFF; the rollout path is ArgoCD sync of the already-reviewed manifests.