HOMELAB-1728: LD4.7 — signals-from-production intake (collector arm, SLO as data, burn in Go) #599

Merged

aaron merged 6 commits from plane/HOMELAB-1728-signals into epic/LD4 2026-10-03 05:58:12 +00:00

Conversation 2 Commits 6 Files changed 35 +7166 -18

aaron commented 2026-10-03 05:01:46 +00:00

Owner

Copy link

Closes the builder half of HOMELAB-1728 (LD4.7 · signals-from-production intake), built to .rpiv/artifacts/epic-ld4/design-LD4.7-signals.md and the orchestrator adjudication on the ticket (SLO substitute accepted: probe_success availability, target 0.99, burn computed in Go, zero monitoring writes; collector = in-dispatcher arm signalintake/; R-23 active/300s/go:in-process/consumer:architect; DigestHash additive ledger field; no cross-project SEND read in v1).

Companion PR: Homelab/prod#— (fixtures ns + config + the one DANGEROUS CNP rule).

What this is

The collector arm (R-23) that turns application telemetry into direction: once per window it measures the availability SLI per surface, computes the error-budget burn in Go, correlates signals per surface, and emits one cited direction item per group above P4 into 4.1's inbox. Signals propose; the architect drafts; the operator ratifies.

lab-director/dispatcher/signalintake/     NEW arm (pure core + I/O + writer + tests + README)
lab-director/signals/{slo-policy,priority-ladder}.yaml, README.md   NEW policy as data
lab-director/dispatcher/direction/ledger.go   +1 additive omitempty field (DigestHash)
lab-director/dispatcher/routines/registry.yaml         R-23 + reserved-ids header reconciliation
lab-director/dispatcher/routines/testdata/registry-4x-shapes.yaml   one comment (superseded trigger)
lab-director/dispatcher/config/{config,example}.go     signals block + arms.signals (default OFF)
lab-director/dispatcher/cmd/dispatcher/arms_signals.go + app.go     the arm, non-fatal

The three Verify lines, mechanized

Verify	Fixture leg (deterministic)	Live leg
1. a synthetic error-budget burn in sendrast-test produces a digested signal with evidence link + priority proposal	TestBurnScaleZeroFullOutage (120 failing scrapes → error ratio 1.0 → burn 100 at the 0.99 target) → TestAC1ScaleZeroBurnProducesADigestedCitedSignal (one direction.Item, source: application-signal, P1, non-empty PriorityCitation, body carries signal_digest_hash=… + the Grafana link, sink idempotent on a second emit) → TestArmPassEmitsOneCitedItem (the same end-to-end through the arm)	scale deploy/sendrast-test-canary --replicas=0, one pass, read the item back, --replicas=1 (prod PR ships the fixture), plus TestBurnReplayRealProductionBurn: the real 2026-10-02T01:29Z burn re-derived from retained samples (30 failing scrapes in the 120-scrape hour → ratio 0.25 → burn 25 → above the 14.4 rung)
2. a signal without a measured evidence link is refused at intake	TestAC2SignalWithoutMeasuredEvidenceIsRefused: empty evidence → ErrUncitedSignal; a link with no observed number → ErrUncitedSignal (a link is an invitation to look, not a measurement); 4.1's FixtureSink.Emit independently refuses a hand-built priority with no citation; zero items land	TestArmRecordsTheLogCoverageGap: a live pass whose Loki source has no application coverage writes a named no-coverage refusal row — the absence of evidence is loud in production too
3. the architect's drafted ticket cites the signal digest hash	CitesDigest(draftBody, ledger) — true when the draft quotes a hash a KindDigest row holds, false for an unknown/empty hash, error on an unreadable ledger (cites_test.go); the hash is on the item body, in the ld:signal-digest marker payload and on the ledger row	the drafted ticket quoting signal_digest_hash=<hash> after the intake sweep runs (awaits the routine executor R-11 is paused on — stated, not claimed)

Supporting: go test -race ./... + go vet ./... green; one-writer-per-class (WriterFor is structural: this is the only application-signal writer); no priority without a citation is a signature property (Propose takes the citation as a parameter and refuses an unmeasured one); quiet windows write nothing; a re-planned window issues no query at all.

Deviations from the brief (each deliberate, all reported to the ticket)

1. burn_rate 200 is 100 at the adjudicated target. The brief's §7.2 figure assumed a 0.995 target; the orchestrator ruled 0.99, so a full outage burns 1.0/0.01 = 100. Both rows are asserted in TestBurnScaleZeroFullOutage so the arithmetic is visible rather than remembered.
2. The marker is ld:signal-digest, not 4.1's ld:direction-digest. A signal item's external id contains the window, and 4.1's guard 3 (FindDigestIssue) matches its marker plus the window string — a signal item carrying 4.1's marker would make the daily sweep believe the digest execution issue already existed and it would write nothing. The distinct marker is the difference between a working loop and a silently starving one (asserted in plane_test.go).
3. The fixture Probe uses the existing tcp_connect blackbox module. Every http_edge_* module asserts TLS and would fail against an in-cluster plain-HTTP Service, and adding an in-cluster HTTP module would be a write into monitoring — the change the design forbids. probe_success and the burn arithmetic are identical, so the code path is the same one production uses. (Prod PR detail.)
4. Ownership is a seam, not a read (deferral). Brief Q3's ownership/verdict.go extraction would refactor 4.5b's landed suppression.Plan. The arm takes an OwnershipReader interface and degrades to unknown (logged once), which proposes a draft; the architect's own reads still stand between that and work. Wiring it later is not a rewrite. No 4.5b code was touched.
5. LogCoverage ships ON, and the anomaly leg claims nothing. No application logs are shipped to Loki anywhere in prod/; the adapter + the pure z-score planner ship, and the live surface is the named refusal. The enabling change (a log shipper) is a separate E-adj ticket.

Not in this PR (stated, not hidden)

• no cross-project SEND read in v1 (adjudicated);
• the JetStream ledger adapter (the dispatcher is not yet an admitted bus client — 4.1/4.5b's named follow-up); the arm uses the direction.Ledger interface;
• R-11 intake-daily-sweep is paused on dep:role-intake, dep:LD4.1-intake-arm, so signal items land in Plane but are not yet batch-digested — emits: [R-11] declares the edge and the registry's cycle check runs on it.

Reviewer notes

signalintake/plan.go is sha256-pinned by R-23's scriptRef and TestInProcessPinsMatchTheCode re-derives it — any change to the planner must re-pin.

Closes the builder half of HOMELAB-1728 (LD4.7 · signals-from-production intake), built to `.rpiv/artifacts/epic-ld4/design-LD4.7-signals.md` and the orchestrator adjudication on the ticket (SLO substitute accepted: `probe_success` availability, target 0.99, burn computed in Go, **zero monitoring writes**; collector = in-dispatcher arm `signalintake/`; R-23 active/300s/go:in-process/consumer:architect; `DigestHash` additive ledger field; no cross-project SEND read in v1). Companion PR: Homelab/prod#— (fixtures ns + config + the one DANGEROUS CNP rule). ## What this is The collector arm (R-23) that turns application telemetry into **direction**: once per window it measures the availability SLI per surface, computes the error-budget **burn in Go**, correlates signals per surface, and emits **one cited direction item per group above P4** into 4.1's inbox. Signals propose; the architect drafts; the operator ratifies. ``` lab-director/dispatcher/signalintake/ NEW arm (pure core + I/O + writer + tests + README) lab-director/signals/{slo-policy,priority-ladder}.yaml, README.md NEW policy as data lab-director/dispatcher/direction/ledger.go +1 additive omitempty field (DigestHash) lab-director/dispatcher/routines/registry.yaml R-23 + reserved-ids header reconciliation lab-director/dispatcher/routines/testdata/registry-4x-shapes.yaml one comment (superseded trigger) lab-director/dispatcher/config/{config,example}.go signals block + arms.signals (default OFF) lab-director/dispatcher/cmd/dispatcher/arms_signals.go + app.go the arm, non-fatal ``` ## The three Verify lines, mechanized | Verify | Fixture leg (deterministic) | Live leg | |---|---|---| | 1. a synthetic error-budget burn in `sendrast-test` produces a digested signal with evidence link + priority proposal | `TestBurnScaleZeroFullOutage` (120 failing scrapes → error ratio 1.0 → **burn 100** at the 0.99 target) → `TestAC1ScaleZeroBurnProducesADigestedCitedSignal` (one `direction.Item`, `source: application-signal`, `P1`, non-empty `PriorityCitation`, body carries `signal_digest_hash=…` + the Grafana link, sink idempotent on a second emit) → `TestArmPassEmitsOneCitedItem` (the same end-to-end **through the arm**) | `scale deploy/sendrast-test-canary --replicas=0`, one pass, read the item back, `--replicas=1` (prod PR ships the fixture), **plus** `TestBurnReplayRealProductionBurn`: the real 2026-10-02T01:29Z burn re-derived from retained samples (30 failing scrapes in the 120-scrape hour → ratio 0.25 → burn 25 → above the 14.4 rung) | | 2. a signal without a measured evidence link is refused at intake | `TestAC2SignalWithoutMeasuredEvidenceIsRefused`: empty evidence → `ErrUncitedSignal`; **a link with no observed number** → `ErrUncitedSignal` (a link is an invitation to look, not a measurement); 4.1's `FixtureSink.Emit` independently refuses a hand-built priority with no citation; zero items land | `TestArmRecordsTheLogCoverageGap`: a live pass whose Loki source has no application coverage writes a **named `no-coverage` refusal row** — the absence of evidence is loud in production too | | 3. the architect's drafted ticket cites the signal digest hash | `CitesDigest(draftBody, ledger)` — true when the draft quotes a hash a `KindDigest` row holds, false for an unknown/empty hash, error on an unreadable ledger (`cites_test.go`); the hash is on the item body, in the `ld:signal-digest` marker payload **and** on the ledger row | the drafted ticket quoting `signal_digest_hash=<hash>` after the intake sweep runs (awaits the routine executor R-11 is paused on — stated, not claimed) | Supporting: `go test -race ./...` + `go vet ./...` green; one-writer-per-class (`WriterFor` is structural: this is the only `application-signal` writer); no priority without a citation is a **signature property** (`Propose` takes the citation as a parameter and refuses an unmeasured one); quiet windows write nothing; a re-planned window issues **no query at all**. ## Deviations from the brief (each deliberate, all reported to the ticket) 1. **`burn_rate 200` is 100 at the adjudicated target.** The brief's §7.2 figure assumed a 0.995 target; the orchestrator ruled 0.99, so a full outage burns `1.0/0.01 = 100`. Both rows are asserted in `TestBurnScaleZeroFullOutage` so the arithmetic is visible rather than remembered. 2. **The marker is `ld:signal-digest`, not 4.1's `ld:direction-digest`.** A signal item's external id **contains the window**, and 4.1's guard 3 (`FindDigestIssue`) matches its marker *plus the window string* — a signal item carrying 4.1's marker would make the daily sweep believe the digest execution issue already existed and it would write **nothing**. The distinct marker is the difference between a working loop and a silently starving one (asserted in `plane_test.go`). 3. **The fixture Probe uses the existing `tcp_connect` blackbox module.** Every `http_edge_*` module asserts TLS and would fail against an in-cluster plain-HTTP Service, and adding an in-cluster HTTP module would be a **write into `monitoring`** — the change the design forbids. `probe_success` and the burn arithmetic are identical, so the code path is the same one production uses. (Prod PR detail.) 4. **Ownership is a seam, not a read (deferral).** Brief Q3's `ownership/verdict.go` extraction would refactor 4.5b's landed `suppression.Plan`. The arm takes an `OwnershipReader` interface and degrades to `unknown` (logged once), which proposes a draft; the architect's own reads still stand between that and work. Wiring it later is not a rewrite. **No 4.5b code was touched.** 5. **`LogCoverage` ships ON, and the anomaly leg claims nothing.** No application logs are shipped to Loki anywhere in `prod/`; the adapter + the pure z-score planner ship, and the live surface is the **named refusal**. The enabling change (a log shipper) is a separate E-adj ticket. ## Not in this PR (stated, not hidden) - no cross-project `SEND` read in v1 (adjudicated); - the JetStream ledger adapter (the dispatcher is not yet an admitted bus client — 4.1/4.5b's named follow-up); the arm uses the `direction.Ledger` interface; - R-11 `intake-daily-sweep` is `paused` on `dep:role-intake, dep:LD4.1-intake-arm`, so signal items land in Plane but are not yet batch-digested — `emits: [R-11]` declares the edge and the registry's cycle check runs on it. ## Reviewer notes `signalintake/plan.go` is **sha256-pinned** by R-23's `scriptRef` and `TestInProcessPinsMatchTheCode` re-derives it — any change to the planner must re-pin.

aaron added 4 commits 2026-10-03 05:01:46 +00:00

HOMELAB-1728: feat(signals): production-signal collector core (SLO as data, burn in Go, citation rule) ... 16cb285d60

LD4.7's pure half: the signal schema, the gauge burn arithmetic over
probe_success, the priority ladder, the planner's ErrUncitedSignal seam, the
digest hash and CitesDigest.

- burn.go owns avg_over_time SEMANTICS as code (a 0/1 gauge is a MEAN, not a
  rate()), half-open windows, ErrNoSamples for an empty window (a coverage gap
  must never read as a quiet window), and budget_consumed_ratio.
- priority.go: Propose takes the citation as a PARAMETER and refuses one that is
  not measured, so there is no code path that yields a priority without
  evidence. Retiring surfaces are forced to P4 with the retire ticket as the
  citation; an edge-observed fault caps at P3.
- plan.go reuses 4.1's WindowKey/Plan shape and refuses the whole window on an
  unmeasured group (batch-wide, the 4.1 ruling).
- digest.go renders the item body with signal_digest_hash=..., cites.go decides
  whether a draft quotes a hash the ledger actually holds (Verify line 3).
- direction.Entry gains ONE additive omitempty DigestHash field (the transition
  4.1's KindDigest comment reserves for 4.7; the 4.8 Run.Target pattern).
- Policy as data: lab-director/signals/{slo-policy,priority-ladder}.yaml
  (target 0.99, retire surfaces flagged with SEND-546, fixture SLO flagged).

HOMELAB-1728: feat(signals): the arm — Prom/Loki readers, the Plane item writer, the pass loop ... b7eba3ef4f

The I/O half: one range query per surface, the burn in Go, one direction item per
group above P4 into 4.1's direction sink, and the window row carrying the digest
hash.

- prom.go: the live Prometheus client (matrix-only; a vector/scalar answer to a
  range query is the wrong query and is refused), ProbeExpr so the citation's ref
  IS the query that ran, and an EMPTY result returned empty (ComputeBurn refuses
  it — a coverage gap is never a healthy window).
- logsource.go: the Loki coverage probe + the pure z-score anomaly planner, and
  ErrNoLogCoverage BY NAME when the only streams are the canary. No live
  application logs are shipped: the leg claims nothing about anomalies.
- plane.go: the sole writer for the application-signal class, idempotent on
  DedupKey by a marker read (the sink is the durable half of the dedup), labels
  and state resolved by name with a named refusal when either is missing.
- server.go: 4.1's arm contract (Now/After seams, sync-then-interval, every
  failure logged and skipped). The window guard runs BEFORE the queries: an
  already-open window issues no query at all. Counters for passes, queries,
  source failures, groups, items, recordals, refusals, no-coverage.
- The log-coverage gap and every refusal are LEDGER ROWS, so the absence of
  evidence is visible in production rather than inferred from a test.

HOMELAB-1728: feat(routines,config): register R-23 and arm the collector ... 34d0fdd157

- routines/registry.yaml: R-23 signals-from-production, ACTIVE, intervalSeconds
  300, go:in-process pinned on signalintake/plan.go (the pure half: the pin is
  worth a sha256 where the arithmetic and the refusals live), consumer
  architect, effects class A reads [prometheus, loki, plane], emitsWork with
  emits: [R-11]. The reserved-ids header is RECONCILED: R-23's event form was
  aspirational prose, no producer emits signal.production, and driving it off
  Alertmanager would route an application signal through the dispatch altitude
  (contract §1 Amendment 2 keeps it at proposal altitude).
- testdata/registry-4x-shapes.yaml: the R-23 row stays as the SCHEMA proof, with
  a comment saying it is superseded in TRIGGER, not in shape (registry_test's
  shape assertion still passes and is untouched).
- config: a signals block (URLs, policy paths, interval, lookback, logCoverage,
  lokiMatch, labels, state) + arms.signals (OFF by default: the arm needs one
  additive CNP egress rule, a dangerous-class change). Defaults and the example
  schema updated; two tests pin them.
- cmd/dispatcher: arms_signals.go builds the arm (Prom/Loki/Plane sink + the
  direction ledger), registered as a NON-FATAL runner so a Prometheus outage is a
  signals outage, not a dispatch outage.

HOMELAB-1728: fix(signals): align the fixture SLI with the shipped Probe, flat policy mount ... d7a0ba1cd8

Two corrections found while writing the prod half -- both are cases where the code
and the cluster would have disagreed:

- The fixtures Probe uses the EXISTING tcp_connect blackbox module (every
  http_edge_* module asserts TLS and would fail against an in-cluster plain-HTTP
  Service; adding an in-cluster HTTP module would be a write into the protected
  monitoring namespace). So the fixture surface id is the TCP target string
  "sendrast-test-canary.sendrast-test.svc.cluster.local:8080", not an http URL --
  otherwise the collector's query would select a series that does not exist and the
  fixture would be refused as a coverage gap. probe_success and the burn arithmetic
  are identical to production's.
- The policy documents are FLAT ConfigMap keys (/etc/lab-director/slo-policy.yaml,
  /etc/lab-director/priority-ladder.yaml), the identity-registry.yaml / policy.md
  precedent: a ConfigMap cannot carry a directory, so a signals/ subdirectory would
  have been a mount change rather than a key. Defaults, the example schema and the
  deployment's config all agree now.

Verified end to end against the DEPLOYMENT's own document (a throwaway check, not
committed): the prod ConfigMap's config.yaml loads under the strict decoder with
arms.signals on, and its embedded slo-policy.yaml + priority-ladder.yaml load
through this package's loaders including the ladder-vs-SLO band check.

aaron added 1 commit 2026-10-03 05:15:22 +00:00

HOMELAB-1728: fix(signals): move the policy documents inside the Go module (CI mounts only dispatcher/) ... 5d56312b2d

CI run #1806 failed on dispatcher-test and the failure was real: the job mounts
ONLY lab-director/dispatcher into the golang container (`-v $PWD/lab-director/dispatcher:/src`),
so `../../signals/slo-policy.yaml` resolved to a path outside the mount and the
policy tests could not read the shipped documents. Reproduced locally by copying
just the dispatcher tree to an empty directory (the same shape as the mount):
the file is absent, the tests fail.

So the policy lives at lab-director/dispatcher/signals/{slo-policy,priority-ladder}.yaml
plus its README. Nothing about the design changes -- policy as data, read by R-23,
validated by the loader and pinned by tests -- but "adjacent to the dispatcher"
now means INSIDE the module, which is the only place the CI that must prove it
parses can see. A test that could not run in CI would have been a test that does
not exist.

Also aligns signalintake's own default policy paths with the deployment's flat
ConfigMap mount (/etc/lab-director/*.yaml), so the arm's defaults and the config
package's defaults are the same two paths.

aaron referenced this pull request 2026-10-03 05:18:57 +00:00

HOMELAB-1731: LD4.11 · maintainability fitness routine (weekly, Sendrast first) #598

aaron commented 2026-10-03 05:21:57 +00:00

Author

Owner

Copy link

CI red → green, plus one path correction to the tree map in the description.

Run #1806 (dispatcher-test) failed and the failure was real, not flake: the job mounts only lab-director/dispatcher into the golang container (-v $PWD/lab-director/dispatcher:/src), so ../../signals/slo-policy.yaml resolved outside the mount and the policy tests could not read the shipped documents. I reproduced it locally by copying just the dispatcher tree into an empty directory — the same shape as the mount — which is the check local go test cannot give you.

Fix (commit 5d56312): the policy documents now live at lab-director/dispatcher/signals/{slo-policy,priority-ladder}.yaml (+ their README) — i.e. lab-director/signals/ in the description becomes lab-director/dispatcher/signals/. Nothing about the design changes (policy as data, read by R-23, validated by the loader, pinned by tests); "adjacent to the dispatcher" now means inside the module, which is the only place the CI that must prove it parses can see it. A test that cannot run in CI is a test that does not exist.

signalintake's own default policy paths are also aligned with the deployment's flat ConfigMap mount (/etc/lab-director/*.yaml), so the package defaults and the config package defaults are the same two paths.

Run #1809 at head 5d56312b2d: dispatcher-test success, all four jobs green.

**CI red → green, plus one path correction to the tree map in the description.** Run #1806 (`dispatcher-test`) failed and the failure was real, not flake: the job mounts **only `lab-director/dispatcher`** into the golang container (`-v $PWD/lab-director/dispatcher:/src`), so `../../signals/slo-policy.yaml` resolved outside the mount and the policy tests could not read the shipped documents. I reproduced it locally by copying just the dispatcher tree into an empty directory — the same shape as the mount — which is the check local `go test` cannot give you. Fix (commit `5d56312`): the policy documents now live at **`lab-director/dispatcher/signals/{slo-policy,priority-ladder}.yaml`** (+ their README) — i.e. `lab-director/signals/` in the description becomes `lab-director/dispatcher/signals/`. Nothing about the design changes (policy as data, read by R-23, validated by the loader, pinned by tests); "adjacent to the dispatcher" now means *inside the module*, which is the only place the CI that must prove it parses can see it. A test that cannot run in CI is a test that does not exist. `signalintake`'s own default policy paths are also aligned with the deployment's flat ConfigMap mount (`/etc/lab-director/*.yaml`), so the package defaults and the `config` package defaults are the same two paths. Run #1809 at head `5d56312b2d`: **dispatcher-test success**, all four jobs green.

aaron added 1 commit 2026-10-03 05:51:13 +00:00

HOMELAB-1728: fix(signals): emit the item's own group and write the marker description 57e3e418f7

aaron referenced this pull request 2026-10-03 05:55:31 +00:00

HOMELAB-1727: LD4.10 — skill &amp; lore health scan (obsolescence probes + sampled observer) #600

aaron commented 2026-10-03 05:56:58 +00:00

Author

Owner

Copy link

Landing review — VERDICT: PASS (2 blockers found, both fixed; head reviewed 57e3e418f7)

Fresh-context review of the diff + ticket HOMELAB-1728 + design-LD4.7-signals.md. CI green on the reviewed head: run #1818, 4/4 jobs (alert-receiver-test, dispatcher-test, helm-validate, receiver-test). go vet + go test -race -count=1 ./... green locally (31 packages).

Verified as claimed (primary sources)

• The citation rule (the heart). Planner.PlanWindow refuses per group before any write, and the refusal is batch-wide: empty evidence, a link with no observed number, and zero samples all return ErrUncitedSignal, so a refused window emits nothing. Propose takes the citation as a parameter (priority.go:229) and ItemFor refuses an empty rendered citation; 4.1's Item.Validate/sink refuses the illegal pair a second time. Asserted by TestAC2SignalWithoutMeasuredEvidenceIsRefused (incl. direction.ErrInvalidItem from the sink, sink.Len()==0). ZERO Plane writes on refusal: confirmed.
• Burn mechanics. TestBurnScaleZeroFullOutage: 120/120 failing → ratio 1.0 → burn 100 at the adjudicated 0.99, with the brief's 0.995 row (200) asserted beside it. TestBurnReplayRealProductionBurn: the real 2026-10-02T01:29Z series (30/120 → 0.25 → 25) crosses the 14.4 rung; I re-derived the rung ordering by hand — the replay's ~2.03d series keeps slowest at ~0.51 (<1.0), so the fast rung is genuinely what fires, and the scale-0 fixture's 3d window makes slowest fire first (P1 either way, per the ladder). Half-open windows and ErrNoSamples on an empty window are asserted.
• CitesDigest true / unknown / none / truncated / second-hash / mixed-markup, nil ledger → error, empty ledger → false, CitesDigestWhich identity; digest hash order-independent and measurement-sensitive.
• R-23 pin: sha256sum signalintake/plan.go = 49da6976… — matches the registry scriptRef exactly, and plan.go is untouched by the fix commit. Entry is active / 300s / go:in-process / consumer: architect / emits: [R-11], and the reserved-ids header is reconciled in the same commit. Entry.DigestHash is additive + omitempty (the LD4.8 Run.Target pattern): no other entry kind sets it.
• The ld:signal-digest deviation is correct. 4.1's FindDigestIssue (intake/plane.go:374) matches its marker plus the window string, and a signal item's external id is <window>#<group_id> — sharing the marker would make the daily sweep believe the digest issue existed and write nothing. The deviation is right; see B2 for why it was not yet effective.
• Arm is non-fatal (buildRunners ⇒ go func, fatal:false), the pass is pure-function dominated with bounded HTTP, and signals:/arms.signals are additive with default OFF. No test contacts a real service.

BLOCKERS found in review — fixed at 57e3e418f7 (fix worker, fresh context; delta re-verified by me)

Both in the emission path, both reproduced with scratch tests (since deleted):

1. Misattribution. SweepWindow indexed plan.Groups[i] against plan.Items[i], but plan.Groups holds every planned group (recorded included) while plan.Items holds only those above P4 — so a recorded group sorting first (the shipped policy forces its two retiring surfaces to P4) made the emitted Plane item's title name another surface. Fixed by a group-id→Group lookup off item.ExternalID, failing loudly on a miss, deliberately without touching the sha256-pinned plan.go.
2. The marker description was never written. PlaneSink.EmitTitled sent item.Body (human HTML), so RenderItemDescription — the function that prepends 4.1's ld:direction marker and appends ld:signal-digest — had no production caller. Consequence: intake.ParseDirectionItem could not read what the sink wrote, so Exists/FindSignalDigest never matched (read-before-write dedup and guard 3 inert → a retry or restart created a duplicate Plane item), and R-11 (the declared emits target) could not read the items at all. Fixed via an EmitDescribed seam; item.Body stays the body inside the payload. The AC tests passed only because direction.FixtureSink dedups in memory — the new tests drive the real PlaneSink over httptest and assert the description parses with intake.ParseDirectionItem, carries MarkerSignalDigest, and sends 0 further POSTs on re-emit.

Concerns (do not block; recommend follow-up)

• §4.4 proxy legs are not wired, and were not in the PR's deviation list. The design calls this the single most likely path to 4.7 becoming a noise generator, and the shipped policy declares every surface app-endpoint, so edgePathCap is inert for the real surfaces. The ticket's three Verify lines are met, so this is a scope reduction that needs a stated deviation + a follow-up ticket, not a silent one.
• Only the long window is consulted (Propose reads LongBurn); ShortBurn/ShortRatio/Band.Threshold are measured but dead, so burn.go's "both windows must agree" prose is not what the code does.
• The citation window is always the fast band's (now.Add(-Bands[0].Long)), even when the slowest rung fired — a P1 from the 3d rung can carry a citation whose numbers do not contain the burn that fired.
• The fixture SLO ships in the production policy and fixture: true is validated but never rendered, so a fixture burn would be presented as an objective (the yaml comment claims otherwise).
• PlaneSink.Bind has no caller (sink HTTP uses context.Background(), so no pass cancellation); warnOnce/Warned only set/read a bool and are never logged (the lost read-before-write guard is silent); ObservationSpan is the fetched-series span capped at the ladder lookback rather than the SLI's history; CitesDigest→LedgerDigestHashes inherits List()'s 500-row oldest-first cap (a false negative once the process ledger exceeds 500 rows — the deterministic leg of Verify line 3, currently only called from tests).
• Package doc "not one test opens a real socket" is literally false (httptest.NewServer binds localhost); the README states it accurately.

Merging on green CI with no open blocker findings.

## Landing review — VERDICT: PASS (2 blockers found, both fixed; head reviewed `57e3e418f7`) Fresh-context review of the diff + ticket HOMELAB-1728 + `design-LD4.7-signals.md`. **CI green on the reviewed head: run #1818, 4/4 jobs** (`alert-receiver-test`, `dispatcher-test`, `helm-validate`, `receiver-test`). `go vet` + `go test -race -count=1 ./...` green locally (31 packages). ### Verified as claimed (primary sources) - **The citation rule (the heart).** `Planner.PlanWindow` refuses per group *before* any write, and the refusal is batch-wide: empty evidence, a link with no observed number, and zero samples all return `ErrUncitedSignal`, so a refused window emits nothing. `Propose` takes the citation as a parameter (`priority.go:229`) and `ItemFor` refuses an empty rendered citation; 4.1's `Item.Validate`/sink refuses the illegal pair a second time. Asserted by `TestAC2SignalWithoutMeasuredEvidenceIsRefused` (incl. `direction.ErrInvalidItem` from the sink, `sink.Len()==0`). **ZERO Plane writes on refusal: confirmed.** - **Burn mechanics.** `TestBurnScaleZeroFullOutage`: 120/120 failing → ratio 1.0 → **burn 100** at the adjudicated 0.99, with the brief's 0.995 row (200) asserted beside it. `TestBurnReplayRealProductionBurn`: the real 2026-10-02T01:29Z series (30/120 → 0.25 → 25) crosses the 14.4 rung; I re-derived the rung ordering by hand — the replay's ~2.03d series keeps `slowest` at ~0.51 (<1.0), so the **fast** rung is genuinely what fires, and the scale-0 fixture's 3d window makes `slowest` fire first (P1 either way, per the ladder). Half-open windows and `ErrNoSamples` on an empty window are asserted. - **CitesDigest** true / unknown / none / truncated / second-hash / mixed-markup, nil ledger → error, empty ledger → false, `CitesDigestWhich` identity; digest hash order-independent and measurement-sensitive. - **R-23 pin**: `sha256sum signalintake/plan.go` = `49da6976…` — matches the registry `scriptRef` exactly, and `plan.go` is untouched by the fix commit. Entry is active / 300s / `go:in-process` / `consumer: architect` / `emits: [R-11]`, and the reserved-ids header is reconciled in the same commit. `Entry.DigestHash` is additive + `omitempty` (the LD4.8 `Run.Target` pattern): no other entry kind sets it. - **The `ld:signal-digest` deviation is correct.** 4.1's `FindDigestIssue` (`intake/plane.go:374`) matches its marker **plus the window string**, and a signal item's external id is `<window>#<group_id>` — sharing the marker would make the daily sweep believe the digest issue existed and write nothing. The deviation is right; see B2 for why it was not yet effective. - **Arm is non-fatal** (`buildRunners` ⇒ `go func`, `fatal:false`), the pass is pure-function dominated with bounded HTTP, and `signals:`/`arms.signals` are additive with **default OFF**. No test contacts a real service. ### BLOCKERS found in review — fixed at `57e3e418f7` (fix worker, fresh context; delta re-verified by me) Both in the emission path, both reproduced with scratch tests (since deleted): 1. **Misattribution.** `SweepWindow` indexed `plan.Groups[i]` against `plan.Items[i]`, but `plan.Groups` holds **every** planned group (recorded included) while `plan.Items` holds only those above P4 — so a recorded group sorting first (the shipped policy forces its two `retiring` surfaces to P4) made the emitted Plane item's **title** name another surface. Fixed by a group-id→`Group` lookup off `item.ExternalID`, failing loudly on a miss, deliberately **without touching the sha256-pinned `plan.go`**. 2. **The marker description was never written.** `PlaneSink.EmitTitled` sent `item.Body` (human HTML), so `RenderItemDescription` — the function that prepends 4.1's `ld:direction` marker and appends `ld:signal-digest` — had **no production caller**. Consequence: `intake.ParseDirectionItem` could not read what the sink wrote, so `Exists`/`FindSignalDigest` never matched (read-before-write dedup and guard 3 inert → a retry or restart created a **duplicate** Plane item), and R-11 (the declared `emits` target) could not read the items at all. Fixed via an `EmitDescribed` seam; `item.Body` stays the body inside the payload. The AC tests passed only because `direction.FixtureSink` dedups in memory — the new tests drive the **real** `PlaneSink` over `httptest` and assert the description parses with `intake.ParseDirectionItem`, carries `MarkerSignalDigest`, and sends 0 further POSTs on re-emit. ### Concerns (do not block; recommend follow-up) - **§4.4 proxy legs are not wired, and were not in the PR's deviation list.** The design calls this the single most likely path to 4.7 becoming a noise generator, and the shipped policy declares every surface `app-endpoint`, so `edgePathCap` is inert for the real surfaces. The ticket's three Verify lines are met, so this is a scope reduction that needs a stated deviation + a follow-up ticket, not a silent one. - Only the **long** window is consulted (`Propose` reads `LongBurn`); `ShortBurn`/`ShortRatio`/`Band.Threshold` are measured but dead, so `burn.go`'s "both windows must agree" prose is not what the code does. - The citation window is always the **fast** band's (`now.Add(-Bands[0].Long)`), even when the `slowest` rung fired — a P1 from the 3d rung can carry a citation whose numbers do not contain the burn that fired. - The **fixture SLO ships in the production policy** and `fixture: true` is validated but never rendered, so a fixture burn would be presented as an objective (the yaml comment claims otherwise). - `PlaneSink.Bind` has no caller (sink HTTP uses `context.Background()`, so no pass cancellation); `warnOnce`/`Warned` only set/read a bool and are never logged (the lost read-before-write guard is silent); `ObservationSpan` is the fetched-series span capped at the ladder lookback rather than the SLI's history; `CitesDigest`→`LedgerDigestHashes` inherits `List()`'s 500-row **oldest-first** cap (a false negative once the process ledger exceeds 500 rows — the deterministic leg of Verify line 3, currently only called from tests). - Package doc "not one test opens a real socket" is literally false (`httptest.NewServer` binds localhost); the README states it accurately. Merging on green CI with no open blocker findings.

aaron force-pushed plane/HOMELAB-1728-signals from 57e3e418f7 to b8a7234d24 2026-10-03 05:58:09 +00:00 Compare

aaron merged commit a3d3b5a049 into epic/LD4 2026-10-03 05:58:12 +00:00

aaron referenced this pull request from a commit 2026-10-03 05:58:12 +00:00

Merge pull request 'HOMELAB-1728: LD4.7 — signals-from-production intake (collector arm, SLO as data, burn in Go)' (#599) from plane/HOMELAB-1728-signals into epic/LD4

aaron referenced this pull request 2026-10-03 18:17:54 +00:00

LD4 (HOMELAB-1713): The Loops — intake, El Director, routines, readiness, plan-freshness, meetings, alert audit → live #591

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

operator-review

Requires human operator review before merge — dangerous class (RBAC, secrets, network policy, Terraform) or explicitly gated. Never auto-merge.

No labels

operator-review

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Homelab/infra-core!599

No description provided.