HOMELAB-2634: feat(herdr): add session janitor #87
No reviewers
ld-reviewer
Labels
No labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Homelab/pi-extensions!87
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "plane/HOMELAB-2634-herdr-janitor"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What
Adds a conservative, dry-run-by-default herdr session janitor, a 15-minute systemd
--userservice/timer, canonical supervisor/overseer skill snapshots using retire markers, focused stdlib tests, and a Python CI job.Safety boundary
--applyand an exactHOMELAB-2634activation file, plus fresh revalidation immediately beforeherdr pane close.ld-hypervisor, working/blocked old panes, non-allowlisted workspaces, and alwayswS/wHare report-only.HANDOFF READY: /<path>/ approved-successor evidence, or an explicit retire marker. Retire markers are deleted only after a successful close.Verification
python3 -m unittest discover -s tests -v: 26 tests OK in this repo (safe handoff, missing evidence, live-work forms, age/blocked successor, retirement lifecycle, protections, idle-with-workers, dry-run non-mutation, activation and close failure).python3 -m py_compile scripts/herdr_janitor/janitor.py tests/test_herdr_janitor.py: pass.systemd-analyze --user verify deploy/herdr-janitor.service deploy/herdr-janitor.timer: pass.seen.json, marker deletion, unit installation, or activation occurred. State directory contained onlyreport.txtandjanitor.log.Worker evidence:
/home/aaron/repos/homelab/working-files/homelab-2518/HOMELAB-2634-worker.mdandHOMELAB-2634-dry-run.txt.Fresh general + premium exact-head reviews are required. Do not merge, activate, deploy, or close HOMELAB-2634 as part of implementation handoff.
Verdict: APPROVED
model: omniroute/kimi-pool
head:
f5bec606bcGENERAL-seat exact-head review of Homelab/pi-extensions PR #87 for HOMELAB-2634, audited against the ticket safety contract with no exceptions found.
Dry-run is the default and the only mode reachable without both --apply and an exact HOMELAB-2634 activation file, and activation refusal happens before any herdr call.
Successor closes require at least 600 seconds of continuous same pane-ID plus name observation with no fallback when the highest successor is blocked or of unknown status, and first sight always fails closed.
No close proceeds with live agents, workers, or shells in the recent footer, and every close requires HANDOFF READY with a real path, approved-successor evidence, or an explicit retire marker, and each close is revalidated from fresh herdr state immediately before herdr pane close.
Unnamed, duplicate, ld-hypervisor, working, blocked, non-allowlisted, and never-workspace panes are report-only, wS and wH cannot be allowlisted, idle-with-workers refuses, and retire markers are deleted only after a successful close with failed closes retaining the marker.
No timer or service was installed or activated by this change and my independent dry-run wrote only report.txt and janitor.log while closing nothing and changing no pane IDs, with all 17 worker dry-run pane IDs confirmed still live by a read-only probe.
CI run #212 passed all 14 jobs and run #213 passed python on this exact head, and I re-ran 26 of 26 unit tests plus py_compile, systemd-analyze verify, and git diff --check at the pinned head with all green.
No blocker open.
Non-blocking nits only: the service Documentation field still references the archived Homelab/hermes-automation repository, and the 600-second successor floor starts accumulating only after activation, which is the documented fail-closed behavior.
Full findings: /home/aaron/repos/homelab/working-files/homelab-2518/HOMELAB-2634-review-general.md
Verdict: REQUEST_CHANGES
model: omniroute/k3.1-pro
head:
f5bec606bctier: premium
Audited the janitor implementation, ticket safety contract, worker evidence, and exact-head CI. Independently confirmed CI #212 with 14 green jobs and #213 with the green Python job. Locally ran 26/26 unit tests successfully, plus py_compile, systemd-analyze --user verify, and diff-check. No timer or service was installed or activated, and no real pane mutation was performed.
Blocker — K1: Successor age accepts non-finite or boolean persisted timestamps
BLOCKER OPEN: K1
load_seenaccepts booleans and non-finite floats because bool is an int subclass and Python JSON accepts NaN.successor_agedoes not reject either, andage < 600is false for NaN. Read-only in-memory probes withfirst_seen=float('nan')andfirst_seen=Trueboth returnedCLOSE_CANDIDATEwithsafe-successor-handoff, so the required ten-minute continuous observation is not proven on malformed persisted state. Reject booleans and non-finite values when loading and at the decision boundary, with regression tests.Blocker — K2: Whitespace-only status output authorizes marker retirement
BLOCKER OPEN: K2
Both close branches use
if not textfor status-read failure. Whitespace-only output is truthy and contains no positive live-work match, so a retire marker alone producedCLOSE_CANDIDATEfor" \n\t\n"in a read-only in-memory probe. This is equivalent to the empty output already refused and does not provide a usable status/footer proving no live work. Treatnot text.strip()as a failed read and add successor and retire-marker regression tests.Round-2 implementation head:
7a01bea0aa61c4b665322d14e7fc5ade9ba6d011.Blocker K1 resolved —
first_seennow rejects booleans and non-finite numeric values at both persisted-state load and the successor-age decision boundary; valid integer/float 600-second boundary controls pass.Blocker K2 resolved — empty and whitespace-only agent-read output now produces
status-read-failedin both successor and retire-marker branches; read-error and valid no-live-footer controls pass.RED FIRST captured 31 tests with 7 failing subtests before implementation. GREEN: 31/31 focused and full janitor tests pass, including all original 26 tests; py_compile, systemd user-unit verify, and diff-check pass. Fresh real dry-run: 16 REPORT / 0 WOULD_CLOSE, with all 16 pane IDs unchanged and no seen-state, marker, unit, activation, or pane mutation.
CI on the new head: main #214 and Python #215 are currently running. Prior verdicts are invalidated by this head; a fresh general + premium exact-head panel is required. No self-review, merge, deploy, activation, or Done transition performed.
7a01bea0aa175e6dc64bCI recovery only — no product change and no source edit.
Main CI #214 failed only
pi-swarm-suite:task-store-wiring > store failure is fail-open(full test:task store wiring > store failure is fail-open: spawn still works when storeDir is unwritable). The exact spec passed independently on old head7a01bea0aa61c4b665322d14e7fc5ade9ba6d011and parent/basef5bec606bc3f40b7db8e1ba6a71819e306d59296(1 passed, 23 skipped each). The PR changes nopi-swarm-suitefiles; its executable/source changes outside that suite are Python-only. Classification: CI flake, not a product fix.Retriggered via a NO-TREE-CHANGE amend and force-with-lease push:
7a01bea0aa61c4b665322d14e7fc5ade9ba6d011175e6dc64b25f27275766eb492f109ace6e032bbab454ae5132a538caf0412af70bfd48421436b58ab454ae5132a538caf0412af70bfd48421436b58Recovery CI on
175e6dc64b25f27275766eb492f109ace6e032bbis green: main CI #216 passed (14/14 jobs, includingpi-swarm-suite), and Python CI #217 passed. This remains a no-tree-change CI retrigger, not a product change.Verdict: APPROVED
model: omniroute/kimi-pool
head:
175e6dc64bGENERAL-seat exact-head final review of the round-2 delta (two files, +100/-4 in janitor.py and tests). Head re-fetched from origin immediately before verdict and tree-identical to
7a01bea0aa(tree objects equal), so the round-2 evidence carries to this head and was independently re-verified rather than trusted.K1 (fail-closed malformed first_seen) is fixed at both boundaries: load_seen drops persisted rows whose first_seen is bool, non-numeric, or non-finite, and successor_age refuses bool, NaN, and plus or minus Infinity at the decision boundary, alongside the existing future-timestamp check. Independent probes: a seen.json containing JSON NaN, Infinity, -Infinity, booleans, a string, and None keeps only the valid-int row; every malformed value returns unproven age; 599 seconds is too new, exactly 600 seconds is eligible, a future timestamp is unproven, and a name-matched row mismatch is unproven.
K2 (whitespace-only status output) is fixed in both close branches: the successor-handoff branch and the retire-marker branch both refuse None, empty, spaces-only, tabs-only, and NBSP-only reads with status-read-failed. Valid controls still close: a HANDOFF READY line with a zero-live footer and a proven 600-second successor is a close candidate, and a retire marker with a done and no-workers status is a close candidate. A live footer such as agents still running still refuses.
Protections hold under independent probes: HANDOFF READY or approved-successor evidence is required, live-work footers refuse, wS and wH are never-workspace, ld-hypervisor is a protected name, unknown workspaces are report-only, the blocked highest-numbered successor never falls back to an older generation, retire markers are report-only while their pane shows live work, and dry-run never mutates panes, seen state, or markers. Worker dry-run evidence audited independently: sixteen REPORT and zero WOULD_CLOSE, identical pane sets before and after, state directory holding only report.txt and janitor.log, no seen.json, no retire directory, and no janitor units installed or activated.
Main CI #216 is fourteen of fourteen green by name at this exact head and Python CI #217 is green; the prior pi-swarm-suite failure in #214 was the known pre-existing task-store flake and did not recur.
Follow-up hardening only, not a merge gate: a negative finite first_seen from hand-corrupted state would compute a huge age and pass the 600-second floor; seen.json is written only by save_seen with float(time.time()), so this is corrupt-state-only and unexploitable in practice.
No blocker open.
Verdict: APPROVED
model: omniroute/k3.1-pro
head:
175e6dc64btier: premium
Independent premium exact-head review of Homelab/pi-extensions PR #87 for HOMELAB-2634. The head was re-fetched before the verdict and is tree-identical to
7a01bea0aa. Main CI #216 is 14/14 green and Python CI #217 is green.K1 is resolved at both persisted-load and decision boundaries. Independent probes confirmed that booleans, NaN, positive/negative infinity, strings, null, arrays, and objects fail closed, while valid integer/float 600-second boundary controls remain eligible and future timestamps refuse.
K2 is resolved in both successor and retire-marker branches. Independent probes confirmed None, empty, spaces/tabs/newlines, and NBSP-only status reads produce status-read-failed, while valid zero-live-footer controls remain eligible.
HANDOFF grammar, approved-successor evidence, live agent/worker/shell detection, protected names, old working/blocked statuses, workspace allowlisting, and permanent wS/wH protections all passed independent spot-checks. Dry-run non-mutation passed without a close call or marker/seen-state changes.
The exact-head focused suite passed 31/31, and py_compile, systemd-analyze --user verify, and full-PR diff-check passed. A fresh real dry-run produced 15 REPORT and 0 WOULD_CLOSE with all 15 pane IDs unchanged and no seen.json or retire-directory writes. No activation, pane mutation, unit installation, service restart, merge, or Done transition was performed.
No blocker open.
Full findings: /home/aaron/repos/homelab/working-files/homelab-2518/HOMELAB-2634-review-kimi-final.md