HOMELAB-1741: feat(pi-swarm-suite): challenge-before-reap watchdog #46

Merged

aaron merged 1 commit from plane/HOMELAB-1741-watchdog-challenge into main 2026-09-29 07:09:03 +00:00

Conversation 2 Commits 1 Files changed 13 +1072 -206

aaron commented 2026-09-29 06:48:33 +00:00

Owner

Copy link

What

Redesigns the LD0.5 stall watchdog (HOMELAB-1668, v1) from silence-only reap to challenge-before-reap, per the operator's ruling:

"watchdogs should be able to ping the agent; if it doesn't answer in 10 min post-check, then kill it."

v1 reaped on silence alone (status: running + run in flight + no activity-bus event for thresholdMinutes), which misfires on a healthy agent parked in a long, silent blocking wait (a get_subagent_result(wait) past the threshold, or a single long silent tool call). Silence is a suspicion, not a verdict.

phase 1  silence > thresholdMinutes  ->  inject a liveness CHALLENGE, open grace (graceMinutes, default 10)
grace    ANY activity-bus event from the agent  ->  cancel the reap (the agent answered by doing its job)
phase 2  still silent at grace end   ->  reap via the UNCHANGED v1 path
         (manager.abort + post-abort verification + escalation + durable entry + loud notice)

graceMinutes: 0 restores the v1 single-phase behaviour verbatim. thresholdMinutes: 0 / enabled: false unchanged. Idle-but-live children (HOMELAB-1577: running + no run in flight) are still never challenged or reaped.

Challenge mechanism — chosen: a pi steer

challenge: (id) => manager.steer(id, STALL_WATCHDOG_CHALLENGE_TEXT).

Why, over the alternatives the ticket listed:

1. It is a real probe, not a synthetic event. The text enters the agent's own input queue (session.steer()); only a live run can drain it and emit activity. HOMELAB-1632 proved the exact differentiator — a steer queued behind a stalled provider request is never processed. A synthetic activity-bus ping would only re-arm the watchdog's own clock and prove nothing about liveness (rejected). A "marker the next event clears" is just the grace window; it needs the same real event source, which the steer provides.
2. No AgentManager semantics change. steer() is the existing public verb behind swarm-attach steer. For a record whose status does not change it publishes no activity event (emitAnyChange only publishes status on an actual status change), so the challenge cannot accidentally answer itself on the real wiring. A dedicated sendCustomMessage channel would have required a new AgentManager method; not needed.
3. The probe text is explicit and self-describing, so a healthy agent that eventually reads it knows it is a liveness check and continues its task. It has no leading /, so pi's skill/template expansion is a no-op.

The BLOCKER-1 class, handled (regression test first)

The challenge's own synchronous bus events must not count as the answer — the same class as the HOMELAB-1668 BLOCKER-1 (abort's status event cleared the reap marker). The grace marker is set before the injection, and a challenging set makes observe() ignore every event the manager publishes synchronously from inside challenge().

The guard is load-bearing, and the ticket's test-first requirement is met:

• 12 new tests in tests/core/stall-watchdog-challenge.test.ts fail on the v1 code (12/12 failed, recorded).
• The dedicated guard test fails when the guard is removed (naive wiring: the challenge's own status event cancels grace → the agent is re-challenged instead of reaped, expected [] to deeply equal ["a1"]).

Notifications name the phase

• Phase 1: STALL SUSPECTED — liveness challenge sent (phase 1 of 2) + Challenged: via steer at <ISO> — grace until <ISO> (10m).
• Phase 2: STALL CONFIRMED — challenged, no answer by grace end — agent reaped automatically (phase 2 of 2). A later reap can no longer read as the first silent detection. The stall-escalated notice carries the challenge context too.

Config + surface

stallWatchdog: { enabled, thresholdMinutes, graceMinutes } — same per-field global/project merge, sanitize, decision-time read, and clamping. /agents → Settings gains Stall grace (min) (0 = legacy single-phase). README, docs/architecture.md, docs/attach-protocol.md, src/suite-settings.ts updated.

VERIFY — E2E re-derived (real pi --mode rpc, scratch PI_CODING_AGENT_DIR, worktree --extension, thresholdMinutes: 1, graceMinutes: 1)

Harness /tmp/watchdog-verify-1741/ (scratch; blackhole SSE server + scripted root that spawns a background Agent; nothing touched by hand after the spawn).

Scenario	Timeline	Result
Wedge A blackholed provider keep-alive stream (child-keepalive)	running to t+178s, stopped/wedged=true at t+188s	PASS — challenged at silence 1m59s, reaped at grace end
Wedge B long silent bash sleep 900 (child-bash900)	running to t+178s, stopped/wedged=true at t+188s	PASS — challenged at silence 1m56s (lastActivity: tool_activity(start, bash)), reaped at grace end
False-positive shape bash sleep 130 (silent past threshold, resolves within grace)	challenged at t+122s, completed/wedged=false at t+138s	PASS — activity within grace cancelled the reap

Read back from the scratch session file, Wedge A:

{"customType":"subagents:stall","data":{"kind":"stall-challenged","agentId":"c73ff4bc-…","silenceMs":119086,
 "thresholdMinutes":1,"lastActivity":"session (1m 59s ago)","graceMinutes":1,"graceUntil":…,"via":"steer","delivered":true}}
{"customType":"subagents:stall","data":{"kind":"stall-reaped","agentId":"c73ff4bc-…","silenceMs":179086,
 "lastActivity":"session (2m 59s ago)","action":"abort() issued — record is now stopped","statusAfter":"stopped",
 "challenge":{"at":…,"graceUntil":…,"graceMinutes":1,"via":"steer","delivered":true}}}

"A wedged agent cannot respond to the challenge" is proven directly: in both wedge scenarios the challenge was delivered (delivered: true through the real manager.steer()), and the agent still produced zero activity and was reaped at grace end. If the challenge had counted as activity, the grace would have been cancelled and the reap would never have fired.

Repo gates

• npm run typecheck clean.
• npx vitest run: 1300 tests / 99 files passed (base origin/main = 1287 / 98 → +13 tests, +1 file).
• Main checkout ~/.repos/pi-extensions untouched: git status --porcelain → only pre-existing untracked .rpiv/ and .worktrees/; all changes are on this branch. Staged explicit paths only (node_modules symlink never staged).

Honest gaps / not verified

• Idle-but-live never-challenged is covered by unit + real-AgentManager tests (isStreaming: false on a real record → no challenge, no reap), not end-to-end: the harness's background child terminalizes on completion, and reproducing the HOMELAB-1577 wake shape E2E needs a post-settle steer.
• The challenge is a steer, so on a healthy agent that has gone quiet the probe text eventually enters its conversation. It is written to be ignorable; the alternative (a custom no-LLM channel) needs a new AgentManager method and was judged unnecessary.
• graceMinutes adds up to threshold + grace of extra latency before an actually-wedged agent is reaped. That is the intended trade the operator asked for; graceMinutes: 0 opts out.
• The escalation path (abort refuses / run survives the abort) remains unit-tested only, as in v1.

Not merged — Class C; a landing worker owns review + merge + rollout.

Closes HOMELAB-1741.

## What Redesigns the LD0.5 stall watchdog (HOMELAB-1668, v1) from **silence-only reap** to **challenge-before-reap**, per the operator's ruling: > "watchdogs should be able to ping the agent; if it doesn't answer in 10 min post-check, then kill it." v1 reaped on silence alone (`status: running` + run in flight + no activity-bus event for `thresholdMinutes`), which misfires on a healthy agent parked in a long, silent blocking wait (a `get_subagent_result(wait)` past the threshold, or a single long silent tool call). **Silence is a suspicion, not a verdict.** ``` phase 1 silence > thresholdMinutes -> inject a liveness CHALLENGE, open grace (graceMinutes, default 10) grace ANY activity-bus event from the agent -> cancel the reap (the agent answered by doing its job) phase 2 still silent at grace end -> reap via the UNCHANGED v1 path (manager.abort + post-abort verification + escalation + durable entry + loud notice) ``` `graceMinutes: 0` restores the v1 single-phase behaviour verbatim. `thresholdMinutes: 0` / `enabled: false` unchanged. Idle-but-live children (HOMELAB-1577: `running` + no run in flight) are still never challenged or reaped. ## Challenge mechanism — chosen: a pi **steer** `challenge: (id) => manager.steer(id, STALL_WATCHDOG_CHALLENGE_TEXT)`. Why, over the alternatives the ticket listed: 1. **It is a real probe, not a synthetic event.** The text enters the agent's own input queue (`session.steer()`); only a live run can drain it and emit activity. HOMELAB-1632 proved the exact differentiator — a steer queued behind a stalled provider request is never processed. A synthetic activity-bus ping would only re-arm the watchdog's own clock and prove nothing about liveness (rejected). A "marker the next event clears" is just the grace window; it needs the same real event source, which the steer provides. 2. **No AgentManager semantics change.** `steer()` is the existing public verb behind `swarm-attach steer`. For a record whose status does not change it publishes **no** activity event (`emitAnyChange` only publishes `status` on an actual status change), so the challenge cannot accidentally answer itself on the real wiring. A dedicated `sendCustomMessage` channel would have required a new AgentManager method; not needed. 3. The probe text is explicit and self-describing, so a healthy agent that eventually reads it knows it is a liveness check and continues its task. It has no leading `/`, so pi's skill/template expansion is a no-op. ## The BLOCKER-1 class, handled (regression test first) The challenge's **own** synchronous bus events must not count as the answer — the same class as the HOMELAB-1668 BLOCKER-1 (abort's `status` event cleared the reap marker). The grace marker is set **before** the injection, and a `challenging` set makes `observe()` ignore every event the manager publishes synchronously from inside `challenge()`. The guard is load-bearing, and the ticket's test-first requirement is met: - 12 new tests in `tests/core/stall-watchdog-challenge.test.ts` **fail on the v1 code** (12/12 failed, recorded). - The dedicated guard test **fails when the guard is removed** (naive wiring: the challenge's own `status` event cancels grace → the agent is re-challenged instead of reaped, `expected [] to deeply equal ["a1"]`). ## Notifications name the phase - Phase 1: `STALL SUSPECTED — liveness challenge sent (phase 1 of 2)` + `Challenged: via steer at <ISO> — grace until <ISO> (10m)`. - Phase 2: `STALL CONFIRMED — challenged, no answer by grace end — agent reaped automatically (phase 2 of 2)`. A later reap can no longer read as the first silent detection. The `stall-escalated` notice carries the challenge context too. ## Config + surface `stallWatchdog: { enabled, thresholdMinutes, graceMinutes }` — same per-field global/project merge, sanitize, decision-time read, and clamping. `/agents → Settings` gains **Stall grace (min)** (0 = legacy single-phase). README, `docs/architecture.md`, `docs/attach-protocol.md`, `src/suite-settings.ts` updated. ## VERIFY — E2E re-derived (real `pi --mode rpc`, scratch `PI_CODING_AGENT_DIR`, worktree `--extension`, `thresholdMinutes: 1`, `graceMinutes: 1`) Harness `/tmp/watchdog-verify-1741/` (scratch; blackhole SSE server + scripted root that spawns a background `Agent`; nothing touched by hand after the spawn). | Scenario | Timeline | Result | |---|---|---| | **Wedge A** blackholed provider keep-alive stream (`child-keepalive`) | running to t+178s, `stopped`/`wedged=true` at t+188s | **PASS** — challenged at silence 1m59s, reaped at grace end | | **Wedge B** long silent `bash sleep 900` (`child-bash900`) | running to t+178s, `stopped`/`wedged=true` at t+188s | **PASS** — challenged at silence 1m56s (`lastActivity: tool_activity(start, bash)`), reaped at grace end | | **False-positive shape** `bash sleep 130` (silent past threshold, resolves within grace) | challenged at t+122s, `completed`/`wedged=false` at t+138s | **PASS** — activity within grace cancelled the reap | Read back from the scratch session file, Wedge A: ```json {"customType":"subagents:stall","data":{"kind":"stall-challenged","agentId":"c73ff4bc-…","silenceMs":119086, "thresholdMinutes":1,"lastActivity":"session (1m 59s ago)","graceMinutes":1,"graceUntil":…,"via":"steer","delivered":true}} {"customType":"subagents:stall","data":{"kind":"stall-reaped","agentId":"c73ff4bc-…","silenceMs":179086, "lastActivity":"session (2m 59s ago)","action":"abort() issued — record is now stopped","statusAfter":"stopped", "challenge":{"at":…,"graceUntil":…,"graceMinutes":1,"via":"steer","delivered":true}}} ``` **"A wedged agent cannot respond to the challenge"** is proven directly: in both wedge scenarios the challenge was delivered (`delivered: true` through the real `manager.steer()`), and the agent still produced zero activity and was reaped at grace end. If the challenge had counted as activity, the grace would have been cancelled and the reap would never have fired. ## Repo gates - `npm run typecheck` clean. - `npx vitest run`: **1300 tests / 99 files passed** (base `origin/main` = 1287 / 98 → +13 tests, +1 file). - Main checkout `~/.repos/pi-extensions` untouched: `git status --porcelain` → only pre-existing untracked `.rpiv/` and `.worktrees/`; all changes are on this branch. Staged explicit paths only (`node_modules` symlink never staged). ## Honest gaps / not verified - **Idle-but-live never-challenged** is covered by unit + real-`AgentManager` tests (`isStreaming: false` on a real record → no challenge, no reap), **not** end-to-end: the harness's background child terminalizes on completion, and reproducing the HOMELAB-1577 wake shape E2E needs a post-settle steer. - The challenge is a steer, so on a *healthy* agent that has gone quiet the probe text eventually enters its conversation. It is written to be ignorable; the alternative (a custom no-LLM channel) needs a new AgentManager method and was judged unnecessary. - `graceMinutes` adds up to `threshold + grace` of extra latency before an actually-wedged agent is reaped. That is the intended trade the operator asked for; `graceMinutes: 0` opts out. - The escalation path (abort refuses / run survives the abort) remains unit-tested only, as in v1. Not merged — Class C; a landing worker owns review + merge + rollout. Closes HOMELAB-1741.

aaron added 1 commit 2026-09-29 06:48:33 +00:00

HOMELAB-1741: feat(pi-swarm-suite): challenge-before-reap watchdog ... ba257e58da

v1 (HOMELAB-1668) reaped on silence alone: running + run in flight + no
activity-bus event for thresholdMinutes. That misfires on a healthy agent
parked in a long, silent blocking wait (a `get_subagent_result(wait)` past
the threshold, or a single long silent tool call) — it looks exactly like a
wedge. Silence is a suspicion, not a verdict.

Two-phase detection, per the operator's ruling ("ping the agent; if it
doesn't answer in 10 min post-check, then kill it"):

  phase 1 — a silence breach CHALLENGES the agent with a liveness steer
    (manager.steer, the same verb `swarm-attach steer` uses; a live run
    drains it and emits activity, a frozen run's event loop never does) and
    opens a grace window (new `stallWatchdog.graceMinutes`, default 10);
  phase 2 — ANY activity-bus event from the agent during grace cancels the
    reap; still silent at grace end → the unchanged v1 reap path (abort +
    post-abort verification + escalation + durable entry + loud notice).

The challenge's OWN synchronous bus events are not an answer (the
HOMELAB-1668 BLOCKER-1 class, where abort's status event cleared the reap
marker): while the watchdog is inside `challenge()` every event the manager
publishes is attributed to the challenge. The grace marker is set BEFORE the
injection, so the guard is load-bearing — the new regression test fails when
it is removed.

- `graceMinutes: 0` = the v1 single-phase behaviour, verbatim.
- `thresholdMinutes: 0` / `enabled: false` unchanged.
- Idle-but-live children are still never challenged or reaped.
- Notices name the phase: phase 1 says "challenged at T, grace until T+10m",
  phase 2 says "challenged, no answer by grace end" — a reap can no longer
  read as a first-phase silent detection.
- No AgentManager semantics change: `steer()` is the existing public verb,
  and it publishes no activity event for a record whose status is unchanged.
- `/agents → Settings` gains "Stall grace (min)"; README + architecture +
  attach-protocol + settings docs updated.

Tests: 12 new regression tests (tests/core/stall-watchdog-challenge.test.ts)
that fail on the v1 code and on the naive (unguarded) v2 wiring; existing
watchdog/settings/manager suites adapted to the two-phase default. Full
suite: 1300 tests / 99 files, typecheck clean.

E2E (real pi RPC, scratch agent dir, 1m/1m thresholds): blackholed-provider
keep-alive wedge and `bash sleep 900` wedge each challenged (~1m56s) then
reaped at grace end (stopped, wedged=true); a `bash sleep 130` call that
resolves within grace was challenged and survived (completed, wedged=false).

aaron commented 2026-09-29 07:08:42 +00:00

Author

Owner

Copy link

Independent landing review — HOMELAB-1741 (head ba257e5)

Two reviewers, each in a fresh pi -p --no-session process, briefed only on the PR diff + the ticket's verbatim
acceptance criteria + the frozen constraints (never on the implementer's summary). Different focus, both against
primary sources; the reviewer wrote no file in this repo.

Reviewer	Focus	Verdict
#1	two-phase state-machine correctness · re-entrancy soundness · comment honesty · behavioural diff vs v1	merge with concerns — no blocker
#2	test adequacy (mutation testing, v1-failure proof) · claim honesty · repo conventions	merge with concerns — no blocker

Independently re-derived by the reviewer (not taken from the PR body)

• 12/12 new tests fail on v1. git archive origin/main scratch, new test file copied in → Tests 12 failed (12).
  (Landing worker re-ran this independently: same result, 12/12.) Nuance: 2 of the 12 (#5 idle-but-live, #7
  graceMinutes: 0) also hold on v1 and fail only because StallTickResult.challenged does not exist there —
  the behavioural discriminators are the other ten.
• The challenging guard is load-bearing and pinned. Deleting if (this.challenging.has(id)) return; fails
  exactly one test, the dedicated "a wedged agent cannot answer: the challenge's OWN bus event is not an answer".
  (Landing worker reproduced: 1 failed | 11 passed.)
• Counts. head 1300 / 99 files; base 1287 / 98; +13 tests / +1 file (12 in the new file + 1 new
  manager-level test); tsc --noEmit exit 0.
• graceMinutes: 0 is v1-verbatim. A/B random-walk equivalence run against v1 source and v2 source:
  6,656 identical output lines (reap order, notices, rendered text, checked, wedgedAt, abort calls).
• Marker lifecycle. 40 × 400 randomised ops against the real class: never an abort() without a pending
  challenge ≥ grace old, never inside the threshold window, never for a non-running or no-run-in-flight record.
• 12 mutants in a scratch copy: cancellation (D), grace base (C), status-is-an-answer (E), legacy path (F),
  idle-but-live exemption (G), run-start reset (I) all killed by the suite.
• BLOCKER-1 class, end-to-end. In both wedge E2E runs the real manager.steer() did not cancel its own
  grace window; the stall-reaped entry carries the challenge object exactly 60,000 ms after stall-challenged.

Concerns (mergeable — posted per the landing contract, not blockers)

1. stall-watchdog.ts:646-652 — the phase-2 notice rebuilds graceUntil/graceMinutes from the current
   config instead of the values announced in phase 1, so if graceMinutes changes mid-grace the two notices
   contradict each other in the operator's transcript (StallChallengeInfo's own doc says it is carried forward).
   Minimal fix: store graceUntil/graceMinutes on the ChallengeRecord and carry those.
2. stall-watchdog.ts:638-643 — raising thresholdMinutes mid-grace leaves the pending challenge in place, so a
   later silence breach reaps on the old ping (notice still reads "challenged, no answer by grace end"). Benign
   in direction (the agent got more than its promised grace, never less), but the ping is not fresh. Minimal fix:
   if (silenceMs <= thresholdMs) { this.challenged.delete(record.id); continue; }.
3. Test strength: the phase-1 notice's graceUntil/graceMinutes/via are not asserted anywhere (zeroing
   graceUntil passes all 1300 tests), and the whole delivered: false / challenge()-throws path is unpinned
   (forcing delivered = true passes 1300/1300). Two small tests close both; the behaviour itself was probed and
   is correct.
4. E2E honesty (Verify #2). The bash sleep 130 "false positive survives" run does show the child challenged
   at silence 1m56s and never reaped, but the child completed ≈44 s before the grace deadline, so the reap was
   never due — that run cannot discriminate the cancellation. The "activity cancels the reap" property is carried
   by the unit + real-AgentManager tests (mutants C/D/E killed), not by that E2E row; and driver.mjs's
   survives verdict would also pass if nothing had happened at all. Recorded here so the evidence is not
   over-claimed.
5. Verify #2's named shape get_subagent_result(wait) was exercised only by the bash sleep substitute. Under
   defaults it is safe (toolWaitCapSec 120 s returns and re-emits activity well inside the 10 m threshold), but
   toolWaitCapSec × thresholdMinutes + graceMinutes is now a coupled pair (cap ceiling 3600 s vs 20 m default
   budget) that no doc names. Worth one line in the README/ticket.
6. Pre-existing (v1, not a regression — identical output on v1 and on v2 with graceMinutes: 0): once
   verifyReaps escalates a still-streaming reaped run it does reaped.delete(id), so the only path that clears
   escalated (if (this.reaped.delete(id))) can never fire again — a resumed record then escapes the watchdog
   for the rest of its life. Out of scope here; follow-up filed.

NITs (README "ANY activity" precision re driver; delivered: true also covers a queued steer with no session;
SUITE(HOMELAB-1668/1741) vs the comma form; stale satellite JSDoc in settings.ts/index.ts calling it a
"pair") are not repeated here in full.

Landing worker verdict

No blocker. CI green on the head sha. Proceeding to merge, then GitOps continuation (ff-only update of the
live ~/.pi/agent/extensions checkout) per the repo's rollout contract.

## Independent landing review — HOMELAB-1741 (head `ba257e5`) Two reviewers, each in a **fresh `pi -p --no-session` process**, briefed only on the PR diff + the ticket's verbatim acceptance criteria + the frozen constraints (never on the implementer's summary). Different focus, both against primary sources; the reviewer wrote no file in this repo. | Reviewer | Focus | Verdict | |---|---|---| | #1 | two-phase state-machine correctness · re-entrancy soundness · comment honesty · behavioural diff vs v1 | **merge with concerns — no blocker** | | #2 | test adequacy (mutation testing, v1-failure proof) · claim honesty · repo conventions | **merge with concerns — no blocker** | ### Independently re-derived by the reviewer (not taken from the PR body) - **12/12 new tests fail on v1.** `git archive origin/main` scratch, new test file copied in → `Tests 12 failed (12)`. (Landing worker re-ran this independently: same result, 12/12.) Nuance: 2 of the 12 (#5 idle-but-live, #7 `graceMinutes: 0`) also hold on v1 and fail only because `StallTickResult.challenged` does not exist there — the behavioural discriminators are the other ten. - **The `challenging` guard is load-bearing and pinned.** Deleting `if (this.challenging.has(id)) return;` fails exactly one test, the dedicated *"a wedged agent cannot answer: the challenge's OWN bus event is not an answer"*. (Landing worker reproduced: 1 failed | 11 passed.) - **Counts.** head `1300 / 99 files`; base `1287 / 98`; `+13 tests / +1 file` (12 in the new file + 1 new manager-level test); `tsc --noEmit` exit 0. - **`graceMinutes: 0` is v1-verbatim.** A/B random-walk equivalence run against v1 source and v2 source: 6,656 identical output lines (reap order, notices, rendered text, `checked`, `wedgedAt`, abort calls). - **Marker lifecycle.** 40 × 400 randomised ops against the real class: never an `abort()` without a pending challenge ≥ grace old, never inside the threshold window, never for a non-running or no-run-in-flight record. - **12 mutants in a scratch copy**: cancellation (D), grace base (C), `status`-is-an-answer (E), legacy path (F), idle-but-live exemption (G), run-start reset (I) all killed by the suite. - **BLOCKER-1 class, end-to-end.** In both wedge E2E runs the real `manager.steer()` did **not** cancel its own grace window; the `stall-reaped` entry carries the `challenge` object exactly 60,000 ms after `stall-challenged`. ### Concerns (mergeable — posted per the landing contract, not blockers) 1. `stall-watchdog.ts:646-652` — the phase-2 notice **rebuilds** `graceUntil`/`graceMinutes` from the *current* config instead of the values announced in phase 1, so if `graceMinutes` changes mid-grace the two notices contradict each other in the operator's transcript (`StallChallengeInfo`'s own doc says it is carried forward). Minimal fix: store `graceUntil`/`graceMinutes` on the `ChallengeRecord` and carry those. 2. `stall-watchdog.ts:638-643` — raising `thresholdMinutes` mid-grace leaves the pending challenge in place, so a later silence breach reaps on the *old* ping (notice still reads "challenged, no answer by grace end"). Benign in direction (the agent got *more* than its promised grace, never less), but the ping is not fresh. Minimal fix: `if (silenceMs <= thresholdMs) { this.challenged.delete(record.id); continue; }`. 3. Test strength: the phase-1 notice's `graceUntil`/`graceMinutes`/`via` are **not asserted** anywhere (zeroing `graceUntil` passes all 1300 tests), and the whole `delivered: false` / `challenge()`-throws path is unpinned (forcing `delivered = true` passes 1300/1300). Two small tests close both; the behaviour itself was probed and is correct. 4. **E2E honesty (Verify #2).** The `bash sleep 130` "false positive survives" run does show the child challenged at silence 1m56s and never reaped, but the child **completed ≈44 s before the grace deadline**, so the reap was never due — that run cannot discriminate the cancellation. The "activity cancels the reap" property is carried by the unit + real-`AgentManager` tests (mutants C/D/E killed), not by that E2E row; and `driver.mjs`'s `survives` verdict would also pass if nothing had happened at all. Recorded here so the evidence is not over-claimed. 5. Verify #2's named shape `get_subagent_result(wait)` was exercised only by the `bash sleep` substitute. Under defaults it is safe (`toolWaitCapSec` 120 s returns and re-emits activity well inside the 10 m threshold), but `toolWaitCapSec` × `thresholdMinutes + graceMinutes` is now a coupled pair (cap ceiling 3600 s vs 20 m default budget) that no doc names. Worth one line in the README/ticket. 6. Pre-existing (v1, **not** a regression — identical output on v1 and on v2 with `graceMinutes: 0`): once `verifyReaps` escalates a still-streaming reaped run it does `reaped.delete(id)`, so the only path that clears `escalated` (`if (this.reaped.delete(id))`) can never fire again — a resumed record then escapes the watchdog for the rest of its life. Out of scope here; follow-up filed. NITs (README "ANY activity" precision re `driver`; `delivered: true` also covers a queued steer with no session; `SUITE(HOMELAB-1668/1741)` vs the comma form; stale satellite JSDoc in `settings.ts`/`index.ts` calling it a "pair") are not repeated here in full. ### Landing worker verdict No blocker. CI green on the head sha. Proceeding to **merge**, then GitOps continuation (ff-only update of the live `~/.pi/agent/extensions` checkout) per the repo's rollout contract.

aaron merged commit 4b48981f3f into main 2026-09-29 07:09:03 +00:00

aaron referenced this pull request from a commit 2026-09-29 07:09:04 +00:00

Merge pull request 'HOMELAB-1741: feat(pi-swarm-suite): challenge-before-reap watchdog' (#46) from plane/HOMELAB-1741-watchdog-challenge into main

aaron commented 2026-09-29 07:09:38 +00:00

Author

Owner

Copy link

Landed — HOMELAB-1741 (merge commit 4b48981)

Step	Result
Fresh-context review ×2	2 reviewer processes (pi -p --no-session, briefed only on diff + ticket ACs + frozen constraints): merge with concerns — no blocker in both
CI on head ba257e5	run #80 green (pi-swarm-suite typecheck+vitest, pi-otel, skill-args, todo, ask-user-question, pr-title)
Merge	method merge → 4b48981f3ff20a432ecf0ffb790a8a2c25e6f2e8 (parents eba861b + ba257e5); branch kept
GitOps continuation	live checkout ~/.pi/agent/extensions fast-forwarded eba861b → 4b48981; git rev-list --count HEAD..forgejo/main = 0

Rollout evidence

• ~/.pi/agent/extensions HEAD = 4b48981; the live pi-swarm-suite/src/core/stall-watchdog.ts contains the
  challenge logic; STALL_WATCHDOG_DEFAULT_GRACE_MINUTES = 10 and stallWatchdogDefaults.graceMinutes = 10 are live.
• HOMELAB-1740 local modifications preserved untouched: the 17 dirty tracked files are byte-identical before
  and after the ff (md5 diff clean), and the 3 untracked entries are unchanged.
• No session was restarted. v2 is active for NEW sessions only; sessions already running keep the v1 code
  they loaded. Defaults: thresholdMinutes 10, graceMinutes 10; graceMinutes: 0 reverts to v1 single-phase.

Residual / follow-up

HOMELAB-1743 (Backlog) captures the review findings that were out of scope here: the pre-existing sticky
escalated marker, notice-vs-config fidelity on a mid-grace config change, the stale-challenge window, the
untested delivered: false / phase-1 notice fields, the discriminating false-positive E2E scenario, and the
toolWaitCapSec × threshold + grace coupling to name in the README.

## Landed — HOMELAB-1741 (merge commit `4b48981`) | Step | Result | |---|---| | Fresh-context review ×2 | 2 reviewer processes (`pi -p --no-session`, briefed only on diff + ticket ACs + frozen constraints): **merge with concerns — no blocker** in both | | CI on head `ba257e5` | run #80 green (pi-swarm-suite typecheck+vitest, pi-otel, skill-args, todo, ask-user-question, pr-title) | | Merge | method `merge` → **`4b48981f3ff20a432ecf0ffb790a8a2c25e6f2e8`** (parents `eba861b` + `ba257e5`); branch kept | | GitOps continuation | live checkout `~/.pi/agent/extensions` fast-forwarded `eba861b → 4b48981`; `git rev-list --count HEAD..forgejo/main` = 0 | ### Rollout evidence - `~/.pi/agent/extensions` HEAD = `4b48981`; the live `pi-swarm-suite/src/core/stall-watchdog.ts` contains the challenge logic; `STALL_WATCHDOG_DEFAULT_GRACE_MINUTES = 10` and `stallWatchdogDefaults.graceMinutes = 10` are live. - **HOMELAB-1740 local modifications preserved untouched**: the 17 dirty tracked files are byte-identical before and after the ff (md5 diff clean), and the 3 untracked entries are unchanged. - No session was restarted. **v2 is active for NEW sessions only**; sessions already running keep the v1 code they loaded. Defaults: `thresholdMinutes 10`, `graceMinutes 10`; `graceMinutes: 0` reverts to v1 single-phase. ### Residual / follow-up `HOMELAB-1743` (Backlog) captures the review findings that were out of scope here: the pre-existing sticky `escalated` marker, notice-vs-config fidelity on a mid-grace config change, the stale-challenge window, the untested `delivered: false` / phase-1 notice fields, the discriminating false-positive E2E scenario, and the `toolWaitCapSec` × `threshold + grace` coupling to name in the README.

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Homelab/pi-extensions!46

No description provided.