HOMELAB-2281: unsuspend etcd snapshot mirror (object-lock destination) #684

Merged

pi-landing merged 1 commit from plane/HOMELAB-2281-etcd-mirror into live 2026-10-30 02:21:57 +00:00

Conversation 1 Commits 1 Files changed 1 +1 -1

aaron commented 2026-10-30 02:08:29 +00:00

Owner

Copy link

Rollout step runbook §4.1 #4, third and final consumer on HOMELAB-2281 (order: OpenBao ✅ → OPNsense (blocked on the router 500, untouched) → etcd).

What changes

One functional line in core/manifests/etcd-backup/mirror-cronjob.yaml:

-  suspend: true
+  suspend: false

Non-comment diff lines, filtered: exactly the two lines above. 1 file changed, 1 insertion(+), 1 deletion(-). No other manifest, script, fixture, alert or CRD is touched. Nothing is applied to the cluster from this branch.

Deliberately unchanged (the design invariants this flip must not disturb), verified mechanically after the edit — 29/29 checks:

invariant	value after this PR
concurrencyPolicy	Forbid (one-writer serialization)
restartPolicy	Never (a restart would reuse the emptyDir status marker and could report a stale success)
activeDeadlineSeconds	1800; startingDeadlineSeconds 1800
schedule	10 3 * * * — 40 min after the primary (02:30, deadline 1200s)
s3 sync excludes	both calls carry --exclude "locks/*" and --exclude "*-tmp-*" (pull and push)
--delete	absent from the mirror script entirely
remote locks assertion	s3 ls …/repo/locks/ must be empty; non-empty → ERROR: lock/other objects exist … + exit 1
verify container	every restic invocation carries --no-lock (3/3), and the newest snapshot must equal date -u +%F (today-UTC freshness gate)
EtcdMirrorStale / EtcdMirrorNeverSucceeded	both still carry unless on() (kube_cronjob_spec_suspend{namespace="etcd-backup",cronjob="etcd-snapshot-mirror"} == 1)
Secret refs	ROT_AWS_ACCESS_KEY_ID / ROT_AWS_SECRET_ACCESS_KEY from etcd-snapshot-b2-locked; no secret ref is optional

Gating evidence (already recorded on HOMELAB-2281)

• §6 day-one writer tests PASS for the etcd-snapshot key, including the lock proof — test 4, the pruner key (which has deleteFiles, no bypassGovernance) deleting a freshly uploaded etcd-snapshot/_locktest/… object: An error occurred (AccessDenied) when calling the DeleteObject operation: Access Denied, and the version was still readable by its versionId afterwards ({"Len": 40}). Capability denials read not entitled; this one reads Access Denied — the store refusing, not the key's permissions. Tests 3 (retention-shorten / legal-hold-off → not entitled), 5 (403 Forbidden cross-prefix read), 5b (cross-prefix write refused, 0 such objects) also PASS.
• Secret etcd-backup/etcd-snapshot-b2-locked is Ready=True reason=SecretSynced (ClusterSecretStore openbao, remote b2-root-of-trust/etcd-writer), with exactly the two keys [ROT_AWS_ACCESS_KEY_ID, ROT_AWS_SECRET_ACCESS_KEY] (b64 36/44).
• Consumer #1 (OpenBao raft) proven green 2026-10-30: the 01:30Z Job logged verified_locked s3://reynoza-root-of-trust/openbao-raft/openbao-2026-10-30.snap bytes=221227 sha256=9bb4f0dc…76cdc (append-only, no prune), the locked object was downloaded and hashed independently to the same sha256, and the old reynoza-velero copy is unchanged.

Expected live effect (first unsuspended run: 2026-10-30T03:10Z, ~1 h after this PR)

1. CronJob creates its first Job; container mirror pulls reynoza-velero/etcd-snapshot/ into the emptyDir scratch dir (excluding locks/*, *-tmp-*), then pushes it to reynoza-root-of-trust/etcd-snapshot/repo/ — the first object copy into the locked bucket for etcd.
2. Remote assertion: …/repo/locks/ must list nothing. One lock object there is permanent (30-day governance retention, no cluster key has bypassGovernance), so this is a hard failure, not a warning.
3. Container verify waits for the mirror's status marker, then opens the locked repo read-only (restic --no-lock) and requires the newest snapshot to be today's UTC date.
4. Failure is loud: any non-zero exit leaves a 1 in the status marker, so the verifier fails instead of waiting, and the Job fails as a unit.
5. The primary etcd-snapshot Job is untouched and keeps its own meaning — separate CronJob, no locked-bucket credential in it, its own read-back proof remains the only thing that decides whether etcd is backed up today.

Alerts arm with this flip. The suspend: false guard stops suppressing EtcdMirrorStale and EtcdMirrorNeverSucceeded (both warning, both already loaded in Prometheus with health=ok). EtcdMirrorNeverSucceeded fires if no success timestamp exists 26 h after the flip; EtcdMirrorStale fires when the last success is older than 26 h. A red mirror means "the copy nobody can delete is missing", not "the backup failed".

Non-goals

• Primary etcd repo untouched — etcd-snapshot keeps writing reynoza-velero/etcd-snapshot/ exactly as before; this PR adds a copy, not a migration.
• No retirement of the old reynoza-velero copies. Runbook §4.1 step 5 requires a full week of green nightly runs across all three consumers, and retiring root-of-trust data is its own decision in its own ticket.
• OPNsense producer still absent — consumer #2 remains blocked on the router 500; its writer key has no producer.

Local validation on this branch

• YAML parse: 2 documents (ConfigMap, CronJob); spec.suspend reads the boolean False.
• The 29-check invariant sweep above (comment-insensitive where it matters — the --delete check ignores comment text, and the locks assertion is anchored to the code path, not a grep hit).
• scripts/root-of-trust/tests/test-etcd-mirror-fixture.sh: 12/12 fixtures pass, 0 failures.
• Nothing applied to the cluster; no credential read, printed or created.

One finding for the reviewer (not fixed here, because the brief scoped this PR to the single flag line)

scripts/root-of-trust/tests/test-etcd-mirror-fixture.sh line 211 asserts the parked state with a whole-file grep -q 'suspend: true' "$MANIFEST". After this flip that grep still succeeds — it matches the design comment at line 53 (# ENABLING IT: `suspend: true` is the flag …), not the spec field. So the §6.1 gate "mirror ships suspended" is now a false positive: it passes for an enabled mirror, and would also pass if someone re-suspended. A test that cannot fail is not a test (the same standard runbook §4.1 step 3 applies to the live tests).

Suggested fix, one line in the fixture, in this PR or as a fix-worker commit on this head:

-grep -q 'suspend: true' "$MANIFEST" \
-  && pass "mirror ships suspended (the flag, runbook §4.1)" \
-  || fail "suspended by default" "suspend: true not found"
+grep -qE '^  suspend: false$' "$MANIFEST" \
+  && pass "mirror ships enabled after the §4.1 rollout (spec.suspend, not a comment)" \
+  || fail "suspend flag" "spec.suspend: false not found"

Anchoring with ^ suspend: also makes the assertion immune to the comment in either direction.

---

I did not review or merge this PR. Ticket stays In Progress; it closes on a green first mirror Job (mirror_ok … locks=0, locked_copy_readback newest_snapshot=2026-10-30, locked_copy_verified), not on this merge.

Rollout step runbook §4.1 #4, **third and final consumer** on HOMELAB-2281 (order: OpenBao ✅ → OPNsense (blocked on the router 500, untouched) → etcd). ## What changes **One functional line** in `core/manifests/etcd-backup/mirror-cronjob.yaml`: ```diff - suspend: true + suspend: false ``` Non-comment diff lines, filtered: exactly the two lines above. `1 file changed, 1 insertion(+), 1 deletion(-)`. No other manifest, script, fixture, alert or CRD is touched. Nothing is applied to the cluster from this branch. Deliberately unchanged (the design invariants this flip must not disturb), verified mechanically after the edit — 29/29 checks: | invariant | value after this PR | |---|---| | `concurrencyPolicy` | `Forbid` (one-writer serialization) | | `restartPolicy` | `Never` (a restart would reuse the emptyDir status marker and could report a stale success) | | `activeDeadlineSeconds` | `1800`; `startingDeadlineSeconds` `1800` | | schedule | `10 3 * * *` — 40 min after the primary (02:30, deadline 1200s) | | `s3 sync` excludes | both calls carry `--exclude "locks/*"` **and** `--exclude "*-tmp-*"` (pull and push) | | `--delete` | absent from the mirror script entirely | | remote locks assertion | `s3 ls …/repo/locks/` must be empty; non-empty → `ERROR: lock/other objects exist …` + `exit 1` | | verify container | every restic invocation carries `--no-lock` (3/3), and the newest snapshot must equal `date -u +%F` (today-UTC freshness gate) | | `EtcdMirrorStale` / `EtcdMirrorNeverSucceeded` | both still carry `unless on() (kube_cronjob_spec_suspend{namespace="etcd-backup",cronjob="etcd-snapshot-mirror"} == 1)` | | Secret refs | `ROT_AWS_ACCESS_KEY_ID` / `ROT_AWS_SECRET_ACCESS_KEY` from `etcd-snapshot-b2-locked`; **no** secret ref is `optional` | ## Gating evidence (already recorded on HOMELAB-2281) - **§6 day-one writer tests PASS for the `etcd-snapshot` key**, including the lock proof — test 4, the pruner key (which *has* `deleteFiles`, no `bypassGovernance`) deleting a freshly uploaded `etcd-snapshot/_locktest/…` object: `An error occurred (AccessDenied) when calling the DeleteObject operation: Access Denied`, and the version was still readable by its `versionId` afterwards (`{"Len": 40}`). Capability denials read `not entitled`; this one reads `Access Denied` — the store refusing, not the key's permissions. Tests 3 (retention-shorten / legal-hold-off → `not entitled`), 5 (`403 Forbidden` cross-prefix read), 5b (cross-prefix write refused, 0 such objects) also PASS. - **`Secret etcd-backup/etcd-snapshot-b2-locked` is `Ready=True reason=SecretSynced`** (ClusterSecretStore `openbao`, remote `b2-root-of-trust/etcd-writer`), with exactly the two keys `[ROT_AWS_ACCESS_KEY_ID, ROT_AWS_SECRET_ACCESS_KEY]` (b64 36/44). - **Consumer #1 (OpenBao raft) proven green 2026-10-30**: the 01:30Z Job logged `verified_locked s3://reynoza-root-of-trust/openbao-raft/openbao-2026-10-30.snap bytes=221227 sha256=9bb4f0dc…76cdc (append-only, no prune)`, the locked object was downloaded and hashed independently to the same sha256, and the old `reynoza-velero` copy is unchanged. ## Expected live effect (first unsuspended run: **2026-10-30T03:10Z**, ~1 h after this PR) 1. CronJob creates its first Job; container `mirror` pulls `reynoza-velero/etcd-snapshot/` into the emptyDir scratch dir (excluding `locks/*`, `*-tmp-*`), then pushes it to **`reynoza-root-of-trust/etcd-snapshot/repo/`** — the first object copy into the locked bucket for etcd. 2. Remote assertion: `…/repo/locks/` must list **nothing**. One lock object there is permanent (30-day governance retention, no cluster key has `bypassGovernance`), so this is a hard failure, not a warning. 3. Container `verify` waits for the mirror's status marker, then opens the **locked** repo read-only (`restic --no-lock`) and requires the newest snapshot to be **today's UTC date**. 4. Failure is loud: any non-zero exit leaves a `1` in the status marker, so the verifier fails instead of waiting, and the Job fails as a unit. 5. **The primary `etcd-snapshot` Job is untouched** and keeps its own meaning — separate CronJob, no locked-bucket credential in it, its own read-back proof remains the only thing that decides whether etcd is backed up today. **Alerts arm with this flip.** The `suspend: false` guard stops suppressing `EtcdMirrorStale` and `EtcdMirrorNeverSucceeded` (both `warning`, both already loaded in Prometheus with `health=ok`). `EtcdMirrorNeverSucceeded` fires if no success timestamp exists 26 h after the flip; `EtcdMirrorStale` fires when the last success is older than 26 h. A red mirror means "the copy nobody can delete is missing", not "the backup failed". ## Non-goals - **Primary etcd repo untouched** — `etcd-snapshot` keeps writing `reynoza-velero/etcd-snapshot/` exactly as before; this PR adds a copy, not a migration. - **No retirement of the old `reynoza-velero` copies.** Runbook §4.1 step 5 requires a full week of green nightly runs across all three consumers, and retiring root-of-trust data is its own decision in its own ticket. - **OPNsense producer still absent** — consumer #2 remains blocked on the router 500; its writer key has no producer. ## Local validation on this branch - YAML parse: 2 documents (`ConfigMap`, `CronJob`); `spec.suspend` reads the boolean `False`. - The 29-check invariant sweep above (comment-insensitive where it matters — the `--delete` check ignores comment text, and the locks assertion is anchored to the code path, not a grep hit). - `scripts/root-of-trust/tests/test-etcd-mirror-fixture.sh`: 12/12 fixtures pass, 0 failures. - Nothing applied to the cluster; no credential read, printed or created. ## One finding for the reviewer (not fixed here, because the brief scoped this PR to the single flag line) `scripts/root-of-trust/tests/test-etcd-mirror-fixture.sh` line 211 asserts the parked state with a whole-file `grep -q 'suspend: true' "$MANIFEST"`. After this flip that grep still succeeds — it matches the **design comment** at line 53 (`` # ENABLING IT: `suspend: true` is the flag … ``), not the spec field. So the §6.1 gate "mirror ships suspended" is now a **false positive**: it passes for an enabled mirror, and would also pass if someone re-suspended. A test that cannot fail is not a test (the same standard runbook §4.1 step 3 applies to the live tests). Suggested fix, one line in the fixture, in this PR or as a fix-worker commit on this head: ```diff -grep -q 'suspend: true' "$MANIFEST" \ - && pass "mirror ships suspended (the flag, runbook §4.1)" \ - || fail "suspended by default" "suspend: true not found" +grep -qE '^ suspend: false$' "$MANIFEST" \ + && pass "mirror ships enabled after the §4.1 rollout (spec.suspend, not a comment)" \ + || fail "suspend flag" "spec.suspend: false not found" ``` Anchoring with `^ suspend:` also makes the assertion immune to the comment in either direction. --- I did not review or merge this PR. Ticket stays **In Progress**; it closes on a green first mirror Job (`mirror_ok … locks=0`, `locked_copy_readback newest_snapshot=2026-10-30`, `locked_copy_verified`), not on this merge.

aaron added 1 commit 2026-10-30 02:08:29 +00:00

HOMELAB-2281: chore(etcd-backup): unsuspend the snapshot mirror (object-lock destination) ... f6657d6bfc

Rollout step §4.1 #4, third and final consumer. One functional line:
spec.suspend true -> false on CronJob etcd-backup/etcd-snapshot-mirror.

Gated by: §6 day-one writer tests PASS for the etcd-snapshot key (test 4
pruner delete refused by retention: literal AccessDenied), Secret
etcd-backup/etcd-snapshot-b2-locked SecretSynced with both ROT_ keys, and
consumer #1 (OpenBao raft) proven green 2026-10-30 with the locked copy
sha256-matched.

Unchanged by design: concurrencyPolicy Forbid, restartPolicy Never,
activeDeadlineSeconds 1800, both s3 sync excludes (locks/*, *-tmp-*), no
--delete, the remote locks-empty assertion, --no-lock read-only verify with
the today-UTC freshness gate, and the EtcdMirror* alerts' suspend guard.
The primary etcd-snapshot Job is untouched.

First unsuspended run: 2026-10-30T03:10Z.

ld-reviewer reviewed 2026-10-30 02:17:52 +00:00

ld-reviewer left a comment

Collaborator

Copy link

Review — HOMELAB-2281 unsuspend etcd snapshot mirror (round 1)

Reviewed head f6657d6b from primary sources only: the diff, the full manifest at head (blob 4f55e79, matching the diff's post-image), core/manifests/monitoring/rules/backup-alerts.yaml, the fixture script, and CI.

(1) Diff is exactly the one-line flip

pulls/684.diff is one hunk, one file, 1 insertion(+), 1 deletion(-): core/manifests/etcd-backup/mirror-cronjob.yaml:216 suspend: true → suspend: false. No comment churn, no ad-hoc edits riding along. Parses to the boolean False (not the string).

(2) Design invariants still intact in the head file

• concurrencyPolicy: Forbid (L217), startingDeadlineSeconds: 1800, activeDeadlineSeconds: 1800 (L230), backoffLimit: 1, restartPolicy: Never (L241) with the documented reason (a restart would reuse the emptyDir status marker and could report a stale success).
• Both s3 sync calls carry both excludes (--exclude "locks/*" --exclude "*-tmp-*"), pull L128-129 and push L135-136 — the deliberate second filter is still there.
• --delete appears nowhere as an argument (only in the "Never --delete" comment). Append-only push preserved.
• Remote assertion intact: locks=$(dst s3 ls "${DST_REPO%/}/locks/" || true); non-empty → ERROR: lock/other objects exist … + exit 1 (L140-146).
• Verify container: all three restic invocations carry --no-lock (L175, 176, 191) and the freshness gate is [ "$newest" != "$(date -u +%F)" ] → exit 1 (L193-195).
• Secret wiring unchanged and non-optional everywhere (no optional: in the file): src etcd-snapshot-b2, dst and verify etcd-snapshot-b2-locked with keys ROT_AWS_ACCESS_KEY_ID / ROT_AWS_SECRET_ACCESS_KEY. The preflight's ROT_* names match the env names, so the credentials-file heredoc cannot silently get an empty credential.
• Both containers share the work emptyDir at /work and STATUS_FILE=/work/mirror-status, so the flip does not expose a status-file path mismatch.

(3) Primary job untouched

The diff touches no other path, so core/manifests/etcd-backup/cronjob.yaml (blob 48f6b04, last changed 2026-10-28) is byte-identical. The mirror is additive; the primary's read-back proof still decides whether etcd is backed up today.

(4) CI green on head

Run #2371 (shellcheck, yaml-lint, terraform-validate) and run #2370 (alert-receiver-test, cockpit-e2e, cockpit-test, dispatcher-test, helm-validate, receiver-test, record-sql-test) — all success on f6657d6b.

(5) Stated live effect is honest

• schedule: "10 3 * * *" → the claimed first run at 2026-10-30T03:10Z is right for a UTC controller clock, and the manifest already documents the 40-min gap after the 02:30 primary.
• Alerts arm exactly as described: both EtcdMirrorStale and EtcdMirrorNeverSucceeded carry unless on() (kube_cronjob_spec_suspend{…} == 1), which stops suppressing the moment suspend reads 0. NeverSucceeded (absent(...), for: 26h) fires ~26 h after the flip if no success exists — matching the body. EtcdMirrorStale needs an existing success timestamp, which the body states correctly ("when the last success is older than 26 h").
• The failure-loudness claim matches the code: the trap writes 1 to the status file on any non-zero exit, and verify fails on != 0 instead of waiting out MIRROR_WAIT_SECONDS.

Findings

Blocker: none open.

Concern (pre-existing, correctly excluded from this diff) — the §6.1 fixture gate is now inert. Confirmed independently, not taken on trust: scripts/root-of-trust/tests/test-etcd-mirror-fixture.sh:211 does a whole-file grep -q 'suspend: true', which matches the design comment at mirror-cronjob.yaml:53 (# ENABLING IT: \suspend: true` is the flag …). The gate therefore passes for an enabled mirror **and** would still pass if someone re-suspended it. The diff touches no fixture, so this is neither caused nor worsened by this PR — but it means green CI is not evidence for the parked/enabled state in either direction. The suggested anchored assertion (grep -qE '^ suspend: false$'`) is the right fix and makes the check immune to the comment both ways. Recommendation: fold that one-liner in on this head, or file it and link it from HOMELAB-2281 — I found no existing Plane ticket for it, so as it stands the follow-up has no owner. Not a merge blocker: the flip is a one-line flag change, and the append-only/lock/no-delete invariants that carry the actual risk are asserted elsewhere and unchanged.

Note (no action): NeverSucceeded's 26 h for: starts when the series goes absent at the flip, so the earliest page is ~2026-10-31T05:10Z; the first scheduled run at 03:10Z has a healthy margin inside it.

Verdict: blockers: none open. No edits or verdict from me beyond this COMMENT review — I did not touch the branch.

## Review — HOMELAB-2281 unsuspend etcd snapshot mirror (round 1) Reviewed head `f6657d6b` from primary sources only: the diff, the full manifest at head (blob `4f55e79`, matching the diff's post-image), `core/manifests/monitoring/rules/backup-alerts.yaml`, the fixture script, and CI. ### (1) Diff is exactly the one-line flip `pulls/684.diff` is one hunk, one file, `1 insertion(+), 1 deletion(-)`: `core/manifests/etcd-backup/mirror-cronjob.yaml:216` `suspend: true` → `suspend: false`. No comment churn, no ad-hoc edits riding along. Parses to the boolean `False` (not the string). ### (2) Design invariants still intact in the head file - `concurrencyPolicy: Forbid` (L217), `startingDeadlineSeconds: 1800`, `activeDeadlineSeconds: 1800` (L230), `backoffLimit: 1`, `restartPolicy: Never` (L241) with the documented reason (a restart would reuse the emptyDir status marker and could report a stale success). - Both `s3 sync` calls carry **both** excludes (`--exclude "locks/*" --exclude "*-tmp-*"`), pull L128-129 **and** push L135-136 — the deliberate second filter is still there. - `--delete` appears nowhere as an argument (only in the "Never `--delete`" comment). Append-only push preserved. - Remote assertion intact: `locks=$(dst s3 ls "${DST_REPO%/}/locks/" || true)`; non-empty → `ERROR: lock/other objects exist …` + `exit 1` (L140-146). - Verify container: all three restic invocations carry `--no-lock` (L175, 176, 191) and the freshness gate is `[ "$newest" != "$(date -u +%F)" ]` → `exit 1` (L193-195). - Secret wiring unchanged and non-optional everywhere (no `optional:` in the file): src `etcd-snapshot-b2`, dst and verify `etcd-snapshot-b2-locked` with keys `ROT_AWS_ACCESS_KEY_ID` / `ROT_AWS_SECRET_ACCESS_KEY`. The preflight's `ROT_*` names match the env names, so the credentials-file heredoc cannot silently get an empty credential. - Both containers share the `work` emptyDir at `/work` and `STATUS_FILE=/work/mirror-status`, so the flip does not expose a status-file path mismatch. ### (3) Primary job untouched The diff touches no other path, so `core/manifests/etcd-backup/cronjob.yaml` (blob `48f6b04`, last changed 2026-10-28) is byte-identical. The mirror is additive; the primary's read-back proof still decides whether etcd is backed up today. ### (4) CI green on head Run #2371 (`shellcheck`, `yaml-lint`, `terraform-validate`) and run #2370 (`alert-receiver-test`, `cockpit-e2e`, `cockpit-test`, `dispatcher-test`, `helm-validate`, `receiver-test`, `record-sql-test`) — all success on `f6657d6b`. ### (5) Stated live effect is honest - `schedule: "10 3 * * *"` → the claimed first run at 2026-10-30T03:10Z is right for a UTC controller clock, and the manifest already documents the 40-min gap after the 02:30 primary. - Alerts arm exactly as described: both `EtcdMirrorStale` and `EtcdMirrorNeverSucceeded` carry `unless on() (kube_cronjob_spec_suspend{…} == 1)`, which stops suppressing the moment `suspend` reads `0`. `NeverSucceeded` (`absent(...)`, `for: 26h`) fires ~26 h after the flip if no success exists — matching the body. `EtcdMirrorStale` needs an existing success timestamp, which the body states correctly ("when the last success is older than 26 h"). - The failure-loudness claim matches the code: the `trap` writes `1` to the status file on any non-zero exit, and verify fails on `!= 0` instead of waiting out `MIRROR_WAIT_SECONDS`. ### Findings **Blocker: none open.** **Concern (pre-existing, correctly excluded from this diff) — the §6.1 fixture gate is now inert.** Confirmed independently, not taken on trust: `scripts/root-of-trust/tests/test-etcd-mirror-fixture.sh:211` does a whole-file `grep -q 'suspend: true'`, which matches the design comment at `mirror-cronjob.yaml:53` (`# ENABLING IT: \`suspend: true\` is the flag …`). The gate therefore passes for an enabled mirror **and** would still pass if someone re-suspended it. The diff touches no fixture, so this is neither caused nor worsened by this PR — but it means green CI is not evidence for the parked/enabled state in either direction. The suggested anchored assertion (`grep -qE '^ suspend: false$'`) is the right fix and makes the check immune to the comment both ways. Recommendation: fold that one-liner in on this head, or file it and link it from HOMELAB-2281 — I found no existing Plane ticket for it, so as it stands the follow-up has no owner. Not a merge blocker: the flip is a one-line flag change, and the append-only/lock/no-delete invariants that carry the actual risk are asserted elsewhere and unchanged. **Note (no action):** `NeverSucceeded`'s 26 h `for:` starts when the series goes `absent` at the flip, so the earliest page is ~2026-10-31T05:10Z; the first scheduled run at 03:10Z has a healthy margin inside it. Verdict: blockers: none open. No edits or verdict from me beyond this COMMENT review — I did not touch the branch.

pi-landing merged commit b4dd2f89b6 into live 2026-10-30 02:21:57 +00:00

pi-landing deleted branch plane/HOMELAB-2281-etcd-mirror 2026-10-30 02:21:58 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

operator-review

Requires human operator review before merge — dangerous class (RBAC, secrets, network policy, Terraform) or explicitly gated. Never auto-merge.

No labels

operator-review

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Homelab/infra-core!684

No description provided.