HOMELAB-1878: chore(longhorn): reduce database snapshot retention 14d -> 7d #593

Merged

aaron merged 3 commits from plane/HOMELAB-1878-retention into live 2026-10-03 18:51:59 +00:00

Conversation 3 Commits 3 Files changed 1 +12 -2

aaron commented 2026-10-03 03:40:36 +00:00

Owner

Copy link

Refs HOMELAB-1878 (Longhorn storage leak cleanup).

What the diff does

core/manifests/longhorn-recurring-jobs.yaml:

• daily-snapshot-databases.spec.retain: 14 -> 7
• Updates that job's header comment (the old text claimed "highest retention because RPO matters most", which the change invalidates) and adds a HOMELAB-1878 rationale block.

No other job is touched. Verified byte-for-byte for the rest of the file:

daily-snapshot-databases   task=snapshot retain=7   cron=0 1 * * *    groups=[group-a]
daily-snapshot-apps        task=snapshot retain=7   cron=0 2 * * *    groups=[group-b]
daily-snapshot-bulk        task=snapshot retain=3   cron=0 3 * * *    groups=[group-c]
weekly-snapshot            task=snapshot retain=4   cron=0 5 * * 0    groups=[group-a,group-b,group-c]
daily-backup-b2            task=backup   retain=7   cron=0 4 * * *    groups=[group-a,group-b,group-c,default]

daily-backup-b2 (the off-cluster B2 backup) is unchanged — explicitly out of scope.

Deployment path

prod/apps/longhorn-config.yaml is an ArgoCD Application: repo infra-core, targetRevision: live, path: core/manifests, directory.include: 'longhorn-*.yaml'. Merging this branch to live is what changes the resources; nothing else consumes this file.

Pre-change live cluster state matched the manifest exactly (kubectl get recurringjob -n longhorn-system), confirming this file is the source of truth.

Validation performed

• File is a plain multi-doc manifest (not Helm-templated), so helm lint/template does not apply. Parsed with yaml.safe_load_all: 5 documents, job names and order unchanged, daily-backup-b2 asserted untouched.
• yamllint -d relaxed core/manifests/longhorn-recurring-jobs.yaml -> exit 0. Repo-wide yamllint -d relaxed core/manifests/ -> 0 errors. The 7 remaining warnings are all pre-existing line-length (80-col) warnings, including 4 on lines this PR did not touch.

REVIEWER / OPERATOR ATTENTION REQUIRED — do not auto-merge

This reduces recovery-point retention for database volumes on group-a.

• The change is a retention reduction on a data-protection control. The effect is only realised as existing snapshots age out past the new cap; it is not instantaneous and it is not trivially reversible (snapshots dropped past the new cap cannot be recovered).
• On-cluster recovery granularity for group-a volumes drops from 14 daily points to 7.
• Mitigating context, stated as context and not as a substitute for review: daily-backup-b2 retains 7 days off-cluster for the same groups, and CloudNativePG/barman maintains its own WAL PITR stream independent of Longhorn snapshots. This PR does not modify either.
• weekly-snapshot (retain 4) also targets group-a and continues to provide ~4 weeks of weekly depth; it is deliberately unchanged here.

Given the above this is not a harmless/additive change. Requesting operator review before merge.

Intended effect (rationale, not proof)

Longhorn pins every block referenced by a retained snapshot, so a volume's actual usage scales with retention depth × daily write volume. At 14 days the observed group-a volumes ran well past their PVC specs (outline CNPG WAL 4 GiB spec / 14 GiB actual; zitadel WAL 10/16 GiB; forgejo WAL 10/12 GiB; prometheus DB 50 GiB spec / 94 GiB actual). These figures come from the HOMELAB-1878 investigation; the diff itself only changes the retention integer.

This PR does not reclaim already-written blocks — that is a separate remediation step under HOMELAB-1878.

Refs HOMELAB-1878 (Longhorn storage leak cleanup). ## What the diff does `core/manifests/longhorn-recurring-jobs.yaml`: - `daily-snapshot-databases.spec.retain`: `14` -> `7` - Updates that job's header comment (the old text claimed "highest retention because RPO matters most", which the change invalidates) and adds a HOMELAB-1878 rationale block. **No other job is touched.** Verified byte-for-byte for the rest of the file: ``` daily-snapshot-databases task=snapshot retain=7 cron=0 1 * * * groups=[group-a] daily-snapshot-apps task=snapshot retain=7 cron=0 2 * * * groups=[group-b] daily-snapshot-bulk task=snapshot retain=3 cron=0 3 * * * groups=[group-c] weekly-snapshot task=snapshot retain=4 cron=0 5 * * 0 groups=[group-a,group-b,group-c] daily-backup-b2 task=backup retain=7 cron=0 4 * * * groups=[group-a,group-b,group-c,default] ``` `daily-backup-b2` (the off-cluster B2 backup) is unchanged — explicitly out of scope. ## Deployment path `prod/apps/longhorn-config.yaml` is an ArgoCD Application: repo `infra-core`, `targetRevision: live`, `path: core/manifests`, `directory.include: 'longhorn-*.yaml'`. Merging this branch to `live` is what changes the resources; nothing else consumes this file. Pre-change live cluster state matched the manifest exactly (`kubectl get recurringjob -n longhorn-system`), confirming this file is the source of truth. ## Validation performed - File is a plain multi-doc manifest (not Helm-templated), so `helm lint`/`template` does not apply. Parsed with `yaml.safe_load_all`: 5 documents, job names and order unchanged, `daily-backup-b2` asserted untouched. - `yamllint -d relaxed core/manifests/longhorn-recurring-jobs.yaml` -> exit 0. Repo-wide `yamllint -d relaxed core/manifests/` -> 0 errors. The 7 remaining warnings are all pre-existing `line-length` (80-col) warnings, including 4 on lines this PR did not touch. ## REVIEWER / OPERATOR ATTENTION REQUIRED — do not auto-merge This **reduces recovery-point retention for database volumes on `group-a`**. - The change is a retention *reduction* on a data-protection control. The effect is only realised as existing snapshots age out past the new cap; it is not instantaneous and it is not trivially reversible (snapshots dropped past the new cap cannot be recovered). - On-cluster recovery granularity for `group-a` volumes drops from 14 daily points to 7. - Mitigating context, stated as context and not as a substitute for review: `daily-backup-b2` retains 7 days off-cluster for the same groups, and CloudNativePG/barman maintains its own WAL PITR stream independent of Longhorn snapshots. This PR does not modify either. - `weekly-snapshot` (retain 4) also targets `group-a` and continues to provide ~4 weeks of weekly depth; it is deliberately unchanged here. Given the above this is not a harmless/additive change. Requesting operator review before merge. ## Intended effect (rationale, not proof) Longhorn pins every block referenced by a retained snapshot, so a volume's actual usage scales with retention depth × daily write volume. At 14 days the observed `group-a` volumes ran well past their PVC specs (outline CNPG WAL 4 GiB spec / 14 GiB actual; zitadel WAL 10/16 GiB; forgejo WAL 10/12 GiB; prometheus DB 50 GiB spec / 94 GiB actual). These figures come from the HOMELAB-1878 investigation; the diff itself only changes the retention integer. This PR does not reclaim already-written blocks — that is a separate remediation step under HOMELAB-1878.

aaron added 1 commit 2026-10-03 03:40:36 +00:00

HOMELAB-1878: chore(longhorn): reduce database snapshot retention 14d -> 7d f5e6fc80e8

aaron commented 2026-10-03 03:49:13 +00:00

Author

Owner

Copy link

CI note — cockpit-e2e failure on run #1774 is unrelated to this diff

Two runs were triggered for f5e6fc8:

• run #1775 — PASS (yaml-lint, shellcheck, terraform-validate) — this is the run covering the file changed here.
• run #1774 — FAIL, 1 job: cockpit-e2e.

Evidence the failure is not caused by this PR

1. Zero file overlap. This branch changes exactly one file:

   $ git diff --name-only origin/live...HEAD
   core/manifests/longhorn-recurring-jobs.yaml
   

   No cockpit*, e2e, or telemetry* path is touched. A Longhorn RecurringJob retain integer cannot influence a Svelte SPA DOM assertion.

2. The failing assertion is a cockpit DOM/render-state check. Spec: lab-director/cockpit/web/e2e/telemetry.spec.ts:275 ("the honest row reads as a sentence at 390px, outside any scroller"). It fails at:

   > 295 | expect(measured.noteRight).not.toBeNull();
   

   noteRight is null because document.querySelector('[data-testid="wave-row-note"]') matched no element — i.e. the wave-row-note row did not render in that run. This is a cockpit render-state/data-dependent flake, not a layout regression from this PR.

3. All other jobs on that run passed: helm-validate, receiver-test, dispatcher-test, alert-receiver-test, cockpit-test.

Caveat on the "pre-existing" claim

I could not positively confirm pre-existence from a live-branch run: the most recent live runs (#1776, #1777) do not list cockpit-e2e in their reported job set at all, so there is no same-job comparison datapoint available. What is established from the evidence above is that it is unrelated to this diff (zero overlap, cockpit-only failure surface).

Per scope, I am not attempting a cockpit e2e fix under HOMELAB-1878. Flagging telemetry.spec.ts:275 — the [data-testid="wave-row-note"] selector not resolving — as worth a separate ticket if it recurs.

## CI note — `cockpit-e2e` failure on run #1774 is unrelated to this diff Two runs were triggered for `f5e6fc8`: - **run #1775 — PASS** (`yaml-lint`, `shellcheck`, `terraform-validate`) — this is the run covering the file changed here. - **run #1774 — FAIL**, 1 job: `cockpit-e2e`. ### Evidence the failure is not caused by this PR 1. **Zero file overlap.** This branch changes exactly one file: ``` $ git diff --name-only origin/live...HEAD core/manifests/longhorn-recurring-jobs.yaml ``` No `cockpit*`, `e2e`, or `telemetry*` path is touched. A Longhorn `RecurringJob` retain integer cannot influence a Svelte SPA DOM assertion. 2. **The failing assertion is a cockpit DOM/render-state check.** Spec: `lab-director/cockpit/web/e2e/telemetry.spec.ts:275` ("the honest row reads as a sentence at 390px, outside any scroller"). It fails at: ``` > 295 | expect(measured.noteRight).not.toBeNull(); ``` `noteRight` is `null` because `document.querySelector('[data-testid="wave-row-note"]')` matched no element — i.e. the `wave-row-note` row did not render in that run. This is a cockpit render-state/data-dependent flake, not a layout regression from this PR. 3. **All other jobs on that run passed**: `helm-validate`, `receiver-test`, `dispatcher-test`, `alert-receiver-test`, `cockpit-test`. ### Caveat on the "pre-existing" claim I could not positively confirm pre-existence from a `live`-branch run: the most recent `live` runs (#1776, #1777) do not list `cockpit-e2e` in their reported job set at all, so there is no same-job comparison datapoint available. What is established from the evidence above is that it is **unrelated to this diff** (zero overlap, cockpit-only failure surface). Per scope, I am not attempting a cockpit e2e fix under HOMELAB-1878. Flagging `telemetry.spec.ts:275` — the `[data-testid="wave-row-note"]` selector not resolving — as worth a separate ticket if it recurs.

aaron commented 2026-10-03 17:17:23 +00:00

Author

Owner

Copy link

Review verdict — APPROVED, no blockers

Landing review for HOMELAB-1878 (fresh context, primary sources only: PR diff, ticket acceptance criteria, live branch protection). Reviewed head sha: f5e6fc80e8a10f9579f1b2a932521799cfd6ccaa.

Scope checked. 1 file changed: core/manifests/longhorn-recurring-jobs.yaml (+12 / -2). No other path touched.

Correctness vs acceptance criteria (ticket Verify: "Retention PR merged and applied, snapshot count drops over following days"; scope item: "daily-snapshot-databases retain 14 → lower"):

• daily-snapshot-databases → retain: 7 (was 14) — matches the ticket's stated target; comment block explains the rationale (retained-snapshot block retention × daily write volume; CNPG WAL volumes at 1.6–1.9× spec).
• daily-backup-b2 untouched: retain: 7, task: backup, cron: "0 4 * * *" — the off-cluster RPO floor is unchanged, as the ticket's rationale requires.
• Other jobs unchanged: daily-snapshot-apps 7, daily-snapshot-bulk 3, weekly-snapshot 4 (group-a depth retained deliberately, noted in the new comment).
• Manifest parses as 5 valid YAML docs (yaml.safe_load_all), resource kinds/names intact; yaml-lint + helm-validate green.

CI on head. All four status contexts required by live branch protection are green on f5e6fc8: CI Review / helm-validate, Lint & Validate / terraform-validate, yaml-lint, shellcheck (runs #1774 + #1775). The only red context, CI Review / cockpit-e2e, is not a required check and is unrelated to this diff (zero file overlap — cockpit Svelte DOM assertion, telemetry.spec.ts:295; see the CI note above). Documented and accepted; not a blocker.

Operator approval. Noted — the human operator explicitly reviewed and approved this retention reduction and authorised the merge.

Verdict: approved / no blockers. Landing worker proceeds to merge plane/HOMELAB-1878-retention → live (merge commit, branch retained per repo convention) and watches the longhorn-config ArgoCD continuation.

## Review verdict — APPROVED, no blockers Landing review for **HOMELAB-1878** (fresh context, primary sources only: PR diff, ticket acceptance criteria, `live` branch protection). Reviewed head sha: `f5e6fc80e8a10f9579f1b2a932521799cfd6ccaa`. **Scope checked.** 1 file changed: `core/manifests/longhorn-recurring-jobs.yaml` (+12 / -2). No other path touched. **Correctness vs acceptance criteria** (ticket *Verify*: "Retention PR merged and applied, snapshot count drops over following days"; scope item: "daily-snapshot-databases retain 14 → lower"): - `daily-snapshot-databases` → `retain: 7` (was 14) — matches the ticket's stated target; comment block explains the rationale (retained-snapshot block retention × daily write volume; CNPG WAL volumes at 1.6–1.9× spec). - `daily-backup-b2` **untouched**: `retain: 7`, `task: backup`, `cron: "0 4 * * *"` — the off-cluster RPO floor is unchanged, as the ticket's rationale requires. - Other jobs unchanged: `daily-snapshot-apps` 7, `daily-snapshot-bulk` 3, `weekly-snapshot` 4 (group-a depth retained deliberately, noted in the new comment). - Manifest parses as 5 valid YAML docs (`yaml.safe_load_all`), resource kinds/names intact; `yaml-lint` + `helm-validate` green. **CI on head.** All four status contexts **required** by `live` branch protection are green on `f5e6fc8`: `CI Review / helm-validate`, `Lint & Validate / terraform-validate`, `yaml-lint`, `shellcheck` (runs #1774 + #1775). The only red context, `CI Review / cockpit-e2e`, is **not** a required check and is unrelated to this diff (zero file overlap — cockpit Svelte DOM assertion, `telemetry.spec.ts:295`; see the CI note above). Documented and accepted; not a blocker. **Operator approval.** Noted — the human operator explicitly reviewed and approved this retention reduction and authorised the merge. **Verdict: approved / no blockers.** Landing worker proceeds to merge `plane/HOMELAB-1878-retention` → `live` (merge commit, branch retained per repo convention) and watches the `longhorn-config` ArgoCD continuation.

aaron referenced this pull request 2026-10-03 17:42:39 +00:00

HOMELAB-1885: fix(cockpit-e2e): wait for the SPA to mount before measuring the honest row #601

aaron added 1 commit 2026-10-03 18:09:57 +00:00

Merge remote-tracking branch 'origin/live' into plane/HOMELAB-1878-retention 1ff31ed1d0

aaron added 1 commit 2026-10-03 18:30:03 +00:00

Merge remote-tracking branch 'origin/live' into plane/HOMELAB-1878-retention 48f9b34a17

aaron commented 2026-10-03 18:51:49 +00:00

Author

Owner

Copy link

Review verdict — HOMELAB-1878, PR #593

Reviewed against HOMELAB-1878 acceptance criteria. Verdict: approved. No open findings; all dimensions reviewed.

Scope (verified at head 48f9b34a17c8a35bda85c1f3632e9b8cc0867a66, not from the PR description): diff vs live touches exactly one path — core/manifests/longhorn-recurring-jobs.yaml (+12 / −2). No other file in the PR.

Acceptance criteria

• daily-snapshot-databases.spec.retain 14 → 7 — correct per the ticket's "retain 14 → lower" scope. Independently parsed with yaml.safe_load_all at the head sha: value is 7.
• daily-backup-b2 untouched — confirmed at head: task=backup, retain=7, cron="0 4 * * *", groups [group-a, group-b, group-c, default]. The off-cluster B2 copy that backs the RPO floor is not modified by this change.
• No collateral change — all 5 recurring jobs enumerated at head, names and order stable:

job	task	retain	cron	groups
daily-snapshot-databases	snapshot	7 (was 14)	0 1 * * *	group-a
daily-snapshot-apps	snapshot	7	0 2 * * *	group-b
daily-snapshot-bulk	snapshot	3	0 3 * * *	group-c
weekly-snapshot	snapshot	4	0 5 * * 0	a,b,c
daily-backup-b2	backup	7	0 4 * * *	a,b,c,default

• YAML valid — yamllint -d relaxed exits 0; the 7 warnings are all pre-existing line-length at 81–94 chars.
• No secret, RBAC, network-policy, or schema surface in the diff.
• The updated header comment replaces the now-false claim ("highest retention because RPO matters most") with the HOMELAB-1878 rationale — the comment matches the code.

Approval: this change reduces on-cluster recovery-point depth for group-a database volumes, so it is not harmless/additive. The operator explicitly reviewed and approved it, with daily-backup-b2 plus barman WAL PITR as independent off-cluster recovery paths.

CI: re-verified on the current head (merged up to live tip 32773157ce). Both workflows green — ci.yaml #1886 6/6 (incl. cockpit-e2e, 2m50s) and lint.yaml #1887 3/3 (incl. terraform-validate, which needed the init-retry fix from #591 — ran 20m23s). 9/9 required contexts success.

## Review verdict — HOMELAB-1878, PR #593 **Reviewed against HOMELAB-1878 acceptance criteria. Verdict: approved. No open findings; all dimensions reviewed.** **Scope (verified at head `48f9b34a17c8a35bda85c1f3632e9b8cc0867a66`, not from the PR description):** diff vs `live` touches exactly one path — `core/manifests/longhorn-recurring-jobs.yaml` (+12 / −2). No other file in the PR. **Acceptance criteria** - `daily-snapshot-databases.spec.retain` `14 → 7` — correct per the ticket's "retain 14 → lower" scope. Independently parsed with `yaml.safe_load_all` at the head sha: value is `7`. - `daily-backup-b2` untouched — confirmed at head: `task=backup`, `retain=7`, `cron="0 4 * * *"`, groups `[group-a, group-b, group-c, default]`. The off-cluster B2 copy that backs the RPO floor is not modified by this change. - No collateral change — all 5 recurring jobs enumerated at head, names and order stable: | job | task | retain | cron | groups | |---|---|---|---|---| | daily-snapshot-databases | snapshot | **7** (was 14) | `0 1 * * *` | group-a | | daily-snapshot-apps | snapshot | 7 | `0 2 * * *` | group-b | | daily-snapshot-bulk | snapshot | 3 | `0 3 * * *` | group-c | | weekly-snapshot | snapshot | 4 | `0 5 * * 0` | a,b,c | | daily-backup-b2 | backup | 7 | `0 4 * * *` | a,b,c,default | - YAML valid — `yamllint -d relaxed` exits 0; the 7 warnings are all pre-existing `line-length` at 81–94 chars. - No secret, RBAC, network-policy, or schema surface in the diff. - The updated header comment replaces the now-false claim ("highest retention because RPO matters most") with the HOMELAB-1878 rationale — the comment matches the code. **Approval:** this change reduces on-cluster recovery-point depth for `group-a` database volumes, so it is not harmless/additive. The operator explicitly reviewed and approved it, with `daily-backup-b2` plus barman WAL PITR as independent off-cluster recovery paths. **CI:** re-verified on the current head (merged up to `live` tip `32773157ce`). Both workflows green — `ci.yaml` #1886 6/6 (incl. `cockpit-e2e`, 2m50s) and `lint.yaml` #1887 3/3 (incl. `terraform-validate`, which needed the init-retry fix from #591 — ran 20m23s). 9/9 required contexts success.

aaron merged commit a69440ff0a into live 2026-10-03 18:51:59 +00:00

aaron referenced this pull request from a commit 2026-10-03 18:52:00 +00:00

Merge pull request 'HOMELAB-1878: chore(longhorn): reduce database snapshot retention 14d -> 7d' (#593) from plane/HOMELAB-1878-retention into live

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

operator-review

Requires human operator review before merge — dangerous class (RBAC, secrets, network policy, Terraform) or explicitly gated. Never auto-merge.

No labels

operator-review

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Homelab/infra-core!593

No description provided.