fix(longhorn): single-replica defaults + restore salvage lever #199

Merged

aaron merged 1 commit from plane/longhorn-singlerep-iac into live

2026-05-01 04:27:43 +00:00

aaron commented

2026-05-01 04:27:40 +00:00

Owner

After 2026-04-30 plane recovery: 4 postgres restore attempts on a 17 GB pg_dumpall failed with EROFS partway, on bitnami AND CNPG storage classes.

Root cause is structural: 2-replica Longhorn on top of single ZFS mirror = 2x writes, 0 fault tolerance gain (single Proxmox host).

Changes:

defaultReplicaCount + persistence.defaultClassReplicaCount: 2 -> 1
replicaAutoBalance: best-effort -> disabled
concurrentReplicaRebuildPerNodeLimit: 2 -> 1
disableRevisionCounter: implicit-true -> explicit false (restore salvage lever)
engineReplicaTimeout: 8 -> 15 (tolerates fsync stalls)
longhorn-db SC: numberOfReplicas 2 -> 1

Forward-only on Longhorn-managed defaults. Existing volumes keep current spec; will bulk-patch in follow-up.

Refs HOMELAB-1003.

After 2026-04-30 plane recovery: 4 postgres restore attempts on a 17 GB pg_dumpall failed with EROFS partway, on bitnami AND CNPG storage classes. Root cause is structural: 2-replica Longhorn on top of single ZFS mirror = 2x writes, 0 fault tolerance gain (single Proxmox host). Changes: - defaultReplicaCount + persistence.defaultClassReplicaCount: 2 -> 1 - replicaAutoBalance: best-effort -> disabled - concurrentReplicaRebuildPerNodeLimit: 2 -> 1 - disableRevisionCounter: implicit-true -> explicit false (restore salvage lever) - engineReplicaTimeout: 8 -> 15 (tolerates fsync stalls) - longhorn-db SC: numberOfReplicas 2 -> 1 Forward-only on Longhorn-managed defaults. Existing volumes keep current spec; will bulk-patch in follow-up. Refs HOMELAB-1003.

aaron added 1 commit

2026-05-01 04:27:41 +00:00

fix(longhorn): single-replica defaults + restore salvage lever

0/0 projects applied successfully.

CI Review / pr-title (pull_request) Has been cancelled

Details

CI Review / helm-validate (pull_request) Has been cancelled

Details

CI Review / ai-review (pull_request) Has been cancelled

Details

Lint & Validate / terraform-validate (pull_request) Has been cancelled

Details

Lint & Validate / yaml-lint (pull_request) Has been cancelled

Details

Lint & Validate / shellcheck (pull_request) Has been cancelled

Details

fc8d5e2ced

After 2026-04-30 plane recovery: 4 postgres restore attempts on a 17 GB
pg_dumpall failed with EROFS partway, on bitnami AND CNPG storage classes.
Root cause is structural: 2-replica Longhorn on top of a single ZFS
mirror doubles write I/O for zero fault tolerance gain (single Proxmox
host). Replica rebuilds during the writes saturate the SSDs and cause
guest filesystems to remount read-only.

Changes (per Longhorn specialist + architect review):
- defaultReplicaCount: 2 -> 1, persistence.defaultClassReplicaCount: 2 -> 1
  (kills phantom redundancy)
- replicaAutoBalance: best-effort -> disabled
  (rebalances trigger I/O storms; ZFS handles physical layout)
- concurrentReplicaRebuildPerNodeLimit: 2 -> 1
  (postgres can keep WAL flushed during rebuild)
- disableRevisionCounter: implicit-true -> explicit false
  (restores salvage lever lost during plane-pgdb 2026-04-29)
- engineReplicaTimeout: 8 -> 15
  (tolerates fsync stalls under SSD-mirror saturation)
- longhorn-db SC: numberOfReplicas 2 -> 1, disableRevisionCounter false

This is forward-only on Longhorn-managed defaults. Existing volumes
keep their current spec.numberOfReplicas; we'll bulk-patch them in a
follow-up step after these settings sync.

Refs HOMELAB-1003 (postmortem).