longhorn: throttle rebuild concurrency + disable auto-balance during recovery #197

Merged

aaron merged 1 commit from plane/longhorn-throttle-recovery into live

2026-04-28 03:02:44 +00:00

aaron commented

2026-04-28 03:02:22 +00:00

Owner

Throttle Longhorn during the EROFS-storm recovery on daytona.

Root cause (per K8s + Longhorn specialist diagnosis): both ZFS pools degraded — hdd-mirror single-disk (HOMELAB-860 sdb detach), ssd-mirror has one UNAVAIL Bestoss SSD. Sustained throughput ~30 MB/s. With concurrentReplicaRebuildPerNodeLimit=5, the HDD spindle saturates → engine writes time out → kernel marks PVCs read-only → CNPG/redis/rabbit pods crashloop.

Change: rebuild concurrency 5→1, auto-balance best-effort→disabled (already inert while volumes are degraded; removes controller pressure).

Restore criteria: revert to concurrentReplicaRebuildPerNodeLimit: 2 + replicaAutoBalance: best-effort once (a) both ZFS pools are resilvered and (b) Longhorn rebuild backlog drains to <5 degraded volumes.

Throttle Longhorn during the EROFS-storm recovery on daytona. **Root cause** (per K8s + Longhorn specialist diagnosis): both ZFS pools degraded — hdd-mirror single-disk (HOMELAB-860 sdb detach), ssd-mirror has one UNAVAIL Bestoss SSD. Sustained throughput ~30 MB/s. With `concurrentReplicaRebuildPerNodeLimit=5`, the HDD spindle saturates → engine writes time out → kernel marks PVCs read-only → CNPG/redis/rabbit pods crashloop. **Change**: rebuild concurrency 5→1, auto-balance best-effort→disabled (already inert while volumes are degraded; removes controller pressure). **Restore criteria**: revert to `concurrentReplicaRebuildPerNodeLimit: 2` + `replicaAutoBalance: best-effort` once (a) both ZFS pools are resilvered and (b) Longhorn rebuild backlog drains to <5 degraded volumes.

aaron added 1 commit

2026-04-28 03:02:24 +00:00

fix(longhorn): throttle rebuild concurrency to 1 + disable auto-balance

0/0 projects applied successfully.

CI Review / pr-title (pull_request) Has been cancelled

Details

CI Review / helm-validate (pull_request) Has been cancelled

Details

CI Review / ai-review (pull_request) Has been cancelled

Details

Lint & Validate / terraform-validate (pull_request) Has been cancelled

Details

Lint & Validate / yaml-lint (pull_request) Has been cancelled

Details

Lint & Validate / shellcheck (pull_request) Has been cancelled

Details

8bdfe9f72f

Today's EROFS storm: with both ZFS pools degraded on daytona (hdd-mirror
single-disk after HOMELAB-860 sdb detach, ssd-mirror has one UNAVAIL Bestoss
SSD), the underlying disk sustains ~30 MB/s. concurrentReplicaRebuildPerNodeLimit=5
saturated the hdd-mirror spindle, engine sync writes timed out, kernel marked
multiple PVCs read-only (postgres CNPGs, harbor-redis, plane-rabbitmq Mnesia).

Throttle to 1 in-flight rebuild per node so the disk can serve foreground
sync writes alongside replica rebuilds. Disable auto-balance during the
recovery window — it's already inert while volumes are degraded
("Cannot auto-balance volume in degraded state" log spam) but remove
the controller pressure entirely.

Restore to concurrency=2 + replicaAutoBalance=best-effort once both ZFS
pools are resilvered and the rebuild backlog has drained.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>