aaron/infra-core
1
0
Fork 2
Code Issues Pull requests 12 Projects Releases 1 Packages Wiki Activity Actions

longhorn: restore throttle to 2/node + best-effort auto-balance #198

Merged
aaron merged 1 commit from plane/longhorn-restore-throttle-after-resilver into live 2026-04-28 21:23:11 +00:00
Conversation 0 Commits 1 Files changed 1 +5 -8
aaron commented 2026-04-28 21:22:55 +00:00
Owner
Copy link

Summary

  • Lifts the emergency 1/node + auto-balance=disabled cap from PR #197 (commit 8bdfe9f)
  • Settles at 2/node (chart default is 5) — proven safe on daytona ZFS-backed pools
  • replicaAutoBalance: best-effort re-enabled for normal cluster rebalancing

Why now

  • ssd-mirror redundancy restored: Bestoss UB205-1TB25N00350 cable reseated 2026-04-28, daytona rebooted to force SATA POST, ZFS auto-imported and resilvered 24.7G with 0 errors
  • Longhorn rebuild backlog drained 20 -> 0 degraded
  • Cluster fully recovered: 137 Running, 0 EROFS hits across all 4 un-hibernated CNPGs

Why 2 and not 5

daytona ZFS pools sustain ~30 MB/s sequential. The original incident proved 5 saturates the spindle, times out engine writes, and triggers an EROFS storm. 2 leaves headroom for workload I/O while still allowing reasonable rebuild throughput.

Test plan

  • ArgoCD reconciles longhorn app cleanly
  • kubectl get settings.longhorn.io -n longhorn-system replica-auto-balance concurrent-replica-rebuild-per-node-limit shows new values
  • No new degraded volumes triggered
  • Cluster pod state remains 137+ Running, 0 EROFS

Follow-up to HOMELAB-1003 postmortem ticket. HOMELAB-860 (hdd-mirror sdb gap) tracked separately.

## Summary - Lifts the emergency 1/node + auto-balance=disabled cap from PR #197 (commit 8bdfe9f) - Settles at 2/node (chart default is 5) — proven safe on daytona ZFS-backed pools - replicaAutoBalance: best-effort re-enabled for normal cluster rebalancing ## Why now - ssd-mirror redundancy restored: Bestoss UB205-1TB25N00350 cable reseated 2026-04-28, daytona rebooted to force SATA POST, ZFS auto-imported and resilvered 24.7G with 0 errors - Longhorn rebuild backlog drained 20 -> 0 degraded - Cluster fully recovered: 137 Running, 0 EROFS hits across all 4 un-hibernated CNPGs ## Why 2 and not 5 daytona ZFS pools sustain ~30 MB/s sequential. The original incident proved 5 saturates the spindle, times out engine writes, and triggers an EROFS storm. 2 leaves headroom for workload I/O while still allowing reasonable rebuild throughput. ## Test plan - [ ] ArgoCD reconciles longhorn app cleanly - [ ] kubectl get settings.longhorn.io -n longhorn-system replica-auto-balance concurrent-replica-rebuild-per-node-limit shows new values - [ ] No new degraded volumes triggered - [ ] Cluster pod state remains 137+ Running, 0 EROFS Follow-up to HOMELAB-1003 postmortem ticket. HOMELAB-860 (hdd-mirror sdb gap) tracked separately.
fix(longhorn): restore throttle to 2/node + best-effort auto-balance
Some checks failed
0/0 projects applied successfully.
CI Review / pr-title (pull_request) Has been cancelled
Details
CI Review / helm-validate (pull_request) Has been cancelled
Details
CI Review / ai-review (pull_request) Has been cancelled
Details
Lint & Validate / terraform-validate (pull_request) Has been cancelled
Details
Lint & Validate / yaml-lint (pull_request) Has been cancelled
Details
Lint & Validate / shellcheck (pull_request) Has been cancelled
Details
93f21c61cb 
Both ZFS pools healthy (ssd-mirror resilvered 2026-04-28 with both Bestoss
disks ONLINE, hdd-mirror redundancy work tracked separately). Lifts the
emergency 1/node + auto-balance=disabled cap from 8bdfe9f.

Settles at 2/node (chart default is 5) because daytona's ZFS-backed pools
sustain ~30 MB/s sequential — the 2026-04-27 incident proved 5 saturates
the spindle and triggers an EROFS storm.
aaron merged commit eccd596d28 into live 2026-04-28 21:23:11 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
aaron/infra-core!198
No description provided.