aaron/infra-core
1
0
Fork 2
Code Issues Pull requests 12 Projects Releases 1 Packages Wiki Activity Actions

fix: prevent PG crash cascade from MinIO OOMKill disk saturation #171

Merged
aaron merged 1 commit from fix/minio-oom-pg-cascade into live 2026-04-10 17:15:32 +00:00
Conversation 0 Commits 1 Files changed 2 +8 -1
aaron commented 2026-04-10 17:13:26 +00:00
Owner
Copy link

Summary

  • Bump MinIO memory limit from 384Mi to 1Gi (was getting OOMKilled, saturating vda on prod-wk-02)
  • Bump CNPG WAL storage from 2Gi to 4Gi for additional headroom

Root cause

MinIO OOMKills → containerd overlay rebuild → 112 MB/s reads on system disk → Longhorn 12s write latency → ext4 read-only → PostgreSQL PANIC on all three CNPG clusters for 9 hours

## Summary - Bump MinIO memory limit from 384Mi to 1Gi (was getting OOMKilled, saturating vda on prod-wk-02) - Bump CNPG WAL storage from 2Gi to 4Gi for additional headroom ## Root cause MinIO OOMKills → containerd overlay rebuild → 112 MB/s reads on system disk → Longhorn 12s write latency → ext4 read-only → PostgreSQL PANIC on all three CNPG clusters for 9 hours
fix: prevent PG crash cascade from MinIO OOMKill disk saturation
Some checks failed
0/0 projects applied successfully.
CI Review / pr-title (pull_request) Has been cancelled
Details
CI Review / helm-validate (pull_request) Has been cancelled
Details
CI Review / ai-review (pull_request) Has been cancelled
Details
Lint & Validate / terraform-validate (pull_request) Has been cancelled
Details
Lint & Validate / yaml-lint (pull_request) Has been cancelled
Details
Lint & Validate / shellcheck (pull_request) Has been cancelled
Details
5dbcea18dd 
MinIO (langfuse-s3) was OOMKilled repeatedly due to chart-default 384Mi
memory limit. Each restart triggered 112 MB/s overlay reads on vda,
saturating the system disk on prod-wk-02. This caused Longhorn write
latencies to spike to 12+ seconds, flipping ext4 WAL filesystems
read-only and crashing all three CNPG clusters (forgejo, temporal,
zitadel) for 9 hours.

- Bump MinIO memory limit from 384Mi to 1Gi
- Bump CNPG WAL storage from 2Gi to 4Gi for additional headroom

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
aaron merged commit a2f7bfa5be into live 2026-04-10 17:15:32 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
aaron/infra-core!171
No description provided.