aaron/infra-core
1
0
Fork 2
Code Issues Pull requests 12 Projects Releases 1 Packages Wiki Activity Actions

feat(alerts): add CNPGWALArchiveFailing + remove outline exclusion #205

Merged
aaron merged 1 commit from fix/cnpg-wal-archive-alert into live 2026-05-13 01:26:29 +00:00
Conversation 0 Commits 1 Files changed 1 +9 -1
aaron commented 2026-05-13 01:26:14 +00:00
Owner
Copy link

Adds CNPGWALArchiveFailing alert that fires when WAL archiving is retrying but continuously failing (e.g. barman exit status 4 due to B2 storage cap). The existing CNPGBackupFailing only catches when archiving is completely stopped (is_archiving==0).

Also removes the arbitrary namespace!="outline" exclusion from CNPGLastBackupOld so Outline clusters are monitored too.

Related incident: 2026-05-12 B2 cap depletion caused WAL files to accumulate locally, filling 4Gi PVCs and bringing down Forgejo, Zitadel, and Temporal databases.

Adds `CNPGWALArchiveFailing` alert that fires when WAL archiving is retrying but continuously failing (e.g. barman exit status 4 due to B2 storage cap). The existing `CNPGBackupFailing` only catches when archiving is completely stopped (`is_archiving==0`). Also removes the arbitrary `namespace!="outline"` exclusion from `CNPGLastBackupOld` so Outline clusters are monitored too. **Related incident**: 2026-05-12 B2 cap depletion caused WAL files to accumulate locally, filling 4Gi PVCs and bringing down Forgejo, Zitadel, and Temporal databases.
feat(alerts): add CNPGWALArchiveFailing + remove outline exclusion
Some checks failed
0/0 projects applied successfully.
CI Review / pr-title (pull_request) Has been cancelled
Details
CI Review / helm-validate (pull_request) Has been cancelled
Details
CI Review / ai-review (pull_request) Has been cancelled
Details
Lint & Validate / terraform-validate (pull_request) Has been cancelled
Details
Lint & Validate / yaml-lint (pull_request) Has been cancelled
Details
Lint & Validate / shellcheck (pull_request) Has been cancelled
Details
46b589d5a4 
Adds CNPGWALArchiveFailing alert that fires when WAL archiving
is retrying but continuously failing (e.g. barman exit status 4
due to B2 storage cap). The existing CNPGBackupFailing only
catches when archiving is completely stopped (is_archiving==0).

Also removes the arbitrary namespace!="outline" exclusion from
CNPGLastBackupOld so Outline clusters are monitored too.

Refs: incident 2026-05-12 B2 cap depletion
aaron merged commit d5914122a1 into live 2026-05-13 01:26:29 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
aaron/infra-core!205
No description provided.