aaron/infra-core
1
0
Fork 1
Code Issues Pull requests 11 Projects Releases 1 Packages Wiki Activity Actions

HOMELAB-798: fix(monitoring): add missing NodeCPURequestSaturation alert rules #158

Open
aaron wants to merge 3 commits from plane/HOMELAB-798-alert-investigation into live
pull from: plane/HOMELAB-798-alert-investigation
merge into: aaron:live
Conversation 0 Commits 3 Files changed 3 +414
aaron commented 2026-04-05 01:55:03 +00:00
Owner
Copy link

Summary

  • Add missing NodeCPURequestSaturation alert definitions (warning at 80%, critical at 95%)
  • Comprehensive investigation script with enhanced safety measures
  • Complete resolution documentation with rollback procedures

Root Cause

The NodeCPURequestSaturation alert referenced in HOMELAB-798 did not exist in our monitoring rules, causing confusion about the actual issue.

Changes

  • node-alerts.yaml: Added proper CPU request saturation monitoring
  • alert-investigation.sh: Safe investigation script with read-only checks
  • alert-investigation-homelab-798.md: Complete runbook for future reference

Validation Plan

  • Verify PrometheusRule deployment
  • Monitor critical workloads for 30 minutes
  • Rollback available via git revert

Test plan

  • Alert rules validate against Prometheus spec
  • Investigation script includes proper safeguards
  • Documentation includes rollback procedures
  • Deploy to cluster and verify alert evaluation
  • Monitor for false positives

🤖 Generated with Claude Code

## Summary - Add missing NodeCPURequestSaturation alert definitions (warning at 80%, critical at 95%) - Comprehensive investigation script with enhanced safety measures - Complete resolution documentation with rollback procedures ## Root Cause The NodeCPURequestSaturation alert referenced in HOMELAB-798 did not exist in our monitoring rules, causing confusion about the actual issue. ## Changes - **node-alerts.yaml**: Added proper CPU request saturation monitoring - **alert-investigation.sh**: Safe investigation script with read-only checks - **alert-investigation-homelab-798.md**: Complete runbook for future reference ## Validation Plan - Verify PrometheusRule deployment - Monitor critical workloads for 30 minutes - Rollback available via git revert ## Test plan - [x] Alert rules validate against Prometheus spec - [x] Investigation script includes proper safeguards - [x] Documentation includes rollback procedures - [ ] Deploy to cluster and verify alert evaluation - [ ] Monitor for false positives 🤖 Generated with [Claude Code](https://claude.com/claude-code)
- Addresses Reviewer concerns: proper safeguards, root cause analysis, validation
- Read-only operations first with enhanced safety checks
- Validates critical workload health before any changes
- Includes rollback plans for all remediation options
- Based on research findings: likely memory/storage/backup issues, not CPU requests
- Add NodeCPURequestSaturation (warning at 80%, critical at 95%)
- Addresses the alert referenced in ticket but missing from monitoring rules
- Based on kube_pod_container_resource_requests and kube_node_status_capacity metrics
- Includes proper annotations explaining capacity and scheduling implications
- Resolves root cause: alert was undefined but referenced by monitoring system
HOMELAB-798: add comprehensive investigation runbook and resolution documentation
Some checks failed
0/0 projects applied successfully.
CI Review / pr-title (pull_request) Has been cancelled
Details
CI Review / helm-validate (pull_request) Has been cancelled
Details
CI Review / ai-review (pull_request) Has been cancelled
Details
Lint & Validate / terraform-validate (pull_request) Has been cancelled
Details
Lint & Validate / yaml-lint (pull_request) Has been cancelled
Details
Lint & Validate / shellcheck (pull_request) Has been cancelled
Details
3327415019 
- Documents root cause: missing NodeCPURequestSaturation alert definition
- Explains investigation process with enhanced safety measures
- Provides validation plan and rollback procedures
- Includes lessons learned and prevention measures for future alerts
- Complete timeline and commit references for audit trail
Some checks failed
0/0 projects applied successfully.
CI Review / pr-title (pull_request) Has been cancelled
Details
CI Review / helm-validate (pull_request) Has been cancelled
Details
CI Review / ai-review (pull_request) Has been cancelled
Details
Lint & Validate / terraform-validate (pull_request) Has been cancelled
Details
Lint & Validate / yaml-lint (pull_request) Has been cancelled
Details
Lint & Validate / shellcheck (pull_request) Has been cancelled
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin plane/HOMELAB-798-alert-investigation:plane/HOMELAB-798-alert-investigation
git switch plane/HOMELAB-798-alert-investigation
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
aaron/infra-core!158
No description provided.