Homelab/infra-core
6
0
Fork 2
Code Issues Pull requests 35 Projects Releases 1 Packages Wiki Activity Actions

monitoring: KimiPoolDegraded alert + OmniRoute kimi-pool Vector counters (IR-20261112-kimipool) #965

Open
pi-landing wants to merge 1 commit from ld/IR-20261112-kimipool into live
pull from: ld/IR-20261112-kimipool
merge into: Homelab:live
Conversation 0 Commits 1 Files changed 2 +651
pi-landing commented 2026-11-12 16:17:51 +00:00
Collaborator
Copy link

monitoring: KimiPoolDegraded alert + OmniRoute kimi-pool Vector counters (IR-20261112-kimipool)

Base: 2d837e9bc0e199f14cc5f4927efbaea59edec0de (local origin/live, commit date
2026-11-12 06:42:28 +0000 — 9 h 31 min old when this pack was built, no fetch).
Patch sha256: 318106bbf147933383284d58b7ed4fe962ad99834ed357d26b16ae7d93aa7242
(kimi-pool-alert.patch, one commit 46bed1d53f64c25d9893aea5d04f6ff4bb200b8b).
Built by homelab-88's lane — this pack was built by homelab-88's lane
(orch38 KIMIALERT handoff-packaging worker, 2026-11-12T16:1xZ). Apply with
git am on a fresh clone at the base sha.

What and why

On 2026-11-12 the OmniRoute combo kimi-pool failed calls with upstream 429
("usage limit") while a different Kimi connection with quota served calls
outside the pool. Nothing observed it:

  • the OmniRoute gateway exposes no /metrics endpoint, and Prometheus carries no
    omniroute_* quota series (measured 2026-11-05);
  • the only evidence is the gateway container's own stdout.

So this change ships a log-metric shipper for exactly that log stream and one
PrometheusRule over the counters it produces. Two new files, no other change.

Design summary

core/manifests/monitoring/log-metrics/vector-omniroute-log-metrics.yaml
(ConfigMap + Service + ServiceMonitor + DaemonSet, all named
vector-omniroute-log-metrics, namespace monitoring)

  • Vector 0.61.0-style config in the ConfigMap: file source over
    /var/log/pods (hostPath, readOnly), filtered to the omniroute pod stream,
    then a log_to_metric transform that names every counter explicitly — no
    invented metric names, and vector test gives the parser a fail-capable
    offline control.
  • Three counters, sink namespace omniroute_logs, scraped by a ServiceMonitor in
    edge mode (one target per pod) so node/pod labels are real:
    • omniroute_kimi_pool_all_failed_total{node,pod} — the dispatcher's terminal
      All models failed line with a kimi target in the failure chain;
    • omniroute_kimi_monthly_limit_total{node,pod,connection} — a usage limit
      line whose text says month; connection is the short 8-hex form when the
      log names one, else unknown;
    • omniroute_kimi_success_total{node,pod} — a kimi success line (leg (a) needs
      a "no success in the window" test and the cluster has no other success
      signal).
  • Scope discipline: read-only hostPath, no API access, no RBAC, no service
    account token (automountServiceAccountToken: false), only the omniroute
    stream tailed.
  • Same convention as the existing vector-longhorn.yaml: any ConfigMap edit must
    recompute the checksum/config annotation on the pod template (done; the
    package test script recomputes it and fails on a mismatch).

core/manifests/monitoring/rules/kimi-pool-alerts.yaml (PrometheusRule
kimi-pool-degraded, label release: kube-prometheus-stack, group kimi-pool)

  • KimiPoolDegraded, severity=warning, non-paging, two legs:
    • (a) omniroute_kimi_pool_all_failed_total moved inside the window and
      no kimi success in the window — the live detector;
    • (b) a monthly usage limit on a connection other than the known-exhausted
      c759b8d7, expressed as one clearly-marked regex on the connection label
      with its evidence and a review date (a dated fact, not a constant; unknown
      is excluded by decision, see open items).
  • KimiPoolLogMetricShipperDown — coverage rule: the counters are the only
    signal, so a missing scrape target must be loud rather than silent.
  • Why warning and not critical: one combo with wrong target selection is a
    routing defect, not an outage (the caller can switch to deepseek-flash with
    no operator cost), and the critical paging rules over the gateway itself are
    already in omniroute-alerts.yaml. In the live Alertmanager route tree only
    severity="critical" (plus a few named routes) reaches Signal, so warning
    records and does not page. Do not promote it.

No wiring, and why

core/manifests/monitoring/rules and core/manifests/monitoring/log-metrics
have no kustomization.yaml (only monitoring/factory/ has one, for its own
Application). Both directories are ArgoCD directory sources — prod
apps/monitoring-rules.yaml and apps/monitoring-log-metrics.yaml use a plain
path: with the default recurse: false — so the directory listing is the
manifest set and both new files are picked up by the existing Applications.
No prod change and no kustomization wiring is required; the diff is two new
files and nothing else.

Verification (run against this commit's content)

Check Result Evidence
promtool 3.4.0 check rules + test rules 11/11 PASS rule-tests/promtool-test.sh (extracts spec.groups verbatim, so the expression under test is byte-for-byte the shipped one)
offline Vector config checks 18/18 PASS rule-tests/kimi-pool-vector-config.test.py
Vector e2e, shipped config PASS rule-tests/kimi-pool-vector-e2e.sh vs timberio/vector:0.61.0 — counters 4/2/1, 3 connection tags
e2e non-vacuity 1/1 mutation caught filter guard removed → asserted 2, observed 3
render sanity (guard-sourced) PASS kimi-pool-alert/test.sh: renders both directories before/after → 29 → 34 objects, added set is exactly the 4 Vector resources + the PrometheusRule, nothing removed or altered
guard OK builder-test-harness-guard.sh sourced first; real paths untouched

The 11 promtool cases: increment, flat, birth, birth-aged, with-success,
monthly-other, monthly-birth, monthly-known-exhausted, monthly-unattributed,
no-shipper, shipper-ok. Full table and the generator's provenance are in
working-files/infra-stability/kimi-pool-alert/VERIFICATION.md (sha256 in that
package's SHA256SUMS).

The rule-test assets stay in the working-files package
(kimi-pool-alert/core/manifests/monitoring/rule-tests/); this PR intentionally
carries only the two manifests, matching the brief's minimal-diff scope. Copying
the tests into core/manifests/monitoring/rule-tests/ is a follow-up if wanted —
they are the same promtool shape the repo already keeps there.

Open items (recorded, not blocking)

  1. Live-log wording check post-landing. The month selector in leg (b) keys
    off the recorded wording [429] You've reached your <window> usage limit.
    Confirm against the live stream after the first real monthly-limit line; if the
    gateway rewords it, the transform filter needs the new token.
  2. Leg (b) is dormant by construction until the log names a connection. The
    gateway masks accounts (account: ***…) and Kimi connections were recorded as
    emitting no id at all on stdout. Until the log (or the transform) names one,
    every monthly-limit event is connection="unknown", which is excluded by
    decision — pinned by the monthly-unattributed promtool case so it is a
    recorded decision rather than an accident. This is f5's call.
  3. Minor, informational: c759b8d7 is described as kimi-a-vivace, which the
    2026-11-05 investigation recorded as a weekly limit while the 2026-11-12
    event calls the same upstream 429 a monthly limit. The ID is what the
    exclusion uses, so the window's name does not change the rule — recorded so
    the next reader does not have to rediscover the disagreement.

Safety notes for the reviewer

  • No credential path is read or referenced; no Secret, ExternalSecret or RBAC
    object is added or changed; no Terraform; no namespace, PVC or volume
    (persistent data) change — nothing prunable under ArgoCD except the new
    objects themselves.
  • Rollback is git revert of one commit: ArgoCD prunes the DaemonSet, Service,
    ServiceMonitor, ConfigMap and PrometheusRule by name. No data is involved.
monitoring: KimiPoolDegraded alert + OmniRoute kimi-pool Vector counters (IR-20261112-kimipool) Base: `2d837e9bc0e199f14cc5f4927efbaea59edec0de` (local `origin/live`, commit date 2026-11-12 06:42:28 +0000 — 9 h 31 min old when this pack was built, no fetch). Patch sha256: `318106bbf147933383284d58b7ed4fe962ad99834ed357d26b16ae7d93aa7242` (`kimi-pool-alert.patch`, one commit `46bed1d53f64c25d9893aea5d04f6ff4bb200b8b`). **Built by homelab-88's lane** — this pack was built by homelab-88's lane (orch38 KIMIALERT handoff-packaging worker, 2026-11-12T16:1xZ). Apply with `git am` on a fresh clone at the base sha. ## What and why On 2026-11-12 the OmniRoute combo `kimi-pool` failed calls with upstream 429 ("usage limit") while a *different* Kimi connection with quota served calls outside the pool. Nothing observed it: * the OmniRoute gateway exposes no `/metrics` endpoint, and Prometheus carries no `omniroute_*` quota series (measured 2026-11-05); * the only evidence is the gateway container's own stdout. So this change ships a log-metric shipper for exactly that log stream and one PrometheusRule over the counters it produces. Two new files, no other change. ## Design summary **`core/manifests/monitoring/log-metrics/vector-omniroute-log-metrics.yaml`** (ConfigMap + Service + ServiceMonitor + DaemonSet, all named `vector-omniroute-log-metrics`, namespace `monitoring`) * Vector `0.61.0`-style config in the ConfigMap: `file` source over `/var/log/pods` (hostPath, **readOnly**), filtered to the omniroute pod stream, then a `log_to_metric` transform that names every counter explicitly — no invented metric names, and `vector test` gives the parser a fail-capable offline control. * Three counters, sink namespace `omniroute_logs`, scraped by a ServiceMonitor in edge mode (one target per pod) so `node`/`pod` labels are real: * `omniroute_kimi_pool_all_failed_total{node,pod}` — the dispatcher's terminal `All models failed` line *with* a kimi target in the failure chain; * `omniroute_kimi_monthly_limit_total{node,pod,connection}` — a `usage limit` line whose text says `month`; `connection` is the short 8-hex form when the log names one, else `unknown`; * `omniroute_kimi_success_total{node,pod}` — a kimi success line (leg (a) needs a "no success in the window" test and the cluster has no other success signal). * Scope discipline: read-only hostPath, no API access, no RBAC, no service account token (`automountServiceAccountToken: false`), only the omniroute stream tailed. * Same convention as the existing `vector-longhorn.yaml`: any ConfigMap edit must recompute the `checksum/config` annotation on the pod template (done; the package test script recomputes it and fails on a mismatch). **`core/manifests/monitoring/rules/kimi-pool-alerts.yaml`** (PrometheusRule `kimi-pool-degraded`, label `release: kube-prometheus-stack`, group `kimi-pool`) * `KimiPoolDegraded`, **severity=warning, non-paging**, two legs: * (a) `omniroute_kimi_pool_all_failed_total` moved inside the window **and** no kimi success in the window — the live detector; * (b) a monthly usage limit on a connection **other than** the known-exhausted `c759b8d7`, expressed as one clearly-marked regex on the `connection` label with its evidence and a review date (a dated fact, not a constant; `unknown` is excluded by decision, see open items). * `KimiPoolLogMetricShipperDown` — coverage rule: the counters are the only signal, so a missing scrape target must be loud rather than silent. * Why warning and not critical: one combo with wrong target selection is a routing defect, not an outage (the caller can switch to `deepseek-flash` with no operator cost), and the critical paging rules over the gateway itself are already in `omniroute-alerts.yaml`. In the live Alertmanager route tree only `severity="critical"` (plus a few named routes) reaches Signal, so warning records and does not page. Do not promote it. ## No wiring, and why `core/manifests/monitoring/rules` and `core/manifests/monitoring/log-metrics` have **no** `kustomization.yaml` (only `monitoring/factory/` has one, for its own Application). Both directories are ArgoCD **directory** sources — prod `apps/monitoring-rules.yaml` and `apps/monitoring-log-metrics.yaml` use a plain `path:` with the default `recurse: false` — so the directory listing *is* the manifest set and both new files are picked up by the existing Applications. **No prod change and no kustomization wiring is required**; the diff is two new files and nothing else. ## Verification (run against this commit's content) | Check | Result | Evidence | |---|---|---| | promtool 3.4.0 `check rules` + `test rules` | **11/11 PASS** | `rule-tests/promtool-test.sh` (extracts `spec.groups` verbatim, so the expression under test is byte-for-byte the shipped one) | | offline Vector config checks | **18/18 PASS** | `rule-tests/kimi-pool-vector-config.test.py` | | Vector e2e, shipped config | **PASS** | `rule-tests/kimi-pool-vector-e2e.sh` vs `timberio/vector:0.61.0` — counters 4/2/1, 3 `connection` tags | | e2e non-vacuity | **1/1 mutation caught** | filter guard removed → asserted 2, observed 3 | | render sanity (guard-sourced) | **PASS** | `kimi-pool-alert/test.sh`: renders both directories before/after → 29 → 34 objects, added set is exactly the 4 Vector resources + the PrometheusRule, nothing removed or altered | | guard | **OK** | `builder-test-harness-guard.sh` sourced first; real paths untouched | The 11 promtool cases: increment, flat, birth, birth-aged, with-success, monthly-other, monthly-birth, monthly-known-exhausted, monthly-unattributed, no-shipper, shipper-ok. Full table and the generator's provenance are in `working-files/infra-stability/kimi-pool-alert/VERIFICATION.md` (sha256 in that package's `SHA256SUMS`). The rule-test assets stay in the working-files package (`kimi-pool-alert/core/manifests/monitoring/rule-tests/`); this PR intentionally carries only the two manifests, matching the brief's minimal-diff scope. Copying the tests into `core/manifests/monitoring/rule-tests/` is a follow-up if wanted — they are the same promtool shape the repo already keeps there. ## Open items (recorded, not blocking) 1. **Live-log wording check post-landing.** The `month` selector in leg (b) keys off the recorded wording `[429] You've reached your <window> usage limit`. Confirm against the live stream after the first real monthly-limit line; if the gateway rewords it, the transform filter needs the new token. 2. **Leg (b) is dormant by construction until the log names a connection.** The gateway masks accounts (`account: ***…`) and Kimi connections were recorded as emitting no id at all on stdout. Until the log (or the transform) names one, every monthly-limit event is `connection="unknown"`, which is excluded by decision — pinned by the `monthly-unattributed` promtool case so it is a recorded decision rather than an accident. **This is f5's call.** 3. Minor, informational: `c759b8d7` is described as `kimi-a-vivace`, which the 2026-11-05 investigation recorded as a *weekly* limit while the 2026-11-12 event calls the same upstream 429 a *monthly* limit. The ID is what the exclusion uses, so the window's name does not change the rule — recorded so the next reader does not have to rediscover the disagreement. ## Safety notes for the reviewer * No credential path is read or referenced; no Secret, ExternalSecret or RBAC object is added or changed; no Terraform; no namespace, PVC or volume (persistent data) change — nothing prunable under ArgoCD except the new objects themselves. * Rollback is `git revert` of one commit: ArgoCD prunes the DaemonSet, Service, ServiceMonitor, ConfigMap and PrometheusRule by name. No data is involved.
IR-20261112-kimipool: feat(monitoring): KimiPoolDegraded alert + OmniRoute kimi-pool Vector counters
All checks were successful
CI Review / helm-validate (pull_request) Successful in 10s
Details
Alert Disposition Contract / alert-contract (pull_request) Successful in 10s
Details
CI Review / receiver-test (pull_request) Successful in 4s
Details
CI Review / runner-manifest-test (pull_request) Successful in 9s
Details
CI Review / alert-receiver-test (pull_request) Successful in 23s
Details
CI Review / record-sql-test (pull_request) Successful in 1m31s
Details
Lint & Validate / shellcheck (pull_request) Successful in 6s
Details
Lint & Validate / yaml-lint (pull_request) Successful in 20s
Details
CI Review / cockpit-e2e (pull_request) Successful in 2m53s
Details
CI Review / cockpit-test (pull_request) Successful in 3m5s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 1m50s
Details
CI Review / dispatcher-test (pull_request) Successful in 3m42s
Details
4a5c6cdd7c 
The OmniRoute combo `kimi-pool` failed every call with upstream 429 while a
different Kimi connection with quota was serving calls outside the pool, and
nothing observed it: the gateway exposes no /metrics, Prometheus carries no
`omniroute_*` quota series (measured 2026-11-05), and the only evidence is the
gateway's own stdout.

Two new manifest files, both in directories that are already synced:

  core/manifests/monitoring/log-metrics/vector-omniroute-log-metrics.yaml
      ConfigMap + Service + ServiceMonitor + DaemonSet `vector-omniroute-log-metrics`
      tails the omniroute pod's log (hostPath /var/log/pods, readOnly, no API
      access, no service account token) and a `log_to_metric` transform exports
      three counters, namespace `omniroute_logs`, edge-mode scrape on the pod:
        omniroute_kimi_pool_all_failed_total{node,pod}
        omniroute_kimi_monthly_limit_total{node,pod,connection}
        omniroute_kimi_success_total{node,pod}

  core/manifests/monitoring/rules/kimi-pool-alerts.yaml
      PrometheusRule `kimi-pool-degraded` (label release=kube-prometheus-stack,
      group `kimi-pool`), two alerts, both severity=warning and non-paging:
        KimiPoolDegraded       leg (a): all-failed counter moved in the window and
                               no kimi success in the window; leg (b): a monthly
                               usage limit on a connection other than the known
                               exhausted `c759b8d7` (the exclusion is one marked
                               regex on the `connection` label, with its evidence
                               and review date — a dated fact, not a constant).
        KimiPoolLogMetricShipperDown  coverage: the counters are the only signal,
                               so a missing scrape target must be loud.

why warning and not critical: a single combo with wrong target selection is a
routing defect, not an outage (the caller can switch combo), and the critical
paging rules over the gateway itself are already in omniroute-alerts.yaml. In the
live Alertmanager route tree only severity=critical (plus named routes) reaches
Signal.

NO KUSTOMIZATION WIRING IS ADDED, because none exists to add to:
`core/manifests/monitoring/rules` and `core/manifests/monitoring/log-metrics`
have no kustomization.yaml. They are ArgoCD DIRECTORY sources —
prod `apps/monitoring-rules.yaml` and `apps/monitoring-log-metrics.yaml` use a
plain `path:` with recurse:false — so the directory listing is the manifest set
and both files are picked up by the existing Applications with no prod change.
(Only `monitoring/factory/` has a kustomization, for its own Application.)

Tests, run against this commit's content:
  promtool 3.4.0: check rules SUCCESS (2 rules); test rules PASS (11 cases) —
    11/11 covering increment/flat/birth/birth-aged/with-success, monthly
    other/birth/known-exhausted/unattributed, and shipper down/ok.
  offline Vector config checks: 18/18 (kimi-pool-vector-config.test.py).
  Vector e2e against the shipped config (timberio/vector:0.61.0): counters
    4/2/1 with 3 connection tags; one mutation (filter guard removed) caught.
  render sanity (working-files/.../kimi-pool-alert/test.sh, guard-sourced):
    rendering both directories before/after the change is 29 -> 34 objects, and
    the added set is exactly the 4 Vector resources + the PrometheusRule, with
    nothing removed or altered.

Open items (recorded, not blocking): confirm the live log wording for the
"month" window post-landing; leg (b) is dormant by construction until the log
names a connection (`monthly-unattributed` pins that as a decision — f5's call).

Lab-Run: IR-20261112-kimipool
All checks were successful
CI Review / helm-validate (pull_request) Successful in 10s
Required
Details
Alert Disposition Contract / alert-contract (pull_request) Successful in 10s
Details
CI Review / receiver-test (pull_request) Successful in 4s
Details
CI Review / runner-manifest-test (pull_request) Successful in 9s
Details
CI Review / alert-receiver-test (pull_request) Successful in 23s
Details
CI Review / record-sql-test (pull_request) Successful in 1m31s
Details
Lint & Validate / shellcheck (pull_request) Successful in 6s
Required
Details
Lint & Validate / yaml-lint (pull_request) Successful in 20s
Required
Details
CI Review / cockpit-e2e (pull_request) Successful in 2m53s
Details
CI Review / cockpit-test (pull_request) Successful in 3m5s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 1m50s
Required
Details
CI Review / dispatcher-test (pull_request) Successful in 3m42s
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin ld/IR-20261112-kimipool:ld/IR-20261112-kimipool
git switch ld/IR-20261112-kimipool
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
operator-review

Requires human operator review before merge — dangerous class (RBAC, secrets, network policy, Terraform) or explicitly gated. Never auto-merge.

No labels
operator-review
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Homelab/infra-core!965
No description provided.