WIP: IR-20261112: feat(monitoring): ship the Forgejo audit sink + CNP (R39 PR-B)

pi-landing commented

2026-11-12 13:05:41 +00:00

Collaborator

Scope — R39 FIX8, PR-B of the split (the Vector audit sink + ingress-only CNP)

This is PR-B of the FIX7 split of Homelab/infra-core#953 (review-cd-r6 B9 /
BLOCK-1), updated by the FIX8 builder (review-f1-r7). It carries only the
Vector Forgejo audit sink and its ingress-only CNP (plus the audit alerts and
offline tests) — the writer side of the flip.

⚠️ MERGE ORDER — unmissable

This PR's base is live, and its branch already contains PR-A's commits (the
branch is cut from PR-A's head 7f3581e8). Merging #962 before #961 lands PR-A's
auth change AND the sink in one push, which would ship the audit trail into the
anonymous fake tenant — the leak the split exists to prevent. #962 must merge
ONLY AFTER #961 (PR-A) is merged and synced, and only after the 401 proof below.
The WIP: title prefix is the mechanical guard and stays until that proof is
recorded.

⚠️ Merge gate — do NOT merge this PR on its own

This PR must NOT merge until:

PR-A (plane/R39-forgejo-access-log-auth, head 7f3581e8, PR #961) is
live (merged and synced), and
an unlabeled Loki read has been proven to return 401 on the live Loki (no
X-Scope-OrgID), while the labeled X-Scope-OrgID: main read returns 200 —
the check block below, run against the LIVE Loki after PR-A lands, is the
proof. Paste its outputs here (or in El Director's GO-ledger entry for #962)
before this PR leaves WIP.

The Vector source reads read_from: end (no backfill), so by the time it starts
tailing, Loki already refuses unlabeled writes (401) and Vector's first push
carries X-Scope-OrgID: audit-forgejo into the audit tenant — never into
fake. Merging this before PR-A's 401 proof would ship the audit trail into the
anonymous fake tenant, which is the leak the split exists to prevent.

Check block (run after PR-A is live; paste the outputs here)

kubectl -n monitoring port-forward svc/loki 3100:3100 &
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3100/loki/api/v1/labels                                  # expect 401
curl -s -o /dev/null -w '%{http_code}\n' -H 'X-Scope-OrgID: main' http://127.0.0.1:3100/loki/api/v1/labels         # expect 200
curl -s -o /dev/null -w '%{http_code}\n' -X POST -H 'Content-Type: application/json' \
  --data '{"streams":[]}' http://127.0.0.1:3100/loki/api/v1/push                                                    # expect 401 (r6 NIT-5)
curl -s -H 'X-Scope-OrgID: audit-forgejo' http://127.0.0.1:3100/loki/api/v1/labels                                  # expect {"status":"success"} with no "data" values (tenant empty)
curl -s http://127.0.0.1:3100/runtime_config | grep -A3 'audit-forgejo:'                                            # expect deletion_mode: disabled
# r39-delete-probe POST (NIT-1: PROBE selector, never {job="forgejo-access"}): expect a non-2xx code
curl -s -o /dev/null -w '%{http_code}\n' -X POST -H 'X-Scope-OrgID: audit-forgejo' 'http://127.0.0.1:3100/loki/api/v1/delete?query=%7Bjob%3D%22r39-delete-probe%22%7D&start=<now-1s>&end=<now>'

"expect 200" read-back vs Loki 3.x (FIX8 B2): the "expect 200" leg uses
/loki/api/v1/labels — NOT the instant /loki/api/v1/query endpoint — because
Loki 3.x returns 400 "log queries are not supported as an instant query type"
for a raw log-stream selector on the instant endpoint (grafana/loki#13421).
query_range (with start/end) is the documented equivalent when a log-stream
query is required; /labels is used here because it is not a log query, avoids
the 400, and keeps this block line-for-line identical to the landing runbook's
copy (that copy carries its list's 3-space indent; strip it and diff between
the two blocks is empty). The unlabeled 401 leg is unaffected by the query type
(auth is evaluated before the query is parsed).

What this PR carries

The forgejo_pod_logs source (glob /var/log/pods/forgejo_*/forgejo/*.log*,
read_from: end), the loki_forgejo_audit sink (tenant_id: audit-forgejo),
acknowledgements.enabled, and the checkpoint hostPath — all in
core/manifests/monitoring/log-metrics/vector-longhorn.yaml.
The ingress-only CNP
(core/manifests/monitoring/log-metrics/loki-ingress-networkpolicy.yaml)
admitting the enumerated pods to Loki on 3100/9095/7946 (the network boundary
complementing the tenant boundary). It is ingress-only — there is no egress
rule, and Vector's reach to Loki is limited by its own config, not by a policy.
The audit alerts (forgejo-audit-alerts.yaml) and their offline tests.

Rollback (FIX8 B1 — the rollback KEEPS the CNP)

PR-B is independently revertible as commit R1 of the two-commit rollback:
revert PR-B EXCEPT the CNP file — remove the source + sink + checkpoint
hostPath together, keep loki-ingress-networkpolicy.yaml, then confirm
sum(rate(loki_distributor_lines_received_total{tenant="audit-forgejo"}[5m])) == 0
before PR-A is reverted as R2. Never remove the audit-tenant CNP while the
tenant holds data — the retention is 45 days, and with auth still ON the
ingress-only CNP is the only thing keeping an unenumerated pod from reading the
tenant with X-Scope-OrgID: audit-forgejo. R1 does not delete the Vector
checkpoint data_dir on the nodes (/var/lib/vector-longhorn-log-metrics); a
re-land may resume from those offsets, and that is acceptable — the sink only
starts once auth is on, so the resumed lines go to audit-forgejo and never
to fake, and the end-to-end acks mean no acked line is re-sent. No
kubectl scale ds (DaemonSets have no scale subresource).

Follow-ups / notes (FIX8 NITs)

Grafana main-header CI check stays in PR-B (not moved to PR-A): moving
check-grafana-datasources.py (or its main-header assertion) into PR-A's
loki-auth-pin-test would touch the already-cleared #961, so it is not cheap;
it is left here in PR-B, where it already ships and will assert the main
header.
The LD dispatcher needs a rebuilt and redeployed image before its Deployment
scales from 0 — the running binary predates the X-Scope-OrgID: main header
and would get 401 (report "no coverage") against auth-on Loki.

## Scope — R39 FIX8, PR-B of the split (the Vector audit sink + ingress-only CNP) This is **PR-B** of the FIX7 split of `Homelab/infra-core#953` (review-cd-r6 B9 / BLOCK-1), updated by the FIX8 builder (review-f1-r7). It carries **only** the Vector Forgejo audit sink and its **ingress-only** CNP (plus the audit alerts and offline tests) — the writer side of the flip. ### ⚠️ MERGE ORDER — unmissable **This PR's base is `live`, and its branch already contains PR-A's commits** (the branch is cut from PR-A's head `7f3581e8`). Merging #962 before #961 lands PR-A's auth change AND the sink in one push, which would ship the audit trail into the anonymous `fake` tenant — the leak the split exists to prevent. **#962 must merge ONLY AFTER #961 (PR-A) is merged and synced, and only after the 401 proof below.** The `WIP:` title prefix is the mechanical guard and stays until that proof is recorded. ### ⚠️ Merge gate — do NOT merge this PR on its own **This PR must NOT merge until:** 1. **PR-A** (`plane/R39-forgejo-access-log-auth`, head `7f3581e8`, PR #961) is **live** (merged and synced), **and** 2. an **unlabeled Loki read has been proven to return 401** on the live Loki (no `X-Scope-OrgID`), while the labeled `X-Scope-OrgID: main` read returns 200 — the check block below, run against the LIVE Loki after PR-A lands, is the proof. Paste its outputs here (or in El Director's GO-ledger entry for #962) before this PR leaves WIP. The Vector source reads `read_from: end` (no backfill), so by the time it starts tailing, Loki already refuses unlabeled writes (401) and Vector's first push carries `X-Scope-OrgID: audit-forgejo` into the audit tenant — never into `fake`. Merging this before PR-A's 401 proof would ship the audit trail into the anonymous `fake` tenant, which is the leak the split exists to prevent. ### Check block (run after PR-A is live; paste the outputs here) ``` kubectl -n monitoring port-forward svc/loki 3100:3100 & curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3100/loki/api/v1/labels # expect 401 curl -s -o /dev/null -w '%{http_code}\n' -H 'X-Scope-OrgID: main' http://127.0.0.1:3100/loki/api/v1/labels # expect 200 curl -s -o /dev/null -w '%{http_code}\n' -X POST -H 'Content-Type: application/json' \ --data '{"streams":[]}' http://127.0.0.1:3100/loki/api/v1/push # expect 401 (r6 NIT-5) curl -s -H 'X-Scope-OrgID: audit-forgejo' http://127.0.0.1:3100/loki/api/v1/labels # expect {"status":"success"} with no "data" values (tenant empty) curl -s http://127.0.0.1:3100/runtime_config | grep -A3 'audit-forgejo:' # expect deletion_mode: disabled # r39-delete-probe POST (NIT-1: PROBE selector, never {job="forgejo-access"}): expect a non-2xx code curl -s -o /dev/null -w '%{http_code}\n' -X POST -H 'X-Scope-OrgID: audit-forgejo' 'http://127.0.0.1:3100/loki/api/v1/delete?query=%7Bjob%3D%22r39-delete-probe%22%7D&start=<now-1s>&end=<now>' ``` **"expect 200" read-back vs Loki 3.x (FIX8 B2):** the "expect 200" leg uses `/loki/api/v1/labels` — NOT the instant `/loki/api/v1/query` endpoint — because Loki 3.x returns **400 "log queries are not supported as an instant query type"** for a raw log-stream selector on the instant endpoint (grafana/loki#13421). `query_range` (with `start`/`end`) is the documented equivalent when a log-stream query is required; `/labels` is used here because it is not a log query, avoids the 400, and keeps this block line-for-line identical to the landing runbook's copy (that copy carries its list's 3-space indent; strip it and `diff` between the two blocks is empty). The unlabeled 401 leg is unaffected by the query type (auth is evaluated before the query is parsed). ### What this PR carries - The `forgejo_pod_logs` source (glob `/var/log/pods/forgejo_*/forgejo/*.log*`, `read_from: end`), the `loki_forgejo_audit` sink (`tenant_id: audit-forgejo`), `acknowledgements.enabled`, and the checkpoint `hostPath` — all in `core/manifests/monitoring/log-metrics/vector-longhorn.yaml`. - The **ingress-only** CNP (`core/manifests/monitoring/log-metrics/loki-ingress-networkpolicy.yaml`) admitting the enumerated pods to Loki on 3100/9095/7946 (the network boundary complementing the tenant boundary). It is ingress-only — there is no egress rule, and Vector's reach to Loki is limited by its own config, not by a policy. - The audit alerts (`forgejo-audit-alerts.yaml`) and their offline tests. ### Rollback (FIX8 B1 — the rollback KEEPS the CNP) PR-B is independently revertible as **commit R1** of the two-commit rollback: **revert PR-B EXCEPT the CNP file** — remove the source + sink + checkpoint hostPath together, keep `loki-ingress-networkpolicy.yaml`, then confirm `sum(rate(loki_distributor_lines_received_total{tenant="audit-forgejo"}[5m])) == 0` before PR-A is reverted as R2. **Never remove the audit-tenant CNP while the tenant holds data** — the retention is **45 days**, and with auth still ON the ingress-only CNP is the only thing keeping an unenumerated pod from reading the tenant with `X-Scope-OrgID: audit-forgejo`. R1 does **not** delete the Vector checkpoint `data_dir` on the nodes (`/var/lib/vector-longhorn-log-metrics`); a re-land may resume from those offsets, and that is acceptable — the sink only starts once auth is on, so the resumed lines go to `audit-forgejo` and never to `fake`, and the end-to-end acks mean no acked line is re-sent. No `kubectl scale ds` (DaemonSets have no scale subresource). ### Follow-ups / notes (FIX8 NITs) - **Grafana main-header CI check stays in PR-B** (not moved to PR-A): moving `check-grafana-datasources.py` (or its `main`-header assertion) into PR-A's `loki-auth-pin-test` would touch the already-cleared #961, so it is not cheap; it is left here in PR-B, where it already ships and will assert the `main` header. - **The LD dispatcher needs a rebuilt and redeployed image before its Deployment scales from 0** — the running binary predates the `X-Scope-OrgID: main` header and would get 401 (report "no coverage") against auth-on Loki.

pi-landing added 2 commits

2026-11-12 13:05:42 +00:00

IR-20261112: feat(monitoring): enable Loki multi-tenancy and name every reader's tenant (R39 PR-A)

CI Review / receiver-test (pull_request) Successful in 6s

Details

CI Review / loki-auth-pin-test (pull_request) Successful in 8s

Details

CI Review / helm-validate (pull_request) Successful in 10s

Details

CI Review / runner-manifest-test (pull_request) Successful in 10s

Details

Build Lab Director Dispatcher Image / preflight (pull_request) Successful in 7s

Details

CI Review / alert-receiver-test (pull_request) Successful in 21s

Details

Lint & Validate / shellcheck (pull_request) Successful in 7s

Details

Lint & Validate / yaml-lint (pull_request) Successful in 22s

Details

CI Review / record-sql-test (pull_request) Successful in 1m0s

Details

Lint & Validate / terraform-validate (pull_request) Successful in 2m7s

Details

CI Review / cockpit-test (pull_request) Successful in 2m42s

Details

CI Review / cockpit-e2e (pull_request) Successful in 2m59s

Details

CI Review / dispatcher-test (pull_request) Successful in 3m11s

Details

Build Lab Director Dispatcher Image / build-and-push (pull_request) Successful in 1m55s

Details

7f3581e897

FIX7 splits infra-core #953 (plane/R39-forgejo-access-log) into two PRs so the
tenancy flip cannot leak the audit trail into `fake` (review-cd-r6 B9/BLOCK-1).
This is PR-A: it turns `loki.auth_enabled` on and gives every PRE-EXISTING
writer/reader its tenant header in the SAME change, plus the checks. PR-B (the
Vector audit sink + its CNP) is a separate PR cut from this head and lands only
after a live 401 on an unlabeled read is proven on the real cluster.

- loki/values.yaml: auth_enabled: true; runtimeConfig.overrides.audit-forgejo
  (retention 45d + deletion_mode disabled); self-monitoring canary pin at the
  root path the chart reads (3c N12).
- kube-prometheus-stack/values.yaml: Grafana `Loki` datasource header
  X-Scope-OrgID: main (secureJsonData).
- opentelemetry-collector/values.yaml: otlphttp/loki header X-Scope-OrgID: main.
- lab-director/dispatcher: the Loki coverage read names tenant `main`
  (DefaultLokiOrgID) so it does not go dark after the flip (3c N1).
- check-loki-tenant-pin.sh (PR-A trim): proves the pin lands where the chart
  reads it and the audit delete API is disabled; the CNP canary-scrape check
  (N11) ships in PR-B.
- ci.yaml: new loki-auth-pin-test job runs the check.

The old branch plane/R39-forgejo-access-log is left untouched (not deleted, not
force-pushed); #953 stays open until PR-A and PR-B supersede it.

Lab-Run: IR-20261112-r39a

IR-20261112: feat(monitoring): ship the Forgejo audit sink + CNP (R39 PR-B)

CI Review / helm-validate (pull_request) Successful in 10s

Details

CI Review / loki-auth-pin-test (pull_request) Successful in 21s

Details

CI Review / runner-manifest-test (pull_request) Successful in 11s

Details

CI Review / receiver-test (pull_request) Successful in 5s

Details

CI Review / dispatcher-test (pull_request) Failing after 9s

Details

CI Review / alert-receiver-test (pull_request) Successful in 20s

Details

CI Review / vector-access-log-test (pull_request) Successful in 2m2s

Details

Build Lab Director Dispatcher Image / preflight (pull_request) Successful in 7s

Details

CI Review / record-sql-test (pull_request) Successful in 52s

Details

Lint & Validate / shellcheck (pull_request) Successful in 7s

Details

Lint & Validate / yaml-lint (pull_request) Successful in 23s

Details

CI Review / cockpit-test (pull_request) Successful in 2m44s

Details

CI Review / cockpit-e2e (pull_request) Successful in 3m2s

Details

Lint & Validate / terraform-validate (pull_request) Successful in 1m50s

Details

Build Lab Director Dispatcher Image / build-and-push (pull_request) Successful in 1m42s

Details

Alert Disposition Contract / alert-contract (pull_request) Successful in 5m46s

Details

c0e62366df

FIX7 splits infra-core #953 into two PRs (review-cd-r6 B9/BLOCK-1). This is
PR-B, cut from PR-A's head (plane/R39-forgejo-access-log-auth): the Vector audit
sink + its CNP + the audit alerts + all the offline tests. It lands ONLY after
PR-A is live and a 401 on an unlabeled read is proven on the real cluster.

- vector-longhorn.yaml: forgejo_pod_logs source (glob-scoped, read_from: end),
  parse-first allowlist, credential redaction (field-masked, line kept), CRI
  partial counter, loki_forgejo_audit sink (tenant_id audit-forgejo, unlimited
  retries, end-to-end acks), checkpoint on hostPath.
- loki-ingress-networkpolicy.yaml: CNP closing the "curl the ClusterIP with
  X-Scope-OrgID: audit-forgejo" read path; admits canary scrape 3500 + dispatcher
  3100.
- org2-bootstrap/grafana-datasource-loki-audit.yaml: the org-2 audit datasource
  (NOT ArgoCD-applied; gated ORG2 job).
- forgejo-audit-alerts.yaml: IngestMissing + Partial + UnparsedBaseline +
  CredentialRedacted.
- rule-tests: run-vector-tests.sh, promtool-test.sh, alertmanager-routing.sh,
  check-grafana-datasources.py, check-loki-tenant-pin.sh (CNP section restored).
- NIT-2 (FIX7): T4b in forgejo-audit-alerts.{gen.py,test.yaml} — the isolating
  increase-leg case for ForgejoAuditCredentialRedacted (sibling of N22's T2).
- ci.yaml + alert-contract.yaml: wire the vector/promtool/amtool harnesses into CI.

The old branch plane/R39-forgejo-access-log is left untouched; #953 stays open
until PR-A and PR-B supersede it.

Lab-Run: IR-20261112-r39b

pi-landing changed title from ~~IR-20261112: feat(monitoring): ship the Forgejo audit sink + CNP (R39 PR-B)~~ to WIP: IR-20261112: feat(monitoring): ship the Forgejo audit sink + CNP (R39 PR-B)

2026-11-12 13:24:20 +00:00

CI Review / helm-validate (pull_request) Successful in 10s

Required

Details

CI Review / loki-auth-pin-test (pull_request) Successful in 21s

Details

CI Review / runner-manifest-test (pull_request) Successful in 11s

Details

CI Review / receiver-test (pull_request) Successful in 5s

Details

CI Review / dispatcher-test (pull_request) Failing after 9s

Details

CI Review / alert-receiver-test (pull_request) Successful in 20s

Details

CI Review / vector-access-log-test (pull_request) Successful in 2m2s

Details

Build Lab Director Dispatcher Image / preflight (pull_request) Successful in 7s

Details

CI Review / record-sql-test (pull_request) Successful in 52s

Details

Lint & Validate / shellcheck (pull_request) Successful in 7s

Required

Details

Lint & Validate / yaml-lint (pull_request) Successful in 23s

Required

Details

CI Review / cockpit-test (pull_request) Successful in 2m44s

Details

CI Review / cockpit-e2e (pull_request) Successful in 3m2s

Details

Lint & Validate / terraform-validate (pull_request) Successful in 1m50s

Required

Details

Build Lab Director Dispatcher Image / build-and-push (pull_request) Successful in 1m42s

Details

Alert Disposition Contract / alert-contract (pull_request) Successful in 5m46s

Details

This pull request is marked as a work in progress.

This branch is out-of-date with the base branch

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin plane/R39-forgejo-access-log-sink:plane/R39-forgejo-access-log-sink

git switch plane/R39-forgejo-access-log-sink

Rows
Columns

WIP: IR-20261112: feat(monitoring): ship the Forgejo audit sink + CNP (R39 PR-B) #962

Scope — R39 FIX8, PR-B of the split (the Vector audit sink + ingress-only CNP)

⚠️ MERGE ORDER — unmissable

⚠️ Merge gate — do NOT merge this PR on its own

Check block (run after PR-A is live; paste the outputs here)

What this PR carries

Rollback (FIX8 B1 — the rollback KEEPS the CNP)

Follow-ups / notes (FIX8 NITs)

Checkout