Homelab/infra-core
6
0
Fork 2
Code Issues Pull requests 35 Projects Releases 1 Packages Wiki Activity Actions

IR-20261112: feat(monitoring): ship Forgejo access logs to a 45d audit tenant (R39) #953

Open
pi-landing wants to merge 10 commits from plane/R39-forgejo-access-log into live
pull from: plane/R39-forgejo-access-log
merge into: Homelab:live
Conversation 0 Commits 10 Files changed 24 +2962 -61
pi-landing commented 2026-11-12 00:55:42 +00:00
Collaborator
Copy link

What / why

Forgejo's container log is the only record of who called what, and it answers neither "who" nor "from where": no identity, no User-Agent, RequestURI verbatim, and only ~70 min reachable through the pod-log API (kubelet containerLogMaxSize 10Mi × containerLogMaxFiles 5; measured: current file 3.44 MiB / 14,787 lines / 77 min). R37 could not recover the source of 7 comments because of it. Design note: working-files/infra-stability/r39-forgejo-access-log-20261112/design.md (option (a), approved by 3c).

This PR is the shipper + tenancy half. The Forgejo half (access logger ON, router log OFF, escaped ACCESS_LOG_TEMPLATE) is a separate prod PR; it lands FIRST (order flipped in fix round 1 — see "Landing order" below).

Changes

  1. core/manifests/monitoring/log-metrics/vector-longhorn.yaml — extends the EXISTING Vector DaemonSet (+0 pods):
    • forgejo_pod_logs file source, scoped by GLOB /var/log/pods/forgejo_*/forgejo/*.log* (the shipper never opens a non-Forgejo file), read_from: end — no backfill (3c B3): until the Forgejo PR lands, the router log writes RequestURI verbatim, so a first pass over the existing files would carry OAuth ?code= and /user/recover_account?code= straight into a 45-day store. ignore_older_secs is deliberately not set — Vector reads it literally, so the design's ignore_older_secs: 0 would ignore every file present at startup.
    • forgejo_annotate derives pod/ns/node/container from the log PATH (no API, no RBAC, automountServiceAccountToken: false unchanged) and runs before any drop, so a dropped event still says where it came from.
    • forgejo_parse — PARSE FIRST, and the parse is the ALLOWLIST (3c B2 + B3). Only a line matching the anchored access shape ships; everything else (the main log's panic and webhook-delivery dumps — a Discord/Slack hook URL keeps its secret in the PATH, where no credential regex can see it — xorm, the SSH logger) is dropped under reason="unparsed". Every later decision reads a parsed field, never the raw line, so the client-controlled User-Agent and X-Forwarded-For cannot steer a drop.
    • Credential hits REDACT the field and KEEP the line (3c B2). A UA of bot token=1, or one merely ENDING in token, used to erase the audit record of the request it belonged to; now ua/xff become [redacted] (and a userinfo URL or bare Basic credential in the path becomes [redacted-path]), counted as forgejo_audit_log_redacted_total{node,pod,reason="credential"}. Defence in depth per requirement 1d: scheme://user:…@host and bare Basic <b64> are matched explicitly.
    • Security endpoints are never noise and their parameters are masked: runner registration-token (user/org/repo/admin variants), /login/oauth/access_token, forgot_password/reset_password, /user/recover_account, /user/activate, the OAuth authorize/callback paths and /api/v1/users/*/tokens. These are exactly the requests a forensic review wants, and the old substring rule swallowed them because their paths end in token or password.
    • Noise is decided on the parsed PATH only — exact/prefix /api/healthz, /metrics, /assets/, RunnerService/(FetchTask|UpdateLog|UpdateTask). A UA containing /assets/ no longer deletes its own line. Drops are ROUTED (not aborted) and counted by reason: forgejo_audit_log_dropped_total{reason="unparsed"|"noise"} — an uncounted filter is an unwatched filter.
    • loki_forgejo_audit sink — tenant_id: audit-forgejo (Vector's sink sends it as X-Scope-OrgID), compression: gzip, encoding.codec: text (the audit datum is the access line; pod/ns/node ride in the stream labels), labels job/ns/node/container. Unlimited retries (the retry_attempts: 5 override is gone: five tries at a 1 s backoff gave up in ~30 s, so the Loki restart that performs the tenancy flip itself would have dropped every line in flight) and acknowledgements.enabled: true, so the file checkpoint only advances once Loki has acknowledged the push (3c N5).
    • the checkpoint data_dir moves from an emptyDir to a hostPath (/var/lib/vector-longhorn-log-metrics, DirectoryOrCreate, the pod already runs as uid 0), so a pod recreation resumes instead of re-reading ~8 h of lines with new timestamps (3c N6).
    • requests 20m/64Mi → 30m/96Mi, limits 200m/256Mi → 300m/384Mi (design §5.1: a shipper OOM-killed under node pressure loses the trail silently), checksum/config rebumped.
  2. Loki auth_enabled: true + runtimeConfig.overrides.audit-forgejo.retention_period: 1080h (45 d, 3c's requirement; the global 30 d is untouched). Verified by helm template grafana/loki 6.29.0 → loki-runtime ConfigMap renders the override; Loki 3.4.2 accepts per-tenant retention as a runtime override.
  3. Every pre-existing writer/reader gets its tenant header in THIS change — Grafana Loki datasource (X-Scope-OrgID: main), otel-collector otlphttp/loki.headers. The flip and its headers must be atomic: auth_enabled 401s an unlabeled writer the moment Loki restarts.
  4. A second Grafana datasource carrying the audit tenant (uid: loki-audit, orgId: 2, editable: false) — the only datasource that can name audit-forgejo, so read access is an org-membership (Server-Admin) decision, not an org-Viewer default. It is NOT in the provisioned set (3c B4): it lives at core/manifests/monitoring/log-metrics/org2-bootstrap/grafana-datasource-loki-audit.yaml, a SUBDIRECTORY of the ArgoCD-synced directory, which directory-mode syncs do not apply. The gated ORG2 bootstrap job POSTs that file's datasources: entry verbatim to POST /api/datasources with X-Grafana-Org-Id: 2 after it creates org 2, after checking the file's sha256 against the review. core/manifests/monitoring/log-metrics/rule-tests/check-grafana-datasources.py (run in CI) proves the main set references no org 2 — a single orgId: 2 in the provisioned set fails provisioning for the WHOLE directory (CheckOrgExists), which would strip X-Scope-OrgID: main from the chart's Loki datasource and then crash-loop Grafana on its next restart.
  5. loki-ingress-networkpolicy.yaml — the tenancy flip is a Grafana boundary, not a network one; this closes the "curl the ClusterIP with X-Scope-OrgID: audit-forgejo" read path to the enumerated writers/readers. Narrowed after review (3c N4): fromEntities is now [health, host] — remote-node and unmanaged admitted every hostNetwork pod and the API-server services/proxy path for a reason ("the node's volume path") that is not Loki pod ingress at all — and every rule is port-scoped to 3100/9095/7946. The remaining services/proxy holders are recorded in the file's header (names only).
  6. forgejo-audit-alerts.yaml — ForgejoAuditLogIngestMissing (0 bytes accepted for the tenant for 15 m while Forgejo is Ready — silence is the failure mode that matters) and ForgejoAuditCredentialRedacted (a client-controlled field carried a credential: the line is KEPT with [redacted] in that field, so a live token was in a URL and the request it belonged to is on record).
  7. rule-tests/run-vector-tests.sh — new R39 cases against a local Loki capture stub (below), and a CI job that actually runs this harness, which no workflow did before (the pre-existing Longhorn cases were hand-run only).

Test result (offline, fail-capable, in CI)

ok   credential lines refused are counted: forgejo_audit_log_dropped_total=6
ok   captured 1 Loki push(es) to ['/loki/api/v1/push']   4 lines in the payload
ok   every push carries X-Scope-OrgID: audit-forgejo
ok   every stream is labelled job=forgejo-access
ok   credential marker FAKE-TOKEN-9f3c / FAKE-AT-77b1 / FAKE-PT-5e2a /
     FAKE-SIG-deadbeef / FAKE-SIG2-c0ffee / FAKE-BEARER-1a2b is ABSENT from the Loki payload
ok   no ?token= / ?sig= / ?signature= / access_token= / private_token= anywhere in the payload
ok   the positive control (a comment POST) reached Loki
ok   a git write (git-receive-pack) reached Loki
ok   the negative control (credential-free query string) SURVIVED
ok   the health probe / metrics scrape / Actions runner poller / static assets did not reach Loki
ok   the hostile-UA line reached Loki (so the assertions below are not vacuous)
ok   the rendered access line parses as ONE anchored shape
ok   Identity parses from the REAL field (pi-landing, not the forgery)
ok   Path parses from the REAL field
ok   the XFF parses from its own field
ok   exactly one ` user="` field in the line (no forged field)
ok   the forgery survives only INSIDE the escaped ua value
ok   no second, forged log line was produced
SUCCESS: all R39 audit-stream control cases passed
SUCCESS: all log-metric control cases passed   (the pre-existing Longhorn cases still pass)

The stub gzips nothing — it DECODES the sink's real gzip and asserts on the bytes Loki would receive, including the X-Scope-OrgID header. The three legs that keep it honest: the drop counter must be exactly 6 (a rule that dropped everything would fail), a credential-free query string must survive (the rule drops credentials, not query strings), and the hostile-UA line must still parse to ONE identity and ONE path (a UA of curl/8.5" user="ld-reviewer"<newline>GET /api/v1/admin/evil survives only as escaped text inside ua="…"). The escaping itself is the prod PR's leg, proven by a real Go text/template render of the shipped template.

Verified locally with the SAME binary the DaemonSet runs (vector 0.61.0).

Landing order — which goes first, and why

The Forgejo values PR (prod) lands FIRST; this PR second. (flipped in fix round 1 — 3c B3)

  • auth_enabled: true breaks any writer without X-Scope-OrgID at the instant Loki restarts, so the flip and every header must be in the same change. They are: canary (sends fake itself), otel-collector, Grafana both datasources, and the new Vector sink.
  • The Forgejo PR restarts Forgejo, which must not happen during a runner gap and only after SR072 is LIVE, with notice to a9 — so it cannot be the one that lands first anyway. It is also strictly additive to this one: it only starts producing lines for a tenant that already exists.

Tenancy flip: enumeration (before) and re-check (after)

Live read-only reads, before the flip:

party direction tenant / header evidence
loki-canary (6 pods) writer + reader self-monitoring via -tenant-id (pinned explicitly in loki/values.yaml; it was fake before the flip) loki_distributor_bytes_received_total{tenant="fake"} = 12,967,905 B / 129,679 lines over 6 h ≈ 600 B/s — the ONLY series present
vector-longhorn-log-metrics writer (new) audit-forgejo (sink tenant_id) 9/9 Running; writes no logs today
opentelemetry-collector writer main (header added here) otelcol_receiver_accepted_log_records_total = 0 over 6 h: configured, idle
Grafana Loki datasource reader main (header added here) kube-prometheus-stack-grafana-datasource, no header today
Grafana Loki (audit) reader (new) audit-forgejo this PR
Prometheus → loki:3100/metrics reader none needed /metrics is served outside the tenant middleware; up{job="loki"} must stay 1
agent-tools/lab-director-dispatcher (Loki reader, cut by this change) reader none today signalintake/ calls /loki/api/v1/series with no tenant header (3c N1). After the flip it gets Loki's missing-tenant error AND is refused by the CNP. It fails LOUDLY (an error plus a ledger refusal every 5 min), not silently. Two ways out, neither in this PR: give the dispatcher a tenant header (it only ever needs main) plus a CNP allow, or turn its logCoverage off.

After the flip each must still write/read with its header: canary bytes still rising under fake; otel accepting a log record under main; both Grafana datasources returning data (not 401); the Vector stream landing under audit-forgejo; up{job="loki"} green; and {tenant="audit-forgejo"} not readable through the main-org datasource. The rollout proof re-checks exactly this list.

Grafana auth config (read from values + live grafana.ini; no credential used)

role_attribute_path = contains(groups[], 'admins') && 'Admin' || 'Viewer' · auto_assign_org_role = Viewer · viewers_can_edit = false · disable_login_form not set (absent from the live ConfigMap, so the local admin login form is enabled) · OAuth via Zitadel, allow_sign_up = true.

The Grafana admin census (every main-org user/service account with Admin or isGrafanaAdmin) is a SEPARATE credential job gated on the overseer's start message: grafana-admins=SEPARATE-JOB.

Residual risks

  1. XFF arrival is unproven offline — it needs one live external request (rollout proof step).
  2. Org 2 does not exist yet: OSS Grafana has no org provisioner, so Loki (audit) only appears after the gated ORG2 bootstrap job creates org audit and POSTs the reviewed datasource. Nothing is provisioned from it in the meantime, so the main set cannot fail.
  3. Accepted and NOT papered over: an OSS Grafana main-org Admin could still create a datasource naming this tenant. "No agent or service account holds org-1 Admin" is UNCHECKED until the separate Grafana admin listing job runs (it needs a credential this PR does not have) — 3c's condition is therefore OPEN, not satisfied. Until it runs, treat audit-forgejo as readable by any org-1 Admin and by any Admin-role service-account token.
  4. The credential rule is fail-closed on look-alike text, but it now REDACTES the field and keeps the line (3c B2): a UA containing token " cannot delete its own audit record any more, and the redaction is counted and alerted.
  5. DISABLE_ROUTER_LOG / LOGGER_*_MODE are v14 names; the prod PR's CI guard fails loudly if an upgrade stops rendering them.

Class

dangerous. The diff touches **/.forgejo/** (a CI workflow — an oversight path; the NPMCACHE PR #952 precedent), **/*networkpolic*.yaml (the CiliumNetworkPolicy) and a **/*-config.yaml-adjacent values.yaml under core/charts/platform/. Per landing-8c666fbe.ts dangerousPaths that is two independent dangerous legs (.forgejo/**, *networkpolic*), so it needs the two-reviewer panel and a post-merge live proof. (The prod PR is routine: values/forgejo/values.yaml, a tests/ script and .forgejo/workflows/ci.yaml — the last of which also makes it dangerous by the same rule, so treat prod as dangerous too.)

Head commit carries Lab-Run: IR-20261112-r39 as its last trailer; author pi-landing; branch cut from origin/live.

FIX ROUND 1 — 3c B2, B3, B4 (+ N1, N2, N5, N6)

Review: working-files/security-hypervisor/pr-r39/review-3c.md. One new commit, no amend.

B2 — client text decided what was dropped. Both rules matched the RAW line, which contains the User-Agent and the X-Forwarded-For: anyone could delete their own audit line with User-Agent: x /assets/, uncounted. The credential rule dropped whole lines, so a UA merely ENDING in token erased the request's record, and the security endpoints a review needs most (runner registration-token, /login/oauth/access_token, forgot_password, /api/v1/users/*/tokens) were dropped as false positives. Now: parse_regex! first; noise decided on the parsed .path only; security endpoints exempt and parameter-masked; a credential hit REDACTES the field (ua/xff → [redacted]) and KEEPS the line; every drop is counted by reason.

B3 — the shipper forwarded every stdout line, and the userinfo/Basic rules were missing. The source is now an ALLOWLIST (access-anchor lines only; the rest counted as unparsed), read_from: end (no backfill of pre-PR query strings), plus defence-in-depth rules for scheme://user:pass@host and bare Basic <b64>. Landing order is now prod FIRST — the anchor filter stays as well; the order is the cheap guarantee, the allowlist is the structural one.

B4 — orgId: 2 broke Grafana provisioning. Split: the audit datasource is now org2-bootstrap/grafana-datasource-loki-audit.yaml, which no sync applies, and the gated ORG2 job POSTs it after creating org 2. rule-tests/check-grafana-datasources.py proves the main set has no org 2 (fails if one is added back, including inside a block scalar).

Non-blocking, same commit: N1 (the LD dispatcher is listed in the table as a reader this change cuts, with its two ways out); N2 (ONE tenant scheme, written down below); N4 (CNP narrowed to health+host, port-scoped, services/proxy holders recorded by name); N5 (unlimited retries + end-to-end acknowledgements); N6 (checkpoint data_dir on a hostPath).

Tenant scheme — one name per purpose (3c N2)

tenant who writes it who reads it why that name
main opentelemetry-collector (otlphttp/loki.headers) Grafana Loki datasource every human-facing stream; it is the tenant the viewers' default datasource names
audit-forgejo vector-longhorn-log-metrics (sink tenant_id) Grafana Loki (audit) in org 2 only the audit trail: 45 d retention override, one writer, one reader, admin-only
self-monitoring loki-canary (-tenant-id, now pinned explicitly in loki/values.yaml) the canary itself (read-back verification) a health signal, not history; the chart's own concept, and it is what the canary switches to once auth_enabled is true
fake nothing (legacy) nothing the pre-flip canary tenant. Not migrated: that 26.7 MB of canary data stays where it is and ages out on the global 30 d retention. It is not "the default tenant" and no dashboard should point at it after this lands.

Test result (offline, fail-capable, in CI)

ok   noise lines dropped (healthz/metrics/assets/runner poller): forgejo_audit_log_dropped_total{reason="noise"}=4
ok   non-access (panic/webhook) lines dropped: forgejo_audit_log_dropped_total{reason="unparsed"}=1
ok   credential fields redacted (line KEPT, field masked): forgejo_audit_log_redacted_total=6
ok   a line written BEFORE the shipper started is NOT shipped (no backfill)
ok   credential marker FAKE-UA-9f3c / FAKE-PASS-77b1 / Zm9vYmFyMTIzNA / FAKE-XFF-5e2a /
     FAKE-PATH-c0ffee / FAKE-HOOK-SECRET-8b21 is ABSENT from the Loki payload
ok   no ?token= / ?sig= / ?signature= / access_token= / private_token= anywhere in the payload
ok   the redaction placeholder is what replaced the credential
ok   the positive control (a comment POST) reached Loki
ok   a git write (git-receive-pack) reached Loki
ok   the negative control (credential-free query string) SURVIVED
ok   a UA containing ' /assets/' did NOT get its line dropped
ok   a UA ending in 'token' did NOT get its line dropped
ok   the UA ending in 'token' was REDACTED, not shipped
ok   security endpoint /actions/runners/registration-token reached Loki
ok   security endpoint /login/oauth/access_token reached Loki
ok   security endpoint /user/forgot_password reached Loki
ok   security endpoint /api/v1/users/pi-landing/tokens reached Loki
ok   a non-access line (webhook panic) did not reach Loki
ok   the hostile-UA line reached Loki (so the assertions below are not vacuous)
ok   the rendered access line parses as ONE anchored shape
ok   Identity parses from the REAL field (pi-landing, not the forgery)
ok   exactly one ` user="` field in the line (no forged field)
ok   no second, forged log line was produced
SUCCESS: all R39 audit-stream control cases passed
SUCCESS: all log-metric control cases passed   (the pre-existing Longhorn cases still pass)

$ python3 core/manifests/monitoring/log-metrics/rule-tests/check-grafana-datasources.py
ok   the provisioned datasource set (chart values) references no orgId 2
ok   the Loki datasource carries X-Scope-OrgID: main (paired with auth_enabled)
ok   all 2 ArgoCD-applied manifest(s) in log-metrics reference no orgId 2
ok   the org-2 datasource is in the repo at core/manifests/monitoring/log-metrics/org2-bootstrap/grafana-datasource-loki-audit.yaml
ok   it lives in a SUBDIRECTORY, which ArgoCD directory mode does not apply
ok   the reviewed org-2 body: orgId is 2 / uid is loki-audit / editable is false / tenant is audit-forgejo
ok   no other file under core/ provisions a datasource on org 2
SUCCESS: the main Grafana provisioning set cannot reference org 2, and the org-2 datasource is
         reviewed, unapplied, and ready for the gated bootstrap job

Both scripts run in the vector-access-log-test CI job, against the SHIPPED config.

FIX ROUND 2 — 3c r2: B5, B6 (+ N9, N11, N12, N1)

Review: working-files/security-hypervisor/pr-r39/review-3c-r2.md. ONE new commit on the handover head; no amend.

B5 — an XFF redaction erased the gateway-appended client IP. The rule replaced the WHOLE field when any part looked like a credential, so X-Forwarded-For: token arrived as token, 198.51.100.7 and shipped as [redacted] — with remote only the cilium-envoy node IP for external traffic, that is a request with no recoverable source, and the ATTACKER chose when. Redaction is now ELEMENT BY ELEMENT on the comma-separated field: an IP literal (v4/v6, with or without port/brackets) is KEPT, a non-IP element becomes [redacted], and the LAST hop — the one the gateway appended — is kept unless it is itself credential-shaped (a single element is not a gateway hop, and a credential in it is not an address). token=FAKE, 203.0.113.9 ships xff="[redacted], 203.0.113.9". Both assertions run on the captured Loki push body: the marker is ABSENT, the address is present.

Live confirmation owed, and it is the leg that makes B5 measurable: the gateway must set XFF for external traffic. Rollout proof: send ONE request through the public route carrying X-Forwarded-For: token=… and read the stored audit-forgejo line back — it must show xff="[redacted], <real client>". If the gateway does NOT set XFF, the client address is still unrecoverable and that is a GATEWAY finding to report BEFORE the audit trail is relied on, not something this shipper can fix.

B6 — length, not content. containerd splits any stdout line over max_container_log_line_size (16 KiB) into CRI P records; none match the anchor, so an oversized header turned a real request into dropped "unparsed" fragments with no alert. Fixes:

  • the P/F flag is CARRIED and fragments are counted under their OWN reason, reason="partial" — a zero baseline, instead of being buried among Forgejo's routine non-access lines (the SSH logger writes one per git-over-SSH operation);
  • unparsed now excludes partials explicitly (.cri_partial != true), so a fragment is counted under exactly one reason and the alert cannot double-report it;
  • new rule ForgejoAuditLogDropped: sum by (node, reason) (increase(forgejo_audit_log_dropped_total{reason=~"unparsed|partial"}[15m])) > 0, for: 0m, severity warning, full disposition contract. partial is the regression detector (the prod template caps UA/XFF at 512 B); unparsed carries the git-over-SSH baseline in its own annotation so the routine case is named and not silenced;
  • tests: two CRI P fragments → reason="partial"=2 and neither fragment reaches Loki; a capped 512-byte User-Agent (what %.512q emits for a 20 KiB input) ships as ONE line, under 4 KiB, that parses.
  • The cap itself is the prod PR (printf "%.512q" on UA and XFF), asserted there by rendering a 20 KiB User-Agent through real Go text/template: one line, both fields at 512 bytes, truncation markers gone, and an anti-vacuity leg that re-renders the UNCAPPED template and requires it to fail.

N9 — GET /. The anchor path is *, not +, in all three anchors (VRL match, VRL parse, harness anchor). Forgejo's sanitiser trims the trailing slash, so GET / renders an EMPTY path ("GET "); with + every dashboard home request was unparsed and dropped. New case asserts the empty-path line reaches Loki.

N11 — the canary scrape port. The chart's ServiceMonitor scrapes the canary Service on http-metrics = 3500 (proved by rendering: canary Service ports: {3500: 'http-metrics'}), and this policy's endpointSelector also selects the canary pods, so the port-scoped rule denied the scrape and kube-prometheus-stack TargetDown would page. The CNP now admits monitoring/prometheus → 3500 and health/host → 3500 (the kubelet readiness probe). How it is tested: rule-tests/check-loki-tenant-pin.sh renders the chart and asserts BOTH halves — the rendered canary Service exposes 3500 AND the shipped CNP admits 3500 to prometheus (and that the dispatcher rule is 3100-only). Live leg for the rollout proof: up{job=~".*loki-canary.*"} must stay 1 after the CNP applies.

N12 — the tenant pin was inert. Moved to the root monitoring.selfMonitoring.tenant.name, the path templates/loki-canary/daemonset.yaml reads. rule-tests/check-loki-tenant-pin.sh proves it three ways:

  1. the shipped values render -user=self-monitoring -tenant-id=self-monitoring on the rendered loki-canary DaemonSet;
  2. overriding that SAME root path with a sentinel changes the rendered args (so the assertion reads the pin, not a chart default);
  3. the OLD loki.monitoring.selfMonitoring.tenant.name path is shown to be INERT — a sentinel set there never appears.
ok   shipped pin: -tenant-id=self-monitoring
ok   shipped pin (user): -user=self-monitoring
ok   root monitoring.selfMonitoring.tenant.name is READ: -tenant-id=pin-sentinel-7c1
ok   the shipped value is overridden (defaults are not what we see): -tenant-id=self-monitoring is absent
ok   loki.monitoring.* is ignored by the chart: inert-path-9f2 is absent
ok   the pin still wins over that path: -tenant-id=self-monitoring
ok   canary Service ports: {3500: 'http-metrics'}
ok   CNP admits monitoring/prometheus -> 3500 (the canary scrape)
ok   CNP admits agent-tools/lab-director-dispatcher -> 3100 (tenant main, 3c N1)
ok   the dispatcher rule is port-scoped to 3100
SUCCESS: the tenant pin lands where the chart reads it, and the canary scrape port is admitted

The stale "main is where every pre-existing stream lives" claim is corrected in the kube-prometheus-stack and otel-collector values: main is the HUMAN-FACING tenant; the pre-existing 26.7 MB of canary history stays under fake and is not migrated.

N1 — the LD dispatcher: option 1 (header support + CNP allow for main only), not "turn logCoverage off". logCoverage is the arm's designed "the coverage gap is VISIBLE in production" leg; turning it off would hide a gap to make a 401 disappear. Implemented in this PR:

  • signalintake.LokiSource sends X-Scope-OrgID; the tenant is signals.lokiOrgID / SIGNALS_LOKI_ORG_ID, default main (DefaultLokiOrgID). A test asserts the header is sent and that the default is never audit-forgejo;
  • the CNP admits agent-tools/lab-director-dispatcher → 3100 only, and the render check asserts that rule admits 3100 AND NOTHING ELSE — the probe may never reach the audit tenant, 9095 or 7946;
  • no prod config change is required (dispatcher-config.yaml needs nothing), though signals.lokiOrgID is available if the tenant ever moves.

No code change, body notes:

  • The hostPath checkpoint dir must NEVER hold a disk buffer. It carries checkpoint state only (fingerprints, offsets, paths); memory is the default buffer, so no raw, not-yet-redacted line is written to the host, and the volume has no sizeLimit. The manifest now says this in place.
  • Longhorn alert owner lane (N15 note 2): the Longhorn source also resumes from checkpoints now, so after a shipper outage it counts the lines written while it was down — a Longhorn abort / proxy-check page can arrive LATE for an incident that has already ENDED. The HOMELAB-2489 rule owner should read those as "delayed, not new".
  • N3 (org-1 Admin) stays OPEN until the Grafana admin listing job runs: treat audit-forgejo as readable by any org-1 Admin until then.

Test result (offline, this round, vector 0.61.0 — the DaemonSet's own binary)

ok   noise lines dropped (healthz/metrics/assets/runner poller): forgejo_audit_log_dropped_total{reason="noise"}=4
ok   non-access (panic/webhook) lines dropped: forgejo_audit_log_dropped_total{reason="unparsed"}=1
ok   CRI partial fragments counted separately (zero-baseline alert): forgejo_audit_log_dropped_total{reason="partial"}=2
ok   credential fields redacted (line KEPT, field masked): forgejo_audit_log_redacted_total=7
ok   captured 1 Loki push(es) to ['/loki/api/v1/push']   18 line(s) in the payload
ok   the root path (GET /) reached Loki with an empty path (3c N9)
ok   the gateway-appended client IP SURVIVES an XFF credential redaction (3c B5)
ok   the attacker-chosen XFF element was redacted, not shipped (3c B5)
ok   a capped 512 B User-Agent still reached Loki (3c B6)
ok   the capped line is a single line under 4 KiB (3c B6)
ok   a CRI partial fragment did NOT reach Loki (it is counted, not shipped, 3c B6)
SUCCESS: all log-metric control cases passed
SUCCESS: all R39 audit-stream control cases passed

The LD side: go build ./... && go test ./signalintake/ ./config/ — ok. Class: this commit also edits .forgejo/workflows/ci.yaml (adds the tenant-pin render step), so the CI oversight leg applies.

Round 3 fix (3c N18): the drop alert is split, and the partial leg can never be silenced

3c r3 CLEARED infra-core #953 at d624b5fc and raised N18 as non-blocking — a
warning worth folding in before the alert exists, so this round does it.

The problem. One rule, ForgejoAuditLogDropped, matched
reason=~"unparsed|partial" with increase(...[15m]) > 0, for: 0m. But
unparsed is not exotic: it is every line of Forgejo's own main log that is not
an access line. The design's own measurement (14,787 stdout lines in ~77 min,
13,310 of them router lines) leaves ≈1,477 main-log lines — about 19 lines/min.
So that leg fired continuously, and the first silence anyone wrote on
alertname=ForgejoAuditLogDropped would have muted the partial detector, the
only mechanical evidence that an audit record was lost.

The split (two rules, two names, so silencing one can never mute the other):

rule expression for silence policy
ForgejoAuditLogPartial sum by (node) (increase(forgejo_audit_log_dropped_total{reason="partial"}[15m])) > 0 0m never
ForgejoAuditLogUnparsedBaseline sum(rate(forgejo_audit_log_dropped_total{reason="unparsed"}[15m])) * 60 > 60 15m this alertname only, and only for a declared git-over-SSH / debug-level exercise

The threshold, stated: 60 unparsed lines/min — 3.1× the measured 19.2
lines/min normal baseline (baseline ≈288 lines per 15 m window, threshold ≈900),
evaluated as a 15 m average and required to hold for 15 m. The margin absorbs the
routine git-over-SSH console lines the old text wrongly called "a handful per
day"; a template change, an anchor regression, a debug-level change or an
injection attempt moves the rate well past it.

The partial rule keeps for: 0m (one fragment is already a lost record), stays at
any count > 0, and is additionally the backstop for the prod PR's deliberately
uncapped X-Forwarded-For (3c B7): an oversized XFF loses its record as counted
partials rather than silently losing its source.

Body notes: N17 and N19 (deliberately not changed here)

N17 — the dispatcher's "main only" is a default, not a control. The tenant
comes from SIGNALS_LOKI_ORG_ID / signals.lokiOrgID with no allowlist, and the
CNP admits agent-tools/lab-director-dispatcher to port 3100 header-agnostically.
Not fixed in this PR, because the LD code is another lane's. The follow-up is
filed separately: either hard-code main in NewLokiSource (4a's lane,
defence in depth) or — stronger, because then code review is not the only control
— a Cilium L7 HTTP rule on that egress, allowing only the read paths the
dispatcher actually uses with the header X-Scope-OrgID: main, proven by a denied
request carrying audit-forgejo. Comments claiming "main is the ONLY tenant it
may name" are wrong until that lands and should not be read as a control.

N19 — the cap is per field; the line is still unbounded through the path.
EscapedPath has no cap and 512 non-printable runes per capped field render ≈5 KB,
so a write action with a user-chosen path segment can still cross 16 KiB. Not
fixed here: this PR's partial alert is the backstop — that loss is counted and
alerts on any count > 0. Lowering the UA cap or bounding the path (%.4096s on
EscapedPath, safe because it is ASCII) is a cheap follow-up.

CI note

This round's single commit also serves as the CI re-run for
build-and-push, which flaked on a Go proxy error at d624b5fc (job 3656). There
is no code change behind that rerun.

FIX ROUND 5 (overseer 10:20:12Z): the promtool docker fallback never validated anything in CI

Disclosure of the empty-commit history first. Head a02fe825 was an EMPTY
CI re-run commit (its tree is byte-identical to f474c999) pushed only because
the 06:52Z Forgejo restart killed jobs 3675 (alert-contract) and 3676
(helm-validate) at ~12 min, and Forgejo 14 has no rerun API. That re-run is
what exposed a real bug, not an external event: run #3679 failed the
alert-contract job at the promtool step with

promtool: error: path '/w/agent-sandbox-alerts.yaml' does not exist, try --help

The fault. rule-tests/promtool-test.sh's docker fallback did
mktemp -d (in /tmp) and docker run -v $TMP:/w:ro. On the host-mode runner
only the checkout ($GITHUB_WORKSPACE == $PWD == /workspace) is shared with the
dind sidecar (HOMELAB-1634; the dispatcher smoke step in
dispatcher-image-build.yaml hit the identical fault in #1608), so the daemon
binds an EMPTY directory at /w and promtool reads no config at all. The
promtool step has therefore never validated anything in CI. The FIX4 local
pass used promtool from PATH, so the docker branch was never exercised.

The fix (option (a) — the smaller of the two): create the temp dir INSIDE
the shared workspace (${GITHUB_WORKSPACE:-$PWD}), matching the
dispatcher-image-build.yaml smoke pattern; keep the digest-pinned cluster
promtool so the unit test still cannot speak a different dialect from the
evaluator. Two supporting changes the local docker-branch proof surfaced:
(1) chmod -R a+rX on the temp tree — the extracted rule files were 0600 under
a strict umask, so the container's uid 65534 got permission denied once the
mount worked; (2) a per-file summary of the unit-test case names, because
promtool only prints SUCCESS per FILE on success — the six
forgejo-audit-alerts cases T1/T1b/T2/T3/T4/T5 are now named in the
alert-contract log.

rule-tests/alertmanager-routing.sh — the very next step in the same
alert-contract job — carried the identical mktemp -d + -v $TMP:/w:ro
fault and would have failed the same way; it gets the same workspace-mount +
readability fix.

Local proof (docker branch; PROMTOOL/AMTOOL unset, neither binary on PATH):
a deliberately broken rule makes promtool check rules fail with a real parse
error (exit 1), all rule files and all six forgejo-audit cases pass on the clean
tree, the routing harness stays 9/9 green with A-I red, and the trap removes the
workspace temp dir on both the success and failure exit paths.

FIX ROUND 6 — 3c r5: B10 (delete API), B9 (flip order), N22 (increase-leg case)

Review: working-files/security-hypervisor/pr-r39/review-3c-r5.md. ONE new commit on the FIX5 head, no amend.

B10 — the Loki delete API was open on audit-forgejo. compactor.delete_request_store: filesystem + retention_enabled: true with deletion_mode set nowhere defaults to filter-and-delete in Loki 3.x, so any party that can name the tenant by header (CNP-admitted pods, port-forward/exec holders, the dispatcher) could POST /loki/api/v1/delete and silently erase the forensic log. Fix: deletion_mode: disabled under runtimeConfig.overrides.audit-forgejo in core/charts/platform/loki/values.yaml, asserted in check-loki-tenant-pin.sh (renders the chart, proves the value reaches the loki-runtime ConfigMap, plus a sentinel override proving the path is read). Runbook §6 gained the effective read-back: /runtime_config (and the loki-runtime ConfigMap) must show overrides.audit-forgejo.deletion_mode: disabled, and a delete POST must be refused.

B9 — the flip is NOT atomic, and the old order leaked. Four ArgoCD Applications sync separately (monitoring-log-metrics wave 5; loki / kube-prometheus-stack / opentelemetry-collector wave 10), so with auth off, Vector could ship the audit trail into fake (readable by every Grafana Viewer) before the flip. The runbook order is rewritten: (1) pause monitoring-log-metrics auto-sync, (2) sync loki first (auth on) and prove an unlabeled read returns 401, (3) sync kps + otel, (4) only then log-metrics/Vector. Chosen "order the sync" over "gate the Vector sink" because Vector's loki sink gets HTTP 200 whether auth is on (audit-forgejo) or off (fake), so it cannot gate on its own responses — the only reliable boundary is the ORDER. The rollback is also ordered: stop the Vector sink (scale to zero) BEFORE auth goes off. The §5 claim that audit data "mixes into fake" is corrected: already-written audit-forgejo data stays put, is unaddressable while auth is off, and ages out at global 30 d once the override reverts. A permanent sync-wave reorder is a prod apps/*.yaml follow-up (below).

N22 — T2 now isolates the increase leg. The old T2 (0+1x20, eval 10 m) had no sample 15 m back, so the birth leg fired with the same value and deleting the increase leg would NOT have gone red. New T2 (0+1x16, eval 15 m) is born before eval−15 m, so unless offset 15m is empty and only the increase leg can fire (value 15). Fail-capable proof: with the increase leg removed from the rule the case reports got: [] and promtool exits 1; with it present the full suite is green (158 cases SUCCESS, T1/T1b/T2/T3/T4/T5 named in the alert-contract log).

Follow-ups (listed, NOT implemented in this round):

  • N21 — the alertclass unique label cannot do what its comment claims: a silence whose matchers all match (e.g. component=forgejo or severity=warning) still mutes ForgejoAuditLogPartial. Fix: drop/rename component on the Partial rule (e.g. component: forgejo-audit-loss), correct the rule comment and the alertmanager-routing.md "R39" text, and record the severity=warning residual under the ruling.
  • No Loki gateway (residual) — there is no gateway (gateway.enabled: false), so any CNP-admitted pod, and any pods/portforward / pods/exec holder in monitoring or agent-tools, can read OR write the audit tenant by setting X-Scope-OrgID: audit-forgejo. The CNP is a network boundary, not a tenant boundary. Longer term: a per-tenant auth gateway that maps each client to one tenant.
  • B7 / N17 / N19 (still open from r4): B7 (XFF uncapped on purpose; the live gateway XFF check is owed), N17 (dispatcher tenant is a default, not a control — Cilium L7 rule follow-up), N19 (path uncapped; the partial alert is the backstop).
  • prod sync-wave reorder — move monitoring-log-metrics to a wave after loki / kube-prometheus-stack / opentelemetry-collector in prod apps/*.yaml, so the flip order above is enforced by ArgoCD rather than by hand.

CI at this head (all four runs green): shellcheck, terraform-validate, yaml-lint, build-and-push, preflight, alert-receiver-test, cockpit-e2e, cockpit-test, dispatcher-test, helm-validate, receiver-test, record-sql-test, runner-manifest-test, vector-access-log-test, alert-contract.

## What / why Forgejo's container log is the only record of who called what, and it answers neither "who" nor "from where": no identity, no User-Agent, `RequestURI` verbatim, and only ~70 min reachable through the pod-log API (kubelet `containerLogMaxSize 10Mi × containerLogMaxFiles 5`; measured: current file 3.44 MiB / 14,787 lines / 77 min). R37 could not recover the source of 7 comments because of it. Design note: `working-files/infra-stability/r39-forgejo-access-log-20261112/design.md` (option (a), approved by 3c). This PR is the **shipper + tenancy** half. The Forgejo half (access logger ON, router log OFF, escaped `ACCESS_LOG_TEMPLATE`) is a separate prod PR; **it lands FIRST** (order flipped in fix round 1 — see "Landing order" below). ## Changes 1. **`core/manifests/monitoring/log-metrics/vector-longhorn.yaml`** — extends the EXISTING Vector DaemonSet (+0 pods): - `forgejo_pod_logs` file source, scoped by GLOB `/var/log/pods/forgejo_*/forgejo/*.log*` (the shipper never opens a non-Forgejo file), `read_from: end` — **no backfill** (3c B3): until the Forgejo PR lands, the router log writes `RequestURI` verbatim, so a first pass over the existing files would carry OAuth `?code=` and `/user/recover_account?code=` straight into a 45-day store. `ignore_older_secs` is deliberately **not** set — Vector reads it literally, so the design's `ignore_older_secs: 0` would ignore every file present at startup. - `forgejo_annotate` derives pod/ns/node/container from the log PATH (no API, no RBAC, `automountServiceAccountToken: false` unchanged) and runs **before** any drop, so a dropped event still says where it came from. - **`forgejo_parse` — PARSE FIRST, and the parse is the ALLOWLIST** (3c B2 + B3). Only a line matching the anchored access shape ships; everything else (the main log's panic and webhook-delivery dumps — a Discord/Slack hook URL keeps its secret in the PATH, where no credential regex can see it — xorm, the SSH logger) is dropped under `reason="unparsed"`. Every later decision reads a **parsed field**, never the raw line, so the client-controlled User-Agent and X-Forwarded-For cannot steer a drop. - **Credential hits REDACT the field and KEEP the line** (3c B2). A UA of `bot token=1`, or one merely ENDING in `token`, used to erase the audit record of the request it belonged to; now `ua`/`xff` become `[redacted]` (and a userinfo URL or bare Basic credential in the path becomes `[redacted-path]`), counted as `forgejo_audit_log_redacted_total{node,pod,reason="credential"}`. Defence in depth per requirement 1d: `scheme://user:…@host` and bare `Basic <b64>` are matched explicitly. - **Security endpoints are never noise and their parameters are masked**: runner `registration-token` (user/org/repo/admin variants), `/login/oauth/access_token`, `forgot_password`/`reset_password`, `/user/recover_account`, `/user/activate`, the OAuth authorize/callback paths and `/api/v1/users/*/tokens`. These are exactly the requests a forensic review wants, and the old substring rule swallowed them because their paths end in `token` or `password`. - **Noise is decided on the parsed PATH only** — exact/prefix `/api/healthz`, `/metrics`, `/assets/`, `RunnerService/(FetchTask|UpdateLog|UpdateTask)`. A UA containing ` /assets/` no longer deletes its own line. Drops are ROUTED (not aborted) and counted **by reason**: `forgejo_audit_log_dropped_total{reason="unparsed"|"noise"}` — an uncounted filter is an unwatched filter. - `loki_forgejo_audit` sink — `tenant_id: audit-forgejo` (Vector's sink sends it as `X-Scope-OrgID`), `compression: gzip`, `encoding.codec: text` (the audit datum is the access line; pod/ns/node ride in the stream labels), labels `job/ns/node/container`. **Unlimited retries** (the `retry_attempts: 5` override is gone: five tries at a 1 s backoff gave up in ~30 s, so the Loki restart that performs the tenancy flip itself would have dropped every line in flight) and **`acknowledgements.enabled: true`**, so the file checkpoint only advances once Loki has acknowledged the push (3c N5). - the checkpoint `data_dir` moves from an emptyDir to a **hostPath** (`/var/lib/vector-longhorn-log-metrics`, `DirectoryOrCreate`, the pod already runs as uid 0), so a pod recreation resumes instead of re-reading ~8 h of lines with new timestamps (3c N6). - requests 20m/64Mi → 30m/96Mi, limits 200m/256Mi → 300m/384Mi (design §5.1: a shipper OOM-killed under node pressure loses the trail silently), `checksum/config` rebumped. 2. **Loki `auth_enabled: true`** + `runtimeConfig.overrides.audit-forgejo.retention_period: 1080h` (45 d, 3c's requirement; the global 30 d is untouched). Verified by `helm template grafana/loki 6.29.0` → `loki-runtime` ConfigMap renders the override; Loki 3.4.2 accepts per-tenant retention as a runtime override. 3. **Every pre-existing writer/reader gets its tenant header in THIS change** — Grafana `Loki` datasource (`X-Scope-OrgID: main`), `otel-collector` `otlphttp/loki.headers`. The flip and its headers must be atomic: `auth_enabled` 401s an unlabeled writer the moment Loki restarts. 4. **A second Grafana datasource carrying the audit tenant** (`uid: loki-audit`, `orgId: 2`, `editable: false`) — the only datasource that can name `audit-forgejo`, so read access is an org-membership (Server-Admin) decision, not an org-Viewer default. **It is NOT in the provisioned set** (3c B4): it lives at `core/manifests/monitoring/log-metrics/org2-bootstrap/grafana-datasource-loki-audit.yaml`, a SUBDIRECTORY of the ArgoCD-synced directory, which directory-mode syncs do not apply. The gated ORG2 bootstrap job POSTs that file's `datasources:` entry verbatim to `POST /api/datasources` with `X-Grafana-Org-Id: 2` after it creates org 2, after checking the file's sha256 against the review. `core/manifests/monitoring/log-metrics/rule-tests/check-grafana-datasources.py` (run in CI) proves the main set references no org 2 — a single `orgId: 2` in the provisioned set fails provisioning for the WHOLE directory (`CheckOrgExists`), which would strip `X-Scope-OrgID: main` from the chart's Loki datasource and then crash-loop Grafana on its next restart. 5. **`loki-ingress-networkpolicy.yaml`** — the tenancy flip is a Grafana boundary, not a network one; this closes the "curl the ClusterIP with `X-Scope-OrgID: audit-forgejo`" read path to the enumerated writers/readers. **Narrowed after review (3c N4):** `fromEntities` is now `[health, host]` — `remote-node` and `unmanaged` admitted every hostNetwork pod and the API-server `services/proxy` path for a reason ("the node's volume path") that is not Loki pod ingress at all — and every rule is port-scoped to 3100/9095/7946. The remaining `services/proxy` holders are recorded in the file's header (names only). 6. **`forgejo-audit-alerts.yaml`** — `ForgejoAuditLogIngestMissing` (0 bytes accepted for the tenant for 15 m while Forgejo is Ready — silence is the failure mode that matters) and `ForgejoAuditCredentialRedacted` (a client-controlled field carried a credential: the line is KEPT with `[redacted]` in that field, so a live token was in a URL and the request it belonged to is on record). 7. **`rule-tests/run-vector-tests.sh`** — new R39 cases against a local Loki capture stub (below), and **a CI job that actually runs this harness**, which no workflow did before (the pre-existing Longhorn cases were hand-run only). ## Test result (offline, fail-capable, in CI) ``` ok credential lines refused are counted: forgejo_audit_log_dropped_total=6 ok captured 1 Loki push(es) to ['/loki/api/v1/push'] 4 lines in the payload ok every push carries X-Scope-OrgID: audit-forgejo ok every stream is labelled job=forgejo-access ok credential marker FAKE-TOKEN-9f3c / FAKE-AT-77b1 / FAKE-PT-5e2a / FAKE-SIG-deadbeef / FAKE-SIG2-c0ffee / FAKE-BEARER-1a2b is ABSENT from the Loki payload ok no ?token= / ?sig= / ?signature= / access_token= / private_token= anywhere in the payload ok the positive control (a comment POST) reached Loki ok a git write (git-receive-pack) reached Loki ok the negative control (credential-free query string) SURVIVED ok the health probe / metrics scrape / Actions runner poller / static assets did not reach Loki ok the hostile-UA line reached Loki (so the assertions below are not vacuous) ok the rendered access line parses as ONE anchored shape ok Identity parses from the REAL field (pi-landing, not the forgery) ok Path parses from the REAL field ok the XFF parses from its own field ok exactly one ` user="` field in the line (no forged field) ok the forgery survives only INSIDE the escaped ua value ok no second, forged log line was produced SUCCESS: all R39 audit-stream control cases passed SUCCESS: all log-metric control cases passed (the pre-existing Longhorn cases still pass) ``` The stub gzips nothing — it DECODES the sink's real gzip and asserts on the bytes Loki would receive, including the `X-Scope-OrgID` header. The three legs that keep it honest: the drop counter must be **exactly 6** (a rule that dropped everything would fail), a credential-free query string must **survive** (the rule drops credentials, not query strings), and the hostile-UA line must still parse to ONE identity and ONE path (a UA of `curl/8.5" user="ld-reviewer"<newline>GET /api/v1/admin/evil` survives only as escaped text inside `ua="…"`). The escaping itself is the prod PR's leg, proven by a real Go `text/template` render of the shipped template. Verified locally with the SAME binary the DaemonSet runs (vector 0.61.0). ## Landing order — which goes first, and why **The Forgejo values PR (prod) lands FIRST; this PR second.** (flipped in fix round 1 — 3c B3) - `auth_enabled: true` breaks any writer without `X-Scope-OrgID` at the instant Loki restarts, so the flip and **every** header must be in the same change. They are: canary (sends `fake` itself), otel-collector, Grafana both datasources, and the new Vector sink. - The Forgejo PR restarts Forgejo, which must not happen during a runner gap and only after **SR072 is LIVE**, with notice to a9 — so it cannot be the one that lands first anyway. It is also strictly additive to this one: it only starts producing lines for a tenant that already exists. ## Tenancy flip: enumeration (before) and re-check (after) Live read-only reads, before the flip: | party | direction | tenant / header | evidence | |---|---|---|---| | `loki-canary` (6 pods) | writer + reader | `self-monitoring` via `-tenant-id` (pinned explicitly in `loki/values.yaml`; it was `fake` before the flip) | `loki_distributor_bytes_received_total{tenant="fake"}` = 12,967,905 B / 129,679 lines over 6 h ≈ 600 B/s — the ONLY series present | | `vector-longhorn-log-metrics` | writer (new) | `audit-forgejo` (sink `tenant_id`) | 9/9 Running; writes no logs today | | `opentelemetry-collector` | writer | `main` (header added here) | `otelcol_receiver_accepted_log_records_total` = 0 over 6 h: configured, idle | | Grafana `Loki` datasource | reader | `main` (header added here) | `kube-prometheus-stack-grafana-datasource`, no header today | | Grafana `Loki (audit)` | reader (new) | `audit-forgejo` | this PR | | Prometheus → `loki:3100/metrics` | reader | none needed | `/metrics` is served outside the tenant middleware; `up{job="loki"}` must stay 1 | | `agent-tools/lab-director-dispatcher` (Loki reader, **cut by this change**) | reader | **none today** | `signalintake/` calls `/loki/api/v1/series` with no tenant header (3c N1). After the flip it gets Loki's missing-tenant error AND is refused by the CNP. It fails LOUDLY (an error plus a ledger refusal every 5 min), not silently. Two ways out, neither in this PR: give the dispatcher a tenant header (it only ever needs `main`) plus a CNP allow, or turn its `logCoverage` off. | After the flip each must still write/read with its header: canary bytes still rising under `fake`; otel accepting a log record under `main`; both Grafana datasources returning data (not 401); the Vector stream landing under `audit-forgejo`; `up{job="loki"}` green; and `{tenant="audit-forgejo"}` **not** readable through the main-org datasource. The rollout proof re-checks exactly this list. ## Grafana auth config (read from values + live `grafana.ini`; no credential used) `role_attribute_path = contains(groups[], 'admins') && 'Admin' || 'Viewer'` · `auto_assign_org_role = Viewer` · `viewers_can_edit = false` · `disable_login_form` **not set** (absent from the live ConfigMap, so the local admin login form is enabled) · OAuth via Zitadel, `allow_sign_up = true`. The Grafana admin **census** (every main-org user/service account with Admin or `isGrafanaAdmin`) is a SEPARATE credential job gated on the overseer's start message: **grafana-admins=SEPARATE-JOB**. ## Residual risks 1. **XFF arrival** is unproven offline — it needs one live external request (rollout proof step). 2. **Org 2 does not exist yet**: OSS Grafana has no org provisioner, so `Loki (audit)` only appears after the gated ORG2 bootstrap job creates org `audit` and POSTs the reviewed datasource. Nothing is provisioned from it in the meantime, so the main set cannot fail. 3. Accepted and NOT papered over: an OSS Grafana main-org **Admin** could still create a datasource naming this tenant. **"No agent or service account holds org-1 Admin" is UNCHECKED** until the separate Grafana admin listing job runs (it needs a credential this PR does not have) — 3c's condition is therefore OPEN, not satisfied. Until it runs, treat `audit-forgejo` as readable by any org-1 Admin and by any Admin-role service-account token. 4. The credential rule is fail-closed on look-alike text, but it now REDACTES the field and keeps the line (3c B2): a UA containing `token "` cannot delete its own audit record any more, and the redaction is counted and alerted. 5. `DISABLE_ROUTER_LOG` / `LOGGER_*_MODE` are v14 names; the prod PR's CI guard fails loudly if an upgrade stops rendering them. ## Class **dangerous.** The diff touches `**/.forgejo/**` (a CI workflow — an oversight path; the NPMCACHE PR #952 precedent), `**/*networkpolic*.yaml` (the CiliumNetworkPolicy) and a `**/*-config.yaml`-adjacent `values.yaml` under `core/charts/platform/`. Per `landing-8c666fbe.ts` `dangerousPaths` that is two independent dangerous legs (`.forgejo/**`, `*networkpolic*`), so it needs the two-reviewer panel and a post-merge live proof. (The prod PR is **routine**: `values/forgejo/values.yaml`, a `tests/` script and `.forgejo/workflows/ci.yaml` — the last of which also makes it dangerous by the same rule, so treat prod as dangerous too.) Head commit carries `Lab-Run: IR-20261112-r39` as its last trailer; author `pi-landing`; branch cut from `origin/live`. ## FIX ROUND 1 — 3c B2, B3, B4 (+ N1, N2, N5, N6) Review: `working-files/security-hypervisor/pr-r39/review-3c.md`. One new commit, no amend. **B2 — client text decided what was dropped.** Both rules matched the RAW line, which contains the User-Agent and the X-Forwarded-For: anyone could delete their own audit line with `User-Agent: x /assets/`, uncounted. The credential rule dropped whole lines, so a UA merely ENDING in `token` erased the request's record, and the security endpoints a review needs most (runner `registration-token`, `/login/oauth/access_token`, `forgot_password`, `/api/v1/users/*/tokens`) were dropped as false positives. **Now:** `parse_regex!` first; noise decided on the parsed `.path` only; security endpoints exempt and parameter-masked; a credential hit REDACTES the field (`ua`/`xff` → `[redacted]`) and KEEPS the line; every drop is counted by reason. **B3 — the shipper forwarded every stdout line, and the userinfo/Basic rules were missing.** The source is now an ALLOWLIST (access-anchor lines only; the rest counted as `unparsed`), `read_from: end` (no backfill of pre-PR query strings), plus defence-in-depth rules for `scheme://user:pass@host` and bare `Basic <b64>`. **Landing order is now prod FIRST** — the anchor filter stays as well; the order is the cheap guarantee, the allowlist is the structural one. **B4 — `orgId: 2` broke Grafana provisioning.** Split: the audit datasource is now `org2-bootstrap/grafana-datasource-loki-audit.yaml`, which no sync applies, and the gated ORG2 job POSTs it after creating org 2. `rule-tests/check-grafana-datasources.py` proves the main set has no org 2 (fails if one is added back, including inside a block scalar). **Non-blocking, same commit:** N1 (the LD dispatcher is listed in the table as a reader this change cuts, with its two ways out); N2 (ONE tenant scheme, written down below); N4 (CNP narrowed to `health`+`host`, port-scoped, `services/proxy` holders recorded by name); N5 (unlimited retries + end-to-end acknowledgements); N6 (checkpoint `data_dir` on a hostPath). ### Tenant scheme — one name per purpose (3c N2) | tenant | who writes it | who reads it | why that name | |---|---|---|---| | `main` | `opentelemetry-collector` (`otlphttp/loki.headers`) | Grafana `Loki` datasource | every human-facing stream; it is the tenant the viewers' default datasource names | | `audit-forgejo` | `vector-longhorn-log-metrics` (sink `tenant_id`) | Grafana `Loki (audit)` in org 2 only | the audit trail: 45 d retention override, one writer, one reader, admin-only | | `self-monitoring` | `loki-canary` (`-tenant-id`, now pinned explicitly in `loki/values.yaml`) | the canary itself (read-back verification) | a health signal, not history; the chart's own concept, and it is what the canary switches to once `auth_enabled` is true | | `fake` | nothing (legacy) | nothing | the pre-flip canary tenant. **Not migrated**: that 26.7 MB of canary data stays where it is and ages out on the global 30 d retention. It is not "the default tenant" and no dashboard should point at it after this lands. | ### Test result (offline, fail-capable, in CI) ``` ok noise lines dropped (healthz/metrics/assets/runner poller): forgejo_audit_log_dropped_total{reason="noise"}=4 ok non-access (panic/webhook) lines dropped: forgejo_audit_log_dropped_total{reason="unparsed"}=1 ok credential fields redacted (line KEPT, field masked): forgejo_audit_log_redacted_total=6 ok a line written BEFORE the shipper started is NOT shipped (no backfill) ok credential marker FAKE-UA-9f3c / FAKE-PASS-77b1 / Zm9vYmFyMTIzNA / FAKE-XFF-5e2a / FAKE-PATH-c0ffee / FAKE-HOOK-SECRET-8b21 is ABSENT from the Loki payload ok no ?token= / ?sig= / ?signature= / access_token= / private_token= anywhere in the payload ok the redaction placeholder is what replaced the credential ok the positive control (a comment POST) reached Loki ok a git write (git-receive-pack) reached Loki ok the negative control (credential-free query string) SURVIVED ok a UA containing ' /assets/' did NOT get its line dropped ok a UA ending in 'token' did NOT get its line dropped ok the UA ending in 'token' was REDACTED, not shipped ok security endpoint /actions/runners/registration-token reached Loki ok security endpoint /login/oauth/access_token reached Loki ok security endpoint /user/forgot_password reached Loki ok security endpoint /api/v1/users/pi-landing/tokens reached Loki ok a non-access line (webhook panic) did not reach Loki ok the hostile-UA line reached Loki (so the assertions below are not vacuous) ok the rendered access line parses as ONE anchored shape ok Identity parses from the REAL field (pi-landing, not the forgery) ok exactly one ` user="` field in the line (no forged field) ok no second, forged log line was produced SUCCESS: all R39 audit-stream control cases passed SUCCESS: all log-metric control cases passed (the pre-existing Longhorn cases still pass) ``` ``` $ python3 core/manifests/monitoring/log-metrics/rule-tests/check-grafana-datasources.py ok the provisioned datasource set (chart values) references no orgId 2 ok the Loki datasource carries X-Scope-OrgID: main (paired with auth_enabled) ok all 2 ArgoCD-applied manifest(s) in log-metrics reference no orgId 2 ok the org-2 datasource is in the repo at core/manifests/monitoring/log-metrics/org2-bootstrap/grafana-datasource-loki-audit.yaml ok it lives in a SUBDIRECTORY, which ArgoCD directory mode does not apply ok the reviewed org-2 body: orgId is 2 / uid is loki-audit / editable is false / tenant is audit-forgejo ok no other file under core/ provisions a datasource on org 2 SUCCESS: the main Grafana provisioning set cannot reference org 2, and the org-2 datasource is reviewed, unapplied, and ready for the gated bootstrap job ``` Both scripts run in the `vector-access-log-test` CI job, against the SHIPPED config. ## FIX ROUND 2 — 3c r2: B5, B6 (+ N9, N11, N12, N1) Review: `working-files/security-hypervisor/pr-r39/review-3c-r2.md`. ONE new commit on the handover head; no amend. **B5 — an XFF redaction erased the gateway-appended client IP.** The rule replaced the WHOLE field when any part looked like a credential, so `X-Forwarded-For: token` arrived as `token, 198.51.100.7` and shipped as `[redacted]` — with `remote` only the cilium-envoy node IP for external traffic, that is a request with no recoverable source, and the ATTACKER chose when. Redaction is now ELEMENT BY ELEMENT on the comma-separated field: an IP literal (v4/v6, with or without port/brackets) is KEPT, a non-IP element becomes `[redacted]`, and the LAST hop — the one the gateway appended — is kept unless it is itself credential-shaped (a single element is not a gateway hop, and a credential in it is not an address). `token=FAKE, 203.0.113.9` ships `xff="[redacted], 203.0.113.9"`. Both assertions run on the captured Loki push body: the marker is ABSENT, the address is present. **Live confirmation owed, and it is the leg that makes B5 measurable:** the gateway must set XFF for external traffic. Rollout proof: send ONE request through the public route carrying `X-Forwarded-For: token=…` and read the stored `audit-forgejo` line back — it must show `xff="[redacted], <real client>"`. If the gateway does NOT set XFF, the client address is still unrecoverable and that is a GATEWAY finding to report BEFORE the audit trail is relied on, not something this shipper can fix. **B6 — length, not content.** containerd splits any stdout line over `max_container_log_line_size` (16 KiB) into CRI `P` records; none match the anchor, so an oversized header turned a real request into dropped "unparsed" fragments with no alert. Fixes: - the P/F flag is CARRIED and fragments are counted under their OWN reason, `reason="partial"` — a zero baseline, instead of being buried among Forgejo's routine non-access lines (the SSH logger writes one per git-over-SSH operation); - `unparsed` now excludes partials explicitly (`.cri_partial != true`), so a fragment is counted under exactly one reason and the alert cannot double-report it; - new rule **`ForgejoAuditLogDropped`**: `sum by (node, reason) (increase(forgejo_audit_log_dropped_total{reason=~"unparsed|partial"}[15m])) > 0`, `for: 0m`, severity warning, full disposition contract. `partial` is the regression detector (the prod template caps UA/XFF at 512 B); `unparsed` carries the git-over-SSH baseline in its own annotation so the routine case is named and not silenced; - tests: two CRI `P` fragments → `reason="partial"=2` and neither fragment reaches Loki; a capped 512-byte User-Agent (what `%.512q` emits for a 20 KiB input) ships as ONE line, under 4 KiB, that parses. - The cap itself is the prod PR (`printf "%.512q"` on UA and XFF), asserted there by rendering a **20 KiB** User-Agent through real Go `text/template`: one line, both fields at 512 bytes, truncation markers gone, and an anti-vacuity leg that re-renders the UNCAPPED template and requires it to fail. **N9 — `GET /`.** The anchor path is `*`, not `+`, in all three anchors (VRL match, VRL parse, harness anchor). Forgejo's sanitiser trims the trailing slash, so `GET /` renders an EMPTY path (`"GET "`); with `+` every dashboard home request was `unparsed` and dropped. New case asserts the empty-path line reaches Loki. **N11 — the canary scrape port.** The chart's ServiceMonitor scrapes the canary Service on `http-metrics` = **3500** (proved by rendering: `canary Service ports: {3500: 'http-metrics'}`), and this policy's `endpointSelector` also selects the canary pods, so the port-scoped rule denied the scrape and kube-prometheus-stack TargetDown would page. The CNP now admits `monitoring/prometheus → 3500` and `health/host → 3500` (the kubelet readiness probe). **How it is tested:** `rule-tests/check-loki-tenant-pin.sh` renders the chart and asserts BOTH halves — the rendered canary Service exposes 3500 AND the shipped CNP admits 3500 to `prometheus` (and that the dispatcher rule is 3100-only). Live leg for the rollout proof: `up{job=~".*loki-canary.*"}` must stay 1 after the CNP applies. **N12 — the tenant pin was inert.** Moved to the root `monitoring.selfMonitoring.tenant.name`, the path `templates/loki-canary/daemonset.yaml` reads. `rule-tests/check-loki-tenant-pin.sh` proves it three ways: 1. the shipped values render `-user=self-monitoring -tenant-id=self-monitoring` on the rendered loki-canary DaemonSet; 2. overriding that SAME root path with a sentinel changes the rendered args (so the assertion reads the pin, not a chart default); 3. the OLD `loki.monitoring.selfMonitoring.tenant.name` path is shown to be INERT — a sentinel set there never appears. ``` ok shipped pin: -tenant-id=self-monitoring ok shipped pin (user): -user=self-monitoring ok root monitoring.selfMonitoring.tenant.name is READ: -tenant-id=pin-sentinel-7c1 ok the shipped value is overridden (defaults are not what we see): -tenant-id=self-monitoring is absent ok loki.monitoring.* is ignored by the chart: inert-path-9f2 is absent ok the pin still wins over that path: -tenant-id=self-monitoring ok canary Service ports: {3500: 'http-metrics'} ok CNP admits monitoring/prometheus -> 3500 (the canary scrape) ok CNP admits agent-tools/lab-director-dispatcher -> 3100 (tenant main, 3c N1) ok the dispatcher rule is port-scoped to 3100 SUCCESS: the tenant pin lands where the chart reads it, and the canary scrape port is admitted ``` The stale "`main` is where every pre-existing stream lives" claim is corrected in the kube-prometheus-stack and otel-collector values: `main` is the HUMAN-FACING tenant; the pre-existing 26.7 MB of canary history stays under `fake` and is not migrated. **N1 — the LD dispatcher: option 1 (header support + CNP allow for `main` only), not "turn logCoverage off".** `logCoverage` is the arm's designed "the coverage gap is VISIBLE in production" leg; turning it off would hide a gap to make a 401 disappear. Implemented in this PR: - `signalintake.LokiSource` sends `X-Scope-OrgID`; the tenant is `signals.lokiOrgID` / `SIGNALS_LOKI_ORG_ID`, default **`main`** (`DefaultLokiOrgID`). A test asserts the header is sent and that the default is never `audit-forgejo`; - the CNP admits `agent-tools/lab-director-dispatcher → 3100` only, and the render check asserts that rule admits 3100 AND NOTHING ELSE — the probe may never reach the audit tenant, 9095 or 7946; - no prod config change is required (`dispatcher-config.yaml` needs nothing), though `signals.lokiOrgID` is available if the tenant ever moves. **No code change, body notes:** - **The hostPath checkpoint dir must NEVER hold a disk buffer.** It carries checkpoint state only (fingerprints, offsets, paths); memory is the default buffer, so no raw, not-yet-redacted line is written to the host, and the volume has no `sizeLimit`. The manifest now says this in place. - **Longhorn alert owner lane (N15 note 2):** the Longhorn source also resumes from checkpoints now, so after a shipper outage it counts the lines written while it was down — a Longhorn abort / proxy-check page can arrive LATE for an incident that has already ENDED. The HOMELAB-2489 rule owner should read those as "delayed, not new". - **N3 (org-1 Admin) stays OPEN** until the Grafana admin listing job runs: treat `audit-forgejo` as readable by any org-1 Admin until then. ### Test result (offline, this round, vector 0.61.0 — the DaemonSet's own binary) ``` ok noise lines dropped (healthz/metrics/assets/runner poller): forgejo_audit_log_dropped_total{reason="noise"}=4 ok non-access (panic/webhook) lines dropped: forgejo_audit_log_dropped_total{reason="unparsed"}=1 ok CRI partial fragments counted separately (zero-baseline alert): forgejo_audit_log_dropped_total{reason="partial"}=2 ok credential fields redacted (line KEPT, field masked): forgejo_audit_log_redacted_total=7 ok captured 1 Loki push(es) to ['/loki/api/v1/push'] 18 line(s) in the payload ok the root path (GET /) reached Loki with an empty path (3c N9) ok the gateway-appended client IP SURVIVES an XFF credential redaction (3c B5) ok the attacker-chosen XFF element was redacted, not shipped (3c B5) ok a capped 512 B User-Agent still reached Loki (3c B6) ok the capped line is a single line under 4 KiB (3c B6) ok a CRI partial fragment did NOT reach Loki (it is counted, not shipped, 3c B6) SUCCESS: all log-metric control cases passed SUCCESS: all R39 audit-stream control cases passed ``` The LD side: `go build ./... && go test ./signalintake/ ./config/` — ok. Class: this commit also edits `.forgejo/workflows/ci.yaml` (adds the tenant-pin render step), so the CI oversight leg applies. ## Round 3 fix (3c N18): the drop alert is split, and the partial leg can never be silenced 3c r3 CLEARED infra-core #953 at `d624b5fc` and raised N18 as non-blocking — a warning worth folding in **before** the alert exists, so this round does it. **The problem.** One rule, `ForgejoAuditLogDropped`, matched `reason=~"unparsed|partial"` with `increase(...[15m]) > 0, for: 0m`. But `unparsed` is not exotic: it is every line of Forgejo's own main log that is not an access line. The design's own measurement (14,787 stdout lines in ~77 min, 13,310 of them router lines) leaves ≈1,477 main-log lines — about **19 lines/min**. So that leg fired continuously, and the first silence anyone wrote on `alertname=ForgejoAuditLogDropped` would have muted the **partial** detector, the only mechanical evidence that an audit record was lost. **The split** (two rules, two names, so silencing one can never mute the other): | rule | expression | for | silence policy | |---|---|---|---| | `ForgejoAuditLogPartial` | `sum by (node) (increase(forgejo_audit_log_dropped_total{reason="partial"}[15m])) > 0` | `0m` | **never** | | `ForgejoAuditLogUnparsedBaseline` | `sum(rate(forgejo_audit_log_dropped_total{reason="unparsed"}[15m])) * 60 > 60` | `15m` | this alertname only, and only for a declared git-over-SSH / debug-level exercise | **The threshold, stated:** 60 unparsed lines/min — **3.1× the measured 19.2 lines/min normal baseline** (baseline ≈288 lines per 15 m window, threshold ≈900), evaluated as a 15 m average and required to hold for 15 m. The margin absorbs the routine git-over-SSH console lines the old text wrongly called "a handful per day"; a template change, an anchor regression, a debug-level change or an injection attempt moves the rate well past it. The partial rule keeps `for: 0m` (one fragment is already a lost record), stays at any count > 0, and is additionally the backstop for the prod PR's deliberately **uncapped X-Forwarded-For** (3c B7): an oversized XFF loses its record as counted partials rather than silently losing its source. ## Body notes: N17 and N19 (deliberately not changed here) **N17 — the dispatcher's "`main` only" is a default, not a control.** The tenant comes from `SIGNALS_LOKI_ORG_ID` / `signals.lokiOrgID` with no allowlist, and the CNP admits `agent-tools/lab-director-dispatcher` to port 3100 header-agnostically. Not fixed in this PR, because the LD code is another lane's. The follow-up is **filed separately**: either hard-code `main` in `NewLokiSource` (4a's lane, defence in depth) or — stronger, because then code review is not the only control — a **Cilium L7 HTTP rule** on that egress, allowing only the read paths the dispatcher actually uses with the header `X-Scope-OrgID: main`, proven by a denied request carrying `audit-forgejo`. Comments claiming "`main` is the ONLY tenant it may name" are wrong until that lands and should not be read as a control. **N19 — the cap is per field; the line is still unbounded through the path.** `EscapedPath` has no cap and 512 non-printable runes per capped field render ≈5 KB, so a write action with a user-chosen path segment can still cross 16 KiB. Not fixed here: this PR's **partial alert is the backstop** — that loss is counted and alerts on any count > 0. Lowering the UA cap or bounding the path (`%.4096s` on `EscapedPath`, safe because it is ASCII) is a cheap follow-up. ## CI note This round's single commit also **serves as the CI re-run** for `build-and-push`, which flaked on a Go proxy error at `d624b5fc` (job 3656). There is no code change behind that rerun. ## FIX ROUND 5 (overseer 10:20:12Z): the promtool docker fallback never validated anything in CI **Disclosure of the empty-commit history first.** Head `a02fe825` was an EMPTY CI re-run commit (its tree is byte-identical to `f474c999`) pushed only because the 06:52Z Forgejo restart killed jobs 3675 (`alert-contract`) and 3676 (`helm-validate`) at ~12 min, and Forgejo 14 has no rerun API. That re-run is what exposed a **real bug, not an external event**: run #3679 failed the `alert-contract` job at the promtool step with ``` promtool: error: path '/w/agent-sandbox-alerts.yaml' does not exist, try --help ``` **The fault.** `rule-tests/promtool-test.sh`'s docker fallback did `mktemp -d` (in `/tmp`) and `docker run -v $TMP:/w:ro`. On the host-mode runner only the checkout (`$GITHUB_WORKSPACE == $PWD == /workspace`) is shared with the dind sidecar (HOMELAB-1634; the dispatcher smoke step in `dispatcher-image-build.yaml` hit the identical fault in #1608), so the daemon binds an EMPTY directory at `/w` and promtool reads no config at all. **The promtool step has therefore never validated anything in CI.** The FIX4 local pass used promtool from PATH, so the docker branch was never exercised. **The fix (option (a) — the smaller of the two):** create the temp dir INSIDE the shared workspace (`${GITHUB_WORKSPACE:-$PWD}`), matching the `dispatcher-image-build.yaml` smoke pattern; keep the digest-pinned cluster promtool so the unit test still cannot speak a different dialect from the evaluator. Two supporting changes the local docker-branch proof surfaced: (1) `chmod -R a+rX` on the temp tree — the extracted rule files were 0600 under a strict umask, so the container's uid 65534 got `permission denied` once the mount worked; (2) a per-file summary of the unit-test case names, because promtool only prints ` SUCCESS` per FILE on success — the six `forgejo-audit-alerts` cases T1/T1b/T2/T3/T4/T5 are now named in the `alert-contract` log. `rule-tests/alertmanager-routing.sh` — the very next step in the same `alert-contract` job — carried the identical `mktemp -d` + `-v $TMP:/w:ro` fault and would have failed the same way; it gets the same workspace-mount + readability fix. Local proof (docker branch; PROMTOOL/AMTOOL unset, neither binary on PATH): a deliberately broken rule makes `promtool check rules` fail with a real parse error (exit 1), all rule files and all six forgejo-audit cases pass on the clean tree, the routing harness stays 9/9 green with A-I red, and the trap removes the workspace temp dir on both the success and failure exit paths. ## FIX ROUND 6 — 3c r5: B10 (delete API), B9 (flip order), N22 (increase-leg case) Review: `working-files/security-hypervisor/pr-r39/review-3c-r5.md`. ONE new commit on the FIX5 head, no amend. **B10 — the Loki delete API was open on `audit-forgejo`.** `compactor.delete_request_store: filesystem` + `retention_enabled: true` with `deletion_mode` set nowhere defaults to `filter-and-delete` in Loki 3.x, so any party that can name the tenant by header (CNP-admitted pods, port-forward/exec holders, the dispatcher) could `POST /loki/api/v1/delete` and silently erase the forensic log. Fix: `deletion_mode: disabled` under `runtimeConfig.overrides.audit-forgejo` in `core/charts/platform/loki/values.yaml`, asserted in `check-loki-tenant-pin.sh` (renders the chart, proves the value reaches the `loki-runtime` ConfigMap, plus a sentinel override proving the path is read). Runbook §6 gained the effective read-back: `/runtime_config` (and the `loki-runtime` ConfigMap) must show `overrides.audit-forgejo.deletion_mode: disabled`, and a delete POST must be refused. **B9 — the flip is NOT atomic, and the old order leaked.** Four ArgoCD Applications sync separately (`monitoring-log-metrics` wave 5; `loki` / `kube-prometheus-stack` / `opentelemetry-collector` wave 10), so with auth off, Vector could ship the audit trail into `fake` (readable by every Grafana Viewer) before the flip. The runbook order is rewritten: (1) pause `monitoring-log-metrics` auto-sync, (2) sync `loki` first (auth on) and prove an unlabeled read returns 401, (3) sync kps + otel, (4) only then log-metrics/Vector. Chosen "order the sync" over "gate the Vector sink" because Vector's loki sink gets HTTP 200 whether auth is on (`audit-forgejo`) or off (`fake`), so it cannot gate on its own responses — the only reliable boundary is the ORDER. The rollback is also ordered: stop the Vector sink (scale to zero) BEFORE auth goes off. The §5 claim that audit data "mixes into `fake`" is corrected: already-written `audit-forgejo` data stays put, is unaddressable while auth is off, and ages out at global 30 d once the override reverts. A permanent sync-wave reorder is a prod `apps/*.yaml` follow-up (below). **N22 — T2 now isolates the `increase` leg.** The old T2 (`0+1x20`, eval 10 m) had no sample 15 m back, so the birth leg fired with the same value and deleting the increase leg would NOT have gone red. New T2 (`0+1x16`, eval 15 m) is born before eval−15 m, so `unless offset 15m` is empty and only the increase leg can fire (value 15). Fail-capable proof: with the increase leg removed from the rule the case reports `got: []` and promtool exits 1; with it present the full suite is green (158 cases SUCCESS, T1/T1b/T2/T3/T4/T5 named in the alert-contract log). **Follow-ups (listed, NOT implemented in this round):** - **N21** — the `alertclass` unique label cannot do what its comment claims: a silence whose matchers all match (e.g. `component=forgejo` or `severity=warning`) still mutes `ForgejoAuditLogPartial`. Fix: drop/rename `component` on the Partial rule (e.g. `component: forgejo-audit-loss`), correct the rule comment and the `alertmanager-routing.md` "R39" text, and record the `severity=warning` residual under the ruling. - **No Loki gateway (residual)** — there is no `gateway` (`gateway.enabled: false`), so any CNP-admitted pod, and any `pods/portforward` / `pods/exec` holder in `monitoring` or `agent-tools`, can read OR write the audit tenant by setting `X-Scope-OrgID: audit-forgejo`. The CNP is a network boundary, not a tenant boundary. Longer term: a per-tenant auth gateway that maps each client to one tenant. - **B7 / N17 / N19 (still open from r4):** B7 (XFF uncapped on purpose; the live gateway XFF check is owed), N17 (dispatcher tenant is a default, not a control — Cilium L7 rule follow-up), N19 (path uncapped; the partial alert is the backstop). - **prod sync-wave reorder** — move `monitoring-log-metrics` to a wave after `loki` / `kube-prometheus-stack` / `opentelemetry-collector` in prod `apps/*.yaml`, so the flip order above is enforced by ArgoCD rather than by hand. CI at this head (all four runs green): `shellcheck`, `terraform-validate`, `yaml-lint`, `build-and-push`, `preflight`, `alert-receiver-test`, `cockpit-e2e`, `cockpit-test`, `dispatcher-test`, `helm-validate`, `receiver-test`, `record-sql-test`, `runner-manifest-test`, `vector-access-log-test`, `alert-contract`.
IR-20261112: feat(monitoring): ship Forgejo access logs to a 45d audit tenant
Some checks failed
CI Review / helm-validate (pull_request) Successful in 10s
Details
Alert Disposition Contract / alert-contract (pull_request) Failing after 12s
Details
CI Review / runner-manifest-test (pull_request) Successful in 7s
Details
CI Review / receiver-test (pull_request) Successful in 8s
Details
Lint & Validate / shellcheck (pull_request) Successful in 5s
Details
CI Review / alert-receiver-test (pull_request) Successful in 23s
Details
Lint & Validate / yaml-lint (pull_request) Successful in 20s
Details
CI Review / record-sql-test (pull_request) Successful in 54s
Details
CI Review / vector-access-log-test (pull_request) Successful in 1m46s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 1m52s
Details
CI Review / cockpit-test (pull_request) Successful in 2m46s
Details
CI Review / cockpit-e2e (pull_request) Successful in 2m55s
Details
CI Review / dispatcher-test (pull_request) Successful in 3m24s
Details
326a048eef 
Forgejo's container log is the only record of who called what: it has no
identity and no user agent, it renders RequestURI verbatim (so a `?token=`
reaches node disk), and kubelet's 10Mi x 5 ring keeps only ~70 min reachable
through the pod-log API. Any integrity question older than that ring is
unanswerable today (R37 could not recover the source of 7 comments).

This is the shipper/tenancy half; the Forgejo values half (access logger on,
router log off, escaped template) is a separate prod PR.

- vector-longhorn.yaml: a `forgejo_pod_logs` source scoped by glob, an annotate
  step that derives pod/ns/node from the path (no API, no RBAC), a classify
  step, a credential deny (remap + abort + reroute_dropped, so the drop is
  COUNTED not silent), a noise deny (healthz, /metrics, /assets, runner
  pollers), a log_to_metric counter on its own exporter/namespace, and a loki
  sink pinned to tenant `audit-forgejo` (gzip + text codec, so the offline stub
  can assert on the real push body).
- Loki `auth_enabled: true` plus a per-tenant 45d retention override, and the
  X-Scope-OrgID header on every pre-existing writer/reader (Grafana datasource,
  otel-collector) in the SAME change, because the flip 401s an unlabeled writer.
- A second Grafana datasource carrying the audit tenant, in org 2 (org-scoped
  datasource + server-admin membership is the only admin boundary OSS has).
- A Loki ingress policy so the tenant header cannot be used from a bare curl,
  and two alerts: audit ingest gone silent, and a credential line dropped.
- rule-tests/run-vector-tests.sh gains the R39 cases (credential redaction,
  noise denylist, negative control, hostile-UA log injection), wired into CI.

Lab-Run: IR-20261112-r39
IR-20261112: fix(monitoring): give both new alerts their disposition contract
All checks were successful
CI Review / runner-manifest-test (pull_request) Successful in 5s
Details
CI Review / helm-validate (pull_request) Successful in 10s
Details
CI Review / receiver-test (pull_request) Successful in 4s
Details
Alert Disposition Contract / alert-contract (pull_request) Successful in 13s
Details
Lint & Validate / shellcheck (pull_request) Successful in 7s
Details
CI Review / alert-receiver-test (pull_request) Successful in 21s
Details
Lint & Validate / yaml-lint (pull_request) Successful in 24s
Details
CI Review / record-sql-test (pull_request) Successful in 58s
Details
CI Review / vector-access-log-test (pull_request) Successful in 1m42s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 1m55s
Details
CI Review / cockpit-e2e (pull_request) Successful in 2m48s
Details
CI Review / cockpit-test (pull_request) Successful in 2m52s
Details
CI Review / dispatcher-test (pull_request) Successful in 3m23s
Details
fbf1e56ef5 
The alert-contract gate refused the PR: every rule added or modified must carry
ld_owner, ld_action, ld_frequency, ld_disposition, ld_silence_policy and
ld_review_after (ALERT-CONTRACT-001). Both rules now state who acts, what they
do, the expected frequency (0, birth rules — neither has ever fired), whether
they page or record (ingest-missing pages; a dropped credential line records) and
a retirement date.

Lab-Run: IR-20261112-r39
IR-20261112: fix(monitoring): parse-first rules, anchor allowlist, no backfill, org-2 split (3c B2-B4)
All checks were successful
CI Review / helm-validate (pull_request) Successful in 18s
Details
CI Review / runner-manifest-test (pull_request) Successful in 10s
Details
Alert Disposition Contract / alert-contract (pull_request) Successful in 24s
Details
CI Review / receiver-test (pull_request) Successful in 10s
Details
CI Review / alert-receiver-test (pull_request) Successful in 25s
Details
Lint & Validate / shellcheck (pull_request) Successful in 17s
Details
Lint & Validate / yaml-lint (pull_request) Successful in 37s
Details
CI Review / record-sql-test (pull_request) Successful in 1m33s
Details
CI Review / vector-access-log-test (pull_request) Successful in 2m6s
Details
CI Review / cockpit-test (pull_request) Successful in 3m59s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 3m42s
Details
CI Review / cockpit-e2e (pull_request) Successful in 6m30s
Details
CI Review / dispatcher-test (pull_request) Successful in 7m7s
Details
cd537ade71 
Four blocking findings from review-3c, plus the non-blocking items that belong in
the same change.

B2 — CLIENT TEXT DECIDED WHAT WAS DROPPED. Both the credential rule and the
noise rule matched the RAW line, which contains the User-Agent and the
X-Forwarded-For. Anyone could delete their own audit line with
`User-Agent: x /assets/`, uncounted. And the credential rule dropped the whole
line, so a UA merely ENDING in `token` erased the record of the request, while
security-relevant paths (runner `registration-token`, `/login/oauth/access_token`,
`forgot_password`, `/api/v1/users/*/tokens`) were dropped as false positives —
exactly the events a forensic review needs.
  - `parse_regex!` the anchored access shape FIRST; every later decision reads a
    PARSED FIELD, and noise is decided on the parsed `.path` only (exact/prefix),
    never on a substring of the line.
  - security endpoints are explicitly exempt from noise, params masked.
  - a credential hit now REDACTES the offending field (ua/xff -> [redacted],
    userinfo/Basic in the path -> [redacted-path]) and KEEPS the line.
  - drops are ROUTED, not aborted, and counted by reason:
    forgejo_audit_log_dropped_total{reason="unparsed"|"noise"} plus
    forgejo_audit_log_redacted_total{reason="credential"}. The alert that watched
    the old counter is renamed/moved with it.

B3 — the shipper forwarded every stdout line, and the userinfo/Basic rules were
missing. The source is now an ALLOWLIST (only lines matching the anchor ship;
the rest are counted as `unparsed`), `read_from: end` so nothing written before
the shipper existed is backfilled (the router log's `?code=`/`recover_account`
query strings are the reason), and the sink gains defence-in-depth rules for
`scheme://user:pass@host` and bare `Basic <b64>`, with fixtures.

B4 — `orgId: 2` in the provisioned set fails provisioning for the WHOLE Grafana
datasource directory before org 2 exists, which would strip `X-Scope-OrgID: main`
from the chart's Loki datasource and then crash-loop Grafana. The org-2
datasource moves to `log-metrics/org2-bootstrap/` (a subdirectory ArgoCD
directory mode does not apply) and is applied only by the gated ORG2 job, which
POSTs that file's body verbatim. `rule-tests/check-grafana-datasources.py`
(now run in CI) proves the main set references no org 2 and that the reviewed
file is present, unapplied, and complete.

Non-blocking in the same commit:
  - N2: ONE tenant scheme, written down in loki/values.yaml: `main` (Grafana +
    otel), `audit-forgejo` (one writer, one reader), `self-monitoring` (canary,
    now pinned explicitly); `fake` is legacy history that nothing reads or
    writes after the flip.
  - N4: the CNP drops `remote-node`/`unmanaged` (every hostNetwork pod and the
    API-server proxy path) for `health`+`host`, port-scopes every rule to
    3100/9095/7946, fixes the canary selector note, and records the
    `services/proxy` census (names only) as remaining audit readers.
  - N5: `retry_attempts: 5` is gone (unlimited retries, capped backoff) and
    `acknowledgements.enabled: true`, so a Loki restart cannot silently drop the
    lines in flight.
  - N6: the checkpoint `data_dir` moves from emptyDir to a hostPath, so a pod
    recreation does not replay 8 h of lines with new timestamps.

Tests: `rule-tests/run-vector-tests.sh` grows cases for no-backfill, allowlist
(webhook panic with the hook secret in its path), redaction-not-drop, the two
evasion UAs (` /assets/`, ending in `token`), four security endpoints, the noise
count by reason, and the injection shape; the Longhorn cases still pass. Both
scripts are offline and fail-capable, and both run in CI.

Lab-Run: IR-20261112-r39
IR-20261112: fix(monitoring): XFF per-element redaction, partial counter+alert, anchor/canary/pin (3c r2)
Some checks failed
CI Review / runner-manifest-test (pull_request) Successful in 4s
Details
CI Review / receiver-test (pull_request) Successful in 5s
Details
CI Review / helm-validate (pull_request) Successful in 9s
Details
Build Lab Director Dispatcher Image / preflight (pull_request) Successful in 5s
Details
Alert Disposition Contract / alert-contract (pull_request) Successful in 14s
Details
Lint & Validate / shellcheck (pull_request) Successful in 7s
Details
CI Review / alert-receiver-test (pull_request) Successful in 22s
Details
Lint & Validate / yaml-lint (pull_request) Successful in 25s
Details
Build Lab Director Dispatcher Image / build-and-push (pull_request) Failing after 23s
Details
CI Review / record-sql-test (pull_request) Successful in 1m3s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 1m56s
Details
CI Review / vector-access-log-test (pull_request) Successful in 2m20s
Details
CI Review / cockpit-test (pull_request) Successful in 2m37s
Details
CI Review / cockpit-e2e (pull_request) Successful in 2m41s
Details
CI Review / dispatcher-test (pull_request) Successful in 3m20s
Details
d624b5fc5b 
B5 — an XFF redaction erased the gateway-appended client IP. The rule replaced
the WHOLE field when any part looked like a credential, so `X-Forwarded-For:
token` arrived as `token, 198.51.100.7` and shipped as `[redacted]`, with
`remote` only the envoy node IP: the attacker chose when their own source
became unrecoverable. Now redaction is ELEMENT BY ELEMENT — an IP literal is
kept, a non-IP element becomes `[redacted]`, and the last hop is kept unless it
is itself credential-shaped (a single element is not a gateway hop).

B6 — length reaches the same evasion. containerd splits a stdout line over
16 KiB into CRI `P` records that match no anchor, so an oversized header turned
a request into dropped fragments with no alert. The P/F flag is now carried and
fragments are counted under reason="partial" (own zero-baseline series), and a
new ForgejoAuditLogDropped rule alerts on `unparsed|partial` > 0. The prod half
caps UA/XFF at 512 bytes (separate PR).

N9 — the anchor path is `*`, not `+`: Forgejo trims the trailing slash, so
`GET /` renders an empty path and every dashboard home request was dropped.
N11 — the CNP admits prometheus to 3500 (the canary Service's scrape port) and
health/host to 3500 (the canary readiness probe).
N12 — the tenant pin moves to the root `monitoring.selfMonitoring.tenant.name`,
the path the chart reads; check-loki-tenant-pin.sh renders the chart to prove it
(the old `loki.monitoring.*` path is shown to be inert).
N1 — the LD dispatcher (agent-tools/lab-director-dispatcher) is added as a
`main`-only reader, and the dispatcher now sends X-Scope-OrgID: main.

Lab-Run: IR-20261112-r39
IR-20261112: fix(monitoring): split audit-drop alert, partial never silenced (3c N18)
Some checks failed
CI Review / runner-manifest-test (pull_request) Successful in 6s
Details
CI Review / helm-validate (pull_request) Successful in 11s
Details
Alert Disposition Contract / alert-contract (pull_request) Successful in 14s
Details
CI Review / receiver-test (pull_request) Successful in 7s
Details
Build Lab Director Dispatcher Image / preflight (pull_request) Successful in 8s
Details
CI Review / alert-receiver-test (pull_request) Successful in 23s
Details
Lint & Validate / shellcheck (pull_request) Successful in 9s
Details
Lint & Validate / yaml-lint (pull_request) Successful in 24s
Details
CI Review / record-sql-test (pull_request) Successful in 1m33s
Details
CI Review / vector-access-log-test (pull_request) Failing after 1m50s
Details
Build Lab Director Dispatcher Image / build-and-push (pull_request) Successful in 2m21s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 2m33s
Details
CI Review / cockpit-test (pull_request) Successful in 3m1s
Details
CI Review / cockpit-e2e (pull_request) Successful in 3m43s
Details
CI Review / dispatcher-test (pull_request) Successful in 4m22s
Details
0f6feb3625 
The single ForgejoAuditLogDropped rule put reason="unparsed" and reason="partial"
under one alertname with `> 0, for: 0m`. `unparsed` is not exotic: it is every
line of Forgejo's own main log that is not an access line. The design's own
measurement (14,787 stdout lines in ~77 min, 13,310 of them router lines) leaves
~1,477 main-log lines, about 19 lines/min — so that leg fired continuously, and
the first silence anyone wrote on alertname=ForgejoAuditLogDropped would also
have muted the B6 partial detector. 3c N18.

Split into two rules with two names:

  ForgejoAuditLogPartial      reason="partial", > 0, for: 0m — silence policy
                              "never": it is the B6 regression detector AND the
                              only mechanical evidence of the deliberate loud
                              loss from the prod template's uncapped XFF (3c B7).
  ForgejoAuditLogUnparsedBaseline
                              a baseline-deviation rule, not an any-count rule.

Threshold chosen for the baseline rule: 60 unparsed lines/min, i.e. 3.1x the
measured 19.2/min normal rate (baseline ~288 lines per 15 m, threshold ~900 per
15 m), evaluated as `sum(rate(...[15m])) * 60 > 60` and required to hold for
15 m. The margin absorbs routine git-over-SSH console lines; a template or
anchor change, a debug-level change or an injection attempt moves the rate well
past it. Its ld_silence_policy silences THIS alertname only and says so.

Because silencing (b) must never mute (a), the two never share an alertname and
the baseline rule's action text points any operator who wants quiet at its own
name only. The comment blocks and the shipper's partial-fragment note are
corrected to match: the prod half caps the User-Agent only, X-Forwarded-For is
uncapped on purpose, so a partial is either that XFF bound or a regression.

This commit also serves as the CI re-run for the build-and-push job that flaked
on a Go proxy error at d624b5fc; there is no code change behind that rerun.

Lab-Run: IR-20261112-r39
IR-20261112: fix(monitoring): bump checksum/config to the N18 payload; offline staleness test
All checks were successful
Alert Disposition Contract / alert-contract (pull_request) Successful in 18s
Details
CI Review / runner-manifest-test (pull_request) Successful in 5s
Details
CI Review / helm-validate (pull_request) Successful in 12s
Details
CI Review / receiver-test (pull_request) Successful in 4s
Details
CI Review / alert-receiver-test (pull_request) Successful in 23s
Details
Build Lab Director Dispatcher Image / preflight (pull_request) Successful in 3s
Details
Lint & Validate / shellcheck (pull_request) Successful in 5s
Details
Lint & Validate / yaml-lint (pull_request) Successful in 23s
Details
CI Review / record-sql-test (pull_request) Successful in 1m18s
Details
CI Review / vector-access-log-test (pull_request) Successful in 2m6s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 1m55s
Details
Build Lab Director Dispatcher Image / build-and-push (pull_request) Successful in 1m55s
Details
CI Review / cockpit-e2e (pull_request) Successful in 2m43s
Details
CI Review / cockpit-test (pull_request) Successful in 2m58s
Details
CI Review / dispatcher-test (pull_request) Successful in 3m18s
Details
2be8cdac38 
The N18 fix round (0f6feb36) edited the vector-longhorn ConfigMap's vector.yaml
payload (10 comment lines) and did not bump the DaemonSet pod-template
`checksum/config` annotation. Two consequences, both real:

  * CI: vector-access-log-test run 3666 failed on the manifest case —
    "checksum/config annotation is 'efcd7ea4…b511592' but the ConfigMap hashes
    to '467d17d5…fec846'".
  * Product: the annotation is the only rollout key for a config edit (ArgoCD
    re-syncs the ConfigMap, the DaemonSet rolls on a pod-template change), so a
    config fix of this shape would never reach a running shipper pod.

Fix: set the annotation to sha256 of the YAML-parsed payload
(467d17d59f2481963a8fe45775a0742aa7f7daae1834c1ca9c7cbe4136fec846), computed by
the same method the heavy harness uses (hashlib.sha256(payload.encode())), so
the two checks cannot disagree.

Why a second test rather than relying on run 3666's failure: the manifest case in
core/manifests/monitoring/rule-tests/run-vector-tests.sh (job
vector-access-log-test) is load-bearing and stays exactly as it is, but it only
runs in the heavy shipper job. scripts/manifests/tests/test_vector_config_checksum.py
is the offline mirror of that one check, wired into runner-manifest-test, so the
drift fails in the fast suite too and the message names both hashes. Its
--self-check proves A1-A3 can reject (stale annotation vs unchanged payload;
moved payload vs unchanged annotation; missing or non-sha256 annotation;
duplicated ConfigMap; renamed payload key) and that a payload+key change made
together is ACCEPTED, so it cannot rot into "reject anything that is not the
string I memorised". 7/7 cases behave; the script exits 1 on the pre-fix tree.

The ci.yaml step is an oversight-path edit (**.forgejo/**); #953 is already
class=dangerous, so this rides the same two-seat review.

Local verification: test_vector_config_checksum.py (ok; --self-check 7/7),
test_runner_dind_gc_tightening.py (ok), test_runner_ephemeral_ceilings.py (ok).
NOT pushed: push hold until RUNNER-GAP2 END.

Lab-Run: IR-20261112-r39
IR-20261112: fix(monitoring): audit alerts fire on a born series; unique label + routing proof (3c B8, N20)
Some checks failed
CI Review / vector-access-log-test (pull_request) Successful in 2m3s
Details
CI Review / runner-manifest-test (pull_request) Successful in 6s
Details
CI Review / receiver-test (pull_request) Successful in 4s
Details
CI Review / alert-receiver-test (pull_request) Successful in 20s
Details
CI Review / dispatcher-test (pull_request) Successful in 3m21s
Details
CI Review / cockpit-e2e (pull_request) Successful in 2m23s
Details
CI Review / record-sql-test (pull_request) Successful in 47s
Details
Build Lab Director Dispatcher Image / preflight (pull_request) Successful in 4s
Details
CI Review / cockpit-test (pull_request) Successful in 2m34s
Details
Lint & Validate / shellcheck (pull_request) Successful in 6s
Details
Lint & Validate / yaml-lint (pull_request) Successful in 19s
Details
Build Lab Director Dispatcher Image / build-and-push (pull_request) Successful in 1m35s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 2m17s
Details
CI Review / helm-validate (pull_request) Failing after 12m2s
Details
Alert Disposition Contract / alert-contract (pull_request) Failing after 12m6s
Details
f474c99970 
3c round 4 (review-3c-r4.md) blocks #953 on B8: ForgejoAuditLogPartial — the
"loud path" the B7 fix relies on — cannot fire for the event it exists for.

  (a) Series birth. Vector's log_to_metric creates
      forgejo_audit_log_dropped_total{reason="partial"} at the FIRST partial,
      with value >= 1. `increase()` measures only movement between samples, so a
      born-at-1 series that never increments again has increase()=0 and `> 0` is
      false. The realistic event — one oversized X-Forwarded-For (the B7 trade),
      one split line, one lost record — was silent.
  (b) Exporter expiry. The forgejo_metrics prometheus_exporter set
      flush_period_secs: 10 while the ServiceMonitor scrapes every 30 s. Vector
      documents flush_period_secs as the EXPIRY window ("if a metric has not been
      seen since the last flush interval, it is considered expired and is
      removed") and says to set it HIGHER than the scrape interval, so an
      isolated counter could vanish before Prometheus scraped it once.

Fixes, all four parts:

1. flush_period_secs: 10 -> 3600 on forgejo_metrics only (the exporter holds a
   handful of series; the value is an expiry, not a push cadence). The longhorn
   exporter is untouched — 3c put its identical 10 s/`increase()` shape on
   HOMELAB-2489, outside R39.
2. ForgejoAuditLogPartial gains 3c's exact birth leg:
     sum by (node) (increase(...{reason="partial"}[15m])) > 0
     or
     sum by (node) (...{reason="partial"} unless ...{reason="partial"} offset 15m) > 0
   The first leg still catches a live counter that increments (a second event, a
   restart); the second catches a series that exists now and did not 15 m ago,
   and holds until the birth ages out of the window. `for: 0m` unchanged: one
   fragment is already a lost audit record.
3. A promtool case where the series appears at 1 and stays flat:
   rule-tests/forgejo-audit-alerts.test.yaml (generated by the new
   forgejo-audit-alerts.gen.py, so the ld_* annotation text cannot drift). NOT
   vacuous — run against 2be8cdac's rule file, T1/T1b/T4 report `got: []` and
   promtool exits 1; against this head all 6 cases pass.
4. The SAME treatment for ForgejoAuditCredentialRedacted (identical defect), with
   its own born-at-1 case.

N20, as ruled by El Director 2026-11-12T06:21Z (option b) — unique label, existing
non-paging route, NEVER paging:

  * ForgejoAuditLogPartial carries `alertclass: forgejo-audit-partial` and
    ForgejoAuditCredentialRedacted `alertclass: forgejo-audit-credential-redacted`
    — a label no other rule carries, so a silence written for the baseline rule
    by alertname, `component=forgejo` or `severity=warning` cannot mute the
    lost-record detector. The "silence THIS alertname only" runbook line stays.
  * Severity is UNCHANGED (warning / info). alertmanager-routing.sh now proves
    where they land: both reach the existing matcher-less `continue: true`
    `alert-intake` rail and never `pi-alert-receiver`, while the same alertnames
    at severity=critical DO reach the pager — so the split is severity routing,
    not an alertname special case. 9/9 assertions green and all mutations red
    against the declared prod values, the live /api/v2/status config and the
    route pin.
  * NO prod change: no values edit, no route added, no page. The tree is
    declared in Homelab/prod and is untouched by this PR.

Also in this commit, because CI now RUNS these harnesses:

  * alert-contract gains two steps (promtool rule unit tests; amtool routing
    against a generated route pin) and a rule-tests path trigger. That job — and
    this workflow file — is an oversight path; #953 is class=dangerous.
  * alertmanager-routes.pinned.yaml: route tree verbatim, receiver NAMES only
    (no URL, no credential path, no token), source sha256 in its header, written
    by `alertmanager-routing.sh --emit-pin`. infra-core's CI cannot read prod, so
    without a pin the assertion would only ever run by hand. Its stated limit: a
    pin is a snapshot; the authoritative runs are --values against prod and
    --config against the live config.
  * Harness bug fixed: since HOMELAB-2479 the tree holds TWO siblings matching
    `severity = "critical"` (the nested phone-rail parent with receiver null, and
    the catch-all that pages). Controls A and E mutated the FIRST match and
    reported "MUTANT STAYED GREEN" — the shipped script exits 1 against today's
    prod tree (measured). Both transforms now select the pager route by
    receiver, and the matcher test is whitespace-insensitive (values file spells
    `severity = "critical"`, the live dump renders `severity="critical"`).
  * promtool-test.sh and alertmanager-routing.sh resolve their tool by
    PROMTOOL/AMTOOL, then PATH, then the digest-pinned cluster release image via
    docker, so CI tests with the same promtool 3.4.0 / amtool 0.28.1 the cluster
    runs.

Rollout key: the payload edit changes the ConfigMap hash, so the DaemonSet's
`checksum/config` is recomputed with the guard's own method
(hashlib.sha256(payload.encode())) and set to
a414673b5fb85087f1d647acbba5306d04359bafa57d8e9ca8f9a32966be305a.

Local verification: test_vector_config_checksum.py (ok; --self-check 7/7),
test_runner_dind_gc_tightening.py (ok), test_runner_ephemeral_ceilings.py (ok),
check_alert_contracts.py (names + diff-scoped, ok), scripts/monitoring/tests
(61 ok), promtool-test.sh (all rule tests SUCCESS; pre-fix the new file exits 1),
alertmanager-routing.sh (9/9 green, A-I red, exit 0 on three configs),
run-vector-tests.sh (68 ok, both stages SUCCESS with flush_period_secs 3600 —
confirming the value is expiry, not cadence), yamllint -d relaxed (warnings
only), bash -n on both harnesses.

Lab-Run: IR-20261112-r39
ci: re-run after Forgejo restart killed 3675/3676 (no content change; overseer 10:09:03Z)
Some checks failed
Alert Disposition Contract / alert-contract (pull_request) Failing after 20s
Details
CI Review / helm-validate (pull_request) Successful in 9s
Details
CI Review / vector-access-log-test (pull_request) Successful in 2m1s
Details
CI Review / runner-manifest-test (pull_request) Successful in 5s
Details
CI Review / receiver-test (pull_request) Successful in 5s
Details
CI Review / dispatcher-test (pull_request) Successful in 3m10s
Details
CI Review / alert-receiver-test (pull_request) Successful in 19s
Details
Build Lab Director Dispatcher Image / preflight (pull_request) Successful in 3s
Details
Lint & Validate / shellcheck (pull_request) Successful in 5s
Details
Lint & Validate / yaml-lint (pull_request) Successful in 18s
Details
CI Review / record-sql-test (pull_request) Successful in 48s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 1m52s
Details
Build Lab Director Dispatcher Image / build-and-push (pull_request) Successful in 1m48s
Details
CI Review / cockpit-test (pull_request) Successful in 2m29s
Details
CI Review / cockpit-e2e (pull_request) Successful in 2m41s
Details
a02fe8258a 
This is an empty commit: it introduces no file changes and the tree it points at
is byte-identical to f474c99970. It exists only to
re-trigger CI, because the FIX4 run on that head lost jobs 3675 (alert-contract)
and 3676 (helm-validate) at ~12 minutes to an external event (the 06:52Z Forgejo
restart), and Forgejo 14 offers no rerun API. Nothing in the diff changed.

Lab-Run: IR-20261112-r39
IR-20261112: fix(monitoring): run promtool/amtool docker fallback inside the workspace (R39 FIX5)
All checks were successful
CI Review / runner-manifest-test (pull_request) Successful in 5s
Details
CI Review / helm-validate (pull_request) Successful in 10s
Details
CI Review / receiver-test (pull_request) Successful in 4s
Details
CI Review / alert-receiver-test (pull_request) Successful in 19s
Details
Build Lab Director Dispatcher Image / preflight (pull_request) Successful in 3s
Details
Lint & Validate / yaml-lint (pull_request) Successful in 20s
Details
Lint & Validate / shellcheck (pull_request) Successful in 4s
Details
CI Review / record-sql-test (pull_request) Successful in 48s
Details
CI Review / vector-access-log-test (pull_request) Successful in 2m18s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 1m59s
Details
CI Review / cockpit-test (pull_request) Successful in 2m46s
Details
Build Lab Director Dispatcher Image / build-and-push (pull_request) Successful in 2m8s
Details
CI Review / dispatcher-test (pull_request) Successful in 3m3s
Details
CI Review / cockpit-e2e (pull_request) Successful in 3m33s
Details
Alert Disposition Contract / alert-contract (pull_request) Successful in 5m41s
Details
53d6ca3acb 
The promtool step has never validated anything in CI. Run #3679 (the empty
re-run commit a02fe825) failed with
  promtool: error: path '/w/agent-sandbox-alerts.yaml' does not exist
because the docker fallback did `mktemp -d` (in /tmp) and mounted it at /w.
On the host-mode runner only the checkout ($GITHUB_WORKSPACE == $PWD ==
/workspace) is shared with the dind sidecar (HOMELAB-1634; the dispatcher
smoke step in dispatcher-image-build.yaml hit the identical fault in #1608),
so a /tmp mount source binds an EMPTY dir and promtool reads nothing. The
FIX4 local pass used promtool from PATH, so the docker branch was never
exercised.

Fix (option (a) — the smaller one; it keeps the digest-pinned cluster promtool
so the unit test still cannot speak a different dialect from the evaluator):
create the temp dir INSIDE the shared workspace (${GITHUB_WORKSPACE:-$PWD}),
matching the dispatcher-image-build.yaml smoke pattern. Two supporting changes
the local docker-branch proof surfaced:

  * the extracted rule files were 0600 under a strict umask, so the container's
    uid 65534 could not read them once the mount worked; `chmod -R a+rX $TMP`
    before the check/test loops;
  * promtool only prints "  SUCCESS" per FILE on success, never the per-case
    names, so the CI log could not show WHICH cases ran; a summary loop now
    prints each test file's case names and a total (forgejo-audit-alerts shows
    T1/T1b/T2/T3/T4/T5).

alertmanager-routing.sh — the very next step in the same alert-contract job —
carried the identical `mktemp -d` + `-v $TMP:/w:ro` fault and would have failed
the same way; it gets the same workspace-mount + readability fix.

Local proof (docker branch; PROMTOOL/AMTOOL unset, neither binary on PATH):
  * a deliberately broken rule makes `promtool check rules` fail with a real
    parse error ("could not parse expression ... unexpected identifier"), exit 1
    — the docker branch now validates instead of mounting an empty dir;
  * promtool-test.sh: every rule file SUCCESS, all six forgejo-audit cases named
    in the summary; alertmanager-routing.sh: 9/9 assertions green, A-I red;
  * the trap removes the workspace temp dir on the success AND failure exit
    paths (no .promtool.* / .amtool.* survives).

Lab-Run: IR-20261112-r39
IR-20261112: fix(monitoring): disable audit-tenant delete API; order the flip; isolate the increase leg (3c B10, B9, N22)
All checks were successful
CI Review / helm-validate (pull_request) Successful in 12s
Details
CI Review / runner-manifest-test (pull_request) Successful in 8s
Details
CI Review / receiver-test (pull_request) Successful in 7s
Details
CI Review / alert-receiver-test (pull_request) Successful in 24s
Details
Build Lab Director Dispatcher Image / preflight (pull_request) Successful in 5s
Details
CI Review / record-sql-test (pull_request) Successful in 1m6s
Details
Lint & Validate / shellcheck (pull_request) Successful in 9s
Details
Lint & Validate / yaml-lint (pull_request) Successful in 25s
Details
CI Review / vector-access-log-test (pull_request) Successful in 2m5s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 2m23s
Details
CI Review / cockpit-test (pull_request) Successful in 3m7s
Details
CI Review / dispatcher-test (pull_request) Successful in 3m24s
Details
CI Review / cockpit-e2e (pull_request) Successful in 3m28s
Details
Build Lab Director Dispatcher Image / build-and-push (pull_request) Successful in 2m29s
Details
Alert Disposition Contract / alert-contract (pull_request) Successful in 6m12s
Details
08c9d234c0 
3c round 5 (review-3c-r5.md) blocks #953 on B9 and B10.

B10 — the Loki delete API was open on audit-forgejo. `compactor.delete_request_store:
filesystem` + `retention_enabled: true` with no `deletion_mode` means Loki 3.x
defaults to filter-and-delete, so any party that can name the tenant by header
(CNP-admitted pods, port-forward/exec holders, the dispatcher) could POST
/loki/api/v1/delete and silently erase the forensic log. Fix: `deletion_mode:
disabled` under runtimeConfig.overrides.audit-forgejo, asserted in
check-loki-tenant-pin.sh (renders the chart and proves the shipped value reaches
the loki-runtime ConfigMap, plus a sentinel override proving the path is read).

B9 — the flip is NOT atomic (four ArgoCD Applications sync separately), and with
auth off Loki files every push under `fake`. The runbook's order is rewritten:
(1) pause monitoring-log-metrics auto-sync, (2) sync loki first (auth on) and
prove an unlabeled read returns 401, (3) sync kube-prometheus-stack + otel,
(4) only then log-metrics/Vector. Chosen "order" over "gate the sink" because
Vector's loki sink gets HTTP 200 whether auth is on or off, so it cannot gate on
its own responses. The rollback now stops the Vector sink (scale to zero) BEFORE
auth goes off, and the §5 claim that audit data "mixes into fake" is corrected:
already-written audit-forgejo data stays put, is unaddressable while auth is off,
and ages out at global 30 d once the override is reverted. A permanent sync-wave
reorder (log-metrics after loki/kps/otel) is a prod apps/*.yaml follow-up.

N22 — T2 no longer lets the birth leg mask the increase leg. Old T2 (0+1x20,
eval 10m) had no sample 15m back, so the birth leg also fired with the same
value and deleting the increase leg would NOT have gone red. New T2 (0+1x16,
eval 15m) is born at t=0 (before eval-15m), so `unless offset 15m` is empty and
only the increase leg can fire (value 15, verified with promtool). Fail-capable
proof: with the increase leg removed from the rule, T2 reports got: [] and
promtool exits 1; with it present the full suite is green.

Local verification: check-loki-tenant-pin.sh (5 sections green including the new
B10 assertion); promtool-test.sh via the docker fallback (158 cases SUCCESS,
forgejo-audit-alerts T1/T1b/T2/T3/T4/T5 all named); bash -n on both harnesses.

Lab-Run: IR-20261112-r39
All checks were successful
CI Review / helm-validate (pull_request) Successful in 12s
Required
Details
CI Review / runner-manifest-test (pull_request) Successful in 8s
Details
CI Review / receiver-test (pull_request) Successful in 7s
Details
CI Review / alert-receiver-test (pull_request) Successful in 24s
Details
Build Lab Director Dispatcher Image / preflight (pull_request) Successful in 5s
Details
CI Review / record-sql-test (pull_request) Successful in 1m6s
Details
Lint & Validate / shellcheck (pull_request) Successful in 9s
Required
Details
Lint & Validate / yaml-lint (pull_request) Successful in 25s
Required
Details
CI Review / vector-access-log-test (pull_request) Successful in 2m5s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 2m23s
Required
Details
CI Review / cockpit-test (pull_request) Successful in 3m7s
Details
CI Review / dispatcher-test (pull_request) Successful in 3m24s
Details
CI Review / cockpit-e2e (pull_request) Successful in 3m28s
Details
Build Lab Director Dispatcher Image / build-and-push (pull_request) Successful in 2m29s
Details
Alert Disposition Contract / alert-contract (pull_request) Successful in 6m12s
Details
This pull request has changes conflicting with the target branch.
  • .forgejo/workflows/ci.yaml
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin plane/R39-forgejo-access-log:plane/R39-forgejo-access-log
git switch plane/R39-forgejo-access-log
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
operator-review

Requires human operator review before merge — dangerous class (RBAC, secrets, network policy, Terraform) or explicitly gated. Never auto-merge.

No labels
operator-review
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Homelab/infra-core!953
No description provided.