Homelab/infra-core
6
0
Fork 2
Code Issues Pull requests 35 Projects Releases 1 Packages Wiki Activity Actions

IR-20261112: feat(monitoring): enable Loki multi-tenancy and name every reader's tenant (R39 PR-A) #961

Open
pi-landing wants to merge 1 commit from plane/R39-forgejo-access-log-auth into live
pull from: plane/R39-forgejo-access-log-auth
merge into: Homelab:live
Conversation 2 Commits 1 Files changed 10 +300 -6
pi-landing commented 2026-11-12 13:05:31 +00:00
Collaborator
Copy link

Scope — R39 FIX7, PR-A of the split (the tenancy flip, no sink)

This is PR-A of the FIX7 split of Homelab/infra-core#953 (review-cd-r6
B9 / BLOCK-1). It lands the Loki multi-tenancy flip and names every
pre-existing writer/reader's tenant, and carries nothing that can ship the
audit trail — no Vector forgejo source/sink, no CNP, no audit alerts. Those
live in PR-B (plane/R39-forgejo-access-log-sink), which must not merge
until this PR is live and the unlabeled-401 proof holds.

What this PR does

  • loki.auth_enabled: false → true at core/charts/platform/loki/values.yaml:55 — every request without an X-Scope-OrgID (or Basic-auth) tenant becomes HTTP 401, writer and reader alike.
  • The runtimeConfig.overrides.audit-forgejo (45 d retention + deletion_mode: disabled) and the canary self-monitoring pin at the root path the chart actually reads.
  • The Grafana Loki datasource header (main, kube-prometheus-stack/values.yaml), the otel-collector otlphttp/loki header (main), and the LD dispatcher's X-Scope-OrgID: main (DefaultLokiOrgID).
  • check-loki-tenant-pin.sh (PR-A trim: pin + deletion_mode; no CNP section) wired into the new loki-auth-pin-test CI job.

r6 findings this closes

  • B9 (flip ordering / 401 proof / rollback) — closed by the split itself (below) plus the rollback rewrite (below).
  • BLOCK-1 (the auto-sync pause is reverted by root selfHeal) — closed by splitting: PR-B cannot ship the Vector sink until auth is proven live, so no "pause" mechanism is needed at all.
  • BLOCK-2 (rollback step 1 kubectl scale ds cannot run; single revert replays the leak) — closed by the two-commit rollback below.
  • NIT-1 (delete probe destructive on the real audit tenant) — closed by the probe-selector rule below.
  • NIT-2 (ForgejoAuditCredentialRedacted increase leg unisolated) — closed by the isolating promtool case below.

vecgate removed — and why

The FIX6 "vecgate=wave" claim is dropped. Sync waves order nothing here:
they only order root's apply of the Application CRs (apps-root/root.yaml),
and an infra-core live merge changes no CR in prod.git, so root does not even
sync — each child Application (loki, kube-prometheus-stack,
opentelemetry-collector, monitoring-log-metrics) auto-syncs independently on
its own refresh of infra-core live, and nothing orders those syncs. A manual
auto-sync pause on monitoring-log-metrics is likewise reverted by root
selfHeal within seconds (root re-applies the automated block). The split is
the git-native fix that survives selfHeal.

Rollback — TWO commits (stop the writer BEFORE auth goes off)

No kubectl scale ds anywhere (DaemonSets have no scale subresource).

  • R1 — revert PR-B only: remove the forgejo_pod_logs source, the loki_forgejo_audit sink, and the checkpoint hostPath together (they live in one manifest, vector-longhorn.yaml). Auth stays ON throughout R1.
  • Confirm R1 before R2: kubectl -n monitoring rollout status ds/vector-longhorn-log-metrics --timeout=5m, then wait for sum(rate(loki_distributor_lines_received_total{tenant="audit-forgejo"}[5m])) == 0.
  • R2 — only then revert PR-A (auth off, tenant overrides, Grafana/otel/dispatcher headers).

Checkpoint / e2e-ack replay (BLOCK-2): the hostPath
/var/lib/vector-longhorn-log-metrics carries checkpoint state only
(fingerprints/offsets) — never a disk buffer (memory is the buffer). R1 removes
source + sink + checkpoint hostPath together, so there is no reader and no sink
left to replay anything, and end-to-end acks mean no acked line is ever re-sent.
The sink's tenant_id: audit-forgejo header is NOT the control (once auth is off
in R2, Loki ignores headers and files everything under fake) — removing the
writer is.

Delete probe (NIT-1) — run BEFORE Vector ships

Right after the 401 proof (step 3 of the landing order), while the audit tenant
is still empty, probe the delete refusal with a probe selector, never
{job="forgejo-access"}:

curl -s -o /dev/null -w '%{http_code}\n' -X POST -H 'X-Scope-OrgID: audit-forgejo' \
  'http://127.0.0.1:3100/loki/api/v1/delete?query=%7Bjob%3D%22r39-delete-probe%22%7D&start=<now-1s>&end=<now>'

On any 2xx, cancel at once with DELETE /loki/api/v1/delete?request_id=<id>
(inside the 24 h cancel period) and stop. The expected refusal is "not 2xx"
(4xx). A delete against the real {job="forgejo-access"} must never be issued.

NIT-2 — isolating promtool case

Add T4b: 0+1x16 on reason="credential", eval 15 m, value 15 — the
isolating increase-leg case that turns red if rules:193 (the
ForgejoAuditCredentialRedacted increase leg) is deleted, mirroring the T2
coverage that already protects ForgejoAuditLogPartial.

Landing order (why PR-A alone is safe to merge)

  1. prod PR (Forgejo log values: DISABLE_ROUTER_LOG, LOGGER_ACCESS_MODE: console, safe ACCESS_LOG_TEMPLATE) — already separate.
  2. PR-A (this PR) — auth on, every pre-existing writer/reader still named.
  3. Prove the 401 negative on the LIVE Loki: unlabeled {job="forgejo-access"} query returns 401, X-Scope-OrgID: main returns 200.
  4. Only then PR-B — the Vector sink + CNP + alerts.

Follow-ups (listed, not fixed here)

  • Grafana org-1 Admin / Server-Admin census (who can mint a datasource naming audit-forgejo) — needs the separate GRAFANA-LIST credential job; until it runs, treat audit-forgejo as readable by any org-1 Admin / Admin-role SA token.
  • UI/API-created Grafana datasources not in a provisioning ConfigMap cannot be enumerated read-only — needs the same job.
  • NIT-3 (pin.sh renders only infra-core values, not prod values/loki/values.yaml; live read-back stays mandatory), NIT-4 (sentinel scope), NIT-5 (add an unlabeled push 401 to the 2b proof), NIT-6 (runbook staleness vs FIX6), NIT-7 (main/self-monitoring/fake keep filter-and-delete) — tracked, out of R39 scope.
## Scope — R39 FIX7, PR-A of the split (the tenancy flip, no sink) This is **PR-A** of the FIX7 split of `Homelab/infra-core#953` (review-cd-r6 B9 / BLOCK-1). It lands the Loki multi-tenancy flip and names every pre-existing writer/reader's tenant, and carries **nothing that can ship the audit trail** — no Vector forgejo source/sink, no CNP, no audit alerts. Those live in **PR-B** (`plane/R39-forgejo-access-log-sink`), which must not merge until this PR is live and the unlabeled-401 proof holds. ### What this PR does - `loki.auth_enabled: false → true` at `core/charts/platform/loki/values.yaml:55` — every request without an `X-Scope-OrgID` (or Basic-auth) tenant becomes HTTP 401, writer and reader alike. - The `runtimeConfig.overrides.audit-forgejo` (45 d retention + `deletion_mode: disabled`) and the canary `self-monitoring` pin at the root path the chart actually reads. - The Grafana `Loki` datasource header (`main`, `kube-prometheus-stack/values.yaml`), the otel-collector `otlphttp/loki` header (`main`), and the LD dispatcher's `X-Scope-OrgID: main` (`DefaultLokiOrgID`). - `check-loki-tenant-pin.sh` (PR-A trim: pin + `deletion_mode`; no CNP section) wired into the new `loki-auth-pin-test` CI job. ### r6 findings this closes - **B9 (flip ordering / 401 proof / rollback)** — closed by the split itself (below) plus the rollback rewrite (below). - **BLOCK-1 (the auto-sync pause is reverted by root selfHeal)** — closed by splitting: PR-B cannot ship the Vector sink until auth is proven live, so no "pause" mechanism is needed at all. - **BLOCK-2 (rollback step 1 `kubectl scale ds` cannot run; single revert replays the leak)** — closed by the two-commit rollback below. - **NIT-1 (delete probe destructive on the real audit tenant)** — closed by the probe-selector rule below. - **NIT-2 (ForgejoAuditCredentialRedacted increase leg unisolated)** — closed by the isolating promtool case below. ### `vecgate` removed — and why The FIX6 "vecgate=wave" claim is **dropped**. Sync waves order nothing here: they only order root's apply of the **Application CRs** (`apps-root/root.yaml`), and an infra-core `live` merge changes no CR in prod.git, so root does not even sync — each child Application (`loki`, `kube-prometheus-stack`, `opentelemetry-collector`, `monitoring-log-metrics`) auto-syncs independently on its own refresh of infra-core `live`, and nothing orders those syncs. A manual auto-sync pause on `monitoring-log-metrics` is likewise reverted by root `selfHeal` within seconds (root re-applies the `automated` block). The split is the git-native fix that survives selfHeal. ### Rollback — TWO commits (stop the writer BEFORE auth goes off) No `kubectl scale ds` anywhere (DaemonSets have no scale subresource). - **R1** — revert **PR-B only**: remove the `forgejo_pod_logs` source, the `loki_forgejo_audit` sink, and the checkpoint hostPath together (they live in one manifest, `vector-longhorn.yaml`). Auth stays ON throughout R1. - Confirm R1 before R2: `kubectl -n monitoring rollout status ds/vector-longhorn-log-metrics --timeout=5m`, then wait for `sum(rate(loki_distributor_lines_received_total{tenant="audit-forgejo"}[5m])) == 0`. - **R2** — only then revert PR-A (auth off, tenant overrides, Grafana/otel/dispatcher headers). **Checkpoint / e2e-ack replay (BLOCK-2):** the hostPath `/var/lib/vector-longhorn-log-metrics` carries checkpoint state only (fingerprints/offsets) — never a disk buffer (memory is the buffer). R1 removes source + sink + checkpoint hostPath together, so there is no reader and no sink left to replay anything, and end-to-end acks mean no acked line is ever re-sent. The sink's `tenant_id: audit-forgejo` header is NOT the control (once auth is off in R2, Loki ignores headers and files everything under `fake`) — removing the writer is. ### Delete probe (NIT-1) — run BEFORE Vector ships Right after the 401 proof (step 3 of the landing order), while the audit tenant is still empty, probe the delete refusal with a **probe selector, never `{job="forgejo-access"}`**: ``` curl -s -o /dev/null -w '%{http_code}\n' -X POST -H 'X-Scope-OrgID: audit-forgejo' \ 'http://127.0.0.1:3100/loki/api/v1/delete?query=%7Bjob%3D%22r39-delete-probe%22%7D&start=<now-1s>&end=<now>' ``` On **any 2xx**, cancel at once with `DELETE /loki/api/v1/delete?request_id=<id>` (inside the 24 h cancel period) and stop. The expected refusal is "not 2xx" (4xx). A delete against the real `{job="forgejo-access"}` must never be issued. ### NIT-2 — isolating promtool case Add **T4b**: `0+1x16` on `reason="credential"`, eval 15 m, value `15` — the isolating increase-leg case that turns red if `rules:193` (the `ForgejoAuditCredentialRedacted` increase leg) is deleted, mirroring the T2 coverage that already protects `ForgejoAuditLogPartial`. ### Landing order (why PR-A alone is safe to merge) 1. prod PR (Forgejo `log` values: `DISABLE_ROUTER_LOG`, `LOGGER_ACCESS_MODE: console`, safe `ACCESS_LOG_TEMPLATE`) — already separate. 2. **PR-A** (this PR) — auth on, every pre-existing writer/reader still named. 3. Prove the 401 negative on the LIVE Loki: unlabeled `{job="forgejo-access"}` query returns **401**, `X-Scope-OrgID: main` returns **200**. 4. Only then **PR-B** — the Vector sink + CNP + alerts. ### Follow-ups (listed, not fixed here) - **Grafana org-1 Admin / Server-Admin census** (who can mint a datasource naming `audit-forgejo`) — needs the separate GRAFANA-LIST credential job; until it runs, treat `audit-forgejo` as readable by any org-1 Admin / Admin-role SA token. - **UI/API-created Grafana datasources** not in a provisioning ConfigMap cannot be enumerated read-only — needs the same job. - NIT-3 (pin.sh renders only infra-core values, not prod `values/loki/values.yaml`; live read-back stays mandatory), NIT-4 (sentinel scope), NIT-5 (add an unlabeled **push** 401 to the 2b proof), NIT-6 (runbook staleness vs FIX6), NIT-7 (`main`/`self-monitoring`/`fake` keep `filter-and-delete`) — tracked, out of R39 scope.
IR-20261112: feat(monitoring): enable Loki multi-tenancy and name every reader's tenant (R39 PR-A)
All checks were successful
CI Review / receiver-test (pull_request) Successful in 6s
Details
CI Review / loki-auth-pin-test (pull_request) Successful in 8s
Details
CI Review / helm-validate (pull_request) Successful in 10s
Details
CI Review / runner-manifest-test (pull_request) Successful in 10s
Details
Build Lab Director Dispatcher Image / preflight (pull_request) Successful in 7s
Details
CI Review / alert-receiver-test (pull_request) Successful in 21s
Details
Lint & Validate / shellcheck (pull_request) Successful in 7s
Details
Lint & Validate / yaml-lint (pull_request) Successful in 22s
Details
CI Review / record-sql-test (pull_request) Successful in 1m0s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 2m7s
Details
CI Review / cockpit-test (pull_request) Successful in 2m42s
Details
CI Review / cockpit-e2e (pull_request) Successful in 2m59s
Details
CI Review / dispatcher-test (pull_request) Successful in 3m11s
Details
Build Lab Director Dispatcher Image / build-and-push (pull_request) Successful in 1m55s
Details
7f3581e897 
FIX7 splits infra-core #953 (plane/R39-forgejo-access-log) into two PRs so the
tenancy flip cannot leak the audit trail into `fake` (review-cd-r6 B9/BLOCK-1).
This is PR-A: it turns `loki.auth_enabled` on and gives every PRE-EXISTING
writer/reader its tenant header in the SAME change, plus the checks. PR-B (the
Vector audit sink + its CNP) is a separate PR cut from this head and lands only
after a live 401 on an unlabeled read is proven on the real cluster.

- loki/values.yaml: auth_enabled: true; runtimeConfig.overrides.audit-forgejo
  (retention 45d + deletion_mode disabled); self-monitoring canary pin at the
  root path the chart reads (3c N12).
- kube-prometheus-stack/values.yaml: Grafana `Loki` datasource header
  X-Scope-OrgID: main (secureJsonData).
- opentelemetry-collector/values.yaml: otlphttp/loki header X-Scope-OrgID: main.
- lab-director/dispatcher: the Loki coverage read names tenant `main`
  (DefaultLokiOrgID) so it does not go dark after the flip (3c N1).
- check-loki-tenant-pin.sh (PR-A trim): proves the pin lands where the chart
  reads it and the audit delete API is disabled; the CNP canary-scrape check
  (N11) ships in PR-B.
- ci.yaml: new loki-auth-pin-test job runs the check.

The old branch plane/R39-forgejo-access-log is left untouched (not deleted, not
force-pushed); #953 stays open until PR-A and PR-B supersede it.

Lab-Run: IR-20261112-r39a
ld-reviewer commented 2026-11-12 13:37:17 +00:00
Collaborator
Copy link

LD-VERDICT APPROVED head=7f3581e8970e3a7774def99f1fa342d6b938e7c3 session=IR-20261112-r39a-seatA-20261112T1323Z

model: omniroute/deepseek-flash
session: IR-20261112-r39a-seatA-20261112T1323Z

Seat A, run IR-20261112-r39a (head trailer Lab-Run: IR-20261112-r39a, last trailer, matches). Judged git diff 2d837e9bc0e199f14cc5f4927efbaea59edec0de...7f3581e8 in my own clone (10 files, +300/-6), the chart itself, and the new check. I did not list or read this PR's comments/reviews; I did not run verdict-cli; this comment is posted once, by the sanctioned tool.

LOADING CHECK (the thing that voided the first pair): exactly FOUR extensions and no others — /etc/ld/pi/extensions/pi-session-tag plus private copies of forgejo (9 files), redact (6 files) and herdr-agent-state.ts (1). Tool set exactly read/bash/grep/find/ls/forgejo_comment_as. Nothing under /home/ld/work/pi-agent/extensions/ auto-loaded. One detail I am recording rather than hiding: both private copy paths in my launch command resolve to /home/ld/work/transcripts/c961A.LQNvs4w8x4/ (another run's 0700 folder), not under my own s961A... folder; counts and modes match the brief and I read nothing there but a directory listing. /home/ld/.local absent at launch and at stop.

  1. THE FLIP LANDS WHERE THE CHART READS IT. I fetched the chart the script pins (loki chart 6.29.0, appVersion 3.4.2) and read the templates rather than trusting the comments:
  • loki/values.yaml:55 sets auth_enabled: true INSIDE the loki: block; the chart's embedded config template renders auth_enabled: {{ .Values.loki.auth_enabled }} (chart values.yaml:146), so it reaches the running Loki config. Correct path.
  • loki/values.yaml:92-102 loki.runtimeConfig.overrides.audit-forgejo -> chart templates/runtime-configmap.yaml renders ConfigMap loki-runtime with toYaml .Values.loki.runtimeConfig verbatim, and the chart already ships runtime_config.file: /etc/loki/runtime-config/runtime-config.yaml (chart values.yaml:194-195) with the ConfigMap mounted in SingleBinary mode (templates/single-binary/statefulset.yaml:187-188,230). So the 45 d retention_period and deletion_mode: disabled do reach the single-binary Loki this repo deploys — they are not values-file decoration. (Loki 3.4.2 is 3.x, so deletion_mode really does default to filter-and-delete when unpinned, i.e. the guard is not hypothetical.)
  • 3c N12 is real and is fixed: templates/loki-canary/daemonset.yaml:51-54 renders -user=/-tenant-id= from $.Values.monitoring.selfMonitoring.tenant.name, ONLY when loki.auth_enabled — i.e. this pin becomes load-bearing precisely because of this diff — and nothing in the chart reads loki.monitoring.*. The pin at monitoring.selfMonitoring.tenant.name: self-monitoring (values.yaml:140-141) is therefore at the read path, not the inert one.
  1. READER/WRITER CENSUS — no unnamed path left in this tree. Three header-carrying paths, all in this diff:
  • Grafana Loki datasource: kube-prometheus-stack/values.yaml:72-75, jsonData.httpHeaderName1: X-Scope-OrgID + secureJsonData.httpHeaderValue1: main (the documented Grafana pairing). Tempo's tracesToLogsV2.datasourceUid: loki reuses that same datasource, so it inherits the header.
  • otel-collector otlphttp/loki: opentelemetry-collector/values.yaml:117-119 headers: X-Scope-OrgID: main, and that exporter really is in the logs pipeline (:134-137) — a named writer, not a dead exporter.
  • dispatcher coverage probe: signalintake/logsource.go:113-115 sets the header from settings.LokiOrgID, default main (settings.go:45-49,82, DefaultLokiOrgID), config/env wired in arms_signals.go:44-46,73-79 and config/config.go:717-719. It cannot degrade to an empty header.
    Sweep for anything else: grep -rn "type: loki" -> exactly one datasource; grep -rl ":3100" -> those three paths plus config/example.go (doc text only); no promtail/alloy/fluent/grafana-agent in the tree; core/manifests/monitoring/log-metrics/vector-longhorn.yaml writes NOTHING to Loki (its only sink is prometheus_exporter), so it is not an unnamed pre-existing writer; the only other Loki traffic is Prometheus /metrics scrapes, which are not tenant-guarded. The canary (chart default lokiCanary.enabled: true) is the pre-existing writer the body names, and it lands in self-monitoring. Caveat I cannot close from this repo: prod-repo App manifests and the dispatcher Deployment env are outside the tree; the in-tree default (main) covers the dispatcher even if no env is set.

  1. SPLIT CLAIM CONFIRMED. git diff --name-only live...HEAD: 10 files, and the only manifest touched is core/manifests/monitoring/rule-tests/check-loki-tenant-pin.sh. No Vector sink/source, no CiliumNetworkPolicy/NetworkPolicy, no PrometheusRule/audit alert. audit-forgejo and "Vector" appear only in comments and in the check's own assertion. PR-A carries the flip and the guards only, as claimed.

  2. THE CHECK IS FAIL-CLOSED AND ON THE LIVE GATE. .forgejo/workflows/ci.yaml:90-104 adds loki-auth-pin-test to ci.yaml, which triggers on pull_request to live (ci.yaml:3-7) — same runs-on: k8s as helm-validate, so helm exists there. The script is set -euo pipefail, accumulates FAIL=1 and ends in exit 1; missing helm -> exit 1; chart pull failure -> exit 1 (no || true); the PyYAML block is || FAIL=1. It is not print-and-pass. It asserts the pin, proves the assertion is non-vacuous by sentinel-overriding the SAME path, proves the old loki.monitoring.* path stays inert, and proves deletion_mode: disabled from the rendered loki-runtime ConfigMap plus a sentinel override of that path. That is the right PR-A trim (no N11/CNP canary-scrape check, correctly deferred to PR-B).

  3. WHAT I RAN IN MY CLONE (no cluster, no image harness, no kubectl — the binary is present but never invoked): go test ./config/... ./signalintake/... ./cmd/... in lab-director/dispatcher — all pass, including the new TestLokiSourceNamesItsTenant (it would fail on an empty/absent header, so the guard is live); bash -n on the new script — clean. shellcheck, promtool, kustomize and helm are NOT installed in this sandbox, so I could not render; I substituted reading the pinned 6.29.0 templates directly, which answers the path questions but not the rendered output.

NON-BLOCKING DEFECTS (named, not holds — none of them can produce a false green, and none change the post-flip tenant of any named writer/reader):
a) check-loki-tenant-pin.sh render helper uses --set monitoring.lokiCanary.enabled=true and the comment above it says "The canary is OFF in the shipped values ... so it is switched ON for the render". In chart 6.29.0 the DaemonSet is gated on the TOP-LEVEL lokiCanary.enabled (templates/loki-canary/daemonset.yaml:1-2), whose chart default is true; monitoring.lokiCanary does not exist in that chart. So that --set is inert, the canary renders because of the chart default, and the repo's own monitoring.lokiCanary.enabled: false (loki/values.yaml:142-143, pre-existing, unchanged here) is likewise inert. The assertions are still correct and still fail closed (if the canary ever stops rendering, section 1 goes red), but the stated "switch it on" mechanism does not work, and the values line claims a disable that never happens. Worth a follow-up commit; it also means the rule-tests comment about "no self-monitoring in this cluster" is wrong on both counts.
b) The check needs CI egress (grafana.github.io index + the GitHub release URL it resolves to). I found no other CI job in this repo that depends on outbound fetch (helm-validate lints only local charts; run-vector-tests.sh downloads but is not wired into CI), so runner egress is unproven. If it is blocked the job goes red, i.e. fails closed, not silently green.
c) Cosmetic: the script's sections number 1,2,3,5 (no 4) and grep -A3 'name: loki-canary$' ... || true is dead code.

NOT VERIFIED (stated plainly): the comment's claim that CHART_VERSION=6.29.0 is "the same pin the ArgoCD Application renders" — the ArgoCD Application manifests are not in this repo (core/manifests/argocd does not exist), so if prod renders a different chart version the canary tenant path could differ; the CI check would only catch that drift at 6.29.0. I also did not verify Loki's runtime behaviour of per-tenant deletion_mode/retention_period on a live cluster (out of scope and forbidden here), only that the override reaches the ConfigMap the config points at.

No hold: every reader and writer that would 401 after the flip is named with a tenant in this same diff, the two guards are real and fail-closed, and PR-A is sink-free.

LD-VERDICT APPROVED head=7f3581e8970e3a7774def99f1fa342d6b938e7c3 session=IR-20261112-r39a-seatA-20261112T1323Z model: omniroute/deepseek-flash session: IR-20261112-r39a-seatA-20261112T1323Z Seat A, run IR-20261112-r39a (head trailer `Lab-Run: IR-20261112-r39a`, last trailer, matches). Judged `git diff 2d837e9bc0e199f14cc5f4927efbaea59edec0de...7f3581e8` in my own clone (10 files, +300/-6), the chart itself, and the new check. I did not list or read this PR's comments/reviews; I did not run verdict-cli; this comment is posted once, by the sanctioned tool. LOADING CHECK (the thing that voided the first pair): exactly FOUR extensions and no others — `/etc/ld/pi/extensions/pi-session-tag` plus private copies of `forgejo` (9 files), `redact` (6 files) and `herdr-agent-state.ts` (1). Tool set exactly read/bash/grep/find/ls/forgejo_comment_as. Nothing under `/home/ld/work/pi-agent/extensions/` auto-loaded. One detail I am recording rather than hiding: both private copy paths in my launch command resolve to `/home/ld/work/transcripts/c961A.LQNvs4w8x4/` (another run's 0700 folder), not under my own `s961A...` folder; counts and modes match the brief and I read nothing there but a directory listing. `/home/ld/.local` absent at launch and at stop. 1) THE FLIP LANDS WHERE THE CHART READS IT. I fetched the chart the script pins (loki chart 6.29.0, appVersion 3.4.2) and read the templates rather than trusting the comments: - `loki/values.yaml:55` sets `auth_enabled: true` INSIDE the `loki:` block; the chart's embedded config template renders `auth_enabled: {{ .Values.loki.auth_enabled }}` (chart `values.yaml:146`), so it reaches the running Loki config. Correct path. - `loki/values.yaml:92-102` `loki.runtimeConfig.overrides.audit-forgejo` -> chart `templates/runtime-configmap.yaml` renders ConfigMap `loki-runtime` with `toYaml .Values.loki.runtimeConfig` verbatim, and the chart already ships `runtime_config.file: /etc/loki/runtime-config/runtime-config.yaml` (chart `values.yaml:194-195`) with the ConfigMap mounted in SingleBinary mode (`templates/single-binary/statefulset.yaml:187-188,230`). So the 45 d `retention_period` and `deletion_mode: disabled` do reach the single-binary Loki this repo deploys — they are not values-file decoration. (Loki 3.4.2 is 3.x, so `deletion_mode` really does default to `filter-and-delete` when unpinned, i.e. the guard is not hypothetical.) - 3c N12 is real and is fixed: `templates/loki-canary/daemonset.yaml:51-54` renders `-user=`/`-tenant-id=` from `$.Values.monitoring.selfMonitoring.tenant.name`, ONLY when `loki.auth_enabled` — i.e. this pin becomes load-bearing precisely because of this diff — and nothing in the chart reads `loki.monitoring.*`. The pin at `monitoring.selfMonitoring.tenant.name: self-monitoring` (values.yaml:140-141) is therefore at the read path, not the inert one. 2) READER/WRITER CENSUS — no unnamed path left in this tree. Three header-carrying paths, all in this diff: - Grafana `Loki` datasource: `kube-prometheus-stack/values.yaml:72-75`, `jsonData.httpHeaderName1: X-Scope-OrgID` + `secureJsonData.httpHeaderValue1: main` (the documented Grafana pairing). Tempo's `tracesToLogsV2.datasourceUid: loki` reuses that same datasource, so it inherits the header. - otel-collector `otlphttp/loki`: `opentelemetry-collector/values.yaml:117-119` `headers: X-Scope-OrgID: main`, and that exporter really is in the `logs` pipeline (`:134-137`) — a named writer, not a dead exporter. - dispatcher coverage probe: `signalintake/logsource.go:113-115` sets the header from `settings.LokiOrgID`, default `main` (`settings.go:45-49,82`, `DefaultLokiOrgID`), config/env wired in `arms_signals.go:44-46,73-79` and `config/config.go:717-719`. It cannot degrade to an empty header. Sweep for anything else: `grep -rn "type: loki"` -> exactly one datasource; `grep -rl ":3100"` -> those three paths plus `config/example.go` (doc text only); no promtail/alloy/fluent/grafana-agent in the tree; `core/manifests/monitoring/log-metrics/vector-longhorn.yaml` writes NOTHING to Loki (its only sink is `prometheus_exporter`), so it is not an unnamed pre-existing writer; the only other Loki traffic is Prometheus `/metrics` scrapes, which are not tenant-guarded. The canary (chart default `lokiCanary.enabled: true`) is the pre-existing writer the body names, and it lands in `self-monitoring`. Caveat I cannot close from this repo: prod-repo App manifests and the dispatcher Deployment env are outside the tree; the in-tree default (`main`) covers the dispatcher even if no env is set. 3) SPLIT CLAIM CONFIRMED. `git diff --name-only live...HEAD`: 10 files, and the only manifest touched is `core/manifests/monitoring/rule-tests/check-loki-tenant-pin.sh`. No Vector sink/source, no CiliumNetworkPolicy/NetworkPolicy, no PrometheusRule/audit alert. `audit-forgejo` and "Vector" appear only in comments and in the check's own assertion. PR-A carries the flip and the guards only, as claimed. 4) THE CHECK IS FAIL-CLOSED AND ON THE LIVE GATE. `.forgejo/workflows/ci.yaml:90-104` adds `loki-auth-pin-test` to `ci.yaml`, which triggers on `pull_request` to `live` (ci.yaml:3-7) — same `runs-on: k8s` as `helm-validate`, so helm exists there. The script is `set -euo pipefail`, accumulates `FAIL=1` and ends in `exit 1`; missing helm -> exit 1; chart pull failure -> exit 1 (no `|| true`); the PyYAML block is `|| FAIL=1`. It is not print-and-pass. It asserts the pin, proves the assertion is non-vacuous by sentinel-overriding the SAME path, proves the old `loki.monitoring.*` path stays inert, and proves `deletion_mode: disabled` from the rendered `loki-runtime` ConfigMap plus a sentinel override of that path. That is the right PR-A trim (no N11/CNP canary-scrape check, correctly deferred to PR-B). 5) WHAT I RAN IN MY CLONE (no cluster, no image harness, no kubectl — the binary is present but never invoked): `go test ./config/... ./signalintake/... ./cmd/...` in `lab-director/dispatcher` — all pass, including the new `TestLokiSourceNamesItsTenant` (it would fail on an empty/absent header, so the guard is live); `bash -n` on the new script — clean. shellcheck, promtool, kustomize and helm are NOT installed in this sandbox, so I could not render; I substituted reading the pinned 6.29.0 templates directly, which answers the path questions but not the rendered output. NON-BLOCKING DEFECTS (named, not holds — none of them can produce a false green, and none change the post-flip tenant of any named writer/reader): a) `check-loki-tenant-pin.sh` render helper uses `--set monitoring.lokiCanary.enabled=true` and the comment above it says "The canary is OFF in the shipped values ... so it is switched ON for the render". In chart 6.29.0 the DaemonSet is gated on the TOP-LEVEL `lokiCanary.enabled` (`templates/loki-canary/daemonset.yaml:1-2`), whose chart default is `true`; `monitoring.lokiCanary` does not exist in that chart. So that `--set` is inert, the canary renders because of the chart default, and the repo's own `monitoring.lokiCanary.enabled: false` (loki/values.yaml:142-143, pre-existing, unchanged here) is likewise inert. The assertions are still correct and still fail closed (if the canary ever stops rendering, section 1 goes red), but the stated "switch it on" mechanism does not work, and the values line claims a disable that never happens. Worth a follow-up commit; it also means the rule-tests comment about "no self-monitoring in this cluster" is wrong on both counts. b) The check needs CI egress (grafana.github.io index + the GitHub release URL it resolves to). I found no other CI job in this repo that depends on outbound fetch (helm-validate lints only local charts; `run-vector-tests.sh` downloads but is not wired into CI), so runner egress is unproven. If it is blocked the job goes red, i.e. fails closed, not silently green. c) Cosmetic: the script's sections number 1,2,3,5 (no 4) and `grep -A3 'name: loki-canary$' ... || true` is dead code. NOT VERIFIED (stated plainly): the comment's claim that `CHART_VERSION=6.29.0` is "the same pin the ArgoCD Application renders" — the ArgoCD Application manifests are not in this repo (`core/manifests/argocd` does not exist), so if prod renders a different chart version the canary tenant path could differ; the CI check would only catch that drift at 6.29.0. I also did not verify Loki's runtime behaviour of per-tenant `deletion_mode`/`retention_period` on a live cluster (out of scope and forbidden here), only that the override reaches the ConfigMap the config points at. No hold: every reader and writer that would 401 after the flip is named with a tenant in this same diff, the two guards are real and fail-closed, and PR-A is sink-free.
ld-reviewer commented 2026-11-12 13:37:38 +00:00
Collaborator
Copy link

APPROVE — ld-reviewer — head 7f3581e897

Seat B, run IR-20261112-r39a. Judged git diff 2d837e9bc0e199f14cc5f4927efbaea59edec0de...7f3581e8970e3a7774def99f1fa342d6b938e7c3 in my own clone only. Head trailer Lab-Run: IR-20261112-r39a is the LAST trailer (checked with %(trailers)), so the census pin holds. Tools used: read, bash, grep, find, ls, forgejo_comment_as — nothing else; no extension or tool beyond the four declared was observable.

WHAT I RAN

  • Full diff read, plus git diff --name-only live...HEAD (10 files: ci.yaml, 3 chart values, the new check script, 5 dispatcher Go files). No cluster, no kubectl, no image harness.
  • Chart-path verification against the PINNED upstream chart, read from source, not from a summary: pulled loki-6.29.0.tgz and read values.yaml + templates/. Also pulled kube-prometheus-stack (93.2.2) and grafana (10.5.15) to confirm the datasource passthrough.
  • bash -n core/manifests/monitoring/rule-tests/check-loki-tenant-pin.sh → OK.
  • go test ./signalintake/... ./config/... in lab-director/dispatcher → ok for both packages (includes the new TestLokiSourceNamesItsTenant).
  • Census greps over the whole tree: X-Scope-OrgID, loki.monitoring.svc, every file mentioning loki, every Vector config, every dashboard.

  1. THE FLIP IS AT THE PATH THE CHART READS. loki.auth_enabled is exactly what the chart interpolates into the Loki config: loki.config carries auth_enabled: {{ .Values.loki.auth_enabled }} (chart values.yaml:150; default true at :329). So auth_enabled: true under loki: is live, not inert. (gateway.enabled: false, so no nginx tenant gate in the way; clients talk to the :3100 service directly.)

  2. runtimeConfig.overrides.audit-forgejo LANDS WHERE THE CHART LOOKS. templates/runtime-configmap.yaml renders .Values.loki.runtimeConfig into ConfigMap <loki.name>-runtime → loki-runtime at release name loki, and the chart already points runtime_config.file at /etc/loki/runtime-config/runtime-config.yaml (chart values.yaml:194-195) and mounts that ConfigMap in the single-binary StatefulSet (statefulset.yaml:187-188, 230-232). So retention_period: 1080h and deletion_mode: disabled are read, not decorative. Cross-check on B10's premise: Loki's own flag doc says deletion mode disabled ("When set to 'filter-only' or 'filter-and-delete', and if retention_enabled is true, then the log entry deletion API endpoints are available") — i.e. disabled kills the delete API only; compactor retention (global limits_config.retention_period: 30d, compactor.retention_enabled: true) still enforces the tenant's 45 d. The stated intent and the mechanism agree.

  3. THE CANARY PIN IS AT THE ROOT PATH, AND IS NON-DEFAULT-ONLY-IN-NAME. templates/loki-canary/daemonset.yaml:51-54 emits -user={{ .Values.monitoring.selfMonitoring.tenant.name }} / -tenant-id=… only when $.Values.loki.auth_enabled — so after this flip the canary (the ONE pre-existing writer, the measured tenant="fake" series) writes self-monitoring and cannot 401. Root monitoring.selfMonitoring.tenant.name is the right path; note the chart default is already "self-monitoring" (chart values.yaml:3675-3680), so the pin's value is explicitness, which is what 3c N12 asked for.

  4. READER CENSUS — I found no unnamed fourth reader or writer. The three that need a header get one in this same diff, and each is at the right path:

    • Grafana: jsonData.httpHeaderName1: X-Scope-OrgID + secureJsonData.httpHeaderValue1: main (kube-prometheus-stack/values.yaml:73-76). kube-prometheus-stack renders grafana.additionalDataSources verbatim (toYaml passthrough, configmaps-datasources.yaml:94-95) and the operator path is not enabled in these values, so secureJsonData reaches Grafana's provisioning file — this is the documented custom-header shape.
    • otel-collector: otlphttp/loki.headers.X-Scope-OrgID: main (opentelemetry-collector/values.yaml:117-118); the exporter is the only Loki exporter, and nothing else in that config writes to Loki.
    • Dispatcher: Signals.lokiOrgID → Settings.LokiOrgID → LokiSourceConfig.OrgID, defaulting to DefaultLokiOrgID = "main"; the header is set on the only HTTP path LokiSource.Series (logsource.go:112-115 — Series is the sole method that builds a request). Unset config can never produce an empty header.
    • Otherwise: the only Vector in the tree, core/manifests/monitoring/log-metrics/vector-longhorn.yaml, sinks to prometheus_exporter ONLY ("writes nothing to Loki" is literally its design note) — so PR-B's sink really is the first Loki writer besides the canary, and there is no pre-existing unnamed writer hiding in the tree. No dashboard in core/manifests/monitoring/dashboards/ queries Loki. Loki's own alert rules (loki-storage.rules) read Prometheus metrics, not the log API, so they are unaffected by tenancy.

  5. THE GUARD IS FAIL-CAPABLE AND WIRED. set -euo pipefail; missing helm → exit 1 (no silent skip); a missing loki-runtime ConfigMap → sys.exit(1); wrong deletion_mode → sys.exit(1); the sentinel overrides prove each assertion is not vacuous (a sentinel on the pinned path changes the render; a sentinel on the OLD loki.monitoring.* path changes nothing); and the accumulator ends in a real exit 1. It is wired as its own job loki-auth-pin-test on pull_request to live/epic/** (ci.yaml:4-7, 90-104), in the same runs-on: k8s pool whose helm-validate job already proves helm and python3+PyYAML exist there (ci.yaml:34, 19). Its deletion_mode sentinel also proves the runtime-config path itself.

  6. THE SPLIT CLAIM HOLDS. git diff --name-only contains no Vector manifest/config, no CiliumNetworkPolicy, no alert rule; grepping the added lines for vector|CiliumNetworkPolicy|promtail|sink|prometheus_remote_write|alert: returns exactly one hit, and it is a comment in loki/values.yaml that describes the future Vector writer. Nothing sink-shaped ships here.

NON-BLOCKING DEFECTS (fix in PR-B or a follow-up; neither defeats the flip)
a) core/manifests/monitoring/rule-tests/check-loki-tenant-pin.sh:40-42 — the comment "The canary is OFF in the shipped values (no self-monitoring in this cluster), so it is switched ON for the render" is false, and the --set monitoring.lokiCanary.enabled=true is an inert no-op. The chart's switch is ROOT lokiCanary.enabled (chart values.yaml:747-748, default true; template guard {{- with .Values.lokiCanary -}}{{- if .enabled -}}), and the repo's monitoring.lokiCanary.enabled: false (loki/values.yaml:145) is at a path nothing reads. That is exactly why the canary is running and why the PR body's own census measures a fake-tenant writer — the facts are consistent, the comment is not. Importantly this does NOT weaken the guard: it passes only because the chart default is true, and if the chart ever flipped that default the section-1/2/3 assertions would FAIL loudly (no canary rendered, no -tenant-id=), not pass quietly. The stale monitoring.lokiCanary.enabled: false line predates this PR; I did not hold this PR for it, but the new comments should not assert the opposite of measured behaviour.
b) check-loki-tenant-pin.sh numbers its sections 1,2,3,5 — there is no section 4 (leftover from the N11 CNP check that moved to PR-B). Cosmetic only.
c) The script's CHART_VERSION=6.29.0 is "the same pin the ArgoCD Application renders" — the Application lives in Homelab/prod, so that equality is asserted, not verifiable from this repo. If prod's pin moves, the guard keeps testing 6.29.0 until someone bumps the literal; the pin should ideally be read from the same place prod reads it.

WHAT I COULD NOT VERIFY (stated plainly): no helm in my environment, so I did not execute the render myself — I verified every path it asserts by reading the pinned chart's templates instead, and I did not install a helm binary (R37: leave the image/agent dirs alone). I could not confirm the runs-on: k8s runner has egress to grafana.github.io for helm pull; that failure mode is loud (job red), never a silent green. I could not check Homelab/prod overlay values, so an override there that drops the otel-collector otlphttp/loki headers or the Grafana datasource header would be outside this diff. The tracesToLogsV2: datasourceUid: loki in the Tempo datasource still has no matching uid: on the Loki datasource entry — pre-existing, unrelated to tenancy, not a 401.

ENVIRONMENT / RULES: LD_SESSION_TAG was set to the full gateway tag; the comment carries the bare sid only. /home/ld/.local and /home/ld/.local/bin do not exist (checked at launch and again at stop). I read no credential file, no /proc/*/environ, and no PR/issue comment or review. I started no background process; ps before posting showed nothing of mine still running (one unrelated shell in another run's directory, pid 1272058, was already running and I did not touch it). Scratch chart downloads were under /tmp and are deleted.

model: omniroute/deepseek-flash
session: IR-20261112-r39a-seatB-20261112T1323Z

**APPROVE — ld-reviewer — head 7f3581e8970e3a7774def99f1fa342d6b938e7c3** Seat B, run IR-20261112-r39a. Judged `git diff 2d837e9bc0e199f14cc5f4927efbaea59edec0de...7f3581e8970e3a7774def99f1fa342d6b938e7c3` in my own clone only. Head trailer `Lab-Run: IR-20261112-r39a` is the LAST trailer (checked with `%(trailers)`), so the census pin holds. Tools used: read, bash, grep, find, ls, forgejo_comment_as — nothing else; no extension or tool beyond the four declared was observable. WHAT I RAN - Full diff read, plus `git diff --name-only live...HEAD` (10 files: ci.yaml, 3 chart values, the new check script, 5 dispatcher Go files). No cluster, no kubectl, no image harness. - Chart-path verification against the PINNED upstream chart, read from source, not from a summary: pulled `loki-6.29.0.tgz` and read `values.yaml` + `templates/`. Also pulled `kube-prometheus-stack` (93.2.2) and `grafana` (10.5.15) to confirm the datasource passthrough. - `bash -n core/manifests/monitoring/rule-tests/check-loki-tenant-pin.sh` → OK. - `go test ./signalintake/... ./config/...` in `lab-director/dispatcher` → `ok` for both packages (includes the new `TestLokiSourceNamesItsTenant`). - Census greps over the whole tree: `X-Scope-OrgID`, `loki.monitoring.svc`, every file mentioning loki, every Vector config, every dashboard. 1. THE FLIP IS AT THE PATH THE CHART READS. `loki.auth_enabled` is exactly what the chart interpolates into the Loki config: `loki.config` carries `auth_enabled: {{ .Values.loki.auth_enabled }}` (chart values.yaml:150; default true at :329). So `auth_enabled: true` under `loki:` is live, not inert. (`gateway.enabled: false`, so no nginx tenant gate in the way; clients talk to the :3100 service directly.) 2. `runtimeConfig.overrides.audit-forgejo` LANDS WHERE THE CHART LOOKS. `templates/runtime-configmap.yaml` renders `.Values.loki.runtimeConfig` into ConfigMap `<loki.name>-runtime` → `loki-runtime` at release name `loki`, and the chart already points `runtime_config.file` at `/etc/loki/runtime-config/runtime-config.yaml` (chart values.yaml:194-195) and mounts that ConfigMap in the single-binary StatefulSet (statefulset.yaml:187-188, 230-232). So `retention_period: 1080h` and `deletion_mode: disabled` are read, not decorative. Cross-check on B10's premise: Loki's own flag doc says deletion mode `disabled` ("When set to 'filter-only' or 'filter-and-delete', and if retention_enabled is true, then the log entry deletion API endpoints are available") — i.e. `disabled` kills the delete API only; compactor retention (global `limits_config.retention_period: 30d`, `compactor.retention_enabled: true`) still enforces the tenant's 45 d. The stated intent and the mechanism agree. 3. THE CANARY PIN IS AT THE ROOT PATH, AND IS NON-DEFAULT-ONLY-IN-NAME. `templates/loki-canary/daemonset.yaml:51-54` emits `-user={{ .Values.monitoring.selfMonitoring.tenant.name }}` / `-tenant-id=…` only when `$.Values.loki.auth_enabled` — so after this flip the canary (the ONE pre-existing writer, the measured `tenant="fake"` series) writes `self-monitoring` and cannot 401. Root `monitoring.selfMonitoring.tenant.name` is the right path; note the chart default is already `"self-monitoring"` (chart values.yaml:3675-3680), so the pin's value is explicitness, which is what 3c N12 asked for. 4. READER CENSUS — I found no unnamed fourth reader or writer. The three that need a header get one in this same diff, and each is at the right path: - Grafana: `jsonData.httpHeaderName1: X-Scope-OrgID` + `secureJsonData.httpHeaderValue1: main` (kube-prometheus-stack/values.yaml:73-76). kube-prometheus-stack renders `grafana.additionalDataSources` verbatim (`toYaml` passthrough, configmaps-datasources.yaml:94-95) and the operator path is not enabled in these values, so `secureJsonData` reaches Grafana's provisioning file — this is the documented custom-header shape. - otel-collector: `otlphttp/loki.headers.X-Scope-OrgID: main` (opentelemetry-collector/values.yaml:117-118); the exporter is the only Loki exporter, and nothing else in that config writes to Loki. - Dispatcher: `Signals.lokiOrgID` → `Settings.LokiOrgID` → `LokiSourceConfig.OrgID`, defaulting to `DefaultLokiOrgID = "main"`; the header is set on the only HTTP path `LokiSource.Series` (logsource.go:112-115 — `Series` is the sole method that builds a request). Unset config can never produce an empty header. - Otherwise: the only Vector in the tree, `core/manifests/monitoring/log-metrics/vector-longhorn.yaml`, sinks to `prometheus_exporter` ONLY ("writes nothing to Loki" is literally its design note) — so PR-B's sink really is the first Loki writer besides the canary, and there is no pre-existing unnamed writer hiding in the tree. No dashboard in `core/manifests/monitoring/dashboards/` queries Loki. Loki's own alert rules (`loki-storage.rules`) read Prometheus metrics, not the log API, so they are unaffected by tenancy. 5. THE GUARD IS FAIL-CAPABLE AND WIRED. `set -euo pipefail`; missing helm → `exit 1` (no silent skip); a missing `loki-runtime` ConfigMap → `sys.exit(1)`; wrong `deletion_mode` → `sys.exit(1)`; the sentinel overrides prove each assertion is not vacuous (a sentinel on the pinned path changes the render; a sentinel on the OLD `loki.monitoring.*` path changes nothing); and the accumulator ends in a real `exit 1`. It is wired as its own job `loki-auth-pin-test` on `pull_request` to `live`/`epic/**` (ci.yaml:4-7, 90-104), in the same `runs-on: k8s` pool whose `helm-validate` job already proves `helm` and `python3`+PyYAML exist there (ci.yaml:34, 19). Its `deletion_mode` sentinel also proves the runtime-config path itself. 6. THE SPLIT CLAIM HOLDS. `git diff --name-only` contains no Vector manifest/config, no CiliumNetworkPolicy, no alert rule; grepping the added lines for `vector|CiliumNetworkPolicy|promtail|sink|prometheus_remote_write|alert:` returns exactly one hit, and it is a comment in loki/values.yaml that *describes* the future Vector writer. Nothing sink-shaped ships here. NON-BLOCKING DEFECTS (fix in PR-B or a follow-up; neither defeats the flip) a) `core/manifests/monitoring/rule-tests/check-loki-tenant-pin.sh:40-42` — the comment "The canary is OFF in the shipped values (no self-monitoring in this cluster), so it is switched ON for the render" is false, and the `--set monitoring.lokiCanary.enabled=true` is an inert no-op. The chart's switch is ROOT `lokiCanary.enabled` (chart values.yaml:747-748, default **true**; template guard `{{- with .Values.lokiCanary -}}{{- if .enabled -}}`), and the repo's `monitoring.lokiCanary.enabled: false` (loki/values.yaml:145) is at a path nothing reads. That is exactly why the canary is running and why the PR body's own census measures a `fake`-tenant writer — the facts are consistent, the comment is not. Importantly this does NOT weaken the guard: it passes only because the chart default is true, and if the chart ever flipped that default the section-1/2/3 assertions would FAIL loudly (no canary rendered, no `-tenant-id=`), not pass quietly. The stale `monitoring.lokiCanary.enabled: false` line predates this PR; I did not hold this PR for it, but the new comments should not assert the opposite of measured behaviour. b) `check-loki-tenant-pin.sh` numbers its sections 1,2,3,5 — there is no section 4 (leftover from the N11 CNP check that moved to PR-B). Cosmetic only. c) The script's `CHART_VERSION=6.29.0` is "the same pin the ArgoCD Application renders" — the Application lives in Homelab/prod, so that equality is asserted, not verifiable from this repo. If prod's pin moves, the guard keeps testing 6.29.0 until someone bumps the literal; the pin should ideally be read from the same place prod reads it. WHAT I COULD NOT VERIFY (stated plainly): no `helm` in my environment, so I did not execute the render myself — I verified every path it asserts by reading the pinned chart's templates instead, and I did not install a helm binary (R37: leave the image/agent dirs alone). I could not confirm the `runs-on: k8s` runner has egress to grafana.github.io for `helm pull`; that failure mode is loud (job red), never a silent green. I could not check Homelab/prod overlay values, so an override there that drops the otel-collector `otlphttp/loki` headers or the Grafana datasource header would be outside this diff. The `tracesToLogsV2: datasourceUid: loki` in the Tempo datasource still has no matching `uid:` on the Loki datasource entry — pre-existing, unrelated to tenancy, not a 401. ENVIRONMENT / RULES: LD_SESSION_TAG was set to the full gateway tag; the comment carries the bare sid only. `/home/ld/.local` and `/home/ld/.local/bin` do not exist (checked at launch and again at stop). I read no credential file, no /proc/*/environ, and no PR/issue comment or review. I started no background process; `ps` before posting showed nothing of mine still running (one unrelated shell in another run's directory, pid 1272058, was already running and I did not touch it). Scratch chart downloads were under /tmp and are deleted. model: omniroute/deepseek-flash session: IR-20261112-r39a-seatB-20261112T1323Z
All checks were successful
CI Review / receiver-test (pull_request) Successful in 6s
Details
CI Review / loki-auth-pin-test (pull_request) Successful in 8s
Details
CI Review / helm-validate (pull_request) Successful in 10s
Required
Details
CI Review / runner-manifest-test (pull_request) Successful in 10s
Details
Build Lab Director Dispatcher Image / preflight (pull_request) Successful in 7s
Details
CI Review / alert-receiver-test (pull_request) Successful in 21s
Details
Lint & Validate / shellcheck (pull_request) Successful in 7s
Required
Details
Lint & Validate / yaml-lint (pull_request) Successful in 22s
Required
Details
CI Review / record-sql-test (pull_request) Successful in 1m0s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 2m7s
Required
Details
CI Review / cockpit-test (pull_request) Successful in 2m42s
Details
CI Review / cockpit-e2e (pull_request) Successful in 2m59s
Details
CI Review / dispatcher-test (pull_request) Successful in 3m11s
Details
Build Lab Director Dispatcher Image / build-and-push (pull_request) Successful in 1m55s
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin plane/R39-forgejo-access-log-auth:plane/R39-forgejo-access-log-auth
git switch plane/R39-forgejo-access-log-auth
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
operator-review

Requires human operator review before merge — dangerous class (RBAC, secrets, network policy, Terraform) or explicitly gated. Never auto-merge.

No labels
operator-review
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Homelab/infra-core!961
No description provided.