HOMELAB-1681: fix(zitadel): assert project roles into the cockpit ID token #590

Merged

aaron merged 1 commit from plane/HOMELAB-1681b-roleclaim into live 2026-10-03 03:17:49 +00:00

Conversation 2 Commits 1 Files changed 1 +28 -2

aaron commented 2026-10-03 03:08:46 +00:00

Owner

Copy link

What

id_token_role_assertion = true on zitadel_application_oidc.lab_director (+ comment correction). One resource, one attribute. No other resource touched, 0 to destroy.

Why — the live LD3 break

Operator logs into https://lab-director.aaron.reynoza.org successfully, then gets "Signed in, but not an operator — your roles: none" (subject 363992137722757672).

Diagnosis (read-only, evidence first):

Hypothesis	Verdict	Evidence
Subject is a stranger	no	GET /management/v1/users/363992137722757672 → aaron@reynoza.org, ACTIVE
Gap #1 — the operator has no operator grant	no	user-grant search for that user → grant 364000641170801240, roleKeys [jellyfin, admins, operator], ACTIVE
Gap #2 — project role assertion off	no	GET /management/v1/projects/364000638805213784 → projectRoleAssertion: true
Real gap — the APPLICATION never asserted roles into the ID token	yes	id_token_role_assertion was false; the plan proved it: ~ id_token_role_assertion = false -> true

project_role_assertion (project) only covers the userinfo endpoint. In Zitadel v4.10.1 — the version this cluster runs — the ID token's role claims are gated by the client flag: internal/api/oidc/userinfo.go assertRoles() returns early unless idTokenRoleAssertion is set, and the projection default is false (internal/query/projection/app.go: handler.Default(false)). The cockpit reads urn:zitadel:iam:org:project:roles off the verified ID token and has no userinfo fallback (lab-director/cockpit/auth/roles.go), so it fails closed — exactly the 403 shown.

Scopes are not a substitute: urn:zitadel:iam:org:project:roles is the claim name, not a Zitadel scope (internal/api/oidc/client.go defines urn:zitadel:iam:org:projects:roles and the urn:zitadel:iam:org:project:role:{key} prefix); role scopes are stripped from the ID token while the flag is false (client_converter.go RestrictAdditionalIdTokenScopes); and assertRoles still gates on the flag. Hence a terraform-first app-level fix, not an OIDC_SCOPES change in prod.

Apply (it is already live)

Guarded local path, same discipline as the LD3.1b applies — targeted plan → verify scope → apply the reviewed saved plan:

terraform plan -target=zitadel_application_oidc.lab_director   → 0 to add, 1 to change, 0 to destroy
terraform show -json | jq .resource_changes[].address          → zitadel_application_oidc.lab_director=update
terraform apply <saved plan>                                   → Apply complete! 0 added, 1 changed, 0 destroyed
state serial 59 → 60

Live re-read (non-secret): GET /management/v1/projects/364000638805213784/apps/393380591770272287 → idTokenRoleAssertion: true. Client id/secret unchanged (state hashes == the cockpit's runtime cockpit-oidc Secret), so the running pods keep working.

Full-plan residual after the apply is only the two known items: william@reynoza.org (HOMELAB-1867 drift) and jesus@reynoza.org (an operator-made console deletion — a full apply would resurrect it; deliberately not applied).

Reviewer context

Only CI's own apply leg could conflict: it runs terraform apply with no target, i.e. it would also resurrect jesus. Not a blocker for this diff (pre-existing, applies equally to live today), but worth knowing before suggesting a merge-triggered apply.

## What `id_token_role_assertion = true` on `zitadel_application_oidc.lab_director` (+ comment correction). One resource, one attribute. No other resource touched, 0 to destroy. ## Why — the live LD3 break Operator logs into https://lab-director.aaron.reynoza.org successfully, then gets **"Signed in, but not an operator — your roles: none"** (subject `363992137722757672`). Diagnosis (read-only, evidence first): | Hypothesis | Verdict | Evidence | |---|---|---| | Subject is a stranger | **no** | `GET /management/v1/users/363992137722757672` → `aaron@reynoza.org`, ACTIVE | | Gap #1 — the operator has no `operator` grant | **no** | user-grant search for that user → grant `364000641170801240`, `roleKeys [jellyfin, admins, operator]`, ACTIVE | | Gap #2 — project role assertion off | **no** | `GET /management/v1/projects/364000638805213784` → `projectRoleAssertion: true` | | **Real gap — the APPLICATION never asserted roles into the ID token** | **yes** | `id_token_role_assertion` was `false`; the plan proved it: `~ id_token_role_assertion = false -> true` | `project_role_assertion` (project) only covers the **userinfo** endpoint. In Zitadel v4.10.1 — the version this cluster runs — the ID token's role claims are gated by the **client** flag: `internal/api/oidc/userinfo.go assertRoles()` returns early unless `idTokenRoleAssertion` is set, and the projection default is false (`internal/query/projection/app.go: handler.Default(false)`). The cockpit reads `urn:zitadel:iam:org:project:roles` off the verified ID token and has no userinfo fallback (`lab-director/cockpit/auth/roles.go`), so it fails closed — exactly the 403 shown. Scopes are **not** a substitute: `urn:zitadel:iam:org:project:roles` is the claim name, not a Zitadel scope (`internal/api/oidc/client.go` defines `urn:zitadel:iam:org:projects:roles` and the `urn:zitadel:iam:org:project:role:{key}` prefix); role scopes are stripped from the ID token while the flag is false (`client_converter.go RestrictAdditionalIdTokenScopes`); and `assertRoles` still gates on the flag. Hence a terraform-first app-level fix, not an `OIDC_SCOPES` change in prod. ## Apply (it is already live) Guarded local path, same discipline as the LD3.1b applies — targeted plan → verify scope → apply the reviewed saved plan: ``` terraform plan -target=zitadel_application_oidc.lab_director → 0 to add, 1 to change, 0 to destroy terraform show -json | jq .resource_changes[].address → zitadel_application_oidc.lab_director=update terraform apply <saved plan> → Apply complete! 0 added, 1 changed, 0 destroyed state serial 59 → 60 ``` Live re-read (non-secret): `GET /management/v1/projects/364000638805213784/apps/393380591770272287` → `idTokenRoleAssertion: true`. Client id/secret unchanged (state hashes == the cockpit's runtime `cockpit-oidc` Secret), so the running pods keep working. Full-plan residual after the apply is **only** the two known items: `william@reynoza.org` (HOMELAB-1867 drift) and `jesus@reynoza.org` (an operator-made console deletion — a full apply would resurrect it; deliberately not applied). ## Reviewer context Only CI's own apply leg could conflict: it runs `terraform apply` with no target, i.e. it would also resurrect jesus. Not a blocker for this diff (pre-existing, applies equally to `live` today), but worth knowing before suggesting a merge-triggered apply.

aaron added 1 commit 2026-10-03 03:08:46 +00:00

HOMELAB-1681: fix(zitadel): assert project roles into the cockpit ID token ... 7da8aea762

The operator signed in successfully but the cockpit answered
"Signed in, but not an operator — your roles: none" (subject
363992137722757672 = aaron@reynoza.org). Diagnosis, in order:

* The subject IS the operator, and their HOMELAB project grant already
  carries role_keys [jellyfin, admins, operator] — the grant is not the gap.
* The HOMELAB project already sets project_role_assertion = true — that is
  not the gap either. It asserts roles into the USERINFO endpoint only.
* The gap is APPLICATION-level: zitadel_application_oidc.lab_director never
  set id_token_role_assertion, and Zitadel's default is false
  (internal/query/projection/app.go: handler.Default(false)). In Zitadel
  v4.x the ID token's role claims are gated by the client flag
  (internal/api/oidc/userinfo.go assertRoles returns early when it is
  false), so a request with the cockpit's scopes (openid profile email)
  yields a perfectly valid login whose ID token has no roles claim at all —
  and cockpit auth/roles.go fails closed on a missing claim.

Set id_token_role_assertion = true on the app. Scopes are not a substitute:
urn:zitadel:iam:org:project:roles is the claim name, not a Zitadel scope
(client.go defines urn:zitadel:iam:org:projects:roles and the
urn:zitadel:iam:org:project:role:{key} prefix), role scopes are stripped
from the ID token while this flag is false, and assertRoles still gates on
the flag. Also corrects the role block's comment, whose project-flag-only
rationale was the trap this change fixes.

Applied locally through the guarded path: targeted plan
(0 to add, 1 to change, 0 to destroy), applied as the reviewed saved plan.
State 59 -> 60.

aaron commented 2026-10-03 03:16:22 +00:00

Author

Owner

Copy link

Apply + verification evidence (fix worker, non-secret)

The change is already live (local guarded path; CI's apply leg is a manual workflow_dispatch, not a merge trigger — so this PR is for durability, not for the effect to happen).

Guarded sequence (core/terraform/live/zitadel, provider v3.8.7, state serial 59 → 60):

terraform plan -target=zitadel_application_oidc.lab_director   → 0 to add, 1 to change, 0 to destroy
  ~ id_token_role_assertion = false -> true          (id 393380591770272287, "Lab Director")
  machine-checked: lab_director=update, zitadel_project.homelab=no-op   ← william/jesus NOT in scope
terraform apply <the reviewed saved plan>
  → Modifications complete after 2s → Apply complete! 0 added, 1 changed, 0 destroyed

Read back from live Zitadel after the apply:

GET /management/v1/projects/364000638805213784/apps/393380591770272287
  → clientId 393380591787049503  (unchanged; the client the cockpit redirects to)
    idTokenRoleAssertion: true
    redirectUris: ["https://lab-director.aaron.reynoza.org/auth/callback"]

Client credentials untouched — sha256 prefix of state's lab_director_client_id/_secret equals the cockpit's runtime lab-director/cockpit-oidc Secret (c3e7f31a7b1036d1 / 906cb49dba2cb214). Nothing to re-seed in OpenBao.

Residual full plan (deliberately not applied): jesus@reynoza.org user+grant create (operator-made console deletion — a full apply would resurrect it) and william@reynoza.org update (HOMELAB-1867). No id_token_role_assertion diff remains.

CI: runs #1766/#1767 on head 7da8aea — all green (terraform-validate, cockpit-e2e, cockpit-test, dispatcher-test, helm-validate, shellcheck, yaml-lint, alert-receiver-test, receiver-test).

Operator-facing: sign out on the 403 page (or clear ld_cockpit_session) and sign in again — the session cookie is sealed for 24h, so the stale roles: none cookie must go. Expected after: no 403 at all (shell loads); if the 403 were ever seen again, your roles must now list operator, not none.

### Apply + verification evidence (fix worker, non-secret) The change is **already live** (local guarded path; CI's apply leg is a manual `workflow_dispatch`, not a merge trigger — so this PR is for durability, not for the effect to happen). **Guarded sequence** (`core/terraform/live/zitadel`, provider v3.8.7, state serial 59 → 60): ``` terraform plan -target=zitadel_application_oidc.lab_director → 0 to add, 1 to change, 0 to destroy ~ id_token_role_assertion = false -> true (id 393380591770272287, "Lab Director") machine-checked: lab_director=update, zitadel_project.homelab=no-op ← william/jesus NOT in scope terraform apply <the reviewed saved plan> → Modifications complete after 2s → Apply complete! 0 added, 1 changed, 0 destroyed ``` **Read back from live Zitadel after the apply:** ``` GET /management/v1/projects/364000638805213784/apps/393380591770272287 → clientId 393380591787049503 (unchanged; the client the cockpit redirects to) idTokenRoleAssertion: true redirectUris: ["https://lab-director.aaron.reynoza.org/auth/callback"] ``` **Client credentials untouched** — sha256 prefix of state's `lab_director_client_id`/`_secret` equals the cockpit's runtime `lab-director/cockpit-oidc` Secret (`c3e7f31a7b1036d1` / `906cb49dba2cb214`). Nothing to re-seed in OpenBao. **Residual full plan (deliberately not applied):** `jesus@reynoza.org` user+grant `create` (operator-made console deletion — a full apply would resurrect it) and `william@reynoza.org` `update` (HOMELAB-1867). No `id_token_role_assertion` diff remains. **CI:** runs #1766/#1767 on head `7da8aea` — all green (`terraform-validate`, `cockpit-e2e`, `cockpit-test`, `dispatcher-test`, `helm-validate`, `shellcheck`, `yaml-lint`, `alert-receiver-test`, `receiver-test`). **Operator-facing:** sign out on the 403 page (or clear `ld_cockpit_session`) and sign in again — the session cookie is sealed for 24h, so the stale `roles: none` cookie must go. Expected after: no 403 at all (shell loads); if the 403 were ever seen again, `your roles` must now list `operator`, not `none`.

aaron commented 2026-10-03 03:17:41 +00:00

Author

Owner

Copy link

Review verdict — APPROVE (no blockers)

Reviewed at head 7da8aea7621a4f6658208e2fd46ab4de4f238a29 (base origin/live = 5215bcb0) by a
fresh, independent reviewer (separate context/process, primary sources only: the PR diff, the ticket's
acceptance criteria, and .rpiv/artifacts/build/ld3-operator-role-fix.md).

Diff scope verified. git diff origin/live..7da8aea -- core/terraform/live/zitadel/ = exactly one
file, core/terraform/live/zitadel/main.tf, +28/−2. The only functional line is
id_token_role_assertion = true inside zitadel_application_oidc.lab_director; every other added line is
a comment. origin/live has 0 occurrences of the attribute, head has 1 — no duplicate, no second
resource, nothing outside that resource block, nothing in prod/.

Matches the applied live state. Diff value true == live
GET /management/v1/projects/364000638805213784/apps/393380591770272287 → idTokenRoleAssertion: true
and terraform state show … id_token_role_assertion = true. The apply telemetry (serial 59→60,
0 added, 1 changed, 0 destroyed, clientId 393380591787049503 unchanged, client id/secret hashes equal
the cockpit's runtime secret) shows an in-place update of this one attribute. The git change therefore
makes config match reality; it changes nothing at runtime today.

Not dangerous. No secret, RBAC, network-policy, module, provider, or schema change; no destroy /
force-new path. In-place update on an existing OIDC app is the only possible effect.

Claim honesty. The prose does not claim to fix the project flag, does not claim a scope change, and
does not claim to touch other resources — it explicitly disclaims both (project_role_assertion covers
userinfo only; urn:zitadel:iam:org:project:roles is the claim name, not a Zitadel scope). Corroborated
in-repo: project_role_assertion = true on the HOMELAB project is untouched, and
lab-director/cockpit/auth/roles.go parses the claim from the verified ID token with no userinfo
fallback. The comment correction is accurate: the old wording wrongly attributed the claim to the
project flag, and the new wording matches the attribute in the same block.

Merge-safety clarification (for the record). The full-apply hazard this PR mentions (an untargeted
terraform apply resurrecting jesus@reynoza.org, plus the william@reynoza.org drift) is pre-existing
and cannot trigger from this merge: the zitadel root was removed from Atlantis' remit (#582 /
HOMELAB-1865) and the infra-apply workflow is broken (HOMELAB-1866). That risk is owned by
HOMELAB-1866/1867 and only materialises once the pipeline is repaired — a concern, not a blocker here.
The verbatim diff itself contains no jesus/william text (the attribution in the earlier diagnosis artifact
refers to the CI apply leg, not to lines added by this PR).

CI. Green on head: runs #1766 (alert-receiver-test, cockpit-e2e, cockpit-test, dispatcher-test,
helm-validate, receiver-test) and #1767 (shellcheck, terraform-validate, yaml-lint) all success;
combined commit status success (10 statuses, including atlantis 0/0 projects applied successfully).

Blockers: none.
Concerns (non-blocking, tracked elsewhere): full-plan human-user drift → HOMELAB-1866/1867;
Zitadel provider constraint remains unpinned (>= 2.10.0), pre-existing; the comment cites upstream
Zitadel source paths not vendored in this repo (cosmetic — the behaviour is confirmed by the live
read-back).

Merging as harmless/additive (single in-place attribute, git-to-reality).

## Review verdict — APPROVE (no blockers) Reviewed at head `7da8aea7621a4f6658208e2fd46ab4de4f238a29` (base `origin/live` = `5215bcb0`) by a fresh, independent reviewer (separate context/process, primary sources only: the PR diff, the ticket's acceptance criteria, and `.rpiv/artifacts/build/ld3-operator-role-fix.md`). **Diff scope verified.** `git diff origin/live..7da8aea -- core/terraform/live/zitadel/` = exactly one file, `core/terraform/live/zitadel/main.tf`, +28/−2. The only functional line is `id_token_role_assertion = true` inside `zitadel_application_oidc.lab_director`; every other added line is a comment. `origin/live` has 0 occurrences of the attribute, head has 1 — no duplicate, no second resource, nothing outside that resource block, nothing in `prod/`. **Matches the applied live state.** Diff value `true` == live `GET /management/v1/projects/364000638805213784/apps/393380591770272287 → idTokenRoleAssertion: true` and `terraform state show … id_token_role_assertion = true`. The apply telemetry (serial 59→60, `0 added, 1 changed, 0 destroyed`, `clientId 393380591787049503` unchanged, client id/secret hashes equal the cockpit's runtime secret) shows an in-place update of this one attribute. The git change therefore makes config match reality; it changes nothing at runtime today. **Not dangerous.** No secret, RBAC, network-policy, module, provider, or schema change; no destroy / force-new path. In-place `update` on an existing OIDC app is the only possible effect. **Claim honesty.** The prose does not claim to fix the project flag, does not claim a scope change, and does not claim to touch other resources — it explicitly disclaims both (`project_role_assertion` covers userinfo only; `urn:zitadel:iam:org:project:roles` is the claim name, not a Zitadel scope). Corroborated in-repo: `project_role_assertion = true` on the HOMELAB project is untouched, and `lab-director/cockpit/auth/roles.go` parses the claim from the verified ID token with no userinfo fallback. The comment correction is accurate: the old wording wrongly attributed the claim to the project flag, and the new wording matches the attribute in the same block. **Merge-safety clarification (for the record).** The full-apply hazard this PR mentions (an untargeted `terraform apply` resurrecting `jesus@reynoza.org`, plus the `william@reynoza.org` drift) is pre-existing and **cannot trigger from this merge**: the `zitadel` root was removed from Atlantis' remit (#582 / HOMELAB-1865) and the infra-apply workflow is broken (HOMELAB-1866). That risk is owned by HOMELAB-1866/1867 and only materialises once the pipeline is repaired — a concern, not a blocker here. The verbatim diff itself contains no jesus/william text (the attribution in the earlier diagnosis artifact refers to the CI apply leg, not to lines added by this PR). **CI.** Green on head: runs #1766 (alert-receiver-test, cockpit-e2e, cockpit-test, dispatcher-test, helm-validate, receiver-test) and #1767 (shellcheck, terraform-validate, yaml-lint) all success; combined commit status `success` (10 statuses, including atlantis `0/0 projects applied successfully`). **Blockers: none.** **Concerns (non-blocking, tracked elsewhere):** full-plan human-user drift → HOMELAB-1866/1867; Zitadel provider constraint remains unpinned (`>= 2.10.0`), pre-existing; the comment cites upstream Zitadel source paths not vendored in this repo (cosmetic — the behaviour is confirmed by the live read-back). Merging as harmless/additive (single in-place attribute, git-to-reality).

aaron merged commit b3108c2120 into live 2026-10-03 03:17:49 +00:00

aaron referenced this pull request from a commit 2026-10-03 03:17:50 +00:00

Merge pull request 'HOMELAB-1681: fix(zitadel): assert project roles into the cockpit ID token' (#590) from plane/HOMELAB-1681b-roleclaim into live

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

operator-review

Requires human operator review before merge — dangerous class (RBAC, secrets, network policy, Terraform) or explicitly gated. Never auto-merge.

No labels

operator-review

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Homelab/infra-core!590

No description provided.