Homelab/infra-core
6
0
Fork 2
Code Issues Pull requests 35 Projects Releases 1 Packages Wiki Activity Actions

HOMELAB-2479: fix(zitadel): h2c Service-DNS provider endpoint (gRPC trailers workaround) #911

Open
pi-landing wants to merge 2 commits from plane/HOMELAB-2479-zitadel-h2c into live
pull from: plane/HOMELAB-2479-zitadel-h2c
merge into: Homelab:live
Conversation 0 Commits 2 Files changed 3 +70 -6
pi-landing commented 2026-11-10 11:39:20 +00:00
Collaborator
Copy link

Cause

The Zitadel terraform provider (zitadel provider block in
core/terraform/live/zitadel/main.tf) connects to the Zitadel gRPC management
API. The hardcoded path was HTTPS/443 via the Cilium Gateway
(domain = "zitadel.${var.base_domain}", insecure = false), which trips the
Cilium Gateway gRPC-trailers bug — cilium/cilium#39484. Every plan that
reaches the provider fails against that path.

Evidence

  • Run 13204 (the CI terraform run) and 5 consecutive clean-plan failures:
    every attempt to plan this module through the 443/ALPN path fails; the
    failure mode is the Gateway breaking gRPC trailers for this API, not the
    terraform configuration itself.
  • The plan proof for this change is HELD — see "Proof" below.

Fix

Three files under core/terraform/live/zitadel/, nothing else (no tfvars, no
backend.hcl, no other module):

  1. variables.tf — three new variables:

    • zitadel_api_endpoint (string, default "zitadel.zitadel.svc.cluster.local")
    • zitadel_api_port (string, default "8080")
    • zitadel_api_insecure (bool, default true)

  2. main.tf provider block — replaces the hardcoded
    domain = "zitadel.${var.base_domain}" / port = "443" / insecure = false
    with domain = var.zitadel_api_endpoint, port = var.zitadel_api_port,
    insecure = var.zitadel_api_insecure. jwt_profile_file unchanged
    (var.zitadel_key_file). The adjacent comment is rewritten: in-cluster h2c
    is the default; off-cluster is a kubectl port-forward to localhost:8080;
    the HTTPS/443+ALPN path is kept only as a reference fallback comment.

  3. terraform.tfvars.example — documents both profiles (CI: no overrides;
    mgmt: -var zitadel_api_endpoint="localhost" / port="8080" /
    insecure=true, with the port-forward precondition).

Endpoint table

Profile Endpoint Transport
CI (in-cluster runner) zitadel.zitadel.svc.cluster.local:8080 (defaults) h2c
mgmt VM kubectl port-forward svc/zitadel 8080 → localhost:8080 h2c
NEVER LAN LB 10.10.10.225:443 —

Security

  • insecure=true is transport-only: plain h2c stays inside the cluster
    (Service DNS) or on a localhost port-forward socket. It changes no
    authentication: JWT-profile service-account auth (jwt_profile_file) is
    unchanged, and no credential material is created, moved, or modified by this
    PR.

Proof

PROOF-GATED. This PR is opened as a gate: the 904FIX PROOF run executes
after the overseer names the proof's credentials — it is deliberately not
run here. The proof exercises the plan/apply path against the in-cluster h2c
endpoint.

Audience caveat: if JWT-profile auth fails on audience at proof time, the
fix for that lands in this PR per El Director's ruling — this branch is the
vehicle for the endpoint + any follow-up auth correction.

Verification (of this diff itself)

  • terraform fmt -check: clean.
  • terraform validate (after terraform init -backend=false): Success!
  • plan/apply: NOT run (proof held).
## Cause The Zitadel terraform provider (`zitadel` provider block in `core/terraform/live/zitadel/main.tf`) connects to the Zitadel gRPC management API. The hardcoded path was HTTPS/443 via the Cilium Gateway (`domain = "zitadel.${var.base_domain}"`, `insecure = false`), which trips the Cilium Gateway gRPC-trailers bug — **cilium/cilium#39484**. Every plan that reaches the provider fails against that path. ## Evidence - Run 13204 (the CI terraform run) and **5 consecutive clean-plan failures**: every attempt to plan this module through the 443/ALPN path fails; the failure mode is the Gateway breaking gRPC trailers for this API, not the terraform configuration itself. - The plan proof for this change is **HELD** — see "Proof" below. ## Fix Three files under `core/terraform/live/zitadel/`, nothing else (no tfvars, no backend.hcl, no other module): 1. **`variables.tf`** — three new variables: - `zitadel_api_endpoint` (string, default `"zitadel.zitadel.svc.cluster.local"`) - `zitadel_api_port` (string, default `"8080"`) - `zitadel_api_insecure` (bool, default `true`) 2. **`main.tf` provider block** — replaces the hardcoded `domain = "zitadel.${var.base_domain}" / port = "443" / insecure = false` with `domain = var.zitadel_api_endpoint`, `port = var.zitadel_api_port`, `insecure = var.zitadel_api_insecure`. `jwt_profile_file` unchanged (`var.zitadel_key_file`). The adjacent comment is rewritten: in-cluster h2c is the default; off-cluster is a `kubectl port-forward` to localhost:8080; the HTTPS/443+ALPN path is kept only as a reference fallback comment. 3. **`terraform.tfvars.example`** — documents both profiles (CI: no overrides; mgmt: `-var zitadel_api_endpoint="localhost"` / `port="8080"` / `insecure=true`, with the port-forward precondition). ## Endpoint table | Profile | Endpoint | Transport | |---|---|---| | CI (in-cluster runner) | `zitadel.zitadel.svc.cluster.local:8080` (defaults) | h2c | | mgmt VM | `kubectl port-forward svc/zitadel 8080` → `localhost:8080` | h2c | | **NEVER** | LAN LB `10.10.10.225:443` | — | ## Security - `insecure=true` is **transport-only**: plain h2c stays inside the cluster (Service DNS) or on a localhost port-forward socket. It changes no authentication: JWT-profile service-account auth (`jwt_profile_file`) is unchanged, and no credential material is created, moved, or modified by this PR. ## Proof **PROOF-GATED.** This PR is opened as a gate: the 904FIX PROOF run executes **after the overseer names the proof's credentials** — it is deliberately not run here. The proof exercises the plan/apply path against the in-cluster h2c endpoint. **Audience caveat:** if JWT-profile auth fails on audience at proof time, the fix for that lands **in this PR** per El Director's ruling — this branch is the vehicle for the endpoint + any follow-up auth correction. ## Verification (of this diff itself) - `terraform fmt -check`: clean. - `terraform validate` (after `terraform init -backend=false`): **Success!** - `plan`/`apply`: NOT run (proof held).
HOMELAB-2479: fix(zitadel): provider endpoint vars, in-cluster h2c default (cilium#39484)
All checks were successful
CI Review / helm-validate (pull_request) Successful in 9s
Details
CI Review / runner-manifest-test (pull_request) Successful in 4s
Details
CI Review / receiver-test (pull_request) Successful in 5s
Details
CI Review / alert-receiver-test (pull_request) Successful in 21s
Details
Lint & Validate / yaml-lint (pull_request) Successful in 20s
Details
Lint & Validate / shellcheck (pull_request) Successful in 5s
Details
CI Review / record-sql-test (pull_request) Successful in 1m29s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 1m51s
Details
CI Review / cockpit-e2e (pull_request) Successful in 2m35s
Details
CI Review / cockpit-test (pull_request) Successful in 3m4s
Details
CI Review / dispatcher-test (pull_request) Successful in 3m11s
Details
adc4435d4a
HOMELAB-2479: fix(zitadel): authority=external-domain, dial via job-scoped add-host
All checks were successful
CI Review / runner-manifest-test (pull_request) Successful in 4s
Details
CI Review / receiver-test (pull_request) Successful in 4s
Details
CI Review / helm-validate (pull_request) Successful in 11s
Details
Lint & Validate / shellcheck (pull_request) Successful in 6s
Details
CI Review / alert-receiver-test (pull_request) Successful in 21s
Details
Lint & Validate / yaml-lint (pull_request) Successful in 20s
Details
CI Review / record-sql-test (pull_request) Successful in 1m15s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 2m14s
Details
CI Review / cockpit-e2e (pull_request) Successful in 2m46s
Details
CI Review / dispatcher-test (pull_request) Successful in 3m4s
Details
CI Review / cockpit-test (pull_request) Successful in 3m7s
Details
a6752c7f8a
All checks were successful
CI Review / runner-manifest-test (pull_request) Successful in 4s
Details
CI Review / receiver-test (pull_request) Successful in 4s
Details
CI Review / helm-validate (pull_request) Successful in 11s
Required
Details
Lint & Validate / shellcheck (pull_request) Successful in 6s
Required
Details
CI Review / alert-receiver-test (pull_request) Successful in 21s
Details
Lint & Validate / yaml-lint (pull_request) Successful in 20s
Required
Details
CI Review / record-sql-test (pull_request) Successful in 1m15s
Details
Lint & Validate / terraform-validate (pull_request) Successful in 2m14s
Required
Details
CI Review / cockpit-e2e (pull_request) Successful in 2m46s
Details
CI Review / dispatcher-test (pull_request) Successful in 3m4s
Details
CI Review / cockpit-test (pull_request) Successful in 3m7s
Details
This pull request has changes conflicting with the target branch.
  • core/terraform/live/zitadel/terraform.tfvars.example
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin plane/HOMELAB-2479-zitadel-h2c:plane/HOMELAB-2479-zitadel-h2c
git switch plane/HOMELAB-2479-zitadel-h2c
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
operator-review

Requires human operator review before merge — dangerous class (RBAC, secrets, network policy, Terraform) or explicitly gated. Never auto-merge.

No labels
operator-review
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Homelab/infra-core!911
No description provided.