HOMELAB-1865: ci(atlantis): zitadel root applies via prod infra-apply, not Atlantis #582

Merged

aaron merged 1 commit from plane/HOMELAB-1865-atlantis-leg into live 2026-10-02 07:51:54 +00:00

Conversation 2 Commits 1 Files changed 1 +26 -11

aaron commented 2026-10-02 07:40:14 +00:00

Owner

Copy link

What

Removes core/terraform/live/zitadel from the Atlantis project remit in infra-core:atlantis.yaml. The root's sanctioned apply path is the prod repo's .forgejo/workflows/infra-apply.yaml with target: zitadel.

Closes HOMELAB-1865. Duplicate analysis: HOMELAB-1864.

Evidence (primary source, not inference)

The failing leg's job output, read from the live atlantis-0 pod log for this repo's PR #581 / commit 9c12fb5:

running 'sh -c' '/usr/local/bin/terraform init -input=false -upgrade -backend-config=backend.hcl'
  in '/atlantis-data/repos/Homelab/infra-core/581/default/core/terraform/live/zitadel': exit status 1
Initializing the backend...
│ Error: Failed to read file
│ The file "backend.hcl" could not be read.

Cause chain, all verified:

Fact	Where
backend.hcl is untracked by design	repo-root .gitignore → *.hcl + !*.hcl.example; the root tracks only backend.hcl.example (whose comment says "Real values live in the private prod repo.")
Atlantis clones infra-core alone	probe log: repo is at correct commit "9c12fb5…" in /atlantis-data/repos/Homelab/infra-core/581/
The custom zitadel workflow demands the missing file	atlantis.yaml → init: {extra_args: ["-backend-config=backend.hcl"]}
No server-side escape hatch	atlantis-repo-config ConfigMap repos.yaml declares only id: /.*/ + a sops workflow — no zitadel-specific backend mount
The working reference implementation	prod/.forgejo/workflows/infra-apply.yaml → target: zitadel checks infra-core out at ref: live and inits with -backend-config="$GITHUB_WORKSPACE/zitadel/backend.hcl"

Debt: this leg was already red on merged commits

failure on 648cfd8, c05c0e3, 4d893e4 and 9c12fb5. The zitadel changes in 648cfd8/c05c0e3/4d893e4 are on live via PRs #546/#552 — i.e. they landed while this leg was red. What changed is enforcement: the landing lock (HOMELAB-1679, no override) now requires ci.state === "success" on the head sha, so a leg that was always broken became a hard merge blocker. This is pre-existing debt, not a defect in #581's diff — #581 never reached a plan.

Change

• Removed the zitadel project block, with a comment recording the failure mode, the sanctioned path, and the precondition for re-enabling.
• workflows.zitadel is retained but marked DORMANT — it encodes the broken -backend-config=backend.hcl step and its env, and keeping it makes re-enable one block. es-atlantis.yaml's comment about that workflow therefore stays accurate.
• No committed backend.hcl (backend/state config is prod-side; the alternative was rejected).

Scope / review notes

• Diff is atlantis.yaml only — 1 file, +26/−11.
• Verified programmatically: projects is now exactly [network, bootstrap] with their dir/workflow/autoplan unchanged; version: 3 intact; the file parses.
• network and bootstrap use workflow: default and do not reference an untracked backend file — unaffected.
• Grepped docs/ and root *.md for any claim that Atlantis covers the zitadel root: none. ADR-004's "Atlantis plans only on PRs" is about the cluster root. No doc line needed.

Expected effect

PRs touching only core/terraform/live/zitadel/** get no Atlantis plan context instead of a permanent failure, so the combined status on a downstream PR head is driven by the real legs. terraform-validate (.forgejo/workflows/lint.yaml) still validates this root on every PR, and the apply path is unchanged and untouched.

Note for #581: its head 9c12fb5 already carries the stale failure status. A re-run/refresh (empty push or re-request checks) will be needed after this merges to clear it — the orchestrator can handle that.

Verification

$ python3 -c "import yaml; d=yaml.safe_load(open('atlantis.yaml')); print([p['name'] for p in d['projects']])"
['network', 'bootstrap']

After merge the real confirmation is the negative test: a zitadel-touching PR whose combined status contains no Atlantis failure.

## What Removes `core/terraform/live/zitadel` from the Atlantis project remit in `infra-core:atlantis.yaml`. The root's sanctioned apply path is the **prod** repo's `.forgejo/workflows/infra-apply.yaml` with `target: zitadel`. Closes HOMELAB-1865. Duplicate analysis: HOMELAB-1864. ## Evidence (primary source, not inference) The failing leg's job output, read from the live `atlantis-0` pod log for this repo's PR #581 / commit `9c12fb5`: ``` running 'sh -c' '/usr/local/bin/terraform init -input=false -upgrade -backend-config=backend.hcl' in '/atlantis-data/repos/Homelab/infra-core/581/default/core/terraform/live/zitadel': exit status 1 Initializing the backend... │ Error: Failed to read file │ The file "backend.hcl" could not be read. ``` Cause chain, all verified: | Fact | Where | |---|---| | `backend.hcl` is untracked by design | repo-root `.gitignore` → `*.hcl` + `!*.hcl.example`; the root tracks only `backend.hcl.example` (whose comment says *"Real values live in the private prod repo."*) | | Atlantis clones `infra-core` alone | probe log: `repo is at correct commit "9c12fb5…"` in `/atlantis-data/repos/Homelab/infra-core/581/` | | The custom `zitadel` workflow demands the missing file | `atlantis.yaml` → `init: {extra_args: ["-backend-config=backend.hcl"]}` | | No server-side escape hatch | `atlantis-repo-config` ConfigMap `repos.yaml` declares only `id: /.*/` + a `sops` workflow — no zitadel-specific backend mount | | The working reference implementation | `prod/.forgejo/workflows/infra-apply.yaml` → `target: zitadel` checks infra-core out at `ref: live` and inits with `-backend-config="$GITHUB_WORKSPACE/zitadel/backend.hcl"` | ## Debt: this leg was already red on merged commits `failure` on `648cfd8`, `c05c0e3`, `4d893e4` and `9c12fb5`. The zitadel changes in `648cfd8`/`c05c0e3`/`4d893e4` are **on `live`** via PRs #546/#552 — i.e. they landed while this leg was red. What changed is enforcement: the landing lock (HOMELAB-1679, no override) now requires `ci.state === "success"` on the head sha, so a leg that was always broken became a hard merge blocker. This is pre-existing debt, not a defect in #581's diff — #581 never reached a plan. ## Change - Removed the `zitadel` project block, with a comment recording the failure mode, the sanctioned path, and the precondition for re-enabling. - `workflows.zitadel` is **retained but marked DORMANT** — it encodes the broken `-backend-config=backend.hcl` step and its env, and keeping it makes re-enable one block. `es-atlantis.yaml`'s comment about that workflow therefore stays accurate. - No committed `backend.hcl` (backend/state config is prod-side; the alternative was rejected). ## Scope / review notes - Diff is **`atlantis.yaml` only** — 1 file, +26/−11. - Verified programmatically: `projects` is now exactly `[network, bootstrap]` with their `dir`/`workflow`/`autoplan` unchanged; `version: 3` intact; the file parses. - `network` and `bootstrap` use `workflow: default` and do not reference an untracked backend file — unaffected. - Grepped `docs/` and root `*.md` for any claim that Atlantis covers the zitadel root: **none**. ADR-004's "Atlantis plans only on PRs" is about the cluster root. No doc line needed. ## Expected effect PRs touching only `core/terraform/live/zitadel/**` get **no** Atlantis plan context instead of a permanent `failure`, so the combined status on a downstream PR head is driven by the real legs. `terraform-validate` (`.forgejo/workflows/lint.yaml`) still validates this root on every PR, and the apply path is unchanged and untouched. **Note for #581:** its head `9c12fb5` already carries the stale `failure` status. A re-run/refresh (empty push or re-request checks) will be needed after this merges to clear it — the orchestrator can handle that. ## Verification <pre> $ python3 -c "import yaml; d=yaml.safe_load(open('atlantis.yaml')); print([p['name'] for p in d['projects']])" ['network', 'bootstrap'] </pre> After merge the real confirmation is the negative test: a zitadel-touching PR whose combined status contains no Atlantis `failure`.

aaron added 1 commit 2026-10-02 07:40:14 +00:00

HOMELAB-1865: ci(atlantis): zitadel root applies via prod infra-apply, not Atlantis ... 6ecd475da8

core/terraform/live/zitadel can never be planned by in-cluster Atlantis: its
backend block is partial and the real backend.hcl is gitignored by the repo
root .gitignore (*.hcl / !*.hcl.example) and lives only in the private prod
repo. Atlantis clones infra-core alone, so its custom workflow's
'init -backend-config=backend.hcl' fails with:

  Error: Failed to read file
  The file "backend.hcl" could not be read.

Verified in the live atlantis-0 pod log for PR #581 (commit 9c12fb5). The red
plan status fired on every commit (seen on 648cfd8, c05c0e3, 4d893e4,
9c12fb5) and, via the landing lock (HOMELAB-1679, no override), now blocks
merges of otherwise-green downstream PRs.

The sanctioned path for this root is prod's .forgejo/workflows/infra-apply.yaml
with 'target: zitadel', which checks infra-core out at live on the runner and
uses prod's tracked backend.hcl + terraform.tfvars. Atlantis coverage was
redundant with that path and environmentally impossible.

Remove the zitadel project block; the network and bootstrap projects are
untouched. The zitadel workflow is retained, marked dormant, for one-block
re-enable once an Atlantis-visible backend config exists.

aaron referenced this pull request 2026-10-02 07:42:21 +00:00

SCRATCH PROBE do not merge - HOMELAB-1865 Atlantis head-vs-base config test #583

aaron commented 2026-10-02 07:45:06 +00:00

Author

Owner

Copy link

Verification evidence — negative test executed (head-vs-base finding)

A scratch probe (PR #583, now closed, not merged; branch deleted, worktree removed) was branched off this branch and touched core/terraform/live/zitadel/main.tf with a no-op comment — a change matching the removed project's when_modified: ["*.tf"].

Atlantis log for "pull":"583":

successfully parsed atlantis.yaml file
0 projects are to be planned based on their when_modified config
determined there was no project to run plan in
Updating Gitea check status for 'atlantis/plan' to 'success'

Commit status on 39bd744: '' → success | 0/0 projects applied successfully.

Under origin/live's config (which still declares the zitadel project) that same commit would have planned 1 project and failed. It planned 0 and passed → the fix works.

Finding that matters for the handoff

Atlantis resolves repo-level atlantis.yaml from the PR head, not the base branch. So merging this PR does not by itself clear the stale failure on #581: its head 9c12fb5 still carries the old atlantis.yaml in its own tree. #581 needs one new commit on its branch carrying the fixed config — cheapest is git merge origin/live and push. A bare "re-request checks" will not suffice.

This PR's own CI is green on 6ecd475 (shellcheck, yaml-lint, terraform-validate, helm-validate, dispatcher-test, alert-receiver-test, receiver-test).

## Verification evidence — negative test executed (head-vs-base finding) A scratch probe (PR #583, now **closed, not merged**; branch deleted, worktree removed) was branched **off this branch** and touched `core/terraform/live/zitadel/main.tf` with a no-op comment — a change matching the removed project's `when_modified: ["*.tf"]`. Atlantis log for `"pull":"583"`: ``` successfully parsed atlantis.yaml file 0 projects are to be planned based on their when_modified config determined there was no project to run plan in Updating Gitea check status for 'atlantis/plan' to 'success' ``` Commit status on `39bd744`: `'' → success | 0/0 projects applied successfully.` Under `origin/live`'s config (which still declares the `zitadel` project) that same commit would have planned **1** project and **failed**. It planned 0 and passed → **the fix works**. ### Finding that matters for the handoff **Atlantis resolves repo-level `atlantis.yaml` from the PR *head*, not the base branch.** So merging this PR does *not* by itself clear the stale failure on #581: its head `9c12fb5` still carries the old `atlantis.yaml` in its own tree. #581 needs one new commit on its branch carrying the fixed config — cheapest is `git merge origin/live` and push. A bare "re-request checks" will **not** suffice. This PR's own CI is green on `6ecd475` (shellcheck, yaml-lint, terraform-validate, helm-validate, dispatcher-test, alert-receiver-test, receiver-test).

aaron referenced this pull request 2026-10-02 07:45:11 +00:00

HOMELAB-1681: feat(zitadel): lab-director OIDC application + operator project role #581

aaron commented 2026-10-02 07:51:45 +00:00

Author

Owner

Copy link

Independent review verdict — head 6ecd475

Fresh reviewer, separate context, primary sources only (diff vs origin/live, both file revisions, prod repo, live Atlantis/Forgejo API).

VERDICT: APPROVE-WITH-CONCERNS
BLOCKERS: 0

All 7 required checks PASS:

1. Scope PASS — git diff --name-status origin/live..origin/plane/HOMELAB-1865-atlantis-leg → M atlantis.yaml only; 1 commit ahead; git diff -w identical counts (26/11) → no whitespace-only rewrite; git diff --check exit 0.
2. Removal honest PASS — block deleted, not commented; parsed projects = ['network','bootstrap']; no renamed-dir project pointing at the zitadel root.
3. DORMANT marking honest PASS — workflows.zitadel retained byte-for-byte and still encodes - init: extra_args: ["-backend-config=backend.hcl"]; zero projects reference workflow: zitadel. Real Atlantis accepts the unused workflow: combined status on head is success | 0/0 projects applied successfully. ExternalSecret atlantis-zitadel-key (ns atlantis) verified against core/manifests/external-secrets/es-atlantis.yaml, matching TF_VAR_zitadel_key_file.
4. Remaining projects unchanged PASS — deep dict equality for network/bootstrap (dir, workflow, autoplan) = True/True; version: 3 in both.
5. NOTE correctness PASS (verified, not inferred) — prod/.forgejo/workflows/infra-apply.yaml declares zitadel as a target, checks infra-core out at ref: live, and inits with -backend-config="$GITHUB_WORKSPACE/zitadel/backend.hcl"; prod tracks zitadel/backend.hcl + terraform.tfvars.
6. No smuggled config PASS — no committed backend.hcl/secrets; .forgejo/workflows/* untouched; terraform-validate / shellcheck / yaml-lint intact.
7. Parses PASS — ['network','bootstrap'], workflows: ['zitadel'], version: 3.

Concerns (non-blocking, comment wording only):

• [concern] "Atlantis plan coverage of this root was redundant with that path" overstates it — infra-apply.yaml is workflow_dispatch (apply-only), and the automated plan-zitadel job in prod/.forgejo/workflows/infra-plan.yaml early-exits when zitadel/*.tf is absent. The accurate statement is impossible, not redundant. Not a regression: the leg was never green (648cfd8/c05c0e3/4d893e4/9c12fb5 all failure | 0/1 projects planned).
• [concern] "fired on every commit" is overstated — autoplan was scoped when_modified: ["*.tf"], so unrelated live commits carried no Atlantis leg. The stated consequence for #581 is accurate.
• [suggestion] core/terraform/live/zitadel/versions.tf:5 documents a non-existent backend path (../../../../environments/prod/zitadel/backend.hcl). Pre-existing, untouched here.
• [suggestion] New comment lines exceed 80 chars (no CI impact; yamllint scopes to core/manifests/ core/ansible/).

Claim honesty: yes — the diff does what the PR body says. Both imprecisions are confined to explanatory comments and do not misstate the change's behavior; the body is already candid that Atlantis resolves atlantis.yaml from the PR head, so merging does not itself clear #581's stale failure.

Merging on 0 blockers + green combined status.

## Independent review verdict — head `6ecd475` Fresh reviewer, separate context, primary sources only (diff vs `origin/live`, both file revisions, prod repo, live Atlantis/Forgejo API). ``` VERDICT: APPROVE-WITH-CONCERNS BLOCKERS: 0 ``` **All 7 required checks PASS:** 1. **Scope PASS** — `git diff --name-status origin/live..origin/plane/HOMELAB-1865-atlantis-leg` → `M atlantis.yaml` only; 1 commit ahead; `git diff -w` identical counts (26/11) → no whitespace-only rewrite; `git diff --check` exit 0. 2. **Removal honest PASS** — block deleted, not commented; parsed projects = `['network','bootstrap']`; no renamed-`dir` project pointing at the zitadel root. 3. **DORMANT marking honest PASS** — `workflows.zitadel` retained byte-for-byte and still encodes `- init: extra_args: ["-backend-config=backend.hcl"]`; zero projects reference `workflow: zitadel`. Real Atlantis accepts the unused workflow: combined status on head is `success | 0/0 projects applied successfully.` ExternalSecret `atlantis-zitadel-key` (ns atlantis) verified against `core/manifests/external-secrets/es-atlantis.yaml`, matching `TF_VAR_zitadel_key_file`. 4. **Remaining projects unchanged PASS** — deep dict equality for `network`/`bootstrap` (dir, workflow, autoplan) = True/True; `version: 3` in both. 5. **NOTE correctness PASS (verified, not inferred)** — `prod/.forgejo/workflows/infra-apply.yaml` declares `zitadel` as a `target`, checks infra-core out at `ref: live`, and inits with `-backend-config="$GITHUB_WORKSPACE/zitadel/backend.hcl"`; prod tracks `zitadel/backend.hcl` + `terraform.tfvars`. 6. **No smuggled config PASS** — no committed `backend.hcl`/secrets; `.forgejo/workflows/*` untouched; terraform-validate / shellcheck / yaml-lint intact. 7. **Parses PASS** — `['network','bootstrap']`, `workflows: ['zitadel']`, `version: 3`. **Concerns (non-blocking, comment wording only):** - `[concern]` "Atlantis plan coverage of this root was **redundant** with that path" overstates it — `infra-apply.yaml` is `workflow_dispatch` (apply-only), and the automated `plan-zitadel` job in `prod/.forgejo/workflows/infra-plan.yaml` early-exits when `zitadel/*.tf` is absent. The accurate statement is *impossible*, not *redundant*. Not a regression: the leg was never green (`648cfd8`/`c05c0e3`/`4d893e4`/`9c12fb5` all `failure | 0/1 projects planned`). - `[concern]` "fired on every commit" is overstated — autoplan was scoped `when_modified: ["*.tf"]`, so unrelated live commits carried no Atlantis leg. The stated consequence for #581 is accurate. - `[suggestion]` `core/terraform/live/zitadel/versions.tf:5` documents a non-existent backend path (`../../../../environments/prod/zitadel/backend.hcl`). Pre-existing, untouched here. - `[suggestion]` New comment lines exceed 80 chars (no CI impact; yamllint scopes to `core/manifests/ core/ansible/`). **Claim honesty: yes** — the diff does what the PR body says. Both imprecisions are confined to explanatory comments and do not misstate the change's behavior; the body is already candid that Atlantis resolves `atlantis.yaml` from the PR head, so merging does not itself clear #581's stale failure. Merging on 0 blockers + green combined status.

aaron merged commit 01925e3641 into live 2026-10-02 07:51:54 +00:00

aaron referenced this pull request from a commit 2026-10-02 07:51:56 +00:00

Merge pull request 'HOMELAB-1865: ci(atlantis): zitadel root applies via prod infra-apply, not Atlantis' (#582) from plane/HOMELAB-1865-atlantis-leg into live

aaron referenced this pull request 2026-10-02 07:57:21 +00:00

HOMELAB-1681: feat(zitadel): lab-director OIDC application + operator project role #581

aaron referenced this pull request 2026-10-02 08:00:46 +00:00

HOMELAB-1681: feat(zitadel): lab-director OIDC application + operator project role #581

aaron referenced this pull request 2026-10-03 03:17:41 +00:00

HOMELAB-1681: fix(zitadel): assert project roles into the cockpit ID token #590

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

operator-review

Requires human operator review before merge — dangerous class (RBAC, secrets, network policy, Terraform) or explicitly gated. Never auto-merge.

No labels

operator-review

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Homelab/infra-core!582

No description provided.