HOMELAB-1524: fix(cert): reflect wildcard cert to headscale namespace #498

Merged

aaron merged 1 commit from plane/HOMELAB-1524-cert-reflector into main

2026-09-05 06:57:27 +00:00

aaron commented

2026-09-05 06:56:02 +00:00

Owner

What

Promotes the wildcard-certificate reflector allowlist from live to main: adds headscale, drops ai (already the state on live via HOMELAB-1327 / HOMELAB-1457).

Why

The prod kustomization pins infra-core at ref=main, so this fix never reached the cluster. Headscale's wildcard cert copy was a stale manual apply that expired 2026-08-14. The 2026-09-03 power outage exposed it:

mgmt VM tailscaled (headscale control server) logged out → MagicDNS dead → all DNS on mgmt broken → pi OAuth refresh failed (KIMI_API_KEY error)
in-cluster tailscale-subnet-router CrashLoopBackOff

Valid cert was manually re-applied as a stopgap (expires Oct 13). Without this PR, the copy goes stale again at the next cert-manager renewal (~Sept 13).

After merge

ArgoCD cert-manager-config syncs the Certificate; cert-manager updates source secret annotations
Delete the manual headscale/wildcard-aaron-reynoza-org-tls copy so reflector recreates it under management

Ticket: HOMELAB-1524

## What Promotes the wildcard-certificate reflector allowlist from `live` to `main`: adds `headscale`, drops `ai` (already the state on live via HOMELAB-1327 / HOMELAB-1457). ## Why The prod kustomization pins infra-core at `ref=main`, so this fix never reached the cluster. Headscale's wildcard cert copy was a stale manual apply that expired **2026-08-14**. The 2026-09-03 power outage exposed it: - mgmt VM tailscaled (headscale control server) logged out → MagicDNS dead → all DNS on mgmt broken → pi OAuth refresh failed (`KIMI_API_KEY` error) - in-cluster `tailscale-subnet-router` CrashLoopBackOff Valid cert was manually re-applied as a stopgap (expires Oct 13). Without this PR, the copy goes stale again at the next cert-manager renewal (~Sept 13). ## After merge 1. ArgoCD `cert-manager-config` syncs the Certificate; cert-manager updates source secret annotations 2. Delete the manual `headscale/wildcard-aaron-reynoza-org-tls` copy so reflector recreates it under management Ticket: HOMELAB-1524

aaron added 1 commit

2026-09-05 06:56:02 +00:00

HOMELAB-1524: fix(cert): reflect wildcard cert to headscale namespace

0/0 projects applied successfully.

Release / release (pull_request) Failing after 3s

Details

26dfa8afe3

Add headscale to the reflector allowed/auto namespaces and drop ai
(no longer used), matching the state already on live (HOMELAB-1327,
HOMELAB-1457). The prod kustomization pins infra-core at ref=main, so
headscale never got automatic reflection: its cert copy was a manual
apply that expired 2026-08-14. The 2026-09-03 power outage exposed it —
mgmt VM tailscaled (headscale control) logged out, MagicDNS died, and
pi on mgmt could not refresh OAuth tokens.

aaron merged commit 26990bf701 into main

2026-09-05 06:57:27 +00:00

aaron deleted branch plane/HOMELAB-1524-cert-reflector

2026-09-05 06:57:27 +00:00

homelab-bot approved these changes

2026-09-05 06:57:28 +00:00

homelab-bot left a comment

Collaborator

AI Review (HOMELAB-1524)

Verdict: PASS-WITH-NITS

Acceptance Criteria (HOMELAB-1524)

Criterion	Met?	Evidence
Branch from main, apply live version of `wildcard-certificate.yaml` (adds `headscale`, drops `ai`)	✅	`core/manifests/cert-manager/wildcard-certificate.yaml:23-24` — both `reflection-allowed-namespaces` and `reflection-auto-namespaces` now list `headscale` and no longer list `ai`
PR to main, auto-merge after CI (additive change)	⚠️	Process step — not verifiable from diff; confirm CI green before merge
ArgoCD `cert-manager-config` syncs and reflector manages headscale copy	⚠️	Post-merge verification (ticket Verify item 1: reflector annotations present on `headscale/wildcard-aaron-reynoza-org-tls`)
Delete manual headscale secret copy so reflector adopts it	⚠️	Post-merge manual step — required, see Open Issues
`tailscale-subnet-router` stays Running; mgmt VM DNS + tailscale healthy	⚠️	Post-merge verification only

Design Doc Citations

Headscale VPN — Architecture & Operations (§2.4 Key configuration facts) — confirms "Wildcard *.aaron.reynoza.org cert reflected into the headscale namespace and mounted at /tls" for the LB at 10.10.10.229:4433. This PR restores exactly that documented behavior, which broke because prod pins infra-core at ref=main.
ADR-005: DNS, TLS, and Service Exposure Architecture — cert-manager/DNS-01 TLS strategy context; consistent with wildcard cert + reflector distribution model.

Open Issues

[post-merge] Reflector adoption ordering: the headscale namespace currently holds a manually applied cert copy (2026-09-03 stopgap). Emberstack Reflector will not adopt/overwrite a target secret it didn't create — it logs a warning and skips. The ticket's "delete manual copy" step must happen after ArgoCD syncs this change, or headscale TLS will silently keep serving the stale cert until expiry (Oct 13). Suggest keeping the merge and the manual delete in the same change window.
[core/manifests/cert-manager/wildcard-certificate.yaml:19-21] Nit — comment accuracy: the new comment warns against adding cilium-secrets (HOMELAB-478 loop), but cilium-secrets was never in either list; the comment guards against a hypothetical future edit. Harmless and arguably useful documentation, but a reader may infer it was removed. Not blocking.
[core/manifests/cert-manager/wildcard-certificate.yaml:23-24] Nit — ai removal: dropping ai means reflector will garbage-collect the existing auto-reflected copy in the ai namespace on next sync. Ticket scope explicitly sanctions this ("adds headscale, drops ai"), but confirm nothing in ai still mounts the wildcard cert before merge.
[scope] The two annotation lists must stay identical (auto ⊂ allowed per reflector semantics) — they do here; just note for future edits.

Diff itself is correct, minimal, and matches the ticket exactly. Merge after CI, then execute the two post-merge verification/deletion steps before closing HOMELAB-1524.

Powered by Kimi k3 · advisory.

## AI Review (HOMELAB-1524) **Verdict**: PASS-WITH-NITS ### Acceptance Criteria (HOMELAB-1524) | Criterion | Met? | Evidence | |---|---|---| | Branch from main, apply live version of `wildcard-certificate.yaml` (adds `headscale`, drops `ai`) | ✅ | `core/manifests/cert-manager/wildcard-certificate.yaml:23-24` — both `reflection-allowed-namespaces` and `reflection-auto-namespaces` now list `headscale` and no longer list `ai` | | PR to main, auto-merge after CI (additive change) | ⚠️ | Process step — not verifiable from diff; confirm CI green before merge | | ArgoCD `cert-manager-config` syncs and reflector manages headscale copy | ⚠️ | Post-merge verification (ticket Verify item 1: reflector annotations present on `headscale/wildcard-aaron-reynoza-org-tls`) | | Delete manual headscale secret copy so reflector adopts it | ⚠️ | Post-merge manual step — **required**, see Open Issues | | `tailscale-subnet-router` stays Running; mgmt VM DNS + tailscale healthy | ⚠️ | Post-merge verification only | ### Design Doc Citations - **Headscale VPN — Architecture & Operations** (§2.4 Key configuration facts) — confirms "Wildcard `*.aaron.reynoza.org` cert reflected into the `headscale` namespace and mounted at `/tls`" for the LB at `10.10.10.229:4433`. This PR restores exactly that documented behavior, which broke because prod pins infra-core at `ref=main`. - **ADR-005: DNS, TLS, and Service Exposure Architecture** — cert-manager/DNS-01 TLS strategy context; consistent with wildcard cert + reflector distribution model. ### Open Issues - [post-merge] **Reflector adoption ordering**: the headscale namespace currently holds a *manually applied* cert copy (2026-09-03 stopgap). Emberstack Reflector will not adopt/overwrite a target secret it didn't create — it logs a warning and skips. The ticket's "delete manual copy" step must happen **after** ArgoCD syncs this change, or headscale TLS will silently keep serving the stale cert until expiry (Oct 13). Suggest keeping the merge and the manual delete in the same change window. - [core/manifests/cert-manager/wildcard-certificate.yaml:19-21] **Nit — comment accuracy**: the new comment warns against adding `cilium-secrets` (HOMELAB-478 loop), but `cilium-secrets` was never in either list; the comment guards against a hypothetical future edit. Harmless and arguably useful documentation, but a reader may infer it was removed. Not blocking. - [core/manifests/cert-manager/wildcard-certificate.yaml:23-24] **Nit — `ai` removal**: dropping `ai` means reflector will garbage-collect the existing auto-reflected copy in the `ai` namespace on next sync. Ticket scope explicitly sanctions this ("adds headscale, drops ai"), but confirm nothing in `ai` still mounts the wildcard cert before merge. - [scope] The two annotation lists must stay identical (auto ⊂ allowed per reflector semantics) — they do here; just note for future edits. Diff itself is correct, minimal, and matches the ticket exactly. Merge after CI, then execute the two post-merge verification/deletion steps before closing HOMELAB-1524. _Powered by Kimi k3 · advisory._

aaron referenced this pull request from a commit

2026-09-05 06:57:29 +00:00

Merge pull request 'HOMELAB-1524: fix(cert): reflect wildcard cert to headscale namespace' (#498) from plane/HOMELAB-1524-cert-reflector into main