pi-landing commented

Collaborator

IR-20261112-mgmtdns — mgmt split-horizon /etc/hosts (infra-core, PR 1/2)

Mechanism and why

On mgmt ONLY, the *.aaron.reynoza.org names that agents send credentials to must resolve to the
in-cluster Cilium Gateway VIP 10.10.10.220 (it serves the *.aaron.reynoza.org wildcard, cert
valid to 2026-12-12), not OPNsense and no LAN-wide change. Today mgmt resolves those names to
the public Pangolin edge 204.168.218.246, which TERMINATES TLS — so a credential sent to those
names from mgmt crosses the public edge. That is the ROUTES hazard this change removes.

Mechanism: /etc/hosts entries managed by the mgmt-base role via ansible.builtin.blockinfile,
chosen over a systemd-resolved drop-in because:

mgmt's nsswitch.conf hosts line is files myhostname md4_minimal [NOTFOUND=return] dns — the
files source is /etc/hosts and it is consulted first, so getent/curl pick up the pins
immediately with no nsswitch/resolv.conf change;
systemd-resolved is active but /etc/resolv.conf is a static file (not the 127.0.0.53 stub) and
nsswitch has no resolve module, so a resolved drop-in would not actually change getent output
without two more config changes;
blockinfile keeps the existing cloud-init header and the existing headscale line
(10.10.10.229 headscale.aaron.reynoza.org) untouched, and gives a marker-delimited block with a
one-run rollback.

The SAME marker and the SAME block are written to BOTH /etc/hosts AND
/etc/cloud/templates/hosts.debian.tmpl. This is required, not cosmetic: cloud-init's
manage_etc_hosts rewrites /etc/hosts from that template on every boot, so a one-file fix would be
silently undone by the first reboot. The template's {{fqdn}}/{{hostname}} header lines and its
headscale line are left untouched.

The pin list is data in mgmt-base/defaults/main.yml (mgmt_split_horizon_hosts), so it is
reproducible and idempotent (blockinfile matches its BEGIN/END markers).

Apply

ansible-playbook -i core/ansible/inventories/mgmt/hosts.ini \
  core/ansible/playbooks/setup-mgmt-vm.yml --tags split-horizon

Apply with --tags split-horizon ONLY.

Do NOT use --tags base. --tags base runs the whole mgmt-base role — including a full
apt upgrade: dist on mgmt and the hostname task — not just the split-horizon block. --tags split-horizon runs only the blockinfile task above.

`--check` diff (ansible-playbook --check --diff, connection=local, become, tags split-horizon)

--- before: /etc/hosts (content)
+++ after: /etc/hosts (content)
@@ -14,3 +14,13 @@
 ff02::1 ip6-allnodes
 ff02::2 ip6-allrouters

+# BEGIN ANSIBLE MANAGED — mgmt split-horizon (10.10.10.220)
+10.10.10.220 forgejo.aaron.reynoza.org
+10.10.10.220 openbao.aaron.reynoza.org
+10.10.10.220 harbor.aaron.reynoza.org
+10.10.10.220 zitadel.aaron.reynoza.org
+10.10.10.220 plane.aaron.reynoza.org
+10.10.10.220 omniroute.aaron.reynoza.org
+10.10.10.220 grafana.aaron.reynoza.org
+10.10.10.220 argocd.aaron.reynoza.org
+# END ANSIBLE MANAGED — mgmt split-horizon (10.10.10.220)

The identical marked block is also appended to /etc/cloud/templates/hosts.debian.tmpl
(insertafter: EOF, after its IPv6 lines), leaving the template's {{fqdn}} header lines and its
headscale line untouched.

Per-name pre-check (read-only, `curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result} %{remote_ip}' --resolve <host>:443:10.10.10.220 https://<host>/…`, no `-k`)

name	http_code	remote_ip	included?
forgejo.aaron.reynoza.org	200	10.10.10.220	yes
openbao.aaron.reynoza.org	200 (/v1/sys/health)	10.10.10.220	yes
harbor.aaron.reynoza.org	200 (/api/v2.0/ping)	10.10.10.220	yes
zitadel.aaron.reynoza.org	200 (/.well-known/openid-configuration)	10.10.10.220	yes
plane.aaron.reynoza.org	200	10.10.10.220	yes
omniroute.aaron.reynoza.org	307 → /dashboard	10.10.10.220	yes
grafana.aaron.reynoza.org	200 (/api/health)	10.10.10.220	yes
argocd.aaron.reynoza.org	200	10.10.10.220	yes

All 8 are served by 10.10.10.220 with the wildcard cert (ssl_verify_result 0) and a harmless GET that
answers 2xx/3xx.

Skipped / pending (not added)

name	http_code	why skipped
auth.aaron.reynoza.org	404	no gateway HTTPRoute (cert valid, TLS terminates at 10.10.10.220, but no backend)
prometheus.aaron.reynoza.org	404	no gateway HTTPRoute
loki.aaron.reynoza.org	404	no gateway HTTPRoute

These three names resolve to 10.10.10.220 with a valid wildcard cert (verify 0) but have no HTTPRoute
on the Cilium Gateway, so a harmless GET returns 404. Pinning them to 10.10.10.220 would only move a 404
from the public edge to the gateway — no benefit — and the brief says "skip and list any name that
fails". No tracked caller of auth.aaron.reynoza.org / prometheus.aaron.reynoza.org /
loki.aaron.reynoza.org was found in prod or infra-core.

Rollback

Set the list to [] in one Ansible run: mgmt_split_horizon_hosts: [] (an empty list renders an
empty block, which blockinfile treats as absent, so the markers and block are removed from both files).
Do NOT "remove the list" — an absent variable re-triggers the default list. The blockinfile
markers make this a clean removal; the headscale line, the {{fqdn}} header and the cloud-init header
are not inside the managed block.

PROOF ADDS (how each will be produced at the later apply — acceptance evidence, not done here)

getent hosts <name> for ALL 11 names → 10.10.10.220 for the 8 included (the 3 skipped names
resolve via DNS, not /etc/hosts).
TLS verify through the system resolver (no --resolve): curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result}' https://<host>/… → verify 0 for each of the 8.
Helper-shaped no-credential requests: GET https://openbao.aaron.reynoza.org/v1/sys/health and
GET https://zitadel.aaron.reynoza.org/.well-known/openid-configuration with
curl -w '%{remote_ip}' → 10.10.10.220.
Drift check: every user's ~/.config resolv overrides, /etc/hosts (managed block only),
nsswitch order, nscd/systemd-resolved caches (flush after apply), and any process with a cached
resolution (count only) — nothing may still point at 204.168.218.246.
Why DNS is the only fix: /usr/local/bin/bao-operator-token hardcodes
https://zitadel.aaron.reynoza.org/oauth/v2/token and BAO_ADDR=https://openbao.aaron.reynoza.org,
so a code change cannot route around it — its target names resolve to 10.10.10.220 after this lands.

# IR-20261112-mgmtdns — mgmt split-horizon /etc/hosts (infra-core, PR 1/2) ## Mechanism and why On mgmt ONLY, the `*.aaron.reynoza.org` names that agents send credentials to must resolve to the in-cluster Cilium Gateway VIP **10.10.10.220** (it serves the `*.aaron.reynoza.org` wildcard, cert valid to 2026-12-12), **not** OPNsense and **no** LAN-wide change. Today mgmt resolves those names to the public Pangolin edge **204.168.218.246**, which TERMINATES TLS — so a credential sent to those names from mgmt crosses the public edge. That is the ROUTES hazard this change removes. Mechanism: `/etc/hosts` entries managed by the **`mgmt-base` role** via `ansible.builtin.blockinfile`, chosen over a systemd-resolved drop-in because: - mgmt's `nsswitch.conf` hosts line is `files myhostname md4_minimal [NOTFOUND=return] dns` — the `files` source is `/etc/hosts` and it is consulted first, so `getent`/`curl` pick up the pins immediately with no nsswitch/resolv.conf change; - systemd-resolved is active but `/etc/resolv.conf` is a **static file** (not the 127.0.0.53 stub) and nsswitch has no `resolve` module, so a resolved drop-in would not actually change `getent` output without two more config changes; - `blockinfile` keeps the existing cloud-init header and the existing **headscale line (`10.10.10.229 headscale.aaron.reynoza.org`)** untouched, and gives a marker-delimited block with a one-run rollback. **The SAME marker and the SAME block are written to BOTH `/etc/hosts` AND `/etc/cloud/templates/hosts.debian.tmpl`.** This is required, not cosmetic: cloud-init's `manage_etc_hosts` rewrites `/etc/hosts` from that template on every boot, so a one-file fix would be silently undone by the first reboot. The template's `{{fqdn}}`/`{{hostname}}` header lines and its headscale line are left untouched. The pin list is data in `mgmt-base/defaults/main.yml` (`mgmt_split_horizon_hosts`), so it is reproducible and idempotent (blockinfile matches its BEGIN/END markers). ## Apply ```bash ansible-playbook -i core/ansible/inventories/mgmt/hosts.ini \ core/ansible/playbooks/setup-mgmt-vm.yml --tags split-horizon ``` > **Apply with `--tags split-horizon` ONLY.** > > **Do NOT use `--tags base`.** `--tags base` runs the whole `mgmt-base` role — including a full > `apt upgrade: dist` on mgmt and the hostname task — not just the split-horizon block. `--tags > split-horizon` runs only the blockinfile task above. ## `--check` diff (ansible-playbook --check --diff, connection=local, become, tags split-horizon) ```diff --- before: /etc/hosts (content) +++ after: /etc/hosts (content) @@ -14,3 +14,13 @@ ff02::1 ip6-allnodes ff02::2 ip6-allrouters +# BEGIN ANSIBLE MANAGED — mgmt split-horizon (10.10.10.220) +10.10.10.220 forgejo.aaron.reynoza.org +10.10.10.220 openbao.aaron.reynoza.org +10.10.10.220 harbor.aaron.reynoza.org +10.10.10.220 zitadel.aaron.reynoza.org +10.10.10.220 plane.aaron.reynoza.org +10.10.10.220 omniroute.aaron.reynoza.org +10.10.10.220 grafana.aaron.reynoza.org +10.10.10.220 argocd.aaron.reynoza.org +# END ANSIBLE MANAGED — mgmt split-horizon (10.10.10.220) ``` The identical marked block is also appended to `/etc/cloud/templates/hosts.debian.tmpl` (`insertafter: EOF`, after its IPv6 lines), leaving the template's `{{fqdn}}` header lines and its headscale line untouched. ## Per-name pre-check (read-only, `curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result} %{remote_ip}' --resolve <host>:443:10.10.10.220 https://<host>/…`, no `-k`) | name | http_code | ssl_verify_result | remote_ip | included? | |------|-----------|-------------------|-----------|-----------| | forgejo.aaron.reynoza.org | 200 | 0 | 10.10.10.220 | yes | | openbao.aaron.reynoza.org | 200 (/v1/sys/health) | 0 | 10.10.10.220 | yes | | harbor.aaron.reynoza.org | 200 (/api/v2.0/ping) | 0 | 10.10.10.220 | yes | | zitadel.aaron.reynoza.org | 200 (/.well-known/openid-configuration) | 0 | 10.10.10.220 | yes | | plane.aaron.reynoza.org | 200 | 0 | 10.10.10.220 | yes | | omniroute.aaron.reynoza.org | 307 → /dashboard | 0 | 10.10.10.220 | yes | | grafana.aaron.reynoza.org | 200 (/api/health) | 0 | 10.10.10.220 | yes | | argocd.aaron.reynoza.org | 200 | 0 | 10.10.10.220 | yes | All 8 are served by 10.10.10.220 with the wildcard cert (ssl_verify_result 0) and a harmless GET that answers 2xx/3xx. ## Skipped / pending (not added) | name | http_code | ssl_verify_result | why skipped | |------|-----------|-------------------|-------------| | auth.aaron.reynoza.org | 404 | 0 | no gateway HTTPRoute (cert valid, TLS terminates at 10.10.10.220, but no backend) | | prometheus.aaron.reynoza.org | 404 | 0 | no gateway HTTPRoute | | loki.aaron.reynoza.org | 404 | 0 | no gateway HTTPRoute | These three names resolve to 10.10.10.220 with a valid wildcard cert (verify 0) but have **no HTTPRoute** on the Cilium Gateway, so a harmless GET returns 404. Pinning them to 10.10.10.220 would only move a 404 from the public edge to the gateway — no benefit — and the brief says "skip and list any name that fails". No tracked caller of `auth.aaron.reynoza.org` / `prometheus.aaron.reynoza.org` / `loki.aaron.reynoza.org` was found in prod or infra-core. ## Rollback Set the list to `[]` in one Ansible run: `mgmt_split_horizon_hosts: []` (an empty list renders an empty block, which blockinfile treats as absent, so the markers and block are removed from both files). Do **NOT** "remove the list" — an absent variable re-triggers the default list. The blockinfile markers make this a clean removal; the headscale line, the `{{fqdn}}` header and the cloud-init header are not inside the managed block. ## PROOF ADDS (how each will be produced at the later apply — acceptance evidence, not done here) 1. `getent hosts <name>` for ALL 11 names → `10.10.10.220` for the 8 included (the 3 skipped names resolve via DNS, not /etc/hosts). 2. TLS verify through the **system resolver** (no `--resolve`): `curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result}' https://<host>/…` → verify **0** for each of the 8. 3. Helper-shaped no-credential requests: `GET https://openbao.aaron.reynoza.org/v1/sys/health` and `GET https://zitadel.aaron.reynoza.org/.well-known/openid-configuration` with `curl -w '%{remote_ip}'` → **10.10.10.220**. 4. Drift check: every user's `~/.config` resolv overrides, `/etc/hosts` (managed block only), `nsswitch` order, nscd/systemd-resolved caches (flush after apply), and any process with a cached resolution (count only) — nothing may still point at `204.168.218.246`. 5. Why DNS is the only fix: `/usr/local/bin/bao-operator-token` hardcodes `https://zitadel.aaron.reynoza.org/oauth/v2/token` and `BAO_ADDR=https://openbao.aaron.reynoza.org`, so a code change cannot route around it — its target names resolve to 10.10.10.220 after this lands.

pi-landing added 1 commit

2026-11-12 12:49:08 +00:00

IR-20261112: feat(mgmt-base): split-horizon /etc/hosts pins for credential hostnames

CI Review / receiver-test (pull_request) Successful in 5s

Details

CI Review / runner-manifest-test (pull_request) Successful in 10s

Details

CI Review / helm-validate (pull_request) Successful in 10s

Details

CI Review / alert-receiver-test (pull_request) Successful in 20s

Details

Lint & Validate / shellcheck (pull_request) Successful in 6s

Details

Lint & Validate / yaml-lint (pull_request) Successful in 23s

Details

CI Review / record-sql-test (pull_request) Successful in 55s

Details

Lint & Validate / terraform-validate (pull_request) Successful in 2m16s

Details

CI Review / cockpit-test (pull_request) Successful in 2m44s

Details

CI Review / cockpit-e2e (pull_request) Successful in 4m26s

Details

CI Review / dispatcher-test (pull_request) Successful in 4m48s

Details

1d86694b18

Pin the *.aaron.reynoza.org names agents send credentials to, to the in-cluster
Cilium Gateway VIP 10.10.10.220 (serves the wildcard, cert valid to 2026-12-12),
so credential traffic never leaves the LAN to the public Pangolin edge
204.168.218.246. mgmt-only (/etc/hosts in the mgmt-base role), NOT OPNsense, no
LAN-wide change.

8 names verified served by 10.10.10.220 (ssl_verify_result 0, harmless GET
2xx/3xx). auth/prometheus/loki skipped: no gateway HTTPRoute -> 404.

blockinfile keeps the cloud-init header and the existing headscale line
(10.10.10.229) untouched; rollback = state: absent (one Ansible run).

Lab-Run: IR-20261112-mgmtdns

pi-landing added 1 commit

2026-11-12 13:29:41 +00:00

IR-20261112: fix(mgmt-base): blockinfile on both /etc/hosts and cloud-init template

CI Review / receiver-test (pull_request) Successful in 5s

Details

CI Review / runner-manifest-test (pull_request) Successful in 9s

Details

CI Review / helm-validate (pull_request) Successful in 10s

Details

CI Review / alert-receiver-test (pull_request) Successful in 21s

Details

CI Review / record-sql-test (pull_request) Successful in 48s

Details

Lint & Validate / yaml-lint (pull_request) Successful in 22s

Details

Lint & Validate / shellcheck (pull_request) Successful in 5s

Details

CI Review / cockpit-test (pull_request) Successful in 2m35s

Details

CI Review / dispatcher-test (pull_request) Successful in 3m3s

Details

Lint & Validate / terraform-validate (pull_request) Successful in 2m2s

Details

CI Review / cockpit-e2e (pull_request) Successful in 2m55s

Details

d1d289ef25

pi-landing added 1 commit

2026-11-12 13:39:47 +00:00

chore: add Lab-Run trailer (no content change; overseer)

CI Review / receiver-test (pull_request) Successful in 4s

Details

CI Review / helm-validate (pull_request) Successful in 9s

Details

CI Review / runner-manifest-test (pull_request) Successful in 11s

Details

CI Review / alert-receiver-test (pull_request) Successful in 21s

Details

Lint & Validate / shellcheck (pull_request) Successful in 8s

Details

Lint & Validate / yaml-lint (pull_request) Successful in 25s

Details

CI Review / record-sql-test (pull_request) Successful in 1m19s

Details

Lint & Validate / terraform-validate (pull_request) Successful in 1m56s

Details

CI Review / cockpit-test (pull_request) Successful in 2m52s

Details

CI Review / dispatcher-test (pull_request) Successful in 3m34s

Details

CI Review / cockpit-e2e (pull_request) Successful in 3m55s

Details

d96e1f376a

Lab-Run: IR-20261112-mgmtdns

CI Review / receiver-test (pull_request) Successful in 4s

Details

CI Review / helm-validate (pull_request) Successful in 9s

Required

Details

CI Review / runner-manifest-test (pull_request) Successful in 11s

Details

CI Review / alert-receiver-test (pull_request) Successful in 21s

Details

Lint & Validate / shellcheck (pull_request) Successful in 8s

Required

Details

Lint & Validate / yaml-lint (pull_request) Successful in 25s

Required

Details

CI Review / record-sql-test (pull_request) Successful in 1m19s

Details

Lint & Validate / terraform-validate (pull_request) Successful in 1m56s

Required

Details

CI Review / cockpit-test (pull_request) Successful in 2m52s

Details

CI Review / dispatcher-test (pull_request) Successful in 3m34s

Details

CI Review / cockpit-e2e (pull_request) Successful in 3m55s

Details

This pull request can be merged automatically.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin plane/IR-20261112-mgmtdns:plane/IR-20261112-mgmtdns

git switch plane/IR-20261112-mgmtdns

Rows
Columns

IR-20261112-mgmtdns — mgmt split-horizon /etc/hosts (infra-core, PR 1/2) #960

IR-20261112-mgmtdns — mgmt split-horizon /etc/hosts (infra-core, PR 1/2)

Mechanism and why

Apply

--check diff (ansible-playbook --check --diff, connection=local, become, tags split-horizon)

Per-name pre-check (read-only, curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result} %{remote_ip}' --resolve <host>:443:10.10.10.220 https://<host>/…, no -k)

Skipped / pending (not added)

Rollback

PROOF ADDS (how each will be produced at the later apply — acceptance evidence, not done here)

Checkout

`--check` diff (ansible-playbook --check --diff, connection=local, become, tags split-horizon)

Per-name pre-check (read-only, `curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result} %{remote_ip}' --resolve <host>:443:10.10.10.220 https://<host>/…`, no `-k`)