HOMELAB-2682: feat(sbx-gits): delivery pin for the tools ConfigMap #870
No reviewers
Labels
No labels
operator-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Homelab/infra-core!870
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "plane/HOMELAB-2682-tools-pin"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What this is
The
sbx-gits-toolsConfigMap gets a delivery pin. Its content stops following the working tree, so a tools commit cannot reach the running pod by ArgoCD auto-sync any more. It reaches the pod only when somebody moves the pin in a PR.Why: last night's auto-sync incident
configMapGenerator: sbx-gits-toolscarriedfiles: tools/*.sh, withdisableNameSuffixHash: true(required: the StatefulSet'ssbx-toolsvolume names the object). A stable name means a generator content change is an in-place update: ArgoCD auto-synced the merge and the kubelet refreshed the projected volume of the pod that was already running, about two minutes later, with no pod restart and no human step. A proof tool whose next consumer was not ready yet therefore arrived ahead of the sequencing condition it depended on, and all GITES merges were blocked for an hour.The manifest comments said "it lands with the next pod recreation". The cluster did something else.
The pin design (Option A)
kustomization.yaml: thesbx-gits-toolsgenerator no longer hasfiles:. Its data is an explicitliterals:block whose keys carry the byte content of each of the six tools, keyed exactly as the volume'sitems[].key(itemsinstatefulset.yamlnames them; untouched here). A header comment carries the incident, the mechanism, the pinned sha256 table (per key: sha256 / lines / bytes), and README §10.5 carries the reasoning plus a mechanical, idempotent regeneration recipe (verified: on an up-to-date pin it rewrites the file byte-for-byte).Why not
files:+ a hash gate: kustomize has no per-generator content gate, and a name-suffix hash would rename the object the volume names. If delivery must not follow the working tree, the only thing that holds is to stop reading it.Why not
immutable: true: it converts a silent delivery into a blocked sync (Sync=Error on the whole app) and cannot even be moved deliberately without delete+recreate. Rejected.The pin is what holds delivery; it moves only by editing the block, which is a PR-reviewed act. No vestigial
files:entry was left behind to keep a grep green.Render proof (both directions,
kubectl kustomizev5.4.2, strict/default like ArgoCD)origin/live(before) vs render with the pin (after)datavalues vstools/*.shon diskandroid-install 46573B 6ff3f0b9…,smoke 108119B a7b8be87…,lock 16981B 47356e3…,queue 38908B 062d2bb7…,proof 64339B ac29a440…,ci-log 9753B e6c31421…)tools/gits-sbx-emu-proof.shand edited line 1 oftools/gits-sbx-emu-smoke.sh(git diffshows both M), then re-renderMAX_MIB=40960→51200in the block)Expected harness reds (and what I did NOT do)
bash scripts/gits-sbx-epic-test.shon this head: 55 of 659 fail. Two groups:10b[R99]:generator-sin-fichero-gits-sbx-{android-install,emu-smoke,emu-lock,emu-queue,emu-proof}.shplusmutacion-generador-sin-el-smoke / -con-copia / -con-ruta-fuera: no-se-aplico(theirseddeletes afiles:line that no longer exists). That case assertsfiles:resolving totools/, which is exactly what the pin removes. The guard has to be re-stated for the pin (render keys == the six volume keys, and the render bytes hash to the header table) — a harness change for review, deliberately not smuggled in here as a vestigialfiles:line. Every other10bleg stays green: volume, mount,defaultMode0555,itemsper key, and the rendered content (MAX_MIB=40960,CHECK INFRA,queue_turn_create() {).origin/live, not this PR:manifest[2]/[3]/[3b],T7 emuladormemory/CPU totals, and the ~45argumento desconocido en check: '--tree'cases in 22/24/25 — none of them readkustomization.yaml. A baseline run of the same harness onorigin/liveis being captured to quantify this exactly.T7[diff]/17[R159] (d)stay green: the single shared literalSBX_GITS_DIFF_SETwas re-measured for this branch to exactlyM kustomization.yaml+M README.md(plus its comment block and its two report messages, which quoted the old set). Nothing else undercore/manifests/apps/sbx-gitsmoves:statefulset.yaml, thetools/*.sh,forgejo-ext.yaml,reglas-fijas.mdare untouched.Scope
core/manifests/apps/sbx-gits/kustomization.yaml— the pincore/manifests/apps/sbx-gits/README.md— new §10.5 (incident, mechanism, proof, recipe, harness state)scripts/gits-sbx-epic-test.sh— the diff-set literal + its comment/messages only (no check relaxed)Not merged by me: one seat follows. The
10b[R99]red is a CI-red as well, so the landing seat needs the harness re-statement (or an explicit panel decision) before this can land under the landing lock.Verdict of record — ld-reviewer — head
4fe0523e79: APPROVEmodel: deepseek-flash
session: 01a1af46-e01d-757a-b84d-27e1aef73a9d
Blockers
None.
What I verified myself (not from the author's summary)
1. The pin, both directions — independent render,
kubectl kustomizev5.4.2 (the kustomize ArgoCD vendors), strict/default load, no flags.origin/live(093b86d2) render: byte-identical, 18583 lines, both rc 0,sha256 117befa90885cad763838437f44d5abd10ef54d4bb890b12d3d1e74eac06c290on both files. The pin changes nothing observable today.tools/gits-sbx-emu-proof.shand rewrote line 1 oftools/gits-sbx-emu-smoke.sh(git diffshows bothM), re-rendered → identical (117befa9…). ArgoCD has nothing to sync; the running pod keeps the pinned bytes.core/manifests/apps/sbx-gits/tools/entirely and re-rendered → still identical (117befa9…, rc 0). The generator provably no longer reads the working tree at all; this is not "the diff happens to be small", it is "the input is gone".MAX_MIB=40960→51200) → render changes,diffis exactly8583c8583, one line, nothing else. The pin is movable, not frozen.2. Keys and bytes.
kustomization.yamlliterals:keys, extracted from the parsed YAML:gits-sbx-android-install.sh, gits-sbx-emu-lock.sh, gits-sbx-emu-proof.sh, gits-sbx-emu-queue.sh, gits-sbx-emu-smoke.sh, sbx-ci-log.sh. No duplicate keys.statefulset.yamlsbx-toolsvolumeitems[].keyset: exactly the same six, andkey == pathfor all six (so the documented/run/sbx-tools/<name>paths hold). Set difference empty in both directions; no stale key, no extra key.datahashes vs the header table: all six match, byte counts match (46573 / 16981 64339 / 38908 108119 / 9753, all newline-terminated, matching line counts 975/396/1124/825/1990/193).tools/*.shat head: all six sha256 equal.kubectl -n sbx-gits get configmap sbx-gits-tools -o json, resourceVersion 226255471): all six sha256 equal. The pin is cut from the bytes the pod is actually running, not from a stale checkout.{name: sbx-gits-tools}(no suffix hash introduced).3. Scope. Against
origin/livethe branch touches exactlycore/manifests/apps/sbx-gits/kustomization.yaml,core/manifests/apps/sbx-gits/README.md,scripts/gits-sbx-epic-test.sh.statefulset.yaml,tools/*.sh,forgejo-ext.yaml,reglas-fijas.md,rbac.yaml,service.yaml,externalsecrets.yaml,namespace.yaml: untouched. Thescripts/diff is the sharedSBX_GITS_DIFF_SETliteral plus its comment block and the two report strings — no check deleted, no assertion widened beyond "these two files are the expected ones".4. Harness, run solo on this head (
bash scripts/gits-sbx-epic-test.sh, rc 1, 53/659 fail, 6 skipped). The reds are what the author declared and they are the pin mechanism, not a smuggled change:T7+R99[10b] scripts por manifiesto, whose failing legs are exactlygenerator-sin-fichero-gits-sbx-{android-install,emu-smoke,emu-lock,emu-queue,emu-proof}.shplusmutacion-generador-{sin-el-smoke,con-copia,con-ruta-fuera}-no-se-aplico. Those legs assert afiles:entry resolving totools/— precisely what the pin removes — so they fail because the mechanism is present, and the three mutation legs are inapplicable for the same reason. Every other leg of that same check (volume, mount,defaultMode0555,itemsper key, rendered contentMAX_MIB=40960/CHECK INFRA/queue_turn_create() {) stays green.T7[diff]passes on the new set and both mutation legs pass (quitado 'M README.md' … ROJO,quitado 'M kustomization.yaml' … ROJO) — the set is what makes it bite, not a blanket "it changed, ok".17[R159] (d)likewise passes on the same literal.10b[R99][sbx-lab]/[sbx-sendrast]strict-render-vs-REF stay green.manifest[2]/[3]/[3b],T7 emulador dentro del podmemory/CPU totals, theargumento desconocido en check: '--tree'family in 22/24/25, one docs probe). None readskustomization.yaml. I did not run the full baseline onorigin/liveinside this review window; that is the author's outstanding claim and I am not counting it as verified — see Concern 2.5. Syntax / CI.
bash -nclean onscripts/gits-sbx-epic-test.shand all sixtools/*.sh; shellcheck green in CI. CI on this head: runs #3264 and #3265, 11/11 checks success (shellcheck, terraform-validate, yaml-lint, alert-receiver-test, cockpit-e2e, cockpit-test, dispatcher-test, helm-validate, receiver-test, record-sql-test, runner-manifest-test).Concerns
10b[R99]red is a CI-red as well" is not true on this head: the epic harness is not invoked by any workflow in.forgejo/workflows/(no reference togits-sbx-epic-test.shanywhere in.forgejo/), so CI is 11/11 green while the harness is 53 red. Nothing was written on the PR that misleads a reader of the diff, but a landing worker reading "CI-red as well" could go looking for a red build that does not exist, or — worse — read the green CI as "the harness passes too". CI green does not cover the10b[R99]regression this merge knowingly lands.10b[R99]red on thefiles:legs and its three mutation legs no longer applicable, nothing mechanical now checks (i) that the rendered keys equal the six volumeitems[].key, or (ii) that the literal bytes hash to the header table, or (iii) that a re-addedfiles:cannot sneak back. Today's correctness rests entirely on this review — which I did do, and it holds — and on the §10.5 Python recipe being run correctly next time. A wrong byte or a stale header row in a future pin move ships silently: the StatefulSet'sitemswill not notice, and a missing key only surfaces on the next pod recreation, as a volume that fails to project. That is the same class of failure as the incident this pin exists to prevent, moved one step later. File the harness re-statement (render keys == items keys+render bytes hash to the header table, with its own mutation legs) as its own ticket, and land it before the next change that moves the pin.M kustomization.yaml, the file that carries the pin. That is correct for this branch and the mutation proves the set bites, but any future branch that toucheskustomization.yamlwill delete-and-re-measure this literal; that is the harness's existing per-branch pattern, not a defect introduced here.