HOMELAB-1735: pull-only Harbor robot seed from OpenBao + stop CI workflows stripping the shared docker config #521
No reviewers
Labels
No labels
operator-review
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Homelab/infra-core!521
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "plane/HOMELAB-1735-harbor-robot"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What
Two related fixes for the Forgejo runner fleet's Harbor credential (HOMELAB-1735):
core/manifests/external-secrets/es-forgejo-runner.yaml— therunner-harbor-robotseed (mounted into the runner daemon's/home/runner/.dockerand dind's/root/.docker) now reads a dedicated pull-only robot from OpenBaokv/app-credentialsinstead of sharing sendrast's own workload robot (kv/sendrast/harbor)..forgejo/workflows/runner-image-build.yaml+.forgejo/workflows/docs-build.yaml— log in to a throwawayDOCKER_CONFIGand dropdocker logout; both now use the scoped push robot repo secretsHARBOR_PUSH_USERNAME/HARBOR_PUSH_PASSWORD.statefulset.yamldocumenting the robot's scope.Why (root cause, re-verified)
The
k8s/infrarunner labels are:hostmode, so workflow steps run inside the runner container asrunnerwithHOME=/home/runnerandDOCKER_CONFIGunset.docker logouttherefore rewrites the pod-shared emptyDir seeded once per pod from therunner-harbor-robotsecret and also used by the runner daemon for job-image pre-pulls (which happen before any workflow step candocker login).Re-verified on 2026-09-29:
kubectl get secret runner-harbor-robot -n forgejo-runner(.dockerconfigjsondecoded)stat -c '%s %y' /home/runner/.docker/config.jsonon runner-0/4/7{"auths":{}}), mtimes 2026-09-25 05:27, 2026-09-29 04:03:37, 2026-09-29 04:07:57The mtimes match CI pushes, not pod starts — i.e. login+logout pairs, not a seeding failure. The seed secret itself is intact.
Scope decision (evidence-based)
Harbor robots created (admin API, secrets never printed):
robot$forgejo-runner-pullsendrast→repository:pullrobot$forgejo-runner-pushplatformandlab-director→repository:pull+repository:pushWhy exactly that:
sendrast(5 repos) andpaperclip(0 repos). Every other project (docker-cache,platform,lab-director,k8s-cache,ghcr-cache,apps,library) is public → anonymous pull.harbor.aaron.reynoza.org/sendrast/e2e-runner:*;harbor.aaron.reynoza.org/docker-cache/catthehacker/ubuntu:act-22.04is public. So the daemon needssendrastpull only.platform/forgejo-runner(runner-image-build),platform/docs(docs-build) andlab-director/pi(pi-image-build) → the robot coversplatform+lab-directoronly. It deliberately does not coversendrast/paperclip/anything else, and has no project-create/delete/list/tag permissions (unlike the all-projectsrobot$ci-pushit is meant to retire).:hostjob could read the sendrast namespace's own registry token out of the shared emptyDir.Capability probes against the registry (401/403 = denied, 200/202 = allowed):
Out-of-band state already applied (not in Git)
kv/app-credentials:forgejo_runner_harbor_pull_username/_pull_token/_push_username/_push_token(read-back hash-verified; values never printed).HARBOR_PUSH_USERNAME/HARBOR_PUSH_PASSWORDcreated onHomelab/infra-core(base64 via API). They are inert until this PR merges; rollback = delete those two repo secrets (plus the robots in Harbor).Rollout caution (read before merging)
This changes a secret that all 8 runner pods consume. ArgoCD will sync the ExternalSecret after merge, but pods keep the config they were seeded with at start — existing pods stay on the old credential until they cycle. Until then their
{"auths":{}}-stripped configs remain stripped, so private pre-pulls stay broken on those pods regardless of this PR.Operator sequence (safe; no restart performed by the agent):
external-secrets-configand confirmkubectl get externalsecret -n forgejo-runner runner-harbor-robot -o jsonpath='{.status.conditions[0]}'→SecretSynced.kubectl get secret runner-harbor-robot -n forgejo-runner -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq -r '.auths|keys[]'→harbor.aaron.reynoza.org; the username inside is nowrobot$forgejo-runner-pull).kubectl rollout restart statefulset/forgejo-runner -n forgejo-runner). Pods re-run thedocker-configinitContainer and re-seed from the new secret. Registration state lives on per-pod PVCs, so a cycle is safe.stat -c %y /home/runner/.docker/config.jsonmtime must not move after a push run of either fixed workflow (that is the proof the logout footgun is gone).Verification done here (offline)
yamllint -d relaxedclean of new errors; both workflow YAMLs parse; the modified run steps passbash -n.kubectl apply --dry-run=server -f core/manifests/external-secrets/es-forgejo-runner.yaml→ schema valid, no write.dockerconfigjsontemplate renders valid JSON with the new values (auth round-trips).Not done (deliberately, with rationale)
pi-image-build.yamluntouched (itsDOCKER_CONFIGpattern is already correct and the brief scoped it out) — it still usesHARBOR_*, so retiring that admin-grade repo secret needs a follow-up that migrates it toHARBOR_PUSH_*(the robot already coverslab-director/piso that follow-up is a one-line secret rename). Its header comment still describesHARBOR_*as a shared admin, which will be stale after the swap.livebranches):Sendrast/api-sendrast—ci-app.yml:133,release.yml:100: unisolateddocker login+docker logout(strips the entry).Sendrast/sendrast-web—e2e-image.yml:93: unisolateddocker login+docker logout(strips the entry).Homelab/lab-director—build.yaml+build-sandbox.yaml(4 logins): unisolateddocker login, no logout — it does not delete the entry but overwrites the pod-shared one with a lab-director push credential, which then breaks the daemon's sendrast pre-pull.Separate PR per repo; listed on the ticket rather than drive-by changed here.
/home/runner/.dockerwould break every remaining:hostworkflow that still does a plaindocker login, and a repo-local lint cannot protect other repos. Proposal recorded on the ticket.Landing worker report — NOT MERGED (1 verified blocker)
Convoy: HOMELAB-1735 landing. Review ran in fresh contexts (two independent
pi -p --no-sessionreviewers, briefed only on the diff + ticket + repo contracts).CI: green. PR head
4646fd7→ 6/6 checks success (CI Review / helm-validate,receiver-test;Lint & Validate / shellcheck,yaml-lint,terraform-validate; atlantis). Note:forgejo_ci_status500s onactions/tasks(HOMELAB-1736) — obtained via thecommits/{sha}/statusfallback.Repo secrets verified present (API, names/created_at only — no values read or printed):
Review verdicts
pi-image-build.yamlpattern (sameset -euo pipefail→export DOCKER_CONFIG="$(mktemp -d)"→ login on stdin, no logout); nodocker logoutremains anywhere in.forgejo/workflows/**; bash guaranteed on both labels (one runner class,DESIRED_LABELS="infra:host,k8s:host", image ships bash); no cluster-mutating step; secrets hygiene clean.BLOCKER — the PR's only object-level change sits inside an ArgoCD-ignored path, so auto-sync will never apply it
prodapps/external-secrets-config.yaml:25-34(the app that ownscore/manifests/external-secrets,targetRevision: live,automated: {prune, selfHeal}):The complete object-level delta of this PR in that directory is
spec.data[].remoteRef.{key,property}onrunner-harbor-robot; everything else (secretStoreRef, target name/creationPolicy, template bytes, the other three ExternalSecrets) is identical base↔head.Independently verified (not taken on trust):
gitops-enginepkg/diff/diff.go—Diff()callsNormalize(config, …)andNormalize(live, …);Normalize()runso.normalizer.Normalize(un), the ignoreDifferences normalizer. Both sides lose the ignored path → the resource compares equal → app reports Synced →automatedsync is never triggered (selfHealuses the same diff). When a sync does happen, ignored fields are applied (hence ArgoCD offersRespectIgnoreDifferencesto opt out) — the point is that nothing triggers one.prodapps/omniroute.yaml:42-53— "the former broad.spec.data[].remoteRefpath also swallowed NEW data entries and property changes" — and that app was migrated to field-scoped paths.apps/external-secrets-config.yamlwas not.external-secrets-config= Synced/Healthy, last operation2026-09-22T04:25:37Z; live ES spec stillkey: kv/sendrast/harbor,property: robot_username/robot_token. No ES in that directory has changed in Git since 2026-09-22, so there is no pending OutOfSync to piggyback on.secretStoreRef(kubernetes-secrets→openbao) → OutOfSync → full apply, which incidentally applied the new remoteRefs.Impact. After merge the in-cluster
runner-harbor-robotExternalSecret keeps the old source, so the dedicated pull-only robot never lands and the security objective (runner pods no longer carrying thesendrastnamespace's workload credential) is silently not achieved. The rollout step 1 in the PR body (SecretSynced) is a false pass — the old source still resolves and syncs fine. Only step 2 (the username inside the Secret must berobot$forgejo-runner-pull) would catch it, and only after pods cycle, which would re-seed the old credential.Remediation options (operator decision — this is why the convoy stopped):
apps/omniroute.yaml:50-54(.spec.data[].remoteRef.conversionStrategy|decodingStrategy|metadataPolicy+.spec.target.deletionPolicy). Then this PR's property change becomes visible → OutOfSync → ArgoCD applies it from Git, no manual sync. Trade-off: that app then compareskey/propertyfor ~20 ExternalSecrets across namespaces; any live drift would be patched byselfHeal(git becomes truth) — a GitOps config change with blast radius wider than this PR, so it needs your sign-off, and it is not in this PR's diff.argocd app sync external-secrets-config— then assertkubectl get externalsecret -n forgejo-runner runner-harbor-robot -o jsonpath='{.spec.data[*].remoteRef.key}'printsapp-credentials app-credentialsbefore cycling pods. Refresh alone movesstatus.sync.revisionwithout applying — it looks like success and is not.What merging anyway would still deliver (context for sequencing): the workflow half — the footgun fix and the
HARBOR_PUSH_*swap go live for this repo's two workflows on merge, and becauserunner-image-build.yamllists itself inon.push.paths, the merge itself triggers a real publish run (an immediate end-to-end test of the push robot + the mktemp pattern). The pull-robot decoupling is the part that does not land.Everything else on the frozen constraints checked out
pi-image-build.yaml:128-132, no logout left; theset -ealso removes the "failedmktempsilently falls back to$HOME/.docker" hole.runner-harbor-robotunchanged (= consumer'ssecretName,statefulset.yaml:256-258),.dockerconfigjsonkey unchanged (= consumer'sitems[].key,:260-262), template bytes identical base↔head, and the template still renders because thedata[].secretKeynames it interpolates (username/password) are unchanged.key: app-credentialsresolves tokv/data/app-credentialsunder the store'spath: kv+version: v2, matching the canonical form already used byes-harbor.yaml,es-grafana.yaml,es-forgejo.yaml,es-zitadel.yaml,es-rocketchat.yaml. (The removedkv/sendrast/harborprefixed form was not broken — ESO strips and re-appends the provider path — so this is a genuine source swap, not a path repair.)pi-image-build.yamluntouched (stillHARBOR_*), no guard mounts, no other repos.statefulset.yamldelta is 3 comment lines → no rendered-object change → no rollout; the ES is initContainer-seeded only, so un-cycled pods keep their current configs.HARBOR_PUSH_*"replaces the all-projects admin-grade HARBOR_* credential for this repo's publishes" — not true yet,pi-image-build.yaml:131-132still logs in withHARBOR_*/HARBOR_PASSWORDfor itslab-director/pipush (the PR body already discloses this).statefulset.yaml:126still says the emptyDir "keeps it writable for :host jobs' docker login", three lines above the new "job steps must never write here". Contradictory docs for the next reader.DOCKER_CONFIGdir is never removed (inherited from the proven pattern), so a push token stays readable in/tmpfor the pod's lifetime;trap 'rm -rf "${DOCKER_CONFIG}"' EXITwould close it — deliberately not requested, since C2 asks for behavioural identity withpi-image-build.yaml..dockerconfigjsongets an empty username/password and private pre-pulls failunauthorizedagain — i.e. the exact "flake in someone else's job" symptom. Consider alerting on the ESO status metric (ServiceMonitor is already enabled) — follow-up, not this PR.No cluster mutations were performed by this convoy; no pods were restarted. Ticket HOMELAB-1735 stays In Progress, blocked on the remediation choice above.
BLOCKER DISPOSITIONED — resolved by Homelab/prod #795 (merged)
The blocker from the landing review above was correct and is now fixed. Nothing in this PR changed; the fix landed upstream in the repo that owned the footgun.
Unblocking change —
Homelab/prod#795, merged as40185c32intolive(PR: Homelab/prod#795):prod/apps/external-secrets-config.yamlno longer ignores the broad.spec.data[].remoteRef. It now ignores the field-scoped set, byte-identical to the two in-repo precedents (apps/omniroute.yaml,apps/sendrast.yaml):remoteRef.key/.propertyare therefore diff-compared again, so this PR's only object-level change becomes visible and triggers a sync.Verified live, after #795 merged (read-only kubectl)
spec.ignoreDifferences[0].jqPathExpressions= the 5-path listSynced/HealthyremoteRef.keystillkv/sendrast/harbor×2Pre-merge drift census re-derived independently: 20 ExternalSecrets / 42
spec.data[]entries in the app's source path, git↔live identical onremoteRef.key+.propertyfor 42/42 (0 mismatches), nodataFrom,remoteRef.version/.sourceRefdeclared on neither side — so #795 provably could not change the diff set, only un-hide future remoteRef edits.Partial-automation note: repo secret
HARBOR_PUSH_USERNAME/HARBOR_PUSH_PASSWORDwere confirmed present and this head is green 6/6. The credential itself is inert for the pods until they cycle (pods keep the config seeded at start); the operator sequence at the end of the PR body still stands and is tracked on HOMELAB-1735.Merging now.
Landing report — MERGED
dd99f5c6(infra-corelive) — with a FAILED publish runMerged on green CI:
4646fd7cc→ commit status success, 6/6. Merge commitdd99f5c637461d889b3a182980861940e049066d, merged2026-09-29T07:49:07Z.ESO half — PASS (the blocker fix works end-to-end)
Prerequisite
Homelab/prod#795had already merged (40185c32) so the app no longer ignoredspec.data[].remoteRef.{key,property}. Read-only assertions after merge:external-secrets-configOutOfSyncatt+255s→ auto-synced →Synced/Healthyatt+270s(well inside the 15-min window)type=Ready status=True reason=**SecretSynced**(this ESO version reports theSecretSyncedreason on aReadycondition),refreshTime=2026-09-29T07:53:47Zspec.data[*].remoteRef— both entriesusername -> app-credentials/forgejo_runner_harbor_pull_usernamepassword -> app-credentials/forgejo_runner_harbor_pull_token.dockerconfigjson202 → 216 bytes;auths = ["harbor.aaron.reynoza.org"]; embedded username sha256 matchesrobot$forgejo-runner-pull(verified by hash comparison — no value printed)The pull-only robot is now the real credential in the cluster, and the
sendrastworkload credential is no longer the source. The review blocker is confirmed resolved.ANOMALY — the runner-image publish run FAILED; no image was published
The
Build Runner Imageworkflow did fire on the merge (its ownon.push.pathslists the workflow file), but it failed:Step breakdown (from the job page):
Build imagesuccess (2m15s) →Push to HarborFAILURE (0s) →Print immutable digestskipped.Log evidence (
actions/runs/1490/jobs/0/attempt/1/logs):The failure is at
docker login, not at the push. The image was built and tagged, then the step died 0.08s later — nowhere near enough time for the ~250 MB multi-layer push this image requires. Critically, the successful run of the same workflow on 2026-09-22 (index 1463) logged, at the identical point:Neither the
WARNING!norLogin Succeededline appears in the failed run. Sodocker loginwas rejected by Harbor before any credential was stored.Confirmed no image was published (
crane, the PR's own tool surface):090c8d9is the 2026-09-22 build; no new runner image has been published since then. The trailinglatesttag likewise did not move to this merge.Where the problem is (not elsewhere)
repos/Homelab/infra-core/actions/secretsreturnsHARBOR_PUSH_USERNAMEandHARBOR_PUSH_PASSWORD, bothcreated_at 2026-09-29T06:31:14Z. (The API never returns values, so values could not be checked.){"auths":{}}config is not the cause: this step now uses a throwawayDOCKER_CONFIG, anddocker loginwrites client-side only.code.forgejo.org/forgejo/runner:12.13.1,debian:bookworm-slim), so the successfulBuild imagestep needed no Harbor auth.platform/forgejo-runner), which points at the value copied into the repo secrets rather than the robot's existence or scope.Leading hypothesis: the
HARBOR_PUSH_USERNAMEvalue is missing therobot$prefix (Harbor system robot usernames arerobot$<name>), or the username/token pair was transposed when copying from OpenBaokv/app-credentials.HARBOR_PUSH_USERNAMEshould equalkv/app-credentials.forgejo_runner_harbor_push_usernameverbatim.Not remediated by design — this landing was scoped to the two-PR sequence; no fix was pushed and nothing in the cluster was mutated. The
HARBOR_*secrets are still present, so a revert of the two workflow files would restore the previous publishing path.Still outstanding
pi-image-build.yamlwas deliberately left onHARBOR_*(noted in the PR body) and is unaffected by this failure. The operator pod-cycle for the pull credential remains pending — the 8 runner pods still read16 bytes({"auths":{}}) at/home/runner/.docker/config.json(runner-0mtime 09-25 05:27,runner-409-29 04:03:37,runner-709-29 04:07:57), so the pre-pull fix is not yet effective on them.