HOMELAB-1105: Update Tempo values for chart 1.24.2 #229

Merged

aaron merged 1 commit from plane/HOMELAB-1105-upgrade-tempo-chart into main 2026-06-12 03:01:31 +00:00

Conversation 0 Commits 1 Files changed 1 +59 -10

aaron commented 2026-06-12 02:58:59 +00:00

Owner

Copy link

Updates Tempo Helm values for chart 1.24.2 compatibility and adds host.name resource_dimensions support.

Breaking changes addressed

1. Port 3100 preservation — server.http_listen_port: 3100 prevents Grafana datasource breakage
2. query_frontend → queryFrontend — fixes camelCase mismatch that silently dropped query-frontend config
3. resources under tempo: — fixes resource limits that were silently ignored at top level
4. metricsGenerator.processor.span_metrics.resource_dimensions — adds host.name for dashboard filtering
5. overrides.defaults.metrics_generator.processors — explicitly declares processors (auto-injection removed)
6. Liveness/readiness probes — configured for port 3100 with 60s initial delay for WAL replay

Related

• prod PR: ArgoCD Application targetRevision bump
• HOMELAB-1102, HOMELAB-1103

Updates Tempo Helm values for chart 1.24.2 compatibility and adds `host.name` resource_dimensions support. ## Breaking changes addressed 1. **Port 3100 preservation** — `server.http_listen_port: 3100` prevents Grafana datasource breakage 2. **query_frontend → queryFrontend** — fixes camelCase mismatch that silently dropped query-frontend config 3. **resources under tempo:** — fixes resource limits that were silently ignored at top level 4. **metricsGenerator.processor.span_metrics.resource_dimensions** — adds `host.name` for dashboard filtering 5. **overrides.defaults.metrics_generator.processors** — explicitly declares processors (auto-injection removed) 6. **Liveness/readiness probes** — configured for port 3100 with 60s initial delay for WAL replay ## Related - prod PR: ArgoCD Application targetRevision bump - HOMELAB-1102, HOMELAB-1103

aaron added 321 commits 2026-06-12 02:58:59 +00:00

HOMELAB-276: feat(eso): add ExternalSecret for Lab Director ArgoCD token ... be0e6068b4

Syncs ArgoCD API token from secrets namespace to lab-director namespace.
Used by operator's devenv manager to create per-ticket ArgoCD Applications.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

HOMELAB-274: fix(lab-director): add /auth route to backend + healthz probe path ... 97eef1e9a9

HTTPRoute was missing /auth PathPrefix match for backend rule — browser
requests to /auth/login hit the SvelteKit frontend (404) instead of the
Go backend's OIDC handler.

Frontend probe path changed from / to /healthz — the root path triggers
auth redirect which breaks liveness checks (HTTPS mismatch on redirect).

Merge pull request 'HOMELAB-274: fix(lab-director): /auth routing + healthz probe path' (#38) from plane/HOMELAB-274-fix-deployment into live d4d5255035

HOMELAB-284: feat(scheduling): workload rebalancing — Longhorn auto-balance, Descheduler, PriorityClasses, LimitRanges ... 6604dfc327

- Enable Longhorn replica-auto-balance (best-effort) + data-locality
- Add Descheduler chart (LowNodeUtilization + RemoveDuplicates)
- Create PriorityClasses: cluster-critical (1M), standard (500K), low-priority (100K)
- Create LimitRanges for harbor, monitoring, management, zitadel, forgejo namespaces
- Add resource requests/limits to Cilium agent, operator, Hubble relay/UI
- Add config-reloader resources to kube-prometheus-stack

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-284: workload rebalancing — Longhorn auto-balance, Descheduler, PriorityClasses, LimitRanges' (#39) from plane/HOMELAB-284-resource-scheduling into live e9771f3911

HOMELAB-284: fix(descheduler): use DefaultEvictor for namespace exclusion ... a78699dc2d

evictableNamespaces is not valid in descheduler v1alpha2 API.
Use DefaultEvictor plugin's namespaces.exclude instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-284: fix descheduler policy API (evictableNamespaces → DefaultEvictor)' (#40) from plane/HOMELAB-284-resource-scheduling into live 20da49062f

HOMELAB-284: fix(descheduler): remove invalid namespace fields from policy ... 4053ae5b8c

v0.32.2 DefaultEvictor only supports ignorePvcPods + evictLocalStoragePods.
Namespace exclusion is not available in this API version. System pods are
protected by DaemonSet immunity, PDBs, and priority classes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-284: fix descheduler policy — remove invalid namespace fields' (#41) from plane/HOMELAB-284-resource-scheduling into live 992dc9c9db

HOMELAB-285: feat(monitoring): add capacity alerting PrometheusRules ... 0e23ca665f

8 early-warning alerts for resource capacity: NodeMemoryHigh (70%),
NodeCPUHigh (80%), PodCountImbalance, request saturation (memory/CPU),
NodeMemoryOvercommit (200%), OOMKillRateHigh, ClusterMemoryHeadroom
(4GiB warning, 2GiB critical). All PromQL validated against live
Prometheus.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-285: feat(monitoring): capacity alerting PrometheusRules' (#42) from plane/HOMELAB-285-capacity-alerts into live 2264287cc1

HOMELAB-319: feat(runner): K8s Forgejo runner with DinD sidecar ... 66b8c6d8e6

Custom runner image with Go, Node.js, golangci-lint, crane, Docker CLI.
DinD sidecar for container builds. Registers with 'infra' label to
replace VM 110 runner.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-319: feat(runner): K8s Forgejo runner with DinD sidecar' (#46) from plane/HOMELAB-319-forgejo-runner-k8s into live 91bddda19b

HOMELAB-319: fix(runner): mount docker-certs at /certs for TLS ... af6b4bffa2

Runner needs certs at /certs/client (DOCKER_CERT_PATH), so mount the
shared emptyDir at /certs root, not /certs/client.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-319: fix(runner): mount docker-certs at /certs for TLS' (#47) from plane/HOMELAB-319-forgejo-runner-k8s into live 52e29ff915

HOMELAB-319: fix(runner): disable DinD TLS + add namespace PodSecurity ... f081558629

- Disable TLS for DinD sidecar (same-pod localhost, no security benefit)
- Remove docker-certs volume entirely (eliminates cert race condition)
- Add namespace.yaml with privileged PodSecurity labels
- Use tcp://localhost:2375 (non-TLS) standard CI pattern

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-319: fix(runner): disable DinD TLS + add namespace PodSecurity' (#48) from plane/HOMELAB-319-forgejo-runner-k8s into live 43c272c80a

HOMELAB-319: fix(harbor): increase LimitRange to 1Gi for registry push ... 35b9ea2165

Harbor nginx and core OOM during large image pushes with 512Mi limit.
Registry processes multi-hundred-MB image layers — needs more headroom.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-319: fix(harbor): increase LimitRange to 1Gi for registry push' (#49) from plane/HOMELAB-319-harbor-limits into live 45b9fb7380

HOMELAB-324: security: scope dev-agent credentials for production ... 557767b5ea

- Remove SOPS age key (master decryption key) from dev-environment chart
  entirely — volume mount, volume definition, and values.yaml reference
- Add scoped RBAC manifest: ClusterRole with read-only cluster access +
  RoleBinding with admin within dev-agents namespace only
- Replaces over-privileged cluster-admin kubeconfig pattern

Addresses critical findings #2 (SOPS key) and #1 (kubeconfig scope) from
HOMELAB-312 security audit.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-324: scope dev-agent credentials for production' (#50) from plane/HOMELAB-324-security into live 0b834049b8

HOMELAB-304: fix(images): pin all :latest tags to specific versions ... c46a98917c

Replace :latest with SHA-based tags for custom images and pinned
releases for external images. Prevents stale deployments when CI
fails silently.

- lab-director backend/frontend: seed with current Harbor SHAs
- dev-environment: seed with pinned-20260324 tag
- minio: pin to RELEASE.2025-09-07T16-13-09Z
- minio/mc: pin to RELEASE.2025-08-13T08-35-41Z
- forgejo-runner: pin to SHA digest
- playwright-mcp: pin to SHA digest
- docker dind: pin to 27.5.1-dind

HOMELAB-316: fix(runner): register K8s runner with k8s-runner label ... 4683d4a05f

Change Forgejo runner registration from --labels "infra:host" to
"k8s-runner:host" so workflows using runs-on: k8s-runner are picked
up by the K8s runner (DinD), not the management VM runner.

The emptyDir volume for /data means the pod re-registers on restart,
automatically picking up the new label via ArgoCD self-heal.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-316: fix(runner): register K8s runner with k8s-runner label' (#52) from plane/HOMELAB-316-runner-label into live 750eaadc4f

Merge pull request 'HOMELAB-304: fix(images): pin all :latest tags to specific versions' (#51) from plane/HOMELAB-304-sha-tags into live 668ddcf7a2

HOMELAB-304: fix(outline): pin minio/mc in Helm template to specific version ... b5255237e7

Missed hardcoded minio/mc:latest in outline minio.yaml template.
Pin to RELEASE.2025-08-13T08-35-41Z to match values.yaml.

Merge pull request 'HOMELAB-304: fix(outline): pin minio/mc in Helm template' (#53) from plane/HOMELAB-304-sha-tags-fix into live 5dfdf5d3ba

HOMELAB-377: feat(cnpg): add B2 ExternalSecret for temporal namespace (#54) 850708c62b

HOMELAB-378: feat: Temporal Server base Helm values ... e83d631a05

Adds base configuration for Temporal Server Helm chart deployment:
- numHistoryShards: 4 (permanent, homelab-appropriate)
- 1 replica per service (frontend, history, matching, worker)
- PostgreSQL backend (postgres12 plugin), external CNPG
- Web UI enabled on port 8080
- Resource limits: ~450m CPU / 448Mi RAM total
- Cassandra, MySQL, Elasticsearch disabled

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-378: Temporal Server base Helm values' (#55) from plane/HOMELAB-378-temporal-server into live 3e23c53e85

HOMELAB-378: fix: correct Temporal Helm chart persistence structure ... 0f48b3e5ec

Chart expects server.config.persistence.default.driver = "sql" with
nested sql block — not a datastores wrapper. Fixes "Please specify
cassandra port" error during helm template.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'fix: correct Temporal Helm chart persistence structure' (#56) from fix/temporal-helm-values into live cb6542c9dd

HOMELAB-379: feat(lab-director): add Temporal worker Helm chart templates ... f29193236d

Add worker-deployment.yaml and worker-pvc.yaml templates to the
lab-director chart. Worker deployment mounts the same credentials
as dev-environment pods (SSH, kubeconfig, ArgoCD, Claude OAuth, MCP).
Disabled by default — enabled via prod overrides.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

HOMELAB-379: feat(lab-director): add worker ExternalSecrets, SA, and RBAC ... e3e3134f24

Create ExternalSecrets for lab-director namespace (synced by existing
ArgoCD external-secrets-config app) to provision dev-agent credentials.
Add ServiceAccount and RBAC templates to the lab-director Helm chart
binding to the existing dev-agent-scoped ClusterRole.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-379: feat(lab-director): Temporal worker Helm chart + ExternalSecrets' (#57) from plane/HOMELAB-379-temporal-worker into live 5acb72111f

HOMELAB-394: feat: add per-worker boot_datastore_id and data_disk_gb overrides ... 4f014c3301

Allows individual workers to use different storage backends for boot
disks and different data disk sizes. Needed for prod-wk-03 which uses
HDD storage (hdd-mirror) instead of SSD (local-lvm) due to SSD capacity
constraints.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

HOMELAB-378: fix: disable schema createDatabase — CNPG handles DB creation ... 5defa6dcec

Schema init job fails with "permission denied to create database" because
CNPG already creates the temporal and temporal_visibility databases.
Set schema.createDatabase.enabled: false to skip DB creation and only
run table schema setup.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'fix: disable schema createDatabase — CNPG handles DB creation' (#58) from fix/temporal-schema-no-create-db into live b1c9086bdb

HOMELAB-397: fix(rbac): add claude-agent SA to dev-agent-scoped bindings ... 3c82abba68

The claude-agent ServiceAccount in dev-agents namespace was missing
from the ClusterRoleBinding and RoleBinding, causing kubectl access
failures across all namespaces.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-397: fix(rbac): add claude-agent SA to dev-agent-scoped bindings' (#59) from plane/HOMELAB-397-agent-rbac into live 3f3d5a7575

HOMELAB-395: feat(lab-director): add worker ConfigMap templates ... 604601c433

Provision dev-agent ConfigMaps (gitconfig, claude-settings, env, mcp-config)
in the lab-director namespace via the Helm chart. Required for worker pod
to start — volume mounts reference these ConfigMaps.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-395: feat(lab-director): add worker ConfigMap templates' (#60) from plane/HOMELAB-395-configmaps into live 7f2be36d7c

HOMELAB-395: feat(temporal): enable auto-registration of default namespace ... b242b55ca5

The Temporal worker fails with "Namespace default is not found" because
the namespace was never registered after server deployment. Enable the
chart's built-in namespace creation via server.config.namespaces.create
with 3-day retention.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-395: feat(temporal): enable auto-registration of default namespace' (#61) from plane/HOMELAB-395-temporal-ns into live b732b3bce5

feat(rbac): grant claude-agent cluster-admin access ... 6fef6bcc16

The claude-agent SA was bound to dev-agent-scoped (read-only) which
blocked investigation of cross-namespace issues like Plane 503s.
Elevate to cluster-admin for autonomous infrastructure operations.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'feat(rbac): grant claude-agent cluster-admin access' (#62) from fix/claude-agent-cluster-admin into live 164fb8bcf3

HOMELAB-404: add Temporal env vars to dev-environment Helm chart ... f4780f8402

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

HOMELAB-404: switch lab-director worker to devenv-homelab image + MODE=temporal ... fbb9178774

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-404: add Temporal env vars to dev-env + switch worker to devenv image' (#64) from plane/HOMELAB-404-devenv-temporal into live 0c1845af04

HOMELAB-406: fix cluster health — Talos version, Loki config, Prometheus resources ... fe8d40ff29

- Update Talos default version v1.11.3 → v1.12.5 (IaC drift fix)
- Add Loki compactor.delete_request_store: filesystem (required for retention)
- Increase Prometheus limits: CPU 1→2, memory 2Gi→4Gi (OOM/throttle fix)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-406: fix cluster health — Talos version, Loki config, Prometheus resources' (#65) from fix/HOMELAB-406-cluster-health into live d0c372cb22

HOMELAB-404: delete worker deployment — DispatcherWorkflow moves into backend ... 5447e7999f

The separate lab-director-worker deployment is removed. The DispatcherWorkflow
now runs inside the Lab Director backend process (manager/employee model):
- Lab Director pod = manager (HTTP + Temporal Client + Dispatcher)
- Dev-env pods = employees (PipelineWorkflow + Activities)

Removed: worker-deployment.yaml, worker-pvc.yaml, worker-rbac.yaml,
worker-serviceaccount.yaml, worker-configmaps.yaml, worker-mcp-config.yaml,
and worker section from values.yaml.

Added TEMPORAL_HOST_PORT and TEMPORAL_NAMESPACE to backend env config.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-404: delete worker deployment — Dispatcher moves into backend' (#66) from plane/HOMELAB-404-delete-worker-deployment into live 6f53173711

HOMELAB-404: feat(lab-director): add Temporal env vars to backend Helm chart ... 2627d0cb1c

Add TEMPORAL_HOST_PORT and TEMPORAL_NAMESPACE to configmap and
deployment templates (conditional — only rendered when set in values).
Required for the embedded Dispatcher to connect to Temporal.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-404: feat(lab-director): add Temporal env vars to backend chart' (#68) from plane/HOMELAB-404-temporal-env into live 231623ce32

HOMELAB-404: fix(dev-environment): default MODE to worker instead of interactive ... feef9dc364

The dev-env pod is a unified environment + worker. Default behavior
(empty MODE) starts the Temporal worker. Interactive mode must be
explicitly requested via MODE=interactive.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-404: fix(dev-environment): default MODE to worker' (#69) from plane/HOMELAB-404-chart-mode into live c3a3f30639

HOMELAB-416: refactor(dev-environment): use single devenv image ... 6dbec4f9bc

Update default image repository from devenv-homelab to devenv.
All projects now use the same consolidated image.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-416: refactor(dev-environment): use single devenv image' (#70) from plane/HOMELAB-416-single-devenv-image into live 6fe9c142a7

HOMELAB-419: update Talos nameserver default to OPNsense (10.10.10.1) ... 92d2e1db1f

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

HOMELAB-419: update Talos nameserver default to OPNsense (10.10.10.1) ... fef63538c5

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Merge pull request 'HOMELAB-419: update Talos nameserver default to OPNsense (10.10.10.1)' (#71) from plane/HOMELAB-419-coredns-upstream-opnsense into live be6df5de68

HOMELAB-433: fix(temporal): set explicit prometheus storageClass to resolve ArgoCD sync ... f0b1703d3b

The temporal-prometheus-server PVC was created with storageClass "longhorn"
(cluster default), but the Helm chart renders nil when not explicitly set.
This immutable field mismatch blocked ArgoCD sync. Setting it explicitly
makes desired state match actual state.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

HOMELAB-434: add HTTPRoute for Temporal Web UI ... 463828e29a

Exposes temporal-web service at temporal.aaron.reynoza.org via
the existing homelab-gateway (Cilium Gateway API + wildcard TLS).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-433: fix(temporal): set explicit prometheus storageClass' (#73) from plane/HOMELAB-433-temporal-prometheus-pvc into live 8f739490c8

Merge pull request 'HOMELAB-434: add HTTPRoute for Temporal Web UI' (#74) from plane/HOMELAB-434-temporal-httproute into live 1a6e047b4f

feat(sandbox): scaffold Helm chart with Chart.yaml, values, helpers 5d6e577e92

feat(sandbox): complete Helm chart templates ... 90757fa8c6

- PVC: 25 GiB Longhorn with helm.sh/resource-policy: keep
- ServiceAccount: dedicated SA, no cluster role bindings
- LimitRange: container resource caps (6 CPU / 12 Gi max)
- CiliumNetworkPolicy: D16 egress allowlist (DNS, Temporal, Lab Director,
  Forgejo, Harbor, Langfuse, OTel, Grafana, internet:443, DinD:2376)
- Deployment: main sandbox container + DinD sidecar (privileged)
  with workspace PVC, SSH keys, git credentials, Claude OAuth,
  Claude settings, and MCP config mounts
- No kubeconfig or ArgoCD secrets mounted (D8 security boundary)

HOMELAB-424: fix(sandbox): align chart values with Supervisor ArgoCD parameters ... f0daefd0d6

Restructure values.yaml to match the --set parameters the Supervisor sends:
- profileId -> sandbox.profileId, sandbox.name, sandbox.profileName
- workspace.storageSize -> persistence.size
- Add persistence.existingClaim support (skip PVC creation when set)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-424: feat(sandbox): sandbox Helm chart with DinD sidecar + CiliumNetworkPolicy' (#75) from HOMELAB-424-sandbox-helm-chart into live 8065afae8e

HOMELAB-433: fix(temporal): add alertmanager storageClass to match existing PVC ... 6dbae09c41

Same immutable PVC issue as prometheus-server — the alertmanager StatefulSet
VCT renders without storageClass, causing persistent OutOfSync drift.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-433: fix(temporal): add alertmanager storageClass to match PVC' (#76) from plane/HOMELAB-433-alertmanager-pvc into live eaaa75bc26

HOMELAB-437: disable bundled node-exporter in temporal chart ... 4a7a75eb0b

temporal namespace uses PodSecurity baseline:latest which blocks node-exporter
(hostNetwork, hostPID, hostPath volumes, hostPort 9100). The bundled DaemonSet
was stuck at 0/3 pods causing ArgoCD to show Progressing health.

kube-prometheus-stack already provides cluster-wide node-exporter coverage,
so the bundled one is redundant.

Key: prometheus.prometheus-node-exporter.enabled (prometheus sub-chart key,
NOT temporal's unused prometheus.nodeExporter.enabled wrapper)

Merge pull request 'HOMELAB-437: disable bundled node-exporter (PodSecurity baseline blocks it)' (#79) from plane/HOMELAB-437-disable-node-exporter into live 8c0fc0e9fa

HOMELAB-439: fix(sandbox): fix SSH key permissions + add SANDBOX_MODE env var ... 4d270f88fc

Replace subPath SSH secret mount with full directory mount at .ssh-secret/
and add fsGroup: 1000 to pod securityContext so claude-agent (uid 1000)
can read the keys. Add SANDBOX_MODE=true env var so worker starts in
sandbox mode by default.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-439: fix(sandbox): fix SSH key permissions + add SANDBOX_MODE env var' (#80) from plane/HOMELAB-439-sandbox-ssh-permissions into live 7b7a22b851

HOMELAB-447: add SANDBOX_MODE and SANDBOX_NAME env vars to sandbox chart ... 62d816103b

Worker exits on startup without SANDBOX_NAME (needed for LifecycleWorkflow
ID: sandbox-lifecycle-{name}). SANDBOX_MODE was also missing from the base
chart. SANDBOX_NAME is derived from the existing sandbox.name value field.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Merge pull request 'HOMELAB-447: add SANDBOX_MODE and SANDBOX_NAME env vars to sandbox chart' (#81) from homelab-447-sandbox-env-vars into live 5b13630b38

feat(sso): add Temporal UI OIDC app in Zitadel — HOMELAB-448 ... eb2adad74d

Register Temporal as an OIDC application in Zitadel and create a K8s
secret with client credentials for the Temporal UI native auth.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'feat(sso): add Temporal UI OIDC app in Zitadel — HOMELAB-448' (#83) from feat/HOMELAB-448-temporal-oidc-live into live 2e7f8c5416

fix(sso): enable id_token_userinfo_assertion for Temporal OIDC — HOMELAB-448 ... 0db48031e7

Include user profile info (name, email) in the ID token so the
Temporal UI can display the logged-in user.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'fix(sso): enable id_token_userinfo_assertion for Temporal — HOMELAB-448' (#84) from feat/HOMELAB-448-temporal-oidc-live into live ed922080bf

HOMELAB-460: feat(litellm): add ollama-fast (qwen3:8b) and ollama-mid (qwen3:32b) model routing 9c851a9d16

HOMELAB-460: feat(lab-director): add LITELLM_URL and LITELLM_API_KEY env vars to backend 0f4ff5955d

HOMELAB-460: feat(sandbox): add LiteLLM egress rule to CiliumNetworkPolicy for Ollama model routing 99d144ff25

Merge pull request 'HOMELAB-460: feat(litellm,lab-director,sandbox): LiteLLM + Ollama model routing for pipeline effort matrix' (#85) from plane/HOMELAB-460-litellm-ollama into live 0baeec43f5

HOMELAB-462: increase Cilium memory limits, enable BPF LB acceleration and bandwidth manager ... 05f968fe7b

- Agent memory: 512Mi → 768Mi (workers at 72-74% before)
- Agent memory request: 128Mi → 256Mi
- Operator memory: 256Mi → 512Mi (11 restarts in 8h)
- Operator memory request: 64Mi → 128Mi
- Enable loadBalancer.acceleration: native (XDP for LB traffic)
- Enable bandwidthManager for EDT-based pacing

Merge pull request 'HOMELAB-462: Cilium memory + BPF LB acceleration + bandwidth manager' (#86) from plane/HOMELAB-462-infra-remediation into live 7bd932f88e

HOMELAB-463: add Zookeeper soft pod anti-affinity to spread across workers ... 0166494cb6

Spreads 3 Zookeeper replicas across prod-wk-01 and prod-wk-02 to reduce
I/O concentration on prod-wk-02. Root cause of high CPU is ClickHouse data
volume I/O via Longhorn instance-manager (structural, follows data locality).

Merge pull request 'HOMELAB-463: Zookeeper soft anti-affinity to spread across workers' (#87) from plane/HOMELAB-463-wk02-cpu into live ddc5c47875

HOMELAB-464: remove Mimir — never deployed, stop remote_write spam ... 79dfe6818c

Mimir's monolithic StatefulSet was never rendered by the mimir-distributed
chart (deploymentMode: monolithic misconfiguration). Prometheus has been
logging DNS lookup errors every 60s for 14 days with no data stored.

Remove:
- Prometheus remoteWrite to mimir.monitoring.svc.cluster.local:9009
- Grafana Mimir datasource (pointing to same broken endpoint)

Prometheus 15d retention is sufficient for this homelab. Mimir can be
re-added when long-term retention becomes a requirement.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Merge pull request 'HOMELAB-464: remove Mimir — stop Prometheus remote_write spam' (#88) from plane/HOMELAB-464-remove-mimir into live ce94738454

HOMELAB-465: reset Longhorn rebuild limit to 2 (from temp 3 set in HOMELAB-462) ... fc9464134a

concurrentReplicaRebuildPerNodeLimit: 1 → 2

Was set to 3 as a temporary measure during HOMELAB-462 to accelerate
volume rebuild backlog. Setting 2 provides faster parallel rebuilds than
the original 1 while not flooding the nodes the way 3 did.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Merge pull request 'HOMELAB-465: reset Longhorn rebuild limit to 2 per node' (#89) from plane/HOMELAB-465-longhorn-cleanup into live 37575f7d83

HOMELAB-449: feat(lab-director): consolidate to single-binary Helm chart ... 4114c20556

Remove separate frontend Deployment and Service — lab-director now deploys
as a single pod serving both API and embedded SvelteKit frontend. Backend
image tag updated for unified build. Template names cleaned up (backend- prefix
removed). All traffic routed to single service.

Merge pull request 'HOMELAB-449: feat(lab-director): consolidate to single-binary Helm chart' (#90) from plane/HOMELAB-449-ui-phase1-helm into live 207f09b935

fix: add cilium-secrets to reflector allowed namespaces (HOMELAB-468) ... 7260efc4bd

Cilium Gateway API syncs TLS secrets to cilium-secrets namespace with
inherited reflector annotations. Reflector detected these as unauthorized
reflections and entered a tight delete loop (~10/sec), causing liveness
probe failures and continuous restarts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'fix: add cilium-secrets to reflector allowed namespaces (HOMELAB-468)' (#91) from plane/HOMELAB-468-reflector-fix into live 8687fc5d1d

HOMELAB-469: fix(lab-director): bump image tag to 6a72f10 (embedded frontend) ... 11a3066c2c

The base values had image tag 6192621 which predates the frontend
embedding work (PR #76). Update to 6a72f10 which includes all 7 UI
phases built into the single Go binary.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-469: fix(lab-director): bump image tag to 6a72f10 (embedded frontend)' (#92) from plane/HOMELAB-469-helm-fix into live 014070a5d0

HOMELAB-469: chore(lab-director): bump image to c11df5f (Phase 7 + WS fix) ... 0253831158

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-469: chore(lab-director): bump image to c11df5f' (#93) from plane/HOMELAB-469-bump-c11df5f into live 3962a95729

HOMELAB-469: fix(lab-director): bump to 2fd90c5 — WS routing + missing handlers ... 8adc63675b

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-469: fix(lab-director): bump to 2fd90c5 — WS + handlers fix' (#94) from plane/HOMELAB-469-deploy-ws-fix into live b756a79239

HOMELAB-470: bump lab-director image to ab2dec6 — Temporal-native chat sessions ... 5def617729

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-470: bump lab-director image to ab2dec6' (#95) from bump/lab-director-ab2dec6 into live 5158477e48

HOMELAB-473: perf(forgejo-runner): persistent Docker cache and resource bump ... 7250f36ef7

- Replace emptyDir with 30Gi Longhorn PVC for /var/lib/docker — Docker layer
  cache now survives pod restarts
- Increase DinD resources: 1 CPU / 1Gi request, 4 CPU / 4Gi limit — allows
  Go compiler and npm to use more parallelism during builds

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-473: perf(forgejo-runner): persistent Docker cache + resource bump' (#96) from plane/HOMELAB-473-runner-cache into live 510a0b7ce4

HOMELAB-474: add Plane, Outline, Forgejo config to lab-director chart ... 4c4b6121c5

Adds configmap entries for API URLs and workspace/project identifiers,
plus secretKeyRef entries for API tokens from dev-agent-credentials
(managed by ESO). Enables Lab Director's chat tool executor to query
live service APIs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-474: add Plane, Outline, Forgejo config to lab-director chart' (#97) from homelab-474/lab-director-service-config into live 62d1be1770

HOMELAB-471: feat(chart): manager execution capability — PVC, env vars, secret mounts ... 35fec32d8d

- execution.enabled feature flag (default false)
- 5Gi workspace PVC for pipeline run isolation
- MANAGER_EXECUTION_ENABLED + WORKSPACE_DIR env vars
- Claude OAuth credentials mount (~/.claude/.credentials.json)
- SSH key mount for Forgejo git operations
- Resource bump: 250m/2CPU, 512Mi/4Gi

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-471: feat(chart): manager execution capability — PVC, env vars, secret mounts' (#99) from plane/HOMELAB-471-chart into live 80a641bbe6

HOMELAB-471: fix(chart): use Recreate strategy when execution PVC is mounted ... 9550564227

RWO PVC can't be shared between old and new pods during rolling update.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-471: fix(chart): Recreate strategy for RWO PVC' (#100) from plane/HOMELAB-471-chart into live 3b22148f5d

HOMELAB-476: fix(chart): clear rollingUpdate when strategy is Recreate 2a1947a1ee

Merge pull request 'HOMELAB-476: fix(chart): clear rollingUpdate when Recreate' (#101) from fix/homelab-476-recreate-strategy into live fd1e0dd009

HOMELAB-476: fix(chart): add fsGroup for workspace PVC write access ... 48cdeef600

Workspace PVC mounts as root-owned, but container runs as appuser (uid
1000). Without fsGroup, QueryChat activity cannot create session dirs
under /workspace/chat-sessions/. Only applied when execution is enabled.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-476: fix(chart): add fsGroup for workspace PVC write access' (#102) from fix/homelab-476-recreate-strategy into live ad768003f9

HOMELAB-477: feat(helm): add token-refresh CronJob templates to lab-director chart ... 1469f72c78

Add Helm templates for the Claude OAuth token-refresh CronJob:
- CronJob template with hardened security context (non-root, read-only fs, drop all caps)
- Cross-namespace RBAC (ServiceAccount, Role in secrets ns, RoleBinding)
- tokenRefresh values section (disabled by default, enabled via prod overrides)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-477: feat(helm): add token-refresh CronJob to lab-director chart' (#103) from plane/HOMELAB-477-token-refresh into live ad8fcddf63

HOMELAB-477: fix(helm): add seccompProfile to token-refresh CronJob ... 3facf84d5d

Fixes PodSecurity "restricted:latest" violation warning.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-477: fix(helm): add seccompProfile to token-refresh CronJob' (#104) from plane/HOMELAB-477-seccomp into live ee812009e8

HOMELAB-478: fix(cert): remove cilium-secrets from Reflector namespaces ... c91b897af4

Reflector and Cilium's gateway secret-syncer both writing to
cilium-secrets creates an infinite reconciliation loop (89 operator
restarts). Remove cilium-secrets from Reflector — Cilium's syncer
handles that namespace natively. Reflector still reflects to
cilium-system where the Gateway references the cert.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-478: fix(cert): remove cilium-secrets from Reflector namespaces' (#105) from plane/HOMELAB-478-fix-reflector-loop into live 364b7ef646

HOMELAB-476: add Langfuse env vars to lab-director Helm chart ... 60c45af716

- Add LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY from serviceCredentials secret
- Add LANGFUSE_HOST configmap entry for MCP server URL

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-476: add Langfuse env vars to lab-director Helm chart' (#106) from plane/HOMELAB-476-langfuse-mcp into live f816eea76d

HOMELAB-420: feat(modules): add per-worker overrides and GPU passthrough ... 083ed3ceee

Cherry-pick of b3bf989 (HOMELAB-280) from main to live, resolving
conflicts with boot_datastore_id support added on live.

- talos-cluster: workers now support per-node cpu_cores, memory_mb,
  boot_disk_gb, data_disk_gb, and gpu_pci_id overrides (falling back
  to global defaults via coalesce)
- proxmox-vm: add GPU PCI passthrough support with dynamic hostpci
  block and automatic q35 machine type when GPU is present

Resolves Terraform drift on prod-wk-01: disk size, machine type, hostpci.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-420: feat(modules): add per-worker overrides and GPU passthrough' (#107) from plane/HOMELAB-420-reconcile-tf-drift into live c3f5391a73

HOMELAB-510: fix(gateway): route Harbor traffic to port 443 (SSL listener) ... e1bf6381a1

INCIDENT: All build pipelines blocked — docker login fails with redirect loop.

Cilium Gateway terminates TLS and forwards plain HTTP to Harbor port 80,
which maps to nginx port 8080. Harbor's nginx returns 301 to HTTPS on 8080,
creating an infinite redirect loop. Route to port 443 (→ nginx 8443) instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-510: fix(gateway): Harbor redirect loop — route to port 443' (#108) from plane/HOMELAB-510-harbor-route into live d7655e8506

HOMELAB-510: fix(gateway): remove Harbor HTTPRoute — use direct LB instead ... 52d5bbc195

Harbor has its own LoadBalancer service (10.10.10.223) with TLS
configured. The HTTPRoute through Cilium Gateway creates a redirect
loop because the Gateway terminates TLS and forwards plain HTTP to
Harbor's port 80, which redirects back to HTTPS.

Harbor needs TLS on its own service for kubelet image pulls, so we
can't disable Harbor's TLS. The simplest fix is to remove the
HTTPRoute and let all traffic go through Harbor's LoadBalancer IP
directly. ControlD DNS should resolve harbor.aaron.reynoza.org to
10.10.10.223 (the LB IP) instead of 10.10.10.228 (the Gateway IP).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-510: fix(gateway): remove Harbor HTTPRoute — use direct LB' (#109) from plane/HOMELAB-510-remove-harbor-route into live 35e2c3123c

HOMELAB-510: fix(gateway): Harbor backend TLS via appProtocol + dedicated Service ... dfc08cb05e

Harbor requires TLS on its backend (nginx 8443). The Gateway terminates
external TLS then re-encrypts to Harbor using Cilium's appProtocol support
(enable-gateway-api-app-protocol is already enabled).

Creates harbor-gateway Service with appProtocol: https targeting Harbor's
nginx SSL port (8443). HTTPRoute points to this service on port 443.

Traffic flow: Client → Gateway (TLS terminated) → Envoy (re-encrypts) →
harbor-gateway:443 → nginx:8443 → Harbor core

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-510: fix(gateway): Harbor backend TLS re-encryption via appProtocol' (#110) from plane/HOMELAB-510-harbor-gateway-tls into live a394bf59a5

HOMELAB-510: fix(gateway): simplify Harbor route — TLS disabled in prod ... 887cb58e71

Now that Harbor's internal TLS is disabled (prod PR #128), nginx serves
plain HTTP on port 8080. Simplify route back to harbor:80 and remove
the harbor-gateway Service workaround.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-510: fix(gateway): simplify Harbor route after TLS disable' (#112) from plane/HOMELAB-510-simplify-harbor-route into live b8cbd53426

HOMELAB-516: fix(sandbox): hardcode SANDBOX_MODE=true in deployment template ... 7bb9f46e6f

SANDBOX_MODE should always be "true" for sandbox deployments — remove from
values.yaml env section and hardcode in deployment.yaml to prevent accidental
override. SANDBOX_NAME already correctly sourced from sandbox.name.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-516: Hardcode SANDBOX_MODE=true in sandbox deployment' (#113) from plane/HOMELAB-516-sandbox-env-vars into live 6e1e384a44

feat(lab-director): add ArgoCD env vars for sandbox provisioning ... 7eedce449b

Adds argocd.* values block to wire ARGOCD_SERVER_URL, ARGOCD_AUTH_TOKEN,
ARGOCD_CHART_REPO_URL, ARGOCD_CHART_PATH, ARGOCD_CHART_REVISION, and
ARGOCD_NAMESPACE into the lab-director deployment. Required for
Supervisor.SpawnSandbox() to create ArgoCD Applications.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-516: feat(lab-director): add ArgoCD env vars for sandbox provisioning' (#115) from plane/HOMELAB-516-argocd-env-vars into live 53cad424f1

feat(argocd): grant lab-director RBAC for dev-agents project ... eebf8f07c6

lab-director account needs create/delete/get/sync on dev-agents/*
to manage sandbox ArgoCD Applications via the provisioning API.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-516: grant lab-director ArgoCD RBAC for dev-agents' (#116) from plane/HOMELAB-516-argocd-rbac into live 883c05be0b

fix(sandbox): correct LAB_DIRECTOR_API_URL service name ... eb81d615c1

The lab-director service is named "lab-director", not
"lab-director-backend". Fixes DNS resolution failure in sandbox pods.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-516: fix sandbox LAB_DIRECTOR_API_URL service name' (#117) from plane/HOMELAB-516-sandbox-api-url into live 566d80bbe0

fix(lab-director): prevent token refresh CronJob from hammering OAuth endpoint ... a75b7b9285

Set backoffLimit: 0 and restartPolicy: Never so failed refresh attempts
don't retry with exponential backoff. Each CronJob run gets exactly one
attempt — if it fails, wait for the next scheduled run in 6 hours.

Previous backoffLimit: 2 caused up to 3 retries per failed job, which
combined with the 6-hour CronJob schedule would hit rate limits when the
OAuth endpoint was temporarily unavailable.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix(sandbox): correct lab-director pod label in egress network policy ... 6e08c5afce

The CiliumNetworkPolicy matched app.kubernetes.io/name: lab-director-backend
but the actual pod label is app.kubernetes.io/name: lab-director, blocking
sandbox→manager API traffic and causing CrashLoopBackOff (profile fetch timeout).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix(sandbox): share Docker socket via emptyDir instead of TLS ... c8ab9eb7c3

k3s --docker uses cri-dockerd which hardcodes /var/run/docker.sock.
The DinD sidecar was exposing Docker over TLS (tcp://localhost:2376)
which cri-dockerd can't use.

Switch to Unix socket by setting DOCKER_TLS_CERTDIR="" on DinD sidecar
and sharing /var/run between containers via emptyDir volume.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix(sandbox): share /var/lib/docker with sandbox container for kubelet ... c85568a1f4

kubelet's filesystem info check needs Docker's data directory visible.
Mount dind-storage (emptyDir) as read-only at /var/lib/docker in the
sandbox container.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix(sandbox): mount /dev/kmsg from host for k3s kubelet ... 546f52ecf2

k3s kubelet requires /dev/kmsg which doesn't exist in containers.
Mount the host's /dev/kmsg as a CharDevice hostPath volume.

Ticket: HOMELAB-519

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix(sandbox): run sandbox container as privileged for k3s kubelet ... 2ca1f8e9e1

k3s kubelet requires /dev/kmsg access which is blocked in unprivileged
containers. The sandbox is already network-isolated via CiliumNetworkPolicy.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix(sandbox): make /var/lib/docker writable for kubelet pod creation ... f2bd43472b

kubelet writes resolv.conf inside /var/lib/docker/containers/ when
creating pod sandboxes. readOnly: true was causing all pod creation
to fail with "read-only file system".

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

HOMELAB-533: feat(talos): add nvidia kernel modules for GPU workers ... 7451ff0230

Workers with gpu_pci_id set now get machine.kernel.modules config
for nvidia, nvidia_uvm, nvidia_drm, nvidia_modeset. Without this,
Talos doesn't load the nvidia kernel modules at boot despite the
extension providing the .ko files on disk.

Root cause of Ollama being unable to schedule — node reported
nvidia.com/gpu: 0 because the device plugin couldn't initialize
NVML without loaded drivers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-533: feat(talos): add nvidia kernel modules for GPU workers' (#128) from plane/HOMELAB-533-nvidia-modules into live 5f5c7ddaf0

HOMELAB-540: feat(otel): route lab-director spans to Langfuse ... 618f77cc8e

Update OTel Collector filter/langfuse to accept spans from both
agent-harness and lab-director service names.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-540: feat(otel): route lab-director spans to Langfuse' (#129) from plane/HOMELAB-540-otel-filter into live 74f5027a7f

fix: resolve critical alerts — Talos compat, ArgoCD OOM, Longhorn replicas ... 6b072273bf

- Disable kubeControllerManager, kubeScheduler, kubeProxy ServiceMonitors
  in kube-prometheus-stack — Talos binds these to localhost (unscrappable)
  and uses Cilium instead of kube-proxy, eliminating false critical alerts
- Bump ArgoCD application-controller memory limit 1Gi → 2Gi to prevent
  OOMKills under load (125 restarts observed)
- Set Longhorn defaultReplicaCount to 2 (matches 2-node storage topology)
  and bump concurrent rebuild limit 2 → 5

Part of HOMELAB-542 infrastructure alerts triage.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'fix: resolve critical alerts — Talos compat, ArgoCD OOM, Longhorn replicas (HOMELAB-542)' (#132) from fix/critical-alerts-live-homelab-542 into live e4c1986782

fix: change token-refresh CronJob default to every 2 hours ... 83e6859d97

Claude OAuth tokens have ~4h TTL. A 6h schedule guarantees the token
expires between runs. Changed to 2h with a 2h refresh buffer in the
token-refresh binary.

HOMELAB-538

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'fix: token-refresh CronJob schedule 6h→2h' (#133) from fix/token-refresh-schedule into live 72ae6a7800

fix: add startup probe to kube-state-metrics to stop restart loop (HOMELAB-544) ... f4d06af6b1

KSM lists/watches 30+ resource types on startup. Default liveness probe
(initialDelay=5s) kills the container at ~35s before caches are built,
causing 499 restarts in 6 days. Startup probe gives up to 185s for
initialization before liveness takes over.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'fix: add startup probe to kube-state-metrics to stop restart loop (HOMELAB-544)' (#134) from fix/HOMELAB-544-ksm-restart-loop into live 726be83687

fix: enable startupProbe for kube-state-metrics (HOMELAB-544) ... 09e5cc7441

The kube-state-metrics subchart requires startupProbe.enabled: true —
without it the probe definition is ignored.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: bump plane worker memory limit 1000Mi → 1500Mi ... 9b0de56de3

Celery worker baseline footprint is ~970Mi, leaving only 3% headroom
at the current 1000Mi limit. Increasing to 1500Mi gives ~35% headroom.

Ticket: HOMELAB-545

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'fix: enable startupProbe.enabled for kube-state-metrics (HOMELAB-544)' (#135) from fix/HOMELAB-544-ksm-restart-loop into live 439fa7b41c

fix: bump memory limits for reflector and langfuse zookeeper ... 644a3a005d

Reflector: 117Mi/128Mi (91%) with 76 OOMKill restarts → bump to 256Mi
Langfuse Zookeeper: 357Mi/384Mi (93%) on zk-0 → explicit 512Mi limit
(was using Bitnami subchart defaults of 384Mi)

Ticket: HOMELAB-545

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'fix: bump memory limits for reflector and langfuse zookeeper (HOMELAB-545)' (#136) from fix/HOMELAB-545-memory-limits into live 046016513e

fix: add shareProcessNamespace to sandbox pod for CNI loopback access ... c39911e5c7

The sandbox and dind containers had separate PID namespaces, so the K3s
kubelet couldn't access /proc/<pid>/ns/net for containers created by
dockerd. This caused every inner kube-system pod to fail CNI loopback
setup in a tight retry loop (~12k error lines in 8h, burning 1+ CPU).

HOMELAB-546

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'fix: shareProcessNamespace for sandbox CNI loopback (HOMELAB-546)' (#141) from fix/HOMELAB-546-sandbox-share-pid-namespace into live 2a212388cb

feat(lab-director): replace CronJob token refresh with in-process manager ... b436b6d355

- Add token-manager-rbac.yaml (SA + cross-namespace Role/RoleBinding)
- Add serviceAccountName + TOKEN_* env vars to deployment
- Replace tokenRefresh values with tokenManager values
- Delete token-refresh-cronjob.yaml and token-refresh-rbac.yaml

HOMELAB-558

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'feat(lab-director): replace CronJob token refresh with in-process manager' (#144) from plane/HOMELAB-558-token-manager into live 8767aa2e88

HOMELAB-584: feat(tempo): increase memory limits to fix OOMKilled events ... 9045a2187d

- Memory request: 256Mi → 512Mi
- Memory limit: 1Gi → 2Gi
- Fixes OOMKilled alerts for tempo-0 pod in monitoring namespace
- Tempo running in monolithic mode with 72h retention requires more memory for in-memory trace buffering

Resolves: PodOOMKilled alerts affecting observability stack

Merge pull request 'HOMELAB-584: feat(tempo): increase memory limits to fix OOMKilled events' (#148) from plane/HOMELAB-584-fix-tempo-oom into live 0dffa017c5

fix(backups): reduce B2 API usage to stay within free tier (HOMELAB-590) ... b22c7d07b7

- Velero: backupSyncPeriod + storeValidationFrequency 30m → 24h
  (saves ~1,200 wasted Class C calls/day from broken sync loop)
- CNPG: retentionPolicy 30d → 7d (reduces stored objects + LIST calls)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'fix(backups): reduce B2 API usage — HOMELAB-590' (#149) from fix/HOMELAB-590-b2-usage-cap into live dc8f65739f

HOMELAB-698: fix(monitoring): guard CNPGLastBackupOld alert for clusters without active archiving 181f01c4dc

Merge PR #150: HOMELAB-698: guard CNPGLastBackupOld alert for clusters without active archiving ... 5f1d0e42ef

Merges PR #150 which adds guard clause to CNPGLastBackupOld alert to prevent false positives on clusters without backup archiving configured.

fix: HOMELAB-705 - Exclude outline cluster from CNPGLastBackupOld alert ... cd3fa41b53

- Add condition to exclude job =~ "outline/.*" from CNPGLastBackupOld alert
- Prevents false alerts for outline cluster where B2 backups are intentionally disabled
- Outline cluster relies on Longhorn snapshots instead of WAL archival (HOMELAB-590)

fix: Correct PromQL syntax for CNPGLastBackupOld alert exclusion ... 2c0e0592f4

- Use label selector {namespace!="outline"} instead of complex expressions
- Excludes metrics from outline namespace to prevent false alerts
- Resolves HOMELAB-705 WAL archive delay alert for intentionally disabled backups

fix(sandbox): resolve RWO PVC contention and termination issues ... 0ad014786d

- Increase terminationGracePeriodSeconds from 60s to 300s for proper DinD cleanup
- Add graceful preStop hooks for both sandbox and dind containers
- Add node affinity configuration support for RWO volume placement
- Prevent FailedKillPod errors when stopping nested Docker containers

Resolves: HOMELAB-713

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

fix(lab-director): remove subPath mount for claude OAuth credentials ... 71ff164de0

K8s subPath mounts never auto-update when the underlying secret changes.
The token manager now writes credentials directly to the local filesystem,
making the volume mount unnecessary.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'fix(lab-director): remove subPath mount for claude OAuth credentials' (#151) from fix/lab-director-remove-subpath-mount into live 1f65fbf6c0

feat(lab-director): wire OUTLINE_COLLECTION_ID into Helm chart — HOMELAB-793 ... 42a58944ce

OUTLINE_COLLECTION_ID was missing from the configmap and deployment
templates, so CreateDocument always returned early with
"OUTLINE_COLLECTION_ID not set". Planner stage docs never reached Outline.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Merge pull request 'feat(lab-director): wire OUTLINE_COLLECTION_ID into Helm chart — HOMELAB-793' (#156) from plane/HOMELAB-793 into live 816140852f

feat(terraform): add boot disk I/O tuning and etcd auto-compaction (HOMELAB-811) ... 1569db193b

Post-power-outage incident revealed Kingston A400 (DRAM-less SSD) with
cache=none causes ~194ms etcd fsync latency, leading to controller
crash loops. Adds configurable boot disk cache/iothread/discard settings
to proxmox-vm module, and etcd auto-compaction config patch to
talos-cluster module.

Changes:
- proxmox-vm: boot_disk_cache, boot_disk_iothread, boot_disk_discard vars
- talos-cluster: CP disk tuning pass-through + etcd_auto_compaction vars
- Defaults unchanged (cache=none) to avoid breaking existing deployments

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'feat(terraform): add boot disk I/O tuning and etcd auto-compaction (HOMELAB-811)' (#159) from fix/HOMELAB-811-etcd-disk-tuning into live 2f3686848f

remove redundant PodOOMKilled alert rule (HOMELAB-819) ... 7cad3c550f

OOMKillRateHigh in capacity-alerts.yaml already covers OOM detection
with a smarter rate-based expression. The per-pod PodOOMKilled rule
fires indefinitely on any historical OOM, generating constant noise.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'remove redundant PodOOMKilled alert rule (HOMELAB-819)' (#160) from HOMELAB-819-alert-cleanup into live 52fb43ee4d

feat(lab-director): add claudeSetupToken support — 1-year token via env var ... 91e7fe1765

Adds CLAUDE_CODE_OAUTH_TOKEN env var from K8s secret when
claudeSetupToken.enabled is true. Replaces the OAuth token refresh
approach (14 fix commits, still broken) with a static 1-year
setup token from `claude setup-token`.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'feat(lab-director): add claudeSetupToken Helm support' (#161) from plane/HOMELAB-809-setup-token into live a5f78cc242

HOMELAB-817: fix(etcd): replace narrow compaction vars with generic etcd_extra_args map ... 3dc656b128

The previous etcd_auto_compaction_mode/retention variables only supported 2 args
and gated on mode != null, which meant a null default silently disabled the entire
patch. Replace with a generic map(string) that supports any etcd extraArgs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-817: replace narrow etcd compaction vars with generic etcd_extra_args map' (#162) from plane/HOMELAB-817-etcd-stability into live cf4f0289e0

HOMELAB-817: fix(talos-cluster): add static IP to machine config patches ... be85d9611a

Talos was relying on Proxmox cloud-init for IP assignment, which broke
after a config apply/reboot — the CP node got DHCP address 10.10.10.66
instead of the expected 10.10.10.10, taking the entire cluster offline.

Add explicit machine.network.interfaces config patch for both CP and
worker nodes using the IP from the control_planes/workers variables.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-817: add static IP to Talos machine config patches' (#163) from plane/HOMELAB-817-etcd-stability into live 17aa66f7a0

HOMELAB-817: fix(talos-cluster): add hostname to machine config patches ... c3eac7349d

After config apply/reboot, Talos generated a random hostname (talos-8a7-y35)
instead of the expected prod-cp-01. This caused stale kubelet registrations
and etcd member hostname mismatches. Set hostname explicitly from node name.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-817: add hostname to Talos machine config patches' (#164) from plane/HOMELAB-817-hostname-fix into live fc310b3980

HOMELAB-817: revert hostname from machine config patches ... c05351975b

Talos HostnameConfig document conflicts with machine.network.hostname —
terraform apply fails. Hostname fix needs separate investigation (follow-up
ticket). Static IP remains in place.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-817: revert hostname from machine config (conflicts with HostnameConfig)' (#165) from plane/HOMELAB-817-revert-hostname into live fa152b43d0

fix: add hostname to Talos machine config patches ... 80b8dd4913

After a Talos config apply/reboot, nodes lose the Proxmox-inherited
hostname and fall back to auto-generated names (e.g. talos-8a7-y35).
This caused ghost node registrations, broken Cilium L2 announcements,
and a full cluster outage.

Sets machine.network.hostname explicitly for both control planes and
workers using the node name from variables.

Refs: HOMELAB-817

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'fix: add hostname to Talos machine config patches' (#167) from plane/HOMELAB-817-hostname-iac into live dd47b6a98e

fix(eso): add ExternalSecrets for claude-setup-token and litellm-master-key ... ab02b25ead

Both secrets were manually created and got pruned by ArgoCD.
Now managed via ESO from the secrets namespace like all other credentials.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'fix(eso): add ExternalSecrets for claude-setup-token and litellm-master-key' (#168) from fix/eso-lab-director-secrets into live 690f5a2119

fix: increase ClickHouse resource limits to resolve Langfuse outage ... 5e335afbc1

ClickHouse was pegged at CPU/memory ceiling (992m/1000m CPU, 1434Mi/1536Mi RAM),
causing readiness probe timeouts and preventing langfuse-web from completing
migrations. Doubled limits to give ClickHouse breathing room.

HOMELAB-845

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'fix: increase ClickHouse resource limits to resolve Langfuse outage (HOMELAB-845)' (#169) from fix/HOMELAB-845-clickhouse-resources into live 972b35eebb

HOMELAB-846: fix(langfuse): increase ClickHouse memory limits to prevent OOM ... d6376a1f1d

ClickHouse was OOMing during MergeTree merge operations on system log
tables (trace_log grew to 6.3GB/302M rows). This caused cascading
failures: CH crashes → langfuse-web migration fails → crash loop
(82 restarts in 8 days).

Bump CH memory: requests 1Gi→2Gi, limits 3Gi→5Gi. Also truncated
accumulated system log tables (~8GB freed) as immediate relief.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-846: fix(langfuse): increase ClickHouse memory to prevent OOM' (#170) from plane/HOMELAB-846-ch-memory into live 7ac70a670a

fix: prevent PG crash cascade from MinIO OOMKill disk saturation ... 5dbcea18dd

MinIO (langfuse-s3) was OOMKilled repeatedly due to chart-default 384Mi
memory limit. Each restart triggered 112 MB/s overlay reads on vda,
saturating the system disk on prod-wk-02. This caused Longhorn write
latencies to spike to 12+ seconds, flipping ext4 WAL filesystems
read-only and crashing all three CNPG clusters (forgejo, temporal,
zitadel) for 9 hours.

- Bump MinIO memory limit from 384Mi to 1Gi
- Bump CNPG WAL storage from 2Gi to 4Gi for additional headroom

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'fix: prevent PG crash cascade from MinIO OOMKill disk saturation' (#171) from fix/minio-oom-pg-cascade into live ... a2f7bfa5be

Reviewed-on: #171

HOMELAB-847: feat(nats): add NATS Helm chart base values ... f52538147a

JetStream enabled, 256Mi in-memory storage, single-replica homelab config.
ClusterIP service — Mac connects via port-forward or WireGuard tunnel.
Environment-specific overrides (external access, TLS) go in prod repo.

Refs: HOMELAB-847

HOMELAB-847: feat(chart): add NATS_URL and Langfuse key env vars to lab-director ... 70660ed891

Adds NATS_URL placeholder (enables server-side NATS handlers in MODE=full
when set) and LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY for the Langfuse
HTTP ingester wired in server_handlers.go.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Merge branch 'live' into plane/HOMELAB-847-nats-k8s-deployment 5027294ed2

Merge pull request 'HOMELAB-847: Add NATS Helm chart + lab-director NATS/Langfuse env vars' (#172) from plane/HOMELAB-847-nats-k8s-deployment into live ... 2816e8bad9

Reviewed-on: #172

HOMELAB-854: feat(nats): add base Helm values for NATS JetStream ... 1bb1aaed14

Deploy NATS as Lab Director transport layer. JetStream enabled with
file storage, Prometheus metrics, and NATS Box for debugging.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-854: feat(nats): add base Helm values for NATS JetStream' (#173) from plane/HOMELAB-854-nats-deploy into live 5a5b2b6be3

fix: remove hostname from talos config patches — already in v1alpha1 ... 1feb8dd5fa

Talos rejects config patches that set machine.network.hostname when
the v1alpha1 base config already has it. Remove from patches since
existing nodes already have static hostnames configured.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'fix: remove hostname from talos config patches — already in v1alpha1' (#174) from fix/HOMELAB-860-talos-hostname-conflict into live a8ee9bf7bf

HOMELAB-857: feat(lab-director chart): wire NATS_URL env var ... a0e6f6e675

Adds NATS_URL to configmap + deployment templates.
Set via prod overrides to connect to in-namespace NATS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-857: feat(lab-director chart): wire NATS_URL env var' (#175) from plane/HOMELAB-857-nats-chart into live 18e1a52ba4

feat(infra): expand LB pool to 151 IPs + expose Temporal/PG via LoadBalancer ... 4588eacf4f

- Expand CiliumLoadBalancerIPPool from 10.10.10.220-250 (31 IPs) to
  10.10.10.100-250 (151 IPs)
- Add temporal-frontend-lb (10.10.10.109:7233) for VPN access
- Add cnpg-lab-director-lb (10.10.10.110:5432) for VPN access
- Eliminates need for kubectl port-forward from Mac

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix(infra): pin LB IPs + expand pool with original range first ... 011117069e

Pool expansion must keep the original range (.220-.250) as the first
block to preserve existing IP assignments. New services pinned to the
expansion range (.100-.219) to avoid conflicts.

- Temporal LB: 10.10.10.100
- PG LB: 10.10.10.101

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

HOMELAB-954: feat(workadventure): add base values, HTTPRoute, ExternalSecret ... 4de42e7f73

Adds the infra-core side of the WorkAdventure deploy:
- core/charts/apps/workadventure/values.yaml: base Helm values for the
  upstream workadventure/workadventure v1.30.5 chart (singleDomain,
  bundled Redis + Longhorn PVC, mapstorage Longhorn PVC, root ingress
  disabled in favour of a sibling HTTPRoute, externalSecretName wired
  to ESO-managed Secret).
- core/manifests/apps/workadventure/httproute.yaml: HTTPRoute on
  homelab-gateway (cilium-system), single host play.aaron.reynoza.org,
  path rules mirroring the chart's nginx Ingress (/api, /uploader,
  /icon, /map-storage, /ws, /).
- core/manifests/apps/workadventure/eso.yaml: ExternalSecret pulling
  workadventure_map_storage_api_token + workadventure_map_storage_password
  from the shared app-credentials Secret in the secrets namespace.

Companion ArgoCD Applications + env overrides ship in the prod repo.

Plan: docs/superpowers/plans/2026-04-19-workadventure-hubs-eval.md

Merge pull request 'HOMELAB-954: feat(workadventure): add base values, HTTPRoute, ExternalSecret' (#176) from plane/HOMELAB-954-workadventure into live 80f71340cb

HOMELAB-956: feat(talos): raise worker kubelet maxPods to 150 ... a2714ffa6a

wk-01 is capped at the kubelet default (110 pods/node) and, with wk-02
cordoned pending HDD replacement, new deploys (observed on HOMELAB-954
WorkAdventure) hit FailedScheduling / "Too many pods". Raise the cap via
the worker config patch so there's headroom for the current stack
(harbor, temporal, longhorn, monitoring, etc.) plus new workloads.

Adds `kubelet.extraConfig.maxPods = 150` to the worker_config_patch
local. Applies to every worker via the existing
talos_machine_configuration_apply.worker loop. Talos reloads kubelet
in place — no full node reboot.

Apply path (Atlantis is offline, manual for now):
  cd prod/terraform && terraform plan -out=956.tfplan
  terraform apply 956.tfplan

Verify after apply:
  kubectl get node prod-wk-01 -o jsonpath='{.status.capacity.pods}{"\n"}'
  # expect: 150

Unblocks HOMELAB-954 (WorkAdventure pods currently Pending).

Merge pull request 'HOMELAB-956: feat(talos): raise worker kubelet maxPods to 150' (#177) from plane/HOMELAB-956-kubelet-maxpods into live ... 5799813436

Reviewed-on: #177

HOMELAB-968: feat(intellimation): static posters site behind menu ... e9dfa897a6

Host 3 Claude Design posters (Pipeline, Homelab v2, What Maps to What)
at intellimation.aaron.reynoza.org. nginx serves content from a kustomize-
generated ConfigMap; wrapper index.html provides a top-bar menu that swaps
an iframe between the three posters. HTTPRoute attaches to homelab-gateway.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

HOMELAB-968: feat(intellimation): static posters site behind menu (#178) ef1103b3db

HOMELAB-969: fix(pangolin): set sso=false on resource create ... e5da1c0b73

Pangolin's API defaults new public resources to sso=true (auth wall).
All existing homelab resources are sso=false; new ones require a manual
follow-up POST to flip the flag. Passing sso=false in the create payload
makes resource creation match established intent without the extra step.

Discovered while shipping HOMELAB-968 (intellimation).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

HOMELAB-969: fix(pangolin): set sso=false on resource create (#179) 8b951540bd

HOMELAB-954: fix(workadventure): use correct chart schema for mapstorage PVC ... 5c5390b337

The upstream workadventure chart v1.30.5 expects `mapstorage.persistence.*`
(with `storageSize`), not `mapstorage.persistentVolume.*` (with `size`) — the
wrong keys were silently ignored, so the Deployment fell through to the
chart's default emptyDir volume. This meant map uploads didn't survive a pod
restart, violating the plan's acceptance criterion.

Rename to the chart's real schema so Longhorn provisions a 10Gi RWO PVC
(workadventure-mapstorage) and the pod mounts it at /maps. Upstream template
annotates the PVC with helm.sh/resource-policy: keep so a future uninstall
won't drop the data.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-954: fix(workadventure): use correct chart schema for mapstorage PVC' (#180) from plane/HOMELAB-954-mapstorage-pvc into live 40449d5e9c

HOMELAB-971: fix(intellimation): blend poster letterbox with grid + bg ... 9b30ab400b

The 1920x1080 posters scale contain-style; on a 16:10 viewport this
letterboxes top/bottom. Stage was #05080c while the poster is #0a0e14
with a 48px grid + 0.8 overlay, so the letterbox read as jet-black
bars framing the poster. Extend the poster's exact background (color
+ grid + overlay) onto the stage and iframe surfaces so the letterbox
disappears visually. Applied to all 3 posters and the index shell.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

HOMELAB-971: fix(intellimation): blend poster letterbox with grid + bg (#182) ... 9ef6a0b1d0

Skipping CI per user direction; CSS-only configmap change.

HOMELAB-973: fix(outline): add startupProbe to prevent liveness race during migrations ... 9407a0aad5

Outline's master process runs sequelize migrations + data-migration
checks + sponsor message before throng forks a worker that reloads
plugins and binds port 3000. On a cold boot this easily takes 45–90s.

The previous probe config (liveness initialDelay=30s, readiness
initialDelay=10s, both 3×10s failureThreshold) triggered SIGTERM
mid-worker-fork, causing the container to exit 0 (graceful shutdown)
and CrashLoopBackOff repeatedly without the web service ever listening.

Add a startupProbe with a 5-minute window (30 × 10s failureThreshold)
so Kubernetes suppresses liveness/readiness until the server actually
listens. Once startupProbe succeeds, liveness/readiness take over with
short periods to catch real post-startup failures fast.

Root cause diagnosed from pod logs: OIDC plugin registered → migrations
run → sponsor message → exit 0 (never reached "Listening on port 3000").

HOMELAB-973: fix(outline): add startupProbe to prevent liveness race during migrations (#184) ... 578165e5bf

Autonomous-merge: harmless additive change to Helm chart template. Template rendered cleanly with helm template against prod values. CI pipeline is currently blocked on HOMELAB-972 (Forgejo Actions queue). Per parent approval, merging without CI; safety validated by helm template.

HOMELAB-974: fix(langfuse): gate web/worker startup on ClickHouse TCP readiness (#183) ... 69f45cbdfc

Adds langfuse.extraInitContainers with busybox nc -z wait-loop on langfuse-clickhouse:9000 and :8123. Fixes CrashLoopBackOff race on node disruption. Chart-level key applies to both web and worker pods.

HOMELAB-966: feat(sops-operator): Helm values + age-key bootstrap runbook (#185) c4b797f685

HOMELAB-966: docs(adr-004): amend — declarative SOPS decryption via operator (#186) dbc5ff92b4

HOMELAB-988: feat(forgejo-runner): bake terraform/sops/age + add infra label (#187) 1633d130f4

HOMELAB-988: feat(forgejo-runner): mount age key + AWS creds + Forgejo token (#188) 3545116ada

HOMELAB-988: fix(forgejo-runner): fsGroup 1000 + defaultMode 0440 for mounts (#189) e282493544

HOMELAB-988: fix(es-forgejo-runner): point at real-AWS terraform creds (#190) b54b346a05

HOMELAB-994: feat(es-cnpg-b2): add ExternalSecret for apexdirector-timescale ... 580fc2a8e8

Adds the 7th cnpg-b2-credentials ExternalSecret entry so the
cnpg-cluster-timescale CNPG cluster can mount B2 credentials and emit
WAL/base backups. Mirrors the existing 6 entries' shape exactly; only
the namespace differs.

Re-arms backups for the 4 CNPG clusters whose backups were disabled
(HOMELAB-590) or never enabled (apexdirector-timescale, which lacked
the secret entirely). Companion change in prod repo flips
backups.enabled true on the 4 values.yaml files.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-994: feat(es-cnpg-b2): add ExternalSecret for apexdirector-timescale' (#191) from plane/HOMELAB-994-cnpg-backups into live 3a5e3d90e4

HOMELAB-990: feat(talos-cluster): bring bootstrap helm_releases into module (#192) 2fee10151f

HOMELAB-990: fix(talos-cluster): drop triggers from wait_for_nodes (#193) 46539d936a

Merge branch 'live' into plane/HOMELAB-276-argocd 14bdeb4617

Merge pull request 'HOMELAB-276: feat(eso): add ExternalSecret for Lab Director ArgoCD token' (#37) from plane/HOMELAB-276-argocd into live ... 0c90089816

Reviewed-on: #37

HOMELAB-958: fix(proxmox-vm): ignore Cilium-runtime churn on VM resource (#194) 9317377398

HOMELAB-958: revert maxPods=150 from worker_config_patch ... 35eaa790bc

HOMELAB-956 added maxPods=150 as a workaround when wk-02 was cordoned
and all density landed on wk-01 (single-worker risk of hitting kubelet's
default 110 cap). With wk-03 joined as a 3rd active worker (HOMELAB-958
completion), workload spreads across 3 nodes and the default 110 is
sufficient. Reverting avoids an unnecessary kubelet restart on next apply.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

feat(longhorn): default StorageClass to 2-way replication ... 6d706258c2

Set persistence.defaultClassReplicaCount=2 so the auto-created `longhorn`
StorageClass provisions PVCs at numberOfReplicas=2, matching `longhorn-db`
and defaultSettings.defaultReplicaCount.

Single Proxmox host topology means Longhorn 3-way replication adds no
cross-host fault tolerance — host failure takes out all 3 VMs and all 3
replicas. The ZFS mirror at the storage layer already handles single-disk
failure, and CNPG B2 backups handle durability. 3-way only triples write
I/O on the shared SSD/HDD pool with no real availability gain.

Existing volumes provisioned before this change remain at 3-way (SC
parameters bind only at provisioning time) — those are migrated separately
by patching their volume.longhorn.io spec.numberOfReplicas in-cluster.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Merge pull request 'HOMELAB-958: revert maxPods=150 from worker_config_patch' (#195) from plane/HOMELAB-958-revert-maxpods into live 297ba42b0a

Merge pull request 'longhorn: default StorageClass to 2-way replication' (#196) from plane/longhorn-default-2way into live 431e663139

fix(longhorn): throttle rebuild concurrency to 1 + disable auto-balance ... 8bdfe9f72f

Today's EROFS storm: with both ZFS pools degraded on daytona (hdd-mirror
single-disk after HOMELAB-860 sdb detach, ssd-mirror has one UNAVAIL Bestoss
SSD), the underlying disk sustains ~30 MB/s. concurrentReplicaRebuildPerNodeLimit=5
saturated the hdd-mirror spindle, engine sync writes timed out, kernel marked
multiple PVCs read-only (postgres CNPGs, harbor-redis, plane-rabbitmq Mnesia).

Throttle to 1 in-flight rebuild per node so the disk can serve foreground
sync writes alongside replica rebuilds. Disable auto-balance during the
recovery window — it's already inert while volumes are degraded
("Cannot auto-balance volume in degraded state" log spam) but remove
the controller pressure entirely.

Restore to concurrency=2 + replicaAutoBalance=best-effort once both ZFS
pools are resilvered and the rebuild backlog has drained.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Merge pull request 'longhorn: throttle rebuild concurrency + disable auto-balance during recovery' (#197) from plane/longhorn-throttle-recovery into live 99d70250dd

fix(longhorn): restore throttle to 2/node + best-effort auto-balance ... 93f21c61cb

Both ZFS pools healthy (ssd-mirror resilvered 2026-04-28 with both Bestoss
disks ONLINE, hdd-mirror redundancy work tracked separately). Lifts the
emergency 1/node + auto-balance=disabled cap from 8bdfe9f.

Settles at 2/node (chart default is 5) because daytona's ZFS-backed pools
sustain ~30 MB/s sequential — the 2026-04-27 incident proved 5 saturates
the spindle and triggers an EROFS storm.

Merge pull request 'longhorn: restore throttle to 2/node + best-effort auto-balance' (#198) from plane/longhorn-restore-throttle-after-resilver into live eccd596d28

fix(longhorn): single-replica defaults + restore salvage lever ... fc8d5e2ced

After 2026-04-30 plane recovery: 4 postgres restore attempts on a 17 GB
pg_dumpall failed with EROFS partway, on bitnami AND CNPG storage classes.
Root cause is structural: 2-replica Longhorn on top of a single ZFS
mirror doubles write I/O for zero fault tolerance gain (single Proxmox
host). Replica rebuilds during the writes saturate the SSDs and cause
guest filesystems to remount read-only.

Changes (per Longhorn specialist + architect review):
- defaultReplicaCount: 2 -> 1, persistence.defaultClassReplicaCount: 2 -> 1
  (kills phantom redundancy)
- replicaAutoBalance: best-effort -> disabled
  (rebalances trigger I/O storms; ZFS handles physical layout)
- concurrentReplicaRebuildPerNodeLimit: 2 -> 1
  (postgres can keep WAL flushed during rebuild)
- disableRevisionCounter: implicit-true -> explicit false
  (restores salvage lever lost during plane-pgdb 2026-04-29)
- engineReplicaTimeout: 8 -> 15
  (tolerates fsync stalls under SSD-mirror saturation)
- longhorn-db SC: numberOfReplicas 2 -> 1, disableRevisionCounter false

This is forward-only on Longhorn-managed defaults. Existing volumes
keep their current spec.numberOfReplicas; we'll bulk-patch them in a
follow-up step after these settings sync.

Refs HOMELAB-1003 (postmortem).

Merge pull request 'fix(longhorn): single-replica defaults + restore salvage lever' (#199) from plane/longhorn-singlerep-iac into live fe3fe36bfe

cnpg-cluster: disable WAL archiving to stay under B2 transaction cap ... 151bb46764

Continuous WAL shipping across 7+ CNPG clusters was exhausting the
Backblaze B2 free tier Class C transaction cap (2,500/day). Daily
base backups via ScheduledBackup still run — PITR is now limited
to the last base backup instead of any point in time.

cnpg-cluster: disable WAL archiving to stay under B2 transaction cap (#200) 1db331ed50

Revert "cnpg-cluster: disable WAL archiving to stay under B2 transaction cap" ... 01aaad86bb

This reverts commit 151bb46764.

Merge pull request 'Revert "cnpg-cluster: disable WAL archiving" — CNPG operator enforces archive_mode=on' (#201) from revert-wal-archive-off into live 6d2ecb0578

fix(langfuse): increase ClickHouse memory 5Gi → 8Gi ... 6e0af920c9

Dashboard queries (traces FINAL LEFT JOIN observation_stats) still OOM
after OPTIMIZE TABLE compaction. The query materializes ~4.5 GiB intermediate
results. Observations ARE being ingested successfully (verified last write
2026-05-10 20:18 UTC). Only the query needs more headroom.

Refs: HOMELAB-1052

Merge pull request 'fix(langfuse): increase ClickHouse memory 5Gi → 8Gi' (#202) from plane/HOMELAB-1052-clickhouse-memory into live 1582909363

HOMELAB-1053: remove lab-director — chart, manifests, ESO, ArgoCD RBAC, OTel + dormant refs ... de38095e02

- Delete lab-director Helm chart (core/charts/apps/lab-director/)
- Delete orphan LB Service manifest (lab-director-pg-lb.yaml)
- Delete worker ExternalSecrets (es-lab-director-worker.yaml)
- Remove lab-director ExternalSecret doc from es-cnpg-b2.yaml (6 remaining namespaces)
- Remove lab-director ArgoCD RBAC policy.csv entries (4 lines)
- Remove lab-director from OTel collector span filter
- Clean dormant sandbox chart refs: LAB_DIRECTOR_API_URL, image registry, CNP egress allow
- Clean dormant dev-environment chart ref: Harbor image path

MUST be merged AFTER prod PR has merged AND ArgoCD has pruned lab-director/cnpg-cluster-lab-director.
Confirm 'argocd app list | grep lab-director' returns empty before merging.

HOMELAB-1053: amend — retire NATS chart, delete missed ExternalSecret ... 3e3031562e

- Delete core/manifests/external-secrets/es-lab-director-argocd.yaml (missed in initial scan; would cause external-secrets-config app to go Degraded after ns deletion)
- Delete core/charts/apps/nats/values.yaml (NATS retired alongside lab-director)
- Delete core/charts/platform/nats/values.yaml (duplicate NATS values overlay)

Merge pull request 'HOMELAB-1053: remove lab-director — chart, manifests, ESO, ArgoCD RBAC, OTel + dormant refs' (#203) from homelab-1053-remove-lab-director into live ... cf7421d69d

Reviewed-on: #203

feat(forgejo): enable SSH access on port 22 ... b9746f9af3

Enable Forgejo SSH service, configure SSH_DOMAIN and SSH_PORT in Helm
values, add TCP listener on port 22 to the homelab gateway, and create
a TCPRoute to route SSH traffic to the forgejo-ssh service.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Merge pull request 'feat(forgejo): enable SSH access on port 22' (#204) from jesus/infra-core:feat/forgejo-ssh into live ... d9209bdbd1

Reviewed-on: #204
Reviewed-by: Aaron Reynoza <aaron@reynoza.org>

HOMELAB-1056: add data_disk_discard, ignore image size drift a19cbf2551

feat(alerts): add CNPGWALArchiveFailing + remove outline exclusion ... 46b589d5a4

Adds CNPGWALArchiveFailing alert that fires when WAL archiving
is retrying but continuously failing (e.g. barman exit status 4
due to B2 storage cap). The existing CNPGBackupFailing only
catches when archiving is completely stopped (is_archiving==0).

Also removes the arbitrary namespace!="outline" exclusion from
CNPGLastBackupOld so Outline clusters are monitored too.

Refs: incident 2026-05-12 B2 cap depletion

Merge pull request 'feat(alerts): add CNPGWALArchiveFailing + remove outline exclusion' (#205) from fix/cnpg-wal-archive-alert into live d5914122a1

Merge pull request 'HOMELAB-1056: add data_disk_discard, ignore image size drift' (#206) from homelab-1056-terraform-drift into live 2987ccb57d

HOMELAB-1056: switch overwrite=false, remove ignore_changes on Talos images ... be0ec14f9c

Setting overwrite=false stops the perpetual force-replacement cycle caused
by the provider comparing compressed URL Content-Length against decompressed
datastore size. ignore_changes=[size] was ineffective (size is provider-computed).

Merge pull request 'HOMELAB-1056: switch overwrite=false, remove ignore_changes on Talos images' (#207) from homelab-1056-overwrite-false into live fb13142d7c

feat(paperclip): add infrastructure manifests for HOM-7 ... 55bcf95fab

- Add paperclip Helm chart (PVC, CiliumNetworkPolicy, secrets, service, deployment stub)
- Add HTTPRoute for paperclip.aaron.reynoza.org in gateway routes

Co-Authored-By: Paperclip <noreply@paperclip.ing>

Merge branch 'paperclip/HOM-7-infra-prep' into live ... cf507f8ee4

Infrastructure prep for Paperclip on Homelab (HOM-7).

Co-Authored-By: Paperclip <noreply@paperclip.ing>

Merge branch 'live' of https://forgejo.aaron.reynoza.org/aaron/infra-core into live ecd3b9a7cf

Merge pull request 'feat(paperclip): infrastructure manifests for HOM-7' (#208) from paperclip/HOM-7-infra-prep into live 31359f64fd

fix(paperclip): correct CiliumNetworkPolicy template syntax (HOM-7) ... 0abd2cd0de

- Simplify template to hardcoded rules matching sandbox pattern
- Remove over-templating that produced invalid YAML
- Clean up values.yaml to match simplified template

Co-Authored-By: Paperclip <noreply@paperclip.ing>

Merge pull request 'fix(paperclip): correct CiliumNetworkPolicy template syntax' (#209) from paperclip/HOM-7-fix-cilium-template into live 17343433b5

fix(paperclip): use fromEntities: cluster for cross-namespace ingress (HOM-8) ... 685893229f

The CiliumNetworkPolicy CRD rejects '*' as a value for
io.kubernetes.pod.namespace in matchLabels. Switch to
fromEntities: - cluster to allow all in-cluster pods to reach
the Paperclip API on port 3100.

Refs: HOM-8

Merge pull request 'fix(paperclip): use fromEntities: cluster for cross-namespace ingress (HOM-8)' (#210) from fix/paperclip-cilium-entities into live c0ec659f45

fix(paperclip): correct port and env vars for HOM-13 deployment ... eca69c0d79

- Change app port from 3100 to 8080 (lab-director backend listens on 8080)
- Update CiliumNetworkPolicy ingress port to 8080
- Fix DATABASE_URL env var name (was PAPERCLIP_DATABASE_URL)
- Remove unused PAPERCLIP_AGENT_JWT_SECRET env var
- Update HTTPRoute backend port to 8080

Refs: HOM-13

feat(paperclip): complete Helm chart for HOM-14 ... 660acb574f

- Revert HOM-13 incorrect port changes (8080 → 3100) in HTTPRoute and CiliumNetworkPolicy
- Parameterize CiliumNetworkPolicy ingress port via .Values.port
- Add ServiceAccount template and helper
- Rewrite deployment with:
  - Proper security contexts (pod and container level)
  - Liveness, readiness, and startup probes on /api/health
  - Complete Paperclip env vars (deployment mode, public URL, storage, etc.)
  - DATABASE_URL from CNPG secret
  - PAPERCLIP_AGENT_JWT_SECRET and BETTER_AUTH_SECRET support
  - Correct volume mount: /paperclip (was /data)
  - Master key secret mounted as file at /paperclip/instances/default/secrets/master.key
  - Recreate strategy for RWO PVC safety
- Update service to use .Values.service.type and .Values.service.port
- Add betterAuthSecret placeholder to secret template
- Chart metadata: version 0.2.0, appVersion v2026.529.0

Merge pull request 'HOM-14: Complete Helm chart for Paperclip' (#211) from paperclip/HOM-14-chart into live 1b8e346f64

fix(paperclip): add rollingUpdate: null to Recreate strategy ... d37ca1dd84

Kubernetes requires explicit rollingUpdate: null when switching
a Deployment from RollingUpdate to Recreate strategy.

Refs: HOM-15
Co-Authored-By: Paperclip <noreply@paperclip.ing>

Merge pull request 'fix(paperclip): add rollingUpdate: null to Recreate strategy' (#212) from fix/paperclip-recreate-strategy into live e70b21fec8

feat(observability): connect pi-otel-telemetry to homelab stack ... e05c29f559

- Add Gateway API HTTPRoute for otel.aaron.reynoza.org -> otel-collector:4318
- Add Agent Telemetry Grafana dashboard (TraceQL panels)
- Add Tempo datasource uid 'tempo' and search.hide=false
- Add metrics_generator to Tempo for TraceQL metrics support
- Refs: HOMELAB-1092

Merge pull request 'HOMELAB-1092: Connect pi-otel-telemetry to homelab observability stack' (#213) from plane/HOMELAB-1092-otel-homelab-integration into live b9b4df23c0

fix(tempo): remove broken metricsGenerator config that caused CrashLoopBackOff ... bf74eee8ba

The overrides.defaults.metrics_generator field is not valid for Tempo v2.6.1,
causing all modules to fail. Removing the metricsGenerator block restores
Tempo to working order. TraceQL basic queries work without metrics_generator;
span-metrics aggregation will be added in a follow-up.

Refs: HOMELAB-1092

Merge pull request 'fix(tempo): restore Tempo from CrashLoopBackOff by removing broken metricsGenerator' (#214) from plane/HOMELAB-1092-otel-homelab-integration into live 9aeafb236a

fix(pangolin): remove unsupported 'sso' field from resource creation API call 4d6bd665e0

feat(gateway): add HTTPRoute for Tempo query frontend 40ade44178

HOMELAB-1096: add Pi Sessions Grafana dashboard f287a4b7a0

Merge pull request 'HOMELAB-1096: Add Pi Sessions Grafana dashboard' (#216) from plane/HOMELAB-1096-pi-sessions-dashboard into live c679813df1

HOMELAB-1096: enable Tempo metrics-generator + Prometheus remote-write-receiver 33e62e3632

HOMELAB-1096: add Pi Tools, Pi LLM & Performance dashboards; fix Agent Telemetry 06dccf136c

Merge pull request 'HOMELAB-1096: Pi telemetry dashboards with Tempo metrics generator' (#217) from plane/HOMELAB-1096-pi-dashboards-with-metrics into live b5a23d564c

HOMELAB-1096: fix Prometheus remote-write-receiver config 6a5ab352b0

Merge pull request 'HOMELAB-1096: fix Prometheus remote-write-receiver config' (#218) from plane/HOMELAB-1096-pi-dashboards-with-metrics into live ae3f39e7ff

HOMELAB-1096: rewrite dashboards with PromQL for stats/time series db2110b2da

Merge pull request 'HOMELAB-1096: rewrite dashboards with PromQL queries' (#219) from plane/HOMELAB-1096-pi-dashboards-with-metrics into live a08c51bb35

HOMELAB-1096: fix dashboard YAML indentation 6e5c990892

Merge pull request 'HOMELAB-1096: fix dashboard YAML indentation' (#220) from plane/HOMELAB-1096-pi-dashboards-with-metrics into live e79f2314d6

feat(ansible,helm): enable Paperclip remote VM execution via SSH cdfdc3b482

fix(inventory): populate VM 110 management IP (192.168.1.10) and VLAN IP (10.10.10.5) 6bf3b614f7

fix(ansible): create operator user, populate inventory, allow package downgrades 5d0358b1cf

Merge pull request 'feat: enable Paperclip remote VM execution via SSH' (#222) from feat/paperclip-remote-vm-execution into live 41165afaa4

fix(paperclip): use exec probes to avoid CSRF 403, add TCPRoute CRD ... 49031011d9

- Switch health probes from httpGet to exec (curl localhost)
  Paperclip's authenticated mode CSRF validation rejects kubelet
  HTTP probes that use pod IP as Host header → returns 403.
  exec probes using curl localhost bypass the origin check.

- Add TCPRoute CRD (gateway.networking.k8s.io)
  The forgejo-ssh TCPRoute was blocking the gateway ArgoCD sync
  because the CRD was never installed. Added tcproutes.yaml from
  upstream Gateway API experimental channel.

- Bump forgejo-ssh TCPRoute from v1alpha2 to v1
  Matches the storage version served by the CRD.

Ref: HOMELAB-1097

Merge branch 'live' into plane/HOMELAB-1097-fix-db-permissions 56219545ac

Merge pull request 'HOMELAB-1097: fix Paperclip exec probes + add TCPRoute CRD' (#223) from plane/HOMELAB-1097-fix-db-permissions into live ... 8f3792ed26

Reviewed-on: #223

chore: bump Paperclip image to v2026.609.0 ... 355b03b2f5

Co-Authored-By: Paperclip <noreply@paperclip.ing>

docs(analysis): HOMELAB-1102 pi telemetry multi-machine identification ... 7f7e35b2e4

Add comprehensive analysis of OpenTelemetry trace identification
for multi-host pi deployments. Evaluates 4 approaches, recommends
auto-detecting hostname via custom NodeTracerProvider passed to
Openlit.init().

Refs: HOMELAB-1102

fix(ansible): update mgmt VM docs and config for admin user ... 302539028b

- Change Ansible inventory from operator to admin
- Update mgmt-base role to use admin instead of operator
- Update sshd_config to allow admin and paperclip-operator
- Update forgejo-runner service to use admin
- Update mgmt-vm-setup runbook with correct IPs and user

Refs: HOMELAB-1099

Merge pull request 'fix(ansible): update mgmt VM docs and config for admin user' (#225) from plane/HOMELAB-1099-fix-mgmt-vm-docs into live 885c650649

fix(paperclip): mount dev-agent SSH key at $HOME (/paperclip/.ssh) (HOMELAB-1104) (#226) f289d9beb5

feat(tempo): add host.name resource dimension to span metrics (HOMELAB-1103) ... ac7251acb4

Phase 2: Override Tempo chart config template to add
metrics_generator.processor.span_metrics.resource_dimensions: [host.name]
so Prometheus metrics include host_name labels.

Phase 3: Add host_name dashboard variable and filter to all pi-telemetry
Grafana dashboards (pi-sessions, pi-tools, pi-llm-performance,
agent-telemetry). All PromQL panel queries now include
host_name=~"\" filter, backward-compatible with "All" default.

Merge branch 'live' into plane/HOMELAB-1103-tempo-host-filtering c2407da0f2

Merge pull request 'feat(tempo): add host.name resource dimension to span metrics (HOMELAB-1103)' (#227) from plane/HOMELAB-1103-tempo-host-filtering into live ... 97c083b5f4

Reviewed-on: #227

revert(tempo): remove config override — resource_dimensions unsupported in Tempo 2.6.1 ... 30a37028bd

Tempo 2.6.1 span_metrics processor only supports span-level dimensions,
not resource_dimensions. The host.name resource attribute set by
HOMELAB-1102 requires Tempo >= 2.8.0 (chart >= 1.23.0).

Dashboards remain with host_name variable/filters — they'll populate
after the Tempo upgrade (HOMELAB-XXX).

HOMELAB-1105: Update tempo values for chart 1.24.2 acc9a4526c

aaron force-pushed plane/HOMELAB-1105-upgrade-tempo-chart from acc9a4526c to c85f6823fc 2026-06-12 03:01:05 +00:00 Compare

aaron merged commit 4baaf21d2e into main 2026-06-12 03:01:31 +00:00

aaron referenced this pull request from a commit 2026-06-12 03:01:33 +00:00

Merge pull request 'HOMELAB-1105: Update Tempo values for chart 1.24.2' (#229) from plane/HOMELAB-1105-upgrade-tempo-chart into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

aaron/infra-core!229

No description provided.