HOMELAB-974: fix(langfuse): gate web/worker startup on ClickHouse TCP readiness #183

Merged

aaron merged 1 commit from plane/HOMELAB-974-clickhouse-init-wait into live

2026-04-24 21:25:33 +00:00

aaron commented

2026-04-24 21:19:03 +00:00

Owner

Summary

Adds langfuse.extraInitContainers to the base chart values (core/charts/apps/langfuse/values.yaml) with a busybox nc -z wait-loop on langfuse-clickhouse:9000 and :8123.
Rendered into both web and worker pod specs by the upstream chart (langfuse-k8s 1.5.22). helm template verified locally against base + prod overrides.
Fixes CrashLoopBackOff race after node/CSI disruption where langfuse-web fails ClickHouse migration before CH is network-ready.

Test plan

After Argo sync (blocked on HOMELAB-972): confirm initContainer: wait-for-clickhouse appears in kubectl -n langfuse get deploy langfuse-web -o yaml.
Delete langfuse-clickhouse-shard0-0 pod; observe langfuse-web new pod sits in Init:0/1 until CH :9000 responds, then Running with zero error restarts.
Compare restart count pre/post simulated disruption — expect 0 additional.

Ref HOMELAB-974.

## Summary - Adds `langfuse.extraInitContainers` to the base chart values (`core/charts/apps/langfuse/values.yaml`) with a busybox `nc -z` wait-loop on `langfuse-clickhouse:9000` and `:8123`. - Rendered into both web and worker pod specs by the upstream chart (langfuse-k8s 1.5.22). `helm template` verified locally against base + prod overrides. - Fixes CrashLoopBackOff race after node/CSI disruption where langfuse-web fails ClickHouse migration before CH is network-ready. ## Test plan - [ ] After Argo sync (blocked on HOMELAB-972): confirm `initContainer: wait-for-clickhouse` appears in `kubectl -n langfuse get deploy langfuse-web -o yaml`. - [ ] Delete `langfuse-clickhouse-shard0-0` pod; observe langfuse-web new pod sits in `Init:0/1` until CH :9000 responds, then Running with zero error restarts. - [ ] Compare restart count pre/post simulated disruption — expect 0 additional. Ref HOMELAB-974.

aaron added 1 commit

2026-04-24 21:19:03 +00:00

HOMELAB-974: fix(langfuse): gate web/worker startup on ClickHouse TCP readiness

CI Review / pr-title (pull_request) Successful in 0s

Details

Lint & Validate / terraform-validate (pull_request) Has been cancelled

Details

Lint & Validate / yaml-lint (pull_request) Has been cancelled

Details

Lint & Validate / shellcheck (pull_request) Has been cancelled

Details

CI Review / helm-validate (pull_request) Has been cancelled

Details

CI Review / ai-review (pull_request) Has been cancelled

Details

49a4da6d38

langfuse-web enters CrashLoopBackOff during node disruption when it tries
ClickHouse migrations (clickhouse://langfuse-clickhouse:9000) before CH is
network-ready and fails fast with:
  dial tcp: connect: operation not permitted
  Applying clickhouse migrations failed.
Observed 31 restarts over 2d20h after prod-wk-01 CSI disruption. Pod eventually
self-recovered, but the next node event will trigger the same cascade.

Add a pod-level initContainer that blocks the main container until both
ClickHouse ports (9000 native, 8123 HTTP) respond to TCP. Uses busybox:1.36
`nc -z` loop with a 2s sleep; idempotent and terminates as soon as CH is up.

Chart value key: `langfuse.extraInitContainers` — rendered into both web
and worker pod specs by the upstream template (langfuse-k8s 1.5.22
templates/web/deployment.yaml:54-57 and templates/worker/deployment.yaml:50-52).
Verified with `helm template` against this values file + prod overrides.

Resource request is trivial (10m/16Mi req, 50m/32Mi limit).