aaron/infra-core
1
0
Fork 2
Code Issues Pull requests 12 Projects Releases 1 Packages Wiki Activity Actions

fix(langfuse): increase ClickHouse memory 5Gi → 8Gi #202

Merged
aaron merged 3 commits from plane/HOMELAB-1052-clickhouse-memory into live 2026-05-10 20:39:07 +00:00
Conversation 0 Commits 3 Files changed 4 +56 -1
aaron commented 2026-05-10 20:38:52 +00:00
Owner
Copy link

What

Increase ClickHouse memory limit from 5Gi to 8Gi to fix dashboard query OOMs.

Why

Langfuse dashboard queries (traces FINAL LEFT JOIN observation_stats) materialize ~4.5 GiB intermediate results during the GROUP BY + JOIN. This exceeds the 5Gi container limit. The OPTIMIZE TABLE compaction reduced fragmentation but the query still needs more headroom.

Verified

  • Observations ARE being ingested (last ClickHouse write: 2026-05-10 20:18 UTC)
  • ClickHouse ingestion pipeline is healthy
  • Only the read query is blocked by memory limits

Refs: HOMELAB-1052

## What Increase ClickHouse memory limit from 5Gi to 8Gi to fix dashboard query OOMs. ## Why Langfuse dashboard queries (`traces FINAL LEFT JOIN observation_stats`) materialize ~4.5 GiB intermediate results during the GROUP BY + JOIN. This exceeds the 5Gi container limit. The OPTIMIZE TABLE compaction reduced fragmentation but the query still needs more headroom. ## Verified - Observations ARE being ingested (last ClickHouse write: 2026-05-10 20:18 UTC) - ClickHouse ingestion pipeline is healthy - Only the read query is blocked by memory limits Refs: HOMELAB-1052
- Expand CiliumLoadBalancerIPPool from 10.10.10.220-250 (31 IPs) to
  10.10.10.100-250 (151 IPs)
- Add temporal-frontend-lb (10.10.10.109:7233) for VPN access
- Add cnpg-lab-director-lb (10.10.10.110:5432) for VPN access
- Eliminates need for kubectl port-forward from Mac

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pool expansion must keep the original range (.220-.250) as the first
block to preserve existing IP assignments. New services pinned to the
expansion range (.100-.219) to avoid conflicts.

- Temporal LB: 10.10.10.100
- PG LB: 10.10.10.101

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
fix(langfuse): increase ClickHouse memory 5Gi → 8Gi
Some checks failed
0/0 projects applied successfully.
CI Review / pr-title (pull_request) Has been cancelled
Details
CI Review / helm-validate (pull_request) Has been cancelled
Details
CI Review / ai-review (pull_request) Has been cancelled
Details
Lint & Validate / terraform-validate (pull_request) Has been cancelled
Details
Lint & Validate / yaml-lint (pull_request) Has been cancelled
Details
Lint & Validate / shellcheck (pull_request) Has been cancelled
Details
6e0af920c9 
Dashboard queries (traces FINAL LEFT JOIN observation_stats) still OOM
after OPTIMIZE TABLE compaction. The query materializes ~4.5 GiB intermediate
results. Observations ARE being ingested successfully (verified last write
2026-05-10 20:18 UTC). Only the query needs more headroom.

Refs: HOMELAB-1052
aaron merged commit 1582909363 into live 2026-05-10 20:39:07 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
aaron/infra-core!202
No description provided.