HOMELAB-1736: fix(forgejo): CI status/failure tools — robust endpoint fallback #48
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "plane/HOMELAB-1736-forgejo-ci"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What
Lands the
forgejo-ciextension (forgejo_ci_status,forgejo_ci_failures) into the repo and fixes theactions/tasksHTTP 500 that blinded the landing pipeline.Fixes HOMELAB-1736.
2 commits, reviewable in order:
chore(forgejo-ci): salvage the unlanded HOMELAB-1639 extension into the repo— byte-for-byte (md5-identical) copy of the working tree that lives only at~/.pi/agent/extensions/forgejo-ci/. HOMELAB-1639 (2026-09-22) delivered the extension but recorded "Nothing outside~/.pi/agent/... was modified", so it never landed here. The copy is verbatim so commit 2 is a real diff against known state.fix(forgejo-ci): page-safe CI status/failure tools with endpoint fallback— the fix.Why the tools were broken (root cause, reproduced)
GET /repos/{o}/{r}/actions/taskshonourslimitonly whenpageis also sent:routers/api/v1/repo/action.go:588→db.FindAndCount[ActionTask]{ListOptions: utils.GetListOptions(ctx)}routers/api/v1/utils/page.go→ListOptions{Page: ctx.FormInt("page"), PageSize: ToCorrectPageSize(limit)}—Page=0when the client omits itmodels/db/list.go:186FindAndCount→if !IsListAll() && pageSize > 0 && page >= 1 { sess.Limit(...) }— thepage >= 1guard drops the LIMITMeasured against the running instance (14.0.3+gitea-1.22.0), same auth the extension uses (
Authorization: token <PAT>):?limit=5(no page)?page=1&limit=5The 500 body is
{"message":""}because Forgejo blanks 500 bodies for non-admin callers (services/context/api.go:179-198) — that is why earlier investigators could not see the cause./actions/runshas the same limit-dropping behaviour (?limit=100returned all 401 runs); it only escaped the 500 because its table is smaller. The extension called?limit=100/?limit=200with nopageon both endpoints, so every Homelab repo 500'd while small repos (sendrast-web) worked — matching the "some repos 500, some don't" reports.Reproduced with the deployed tool before the fix:
forgejo_ci_status Homelab/infra-core→Forgejo API 500 on /repos/Homelab/infra-core/actions/tasks?limit=100.The fix
Api.paged()always sendspage+limit, walks pages, clamps an over-long page defensively (protects against the upstream bug recurring), and covers the whole feed where it must (the run list has NO defined ordering, so a partial window cannot be trusted).run_numberfilter → one exact request. PR / branch / sha refs cannot use that filter, so they walk the whole/actions/runsfeed (bounded bytotal_count) and take the newest match, ranked by match precision first (an exactprettyref/PR match outranks a title substring).#Nis read as a PR number first, then as a run index.actions/tasks→ web run page →commits/{sha}/statuscontexts → run-level status. A job-level failure can no longer kill the tool; the degraded source is reported (details.degraded+ anote:line).updated_at(bumped hours after a job ends) is only a fallback, clamped to the run duration.resource does not exist), not 404; cleaned-up/never-started logs are named; a failed job whose log has no failure marker now yields a bounded tail excerpt instead of "log has no failure marker" (a gap noted in.rpiv/artifacts/work/epic0-contracts/review-send139.md); excerpts anchor on runner/cancellation markers so a cancelled log no longer dumps 30 lines of passing specs.detailsis a superset of the previous keys (owner/repo/run/runId/failedJobs;runs[].jobs[].name/status).Evidence
Live replay (real Forgejo, no fixtures) —
FORGEJO_LIVE=1 npx vitest run tests/live.test.ts, 11/11 pass:AC replay from the ticket: sendrast-web #489 / #490 (cancelled e2e jobs → excerpt returned) and #492 (
[pass]), plus infra-core run 1481. Green runs reportdegraded: []andjobsSource: tasks.Durations now match the Forgejo UI (
5m49s/9m49s) instead of the previous2591m52s/43h35m49sartefacts of taskupdated_at.Tests / CI
forgejo-ci/package.json+tsconfig.json+package-lock.json; new CI jobforgejo-ci(npm ci→npm run typecheck→npm test).page+limitalways present), pagination/truncation/clamp,run_numberfast path + candidate fallback, branch/PR precision ranking (an exact ref beats a newer title mention), degradation paths (tasks 500 → run page / commit-status / run-level), partial-task-feed → run-page union surfaced indegraded, case/whitespace-insensitive job-union de-dup,maxPagestruncation surfacing, log unavailability (404,resource does not exist, cleaned up, never started), excerpt anchoring, duration parsing/clamping.tests/live.test.tsis opt-in (FORGEJO_LIVE=1) and self-skips without it — CI has no Forgejo credentials.npm test+npm run typecheckgreen locally, plustsc --noUnusedLocals --noUnusedParametersclean.Territory / drift note
forgejo-ci/**(new),docs/forgejo-ci.md(new),README.md(extension + config rows),.forgejo/workflows/ci.yaml(one job). No other extension touched.~/.pi/agent/extensions/forgejo-ci/is untouched (md5s verified unchanged) for the HOMELAB-1740 custodian. Recommended reconciliation: delete it once this lands — it is the same 1639 source, now tracked and fixed. Note for 1740: its description guesses that dir is "the working tree of the HOMELAB-1736 fix"; it is not — it predates 1736 by a week and is the 1639 deliverable.FindAndCountdropping the LIMIT forpage == 0). The client-side fix is in place; the upstream-shaped detail (endpoint, request shape, reproduction, source lines) is recorded on HOMELAB-1736 rather than left as a broken tool.Root cause (server-side, client-fixable): GET /repos/{o}/{r}/actions/tasks honours `limit` ONLY when `page` is also sent. routers/api/v1/utils/page.go builds ListOptions{Page: FormInt("page")} (0 when omitted) and models/db/list.go FindAndCount applies the SQL LIMIT only under `page >= 1`, so ?limit=N asks for the whole table. Measured on this instance: Sendrast/sendrast-web ?limit=5 returned all 893 rows (443 KB); on repos with a large action_task table the unbounded query fails with HTTP 500 and a blanked message (services/context/api.go hides 500 bodies from non-admin callers) — Homelab/infra-core (2274 tasks) 500 after 6.3 s, Homelab/prod (3056) after 7.1 s. /actions/runs has the same limit-dropping behaviour (?limit=100 returned all 401 runs) and only escaped the 500 because its table is smaller. The extension called ?limit=100 without page on both endpoints, so every Homelab repo — the ones the landing pipeline lands — 500'd. Fix: - Api.paged() always sends page+limit, walks pages, clamps an over-long page defensively, and stops early (tasks are newest-first, so a run lookup stops once the run is covered). - Run lookup uses the server-side run_number filter (exact, one request) instead of hoping the run is inside a fixed page window; #N is read as a PR number first and a run index second. - Degradation ladder: actions/tasks -> web run page -> commits/{sha}/status contexts -> run-level status. Job-level failure never throws; it reports the degraded source in details.degraded and a note: line, so the landing pipeline is never blinded again. - Job durations/indexes now come from the run page (authoritative); task updated_at (bumped hours after a job ends — the 43h/2600m displays) is only a fallback, clamped to the run duration. - Log handling: a missing attempt answers 500 ("resource does not exist"), not 404; cleaned-up/never-started logs are named as such; a failed job whose log has no failure marker now gets a bounded tail excerpt instead of "log has no failure marker", and excerpts anchor on runner/cancellation markers so a cancelled log no longer dumps 30 lines of passing specs. - Response contract preserved: same text layout and a superset of the previous details keys (owner/repo/run/runId/failedJobs; runs[].jobs[].name/status). Tests: 36 unit/tool tests against a fake Forgejo (paging URLs, clamp, fallbacks, excerpt extraction, log availability) plus an opt-in live replay (FORGEJO_LIVE=1) covering infra-core 1481, sendrast-web #489/#490/#492 and infra-core 1226. Typecheck + vitest gated by a new CI job. Salvaged from HOMELAB-1639's unlanded working tree (previous commit).Independent review (landing worker, HOMELAB-1736) — HOLD, 3 blockers
Reviewed at
9db57ba(CI run #85, all jobs green). Reviewers ran as freshpi -p --no-sessionprocesses, primary sources only (diff + ticket ACs + live Forgejo). Three dimensions: correctness/claim-honesty, design/codebase-fit, test-adequacy/security.What is verified and good
/actions/tasks?limit=5(nopage) → 500 on infra-core/prod/lab-director;?page=1&limit=5→ 200. On sendrast-web,limitis silently ignored (443 680 B = the whole table). Thepage+limitfix inApi.paged()is correct.FORGEJO_LIVE=1→ 4/4 (infra-core 1481, sendrast #489/#490/#492, two excerpts). 36 unit tests + typecheck clean.detailscontract genuinely a superset of the old keys; no key removed or retyped.package-lock.json.Blockers — each reproduced against the live server with the post-fix code
Default listing returns stale runs (regression).
forgejo_ci_status {repo_ref:"Sendrast/sendrast-web", limit:5}→[484,478,458,408,407](runsScanned=150 runsTotal=492); the true newest is 492. The pre-fix code on the same input returned[492,491,490,489,488]. Cause:/actions/runsis unordered (?page=1&limit=50first indices[1,311,7,458,379,8,…]); sampling 150 of 492 and sorting gives the max of a random sample.PR/branch/sha lookup returns an older match.
query:"#147"→ run 458; the newest run for PR 147 is 468. Cause:stopWhen: items.some(match)stops at the first page containing any match in an unordered feed. The PR's claim that lookups use the exactrun_numberfilter is true only for run-index refs.Tasks paging drops a job while reporting a healthy, complete result.
{query:"441"}lists onlye2e, withjobsSource:"tasks"anddegraded:[]— run 441 also hastest. Data proof: tasks?page=2ends with run 441e2e(id 12522),?page=3starts with run 441test(id 12521). Cause:stopWhenuses<=(stops exactly at the split run) andresolveJobs' tasks branch never unions the run-page job list. If the dropped job is the failing one,forgejo_ci_failuresnever quotes it — the AC "keep failure-excerpt capability working end-to-end" is defeated.Concerns routed to the same fix
maxAttempt→ quotes attempt #1's log for a job on attempt #2 (reproduced with a fake; latent — the real AC runs are all attempt 1).details.degradedstays[]when jobs came from the run page due to a page-window miss; the "(actions/tasks unavailable)" note text is hardcoded and inaccurate there.forgejo_ci_statusfans out one web request per job index per run to compute amaxAttemptit then discards (measured: 22 requests / 8.7 s for 5 runs on infra-core).page, no row-threshold 500), and the regression is guarded mostly by URL-shape assertions. (A revert experiment does go red — 9 failed / 27 passed — so the suite locks the literal URL, but not the server contract.)Suggestions (follow-up ticket, not this PR): request timeouts, response/log size caps, env-file permission check,
repo_refsegment validation, dead code (durationBetween,buildJobPageUrl,ArtifactInfo,JobSource="none"), no-opconst full = sha ? sha : sha, and thedocs/forgejo-ci.mdexample that labels run index#489as "the PR".Next step: a fresh-context fix worker is producing a follow-up commit on this branch. The PR will be re-reviewed on the delta and merged only if the three blockers are fixed with regression tests that fail on
9db57ba.Follow-up commit — review blockers fixed
Landed
911e0d7on top of9db57ba(fast-forward push, no force-push). All three BLOCKERs (F1–F3) plus concerns F4–F7 are fixed, each with a regression test that fails on9db57ba.F1 — no-query listing returned stale runs (regression)
Root cause:
/actions/runsis unordered andlookupRunssampled only 3 pages (150/492) then sorted the sample.Sendrast/sendrast-web limit:5->484,478,458,408,407;Homelab/infra-coremax1399Sendrast/sendrast-web->[492,491,490,489,488];Homelab/infra-core->[1509,1508,1507,1506,1505]Fix: cover the whole feed (bounded by
total_count) and takemax(index_in_repo).F2 — PR/branch/sha lookup returned an older match
Root cause: early
stopWhenstopped at the first page containing any match in an unordered feed.#147->458;main->311;#157->491,480#147->468;main->492;#157->492Fix: full-feed scan, return the newest match. A run index still uses the exact
run_numberfilter (one request).F3 — tasks paging dropped a job while reporting a healthy complete result
Root cause:
stopWhen <=stopped exactly at the run split across the page boundary, and the tasks branch never unioned run-page jobs.Sendrast/sendrast-web query:441->["e2e"],jobsSource:"tasks",degraded:[]-> ["e2e","test"]Fix: strict
<(fetch one page past the oldest selected run) + union meta-only jobs.Concerns
job-level failure detail unavailable; run status=<x>instead ofNo failed jobs found for run #N.details.degraded(task page window did not cover run N) with an accurate note, instead ofdegraded:[]plus the hardcoded "(actions/tasks unavailable)".maxAttempt, soforgejo_ci_failuresquotes the latest attempt's log (regression test usesdata-attempt-number="2").durationBetween,buildJobPageUrl,ArtifactInfo,JobSource "none"), removed the no-opfull, validatedrepo_refsegments (reject./..and URL metacharacters), encoded path segments, fixed stale docs/comments, and stopped the status path fanning out one web request per job.Tests
tests/regression.test.ts(11 tests): all fail on9db57ba, pass after.tests/fake.tsgains a statefulmodeledForgejo(page-keyed responses, no-page table/500, unordered runs feed, split tasks feed).tests/live.test.tsextended with the F1/F2/F3 live assertions.npm test-> 47 passed / 9 skipped;tsc --noEmit --noUnusedLocals --noUnusedParametersclean;FORGEJO_LIVE=1 vitest run tests/live.test.ts-> 9/9.Deliberately not fixed (r3 concerns, non-blocking)
Request timeouts, response/log size caps, env-file permission warnings, 429 backoff, log redaction. Left for a separate change.
FIX WORKER #2 — delta blocker fixed (
c9ade35)FIX A (blocker) — branch/PR matching returned a run on a different branch.
matchesRunQueryOR'dprettyref === valuewithtitle.includes(value), so the whole-feed newest-match took a run on another branch whose title merely mentioned the queried branch.Ranking is now precision first, recency second:
matchTier()marks an exactprettyref/ exactprNumberOfRunmatch as tier 2 and a loose title substring as tier 1;lookupRuns()keeps only the best tier, then the newest within it.index/shaunchanged.Live before → after (newest returned run, real Forgejo):
liveprettyref "#604", title mentionslive)prettyref === "live")livemain#147FIX B (concern) — a partial task feed was hidden from
degraded. Whenactions/tasksreturned only some of a run's rows, the F3 union completed the list from the run page whilejobsSourcestayed"tasks"anddegradedstayed[].resolveJobs()now reportsfilledFromRunPage; both tools pushtask rows for run N were incomplete; filled from the run pageintodegraded(plus a note). Before:{"degraded":[],"jobsSource":"tasks","jobs":[e2e(tasks),test(run-page)]}; after:degradednon-empty.FIX C (quality). The whole-feed scan test is renamed/rewritten (the early stop is gone); the F6 attempt test returns a distinct body per attempt and asserts the attempt-2 marker; union de-dup keys are normalised (trim+lowercase); and
runsTruncatedis surfaced indetailsplus a text note when themaxPagescap is hit.Verification
npm test→ 52 passed / 11 skipped;tsc --noEmit --noUnusedLocals --noUnusedParametersclean.FORGEJO_LIVE=1 npx vitest run tests/live.test.ts→ 11/11 passed, including the newHomelab/prod live → 1046andHomelab/infra-core live → 1509assertions.911e0d7(FIX A branch + PR precision, FIX Bdegraded, FIX C normalisation + truncation); source delta is911e0d7..c9ade35.Landing review complete at
c9ade35— mergingThree-stage independent review (fresh
pi -p --no-sessionprocesses, primary sources only), two fix cycles, all verified against the live Forgejo.Stage 1 (
9db57ba) — 3 blockers, all reproduced with the post-fix code and one a regression against the pre-fix tool:[484,478,458,408,407]vs true492; the pre-fix tool returned[492,491,490,489,488]);#147-> 458 vs true newest 468);<=exactly at a run split across a page boundary (run 441 -> onlye2e, reported withdegraded: []).Stage 2 (
911e0d7) — F1/F2/F3 fixed and those 11 regression tests verified RED on9db57ba/ GREEN here. But the delta review found a NEW blocker: with the early stop removed, a newer run on a different branch whose title merely contained the queried branch name won (Homelab/prodlive-> run 1163 on#604instead of the true newestliverun 1046).Stage 3 (
c9ade35) —VERDICT: CONCERN(no blockers). Branch/PR matching is now ranked by precision before recency; the 5 new tests are honest (RED on911e0d7, GREEN here); the healthy path is not turned into a false alarm.Independent live verification of this head (landing worker, real registered tool, real server):
Homelab/prodlive[1046, 987, 984, 983, 982]— correct (was 1163)Homelab/infra-corelive[1509, 1508, 1499, 1491, 1490]Sendrast/sendrast-webmain[492, 491, 490, 489, 483]Sendrast/sendrast-web#147[468, 464, 458]— correct (was 458 only)Sendrast/sendrast-webno-query[492, 491, 490, 489, 488], scanned 492/492Homelab/infra-coreno-query[1509, 1508, 1507, 1506, 1505], scanned 401/401["e2e", "test"]— both present (was["e2e"])#489e2e:cancelled,test:success; excerpt still quotes the cancelled-e2e lognpm test52 passed / 11 skipped,tsc --noEmit --noUnusedLocals --noUnusedParametersclean,FORGEJO_LIVE=111/11. CI run #87 green on all 7 jobs.Open concerns carried to HOMELAB-1749 (not blocking)
candidateQueries, so a loose#Ntitle mention can still shadow an available exact run-index match. Pre-existing in all three commits (identical result on9db57ba/911e0d7), no live repo currently triggers it, and#N-as-PR is the documented primary reading — so it merges, but the README's precision invariant should be read as per-interpretation until this is fixed.title.includes()fallback is very broad when a repo has zero exact matches:Sendrast/sendrast-weblivereturns[488,417,416,406,403](488 matched only because"outlive".includes("live")). Worth a token-boundary match or an explicit "no exact branch match" outcome.forgejo_ci_failuresdoes not surfacerunsScanned/runsTotal/runsTruncated(onlyforgejo_ci_statusdoes), so at themaxPagescap it could silently operate on a truncated "newest". Unreachable today (prod 833 / infra-core 401 / sendrast 492 runs).jobKey()lowercasing can merge two genuinely case-distinct jobs.resolveRunFromListis production-dead (test-only helper).modeledForgejo's no-page/blank-500 branch is still not driven by any test.Plus the earlier suggestion-tier items (request timeouts, response/log size caps, env-file permission check, 429 backoff, log redaction) — all tracked in HOMELAB-1749.
LANDED — PR #48 merged
Merge SHA:
3d3171015b4f4428dd00a1fbd983f9cae08ccddb(merge commit onmain, first parent17ecc90)Branch:
plane/HOMELAB-1736-forgejo-cikept (not deleted, matching repo convention)Landed commits:
b1aff66(salvage),9db57ba(fix 1),911e0d7(review blockers),c9ade35(delta blocker)CI: run #87 green on all 7 jobs pre-merge; run #88 on
mainpost-mergeReview: 3 independent fresh-context passes + 2 fix cycles; final verdict
CONCERN(no open blockers)Live checkout reconciliation
~/.pi/agent/extensionshadforgejo-ci/as an untracked dir, so a plain ff-merge would have aborted ("untracked files would be overwritten"). Executed:b1aff66:forgejo-ci/: 5/5 byte-identical(
ea98ffac… README.md,341a4105… forgejo-ci.env.example,cf20ff56… forgejo.ts,ce96e86e… index.ts,70b7b80b… types.ts). Re-verified immediately before the delete. A tarball backup was taken first as belt-and-braces (the content is also in git atb1aff66).forgejo-ci/(behind that gate).git merge --ff-only forgejo/main→17ecc90..3d31710, clean fast-forward.forgejo-ci/now exists as 13 tracked files at HEAD; all 12 non-lockfile files byte-matchc9ade35, the reviewed commit.pi-swarm-suite/**,pi-skills/**) and the 2 untracked dirs (pi-google-docs/skills/,probe-hasui/) are untouched — the onlygit statusdelta versus before is the removal of?? forgejo-ci/. HOMELAB-1740 owns the rest.Post-merge tool verification
Forgejo API 500 on …/actions/tasks?limit=100— expected: extensions are loaded at session start, so the running process holds the pre-fix code in memory. No session was restarted. Running sessions keep old code until they restart; new sessions get the fix.forgejo_ci_status Homelab/infra-core→ no 500; runs #1509/#1508/#1507 withbuild-and-push success 1m57s/28s/1m39s(and 1509 is the correct newest — the pre-fix tool's max was 1399).forgejo_ci_failures Sendrast/sendrast-web 489→ red run with the cancellede2ejob (index 1, 9m49s, #1:cancelled) and a real log excerpt anchored oncontext canceled/🏁 Job failed.~/repos/pi-extensions(the other checkout) is unchanged — byte-identicalgit statusto its pre-merge state. Note it is stale/diverged (8fe0370, a local-only commit absent from Forgejo, 2 PRs behind) — flagged, not touched.GitOps continuation
pi-extensionshas no image build or manifest pipeline — it is consumed as a source checkout (.forgejo/workflows/ci.yamlruns CI only). The continuation is therefore the checkout reconciliation above, which is complete and verified. Post-merge CI onmainis run #88.Residual (non-blocking) findings → HOMELAB-1749
rankMatchestiering is applied per candidate interpretation, not acrosscandidateQueries(a loose#Ntitle mention can still shadow an exact run-index match; pre-existing in all commits, no live repo triggers it); the loosetitle.includes()branch fallback is broad when a repo has zero exact matches;forgejo_ci_failuresomitsrunsScanned/runsTotal/runsTruncated;jobKey()lowercasing can merge case-distinct jobs;resolveRunFromListis production-dead. Plus the suggestion tier: request timeouts, response/log size caps, env-file permission check, 429 backoff, log redaction, and drivingmodeledForgejo's no-page/500 branch from a test.