Zayin/Signage
3
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases 4 Wiki Activity

ci: the integration job removes its stacks even when the runner kills it #8

Merged
Zayin_Admin merged 3 commits from ci/stack-cleanup into main 2026-10-06 02:58:11 +00:00
Conversation 1 Commits 3 Files Changed 4
+73 -2
ekklesia-bot commented 2026-10-06 02:05:16 +00:00
Member
Copy Link
Copy Source

ci: the integration job removes its stacks even when the runner kills it

The platform's rule from Ekklesia #60, applied to the one Signage job that starts Docker stacks. A runner that kills a job (a cancel, or its own time limit; Gitea 1.24 ignores timeout-minutes) never lets that job's cleanup trap run. A platform migration stack leaked for two days that way.

Changes

  • The integration job has its own EKKLESIA_CI_JOB_KEY.
  • It runs ci-sweep.sh --stale (from the assembled platform) before starting anything.
  • It runs ci-sweep.sh --job in a final if: always() step. That step tolerates a failed assembly, since nothing was started then.
  • ci/integration.sh names and labels its image as a CI stack of this run and job, with the time CI built it.
  • ci/check-cleanup.sh (run by lint) checks all of that.
  • ci/platform.env pins 874deab, Ekklesia #60's merge on platform main, which adds the sweep. #60 was rebase-merged, so its PR head 00e5441 is not on main. The sweep, its helper and the smoke script are byte-identical between the two, and no package file changed. The assembled lockfile is unchanged and still passes a frozen install.

Commits

  1. Failing check first: ci/check-cleanup.sh, wired into lint. On main it fails on all five counts.
  2. The fix.

Verified locally

These ran in the exact CI layout: build/workspace assembled from Ekklesia 00e5441 plus this branch (the sweep there is byte-identical to 874deab).

  • Killed mid-run: SIGKILL of the integration script's whole process group (verified: 0 processes left). 20 s later 11 resources were still there: 2 containers, 4 networks, 4 volumes, and the image. ci-sweep.sh --job removed exactly those 11. Nothing is left.
  • Normal run: ci/integration.sh passes (pair, restart, two-pages). The job sweep afterwards removes 0, because the script's own trap had already cleaned up.
  • ci/check-cleanup.sh: all checks pass.

Pin

Ekklesia #60 is merged, and the pin is on platform main, so ci/assemble.sh accepts it.

Note

This branch is independent of #4–#6 and touches none of their files. Their rebases onto the new main are separate.

🤖 Generated with Claude Code

## ci: the integration job removes its stacks even when the runner kills it The platform's rule from Ekklesia #60, applied to the one Signage job that starts Docker stacks. A runner that kills a job (a cancel, or its own time limit; Gitea 1.24 ignores `timeout-minutes`) never lets that job's cleanup trap run. A platform migration stack leaked for two days that way. ### Changes - The `integration` job has its own `EKKLESIA_CI_JOB_KEY`. - It runs `ci-sweep.sh --stale` (from the assembled platform) before starting anything. - It runs `ci-sweep.sh --job` in a final `if: always()` step. That step tolerates a failed assembly, since nothing was started then. - `ci/integration.sh` names and labels its image as a CI stack of this run and job, with the time CI built it. - `ci/check-cleanup.sh` (run by `lint`) checks all of that. - `ci/platform.env` pins `874deab`, Ekklesia #60's merge on platform `main`, which adds the sweep. #60 was rebase-merged, so its PR head `00e5441` is not on `main`. The sweep, its helper and the smoke script are byte-identical between the two, and no package file changed. The assembled lockfile is unchanged and still passes a frozen install. ### Commits 1. **Failing check first:** `ci/check-cleanup.sh`, wired into `lint`. On `main` it fails on all five counts. 2. **The fix.** ### Verified locally These ran in the exact CI layout: `build/workspace` assembled from Ekklesia `00e5441` plus this branch (the sweep there is byte-identical to `874deab`). - **Killed mid-run:** SIGKILL of the integration script's whole process group (verified: 0 processes left). 20 s later **11 resources** were still there: 2 containers, 4 networks, 4 volumes, and the image. `ci-sweep.sh --job` removed exactly those 11. Nothing is left. - **Normal run:** `ci/integration.sh` passes (pair, restart, `two-pages`). The job sweep afterwards removes **0**, because the script's own trap had already cleaned up. - `ci/check-cleanup.sh`: all checks pass. ### Pin Ekklesia #60 is merged, and the pin is on platform `main`, so `ci/assemble.sh` accepts it. ### Note This branch is independent of #4–#6 and touches none of their files. Their rebases onto the new `main` are separate. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
ekklesia-bot added 2 commits 2026-10-06 02:05:16 +00:00
test(ci): the integration job must clean up its stacks even when killed cfa4628f8c 
The platform leaked a whole CI stack for two days: a job the runner
kills never runs its cleanup trap. The platform's rule now applies here:
the job that starts Docker stacks (integration) has its own
EKKLESIA_CI_JOB_KEY, sweeps stale CI stacks first, removes its own in a
final if: always() step, and labels the image it builds.

ci/check-cleanup.sh checks that, and the lint job runs it. It fails
today on all five counts.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
ci: the integration job removes its stacks even when the runner kills it
CI / typecheck (pull_request) Failing after 3s
Details
CI / test (pull_request) Failing after 4s
Details
CI / build (pull_request) Failing after 4s
Details
CI / integration (pull_request) Skipped
Details
CI / lint (pull_request) Failing after 3s
Details
cbf2a8ee39 
The platform's rule (its ci-sweep.sh, docs/development/security.md),
applied to the one Signage job that starts Docker stacks:
- it has its own EKKLESIA_CI_JOB_KEY;
- it sweeps stale CI stacks from the runner before it starts;
- it removes its own stacks in a final `if: always()` step, which
  tolerates a failed assembly, since nothing was started then;
- the image ci/integration.sh builds is named and labeled as a CI stack
  of this run and job, with the time CI built it, so the sweep finds it.

ci/platform.env moves to the platform commit that adds the sweep
(Ekklesia #60). The assembled lockfile is unchanged, and still passes a
frozen install.

Verified: SIGKILL of the integration script's whole process group
mid-run left 11 resources (2 containers, 4 networks, 4 volumes and the
image). `ci-sweep.sh --job` removed exactly those 11. A normal run
passes, and the job sweep after it finds nothing.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Zayin_Admin added 1 commit 2026-10-06 02:20:00 +00:00
ci: pin the platform at the merged sweep, 874deab on main
CI / lint (pull_request) Successful in 14s
Details
CI / typecheck (pull_request) Successful in 14s
Details
CI / test (pull_request) Successful in 19s
Details
CI / build (pull_request) Successful in 18s
Details
CI / integration (pull_request) Successful in 3m49s
Details
6869ae433a 
Ekklesia #60 was rebase-merged, so the PR head 00e5441 never became part
of main, and ci/assemble.sh rightly refuses it. 874deab is #60's merge on
main. ci-sweep.sh, ci-project.sh and smoke-deploy.sh are byte-identical
to the old pin, no package file changed, and the assembled lockfile
still passes a frozen install.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Zayin_Admin merged commit e6c6553afc into main 2026-10-06 02:58:11 +00:00
Zayin_Admin deleted branch ci/stack-cleanup 2026-10-06 02:58:11 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Zayin_Admin
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
Joey Zayin_Admin ekklesia-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Zayin/Signage#8