A migration-smoke stack (PostgreSQL and SeaweedFS) ran on the shared
runner for two days. Every CI script removes its stack in an EXIT trap,
but a trap never runs when the runner kills the job container: a
cancelled job, or the runner's own time limit (Gitea 1.24 ignores
timeout-minutes). Reproduced: SIGKILL of the infrastructure smoke
mid-run leaves exactly that stack running.
Cleanup now has three layers:
- the script's own trap, unchanged;
- a final `if: always()` step in every job that starts stacks
(infrastructure-smoke, migration-smoke, image-build, deploy-smoke)
runs `scripts/ci-sweep.sh --job`, which removes everything that job
started at any age;
- a first step in each of those jobs runs `scripts/ci-sweep.sh --stale`,
which removes CI stacks older than three hours: what a killed job left
before it could clean up.
How stacks are found is defined once, in scripts/lib/ci-project.sh:
- every CI Compose project is named `ekklesia-ci-<run>-<attempt>-<job
key>-<kind>-<unique>`, and Compose labels its containers, networks and
volumes with that name;
- the deploy smoke labels everything it starts outside Compose:
browser, kiosk and driver containers, the kiosk volume, built images;
- images also record when CI built them. An image's own creation time
can be its base's (BuildKit, metadata-only builds), and the sweep
removed a brand-new image as "old" until it stopped trusting that.
The job key is per matrix leg, so one topology's cleanup cannot remove
the other's running stack. The sweep never touches anything outside the
ekklesia-ci- prefix, disconnects rather than removes a job container
attached to a stack's network, and refuses --job outside CI. Network
creation times are read as RFC 3339; Docker's default form was
unparseable.
scripts/test-ci-sweep.sh proves all of that against real Docker, and
CI runs it before relying on it. security.md and testing.md document
the rule, and that it complements, not replaces, a runner-side reaper.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A migration-smoke stack (PostgreSQL and SeaweedFS) leaked on the shared
runner and ran for two days, until it was removed by hand. Every CI
script already tears its stack down in an EXIT/INT/TERM trap, but a
trap never runs when the runner kills the job container: a cancelled
job, or the runner's own time limit, since Gitea 1.24 ignores
timeout-minutes.
Static checks (tests/ci-scripts.test.ts):
- every stack script names its project through one helper: the reserved
ekklesia-ci- prefix, run, attempt, job key, and a unique suffix;
- everything the deploy smoke creates outside Compose carries the CI
label;
- every workflow job that starts stacks sweeps stale CI stacks first,
and removes its own in a final if: always() step. Each matrix leg has
its own key, so one cannot remove the other's running stack.
Behaviour against real Docker (scripts/test-ci-sweep.sh):
- --job removes this job's containers, network, volume, image and
staging directory, and leaves other jobs' and non-CI stacks alone;
- --job disconnects, never removes, a job container on the stack's
network, and refuses to run outside CI;
- --stale spares young stacks and removes old ones, never anything
outside the prefix.
Fails today: there is no helper, no sweep, no label and no step.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
