A migration-smoke stack (PostgreSQL and SeaweedFS) ran on the shared
runner for two days. Every CI script removes its stack in an EXIT trap,
but a trap never runs when the runner kills the job container: a
cancelled job, or the runner's own time limit (Gitea 1.24 ignores
timeout-minutes). Reproduced: SIGKILL of the infrastructure smoke
mid-run leaves exactly that stack running.
Cleanup now has three layers:
- the script's own trap, unchanged;
- a final `if: always()` step in every job that starts stacks
(infrastructure-smoke, migration-smoke, image-build, deploy-smoke)
runs `scripts/ci-sweep.sh --job`, which removes everything that job
started at any age;
- a first step in each of those jobs runs `scripts/ci-sweep.sh --stale`,
which removes CI stacks older than three hours: what a killed job left
before it could clean up.
How stacks are found is defined once, in scripts/lib/ci-project.sh:
- every CI Compose project is named `ekklesia-ci-<run>-<attempt>-<job
key>-<kind>-<unique>`, and Compose labels its containers, networks and
volumes with that name;
- the deploy smoke labels everything it starts outside Compose:
browser, kiosk and driver containers, the kiosk volume, built images;
- images also record when CI built them. An image's own creation time
can be its base's (BuildKit, metadata-only builds), and the sweep
removed a brand-new image as "old" until it stopped trusting that.
The job key is per matrix leg, so one topology's cleanup cannot remove
the other's running stack. The sweep never touches anything outside the
ekklesia-ci- prefix, disconnects rather than removes a job container
attached to a stack's network, and refuses --job outside CI. Network
creation times are read as RFC 3339; Docker's default form was
unparseable.
scripts/test-ci-sweep.sh proves all of that against real Docker, and
CI runs it before relying on it. security.md and testing.md document
the rule, and that it complements, not replaces, a runner-side reaper.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
PR #51's migration-smoke failed 0.7 s after the build: the suite pinned
PostgreSQL to an ephemeral port but inherited S3_PORT=8333 from
.env.example, so concurrent runs on one runner (Gitea ignores workflow
concurrency) contended for one host port. Compose's own error was
discarded, so the failure said only that the services "did not become
ready".
Isolation, from an audit of every script CI runs:
- test-migrations-integration.sh: S3_PORT=0, like POSTGRES_PORT=0.
- test-migrations-integration.sh and smoke-infrastructure.sh: project names
gain a random suffix; run id plus PID alone is not unique when job
containers share low PIDs.
- smoke-deploy.sh: the edge /24 is chosen from those no network on the
daemon uses, and the first `up` chooses again when a concurrent run wins
a race for the same subnet (Docker's address-pool overlap), up to five
times.
- CI image-build: the image tag includes the run id and attempt, so a pull
request and a push of the same commit cannot remove each other's image.
- Already isolated: every other published port (ephemeral), the TLS
suite's container and network (random), smoke staging directories and
image tags (random).
Diagnostics: scripts/lib/compose-diagnostics.sh prints the failed `up`
command's output, `docker compose ps`, and the last 40 log lines of each
service, all redacted (supplied literals, credential-looking NAME=value
pairs, URL user information, Authorization headers) with terminal control
sequences stripped. All three scripts use it at every readiness check; the
infrastructure smoke's unredacted full-log dump is replaced by it.
tests/ci-scripts.test.ts pins ephemeral ports, unique project names, the
subnet choice, the CI image tag, the diagnostics, and the redaction itself
(run through bash).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
