A migration-smoke stack (PostgreSQL and SeaweedFS) leaked on the shared
runner and ran for two days, until it was removed by hand. Every CI
script already tears its stack down in an EXIT/INT/TERM trap, but a
trap never runs when the runner kills the job container: a cancelled
job, or the runner's own time limit, since Gitea 1.24 ignores
timeout-minutes.
Static checks (tests/ci-scripts.test.ts):
- every stack script names its project through one helper: the reserved
ekklesia-ci- prefix, run, attempt, job key, and a unique suffix;
- everything the deploy smoke creates outside Compose carries the CI
label;
- every workflow job that starts stacks sweeps stale CI stacks first,
and removes its own in a final if: always() step. Each matrix leg has
its own key, so one cannot remove the other's running stack.
Behaviour against real Docker (scripts/test-ci-sweep.sh):
- --job removes this job's containers, network, volume, image and
staging directory, and leaves other jobs' and non-CI stacks alone;
- --job disconnects, never removes, a job container on the stack's
network, and refuses to run outside CI;
- --stale spares young stacks and removes old ones, never anything
outside the prefix.
Fails today: there is no helper, no sweep, no label and no step.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
PR #51's migration-smoke failed 0.7 s after the build: the suite pinned
PostgreSQL to an ephemeral port but inherited S3_PORT=8333 from
.env.example, so concurrent runs on one runner (Gitea ignores workflow
concurrency) contended for one host port. Compose's own error was
discarded, so the failure said only that the services "did not become
ready".
Isolation, from an audit of every script CI runs:
- test-migrations-integration.sh: S3_PORT=0, like POSTGRES_PORT=0.
- test-migrations-integration.sh and smoke-infrastructure.sh: project names
gain a random suffix; run id plus PID alone is not unique when job
containers share low PIDs.
- smoke-deploy.sh: the edge /24 is chosen from those no network on the
daemon uses, and the first `up` chooses again when a concurrent run wins
a race for the same subnet (Docker's address-pool overlap), up to five
times.
- CI image-build: the image tag includes the run id and attempt, so a pull
request and a push of the same commit cannot remove each other's image.
- Already isolated: every other published port (ephemeral), the TLS
suite's container and network (random), smoke staging directories and
image tags (random).
Diagnostics: scripts/lib/compose-diagnostics.sh prints the failed `up`
command's output, `docker compose ps`, and the last 40 log lines of each
service, all redacted (supplied literals, credential-looking NAME=value
pairs, URL user information, Authorization headers) with terminal control
sequences stripped. All three scripts use it at every readiness check; the
infrastructure smoke's unredacted full-log dump is replaced by it.
tests/ci-scripts.test.ts pins ephemeral ports, unique project names, the
subnet choice, the CI image tag, the diagnostics, and the redaction itself
(run through bash).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
