fix(ci): nothing a CI job starts outlives it, even when the runner kills the job
CI / verify (pull_request) Successful in 57s
CI / infrastructure-smoke (pull_request) Successful in 28s
CI / migration-smoke (pull_request) Successful in 1m49s
CI / image-build (pull_request) Successful in 19s
CI / deploy-smoke (same-origin) (pull_request) Successful in 48s
CI / deploy-smoke (split) (pull_request) Successful in 53s
CI / verify (pull_request) Successful in 57s
CI / infrastructure-smoke (pull_request) Successful in 28s
CI / migration-smoke (pull_request) Successful in 1m49s
CI / image-build (pull_request) Successful in 19s
CI / deploy-smoke (same-origin) (pull_request) Successful in 48s
CI / deploy-smoke (split) (pull_request) Successful in 53s
A migration-smoke stack (PostgreSQL and SeaweedFS) ran on the shared runner for two days. Every CI script removes its stack in an EXIT trap, but a trap never runs when the runner kills the job container: a cancelled job, or the runner's own time limit (Gitea 1.24 ignores timeout-minutes). Reproduced: SIGKILL of the infrastructure smoke mid-run leaves exactly that stack running. Cleanup now has three layers: - the script's own trap, unchanged; - a final `if: always()` step in every job that starts stacks (infrastructure-smoke, migration-smoke, image-build, deploy-smoke) runs `scripts/ci-sweep.sh --job`, which removes everything that job started at any age; - a first step in each of those jobs runs `scripts/ci-sweep.sh --stale`, which removes CI stacks older than three hours: what a killed job left before it could clean up. How stacks are found is defined once, in scripts/lib/ci-project.sh: - every CI Compose project is named `ekklesia-ci-<run>-<attempt>-<job key>-<kind>-<unique>`, and Compose labels its containers, networks and volumes with that name; - the deploy smoke labels everything it starts outside Compose: browser, kiosk and driver containers, the kiosk volume, built images; - images also record when CI built them. An image's own creation time can be its base's (BuildKit, metadata-only builds), and the sweep removed a brand-new image as "old" until it stopped trusting that. The job key is per matrix leg, so one topology's cleanup cannot remove the other's running stack. The sweep never touches anything outside the ekklesia-ci- prefix, disconnects rather than removes a job container attached to a stack's network, and refuses --job outside CI. Network creation times are read as RFC 3339; Docker's default form was unparseable. scripts/test-ci-sweep.sh proves all of that against real Docker, and CI runs it before relying on it. security.md and testing.md document the rule, and that it complements, not replaces, a runner-side reaper. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
committed by
The_River
co-authored by
Claude Opus 5.5
parent
703d92436b
commit
62951975da
@@ -746,6 +746,33 @@ The declarations are kept deliberately: they are already correct for a supported
|
||||
for GitHub Actions, and keeping them means the upgrade does not require re-deriving the intent. They
|
||||
are documented as ignored, not removed.
|
||||
|
||||
### Disposable stacks never outlive their job
|
||||
|
||||
Every CI script removes its own Compose stack in an `EXIT` trap, but a trap never runs when the runner
|
||||
kills the job container: a cancelled job, or the runner's own time limit. One such stack ran on the
|
||||
shared runner for two days. So stack cleanup has three layers:
|
||||
|
||||
- **The script's own trap**, for every ordinary failure.
|
||||
- **A final `if: always()` step** in every job that starts stacks: `scripts/ci-sweep.sh --job` removes
|
||||
everything that job started, at any age, even after a failed step.
|
||||
- **A sweep before every such job starts**: `scripts/ci-sweep.sh --stale` removes anything a CI job
|
||||
left more than three hours ago, which catches whatever a killed job could not clean up itself.
|
||||
|
||||
A stack is found by name and label, never by guesswork (`scripts/lib/ci-project.sh`):
|
||||
|
||||
- every CI Compose project is named `ekklesia-ci-<run>-<attempt>-<job key>-…`, and Compose labels
|
||||
its containers, networks, and volumes with that name;
|
||||
- anything a script starts outside Compose carries `org.riverofglorymn.ekklesia.ci.project`;
|
||||
- images also carry the time CI built them, because an image's own creation time can be its base's.
|
||||
|
||||
The job key is per matrix leg, so one leg's cleanup cannot match another's running stack. Nothing
|
||||
outside the `ekklesia-ci-` prefix is ever removed. A job container attached to a stack's network is
|
||||
disconnected, never removed.
|
||||
|
||||
**This is not a substitute for a runner-side reaper.** A stack killed with its job is removed only
|
||||
when another job that sweeps runs on the same runner. Bounded execution and teardown are still the
|
||||
runner host's to enforce, as above.
|
||||
|
||||
### Runner topology is not verified
|
||||
|
||||
**The runner topology is not verified in this repository and is not documented here as though it
|
||||
|
||||
@@ -159,6 +159,12 @@ also receives a safely named disposable database. Direct host runs use a Docker-
|
||||
port bound only to `127.0.0.1`; a containerized Gitea job joins the isolated Compose network and
|
||||
uses the PostgreSQL service name.
|
||||
|
||||
In CI, a stack whose job was killed before its own cleanup ran is still removed: by that job's final
|
||||
`if: always()` step, or by the stale sweep before a later job starts (`scripts/ci-sweep.sh`, see
|
||||
[Disposable stacks never outlive their job](security.md#disposable-stacks-never-outlive-their-job)).
|
||||
`bash scripts/test-ci-sweep.sh` proves what the sweep removes and what it leaves alone, against real
|
||||
Docker resources.
|
||||
|
||||
The live suite proves fresh read-only planning, initial and incremental application, genuine
|
||||
no-op behavior, exact applied checksums and history, edited or corrupt history rejection,
|
||||
whole-batch DDL rollback, Drizzle breakpoint execution, real advisory-lock concurrency and
|
||||
|
||||
Reference in New Issue
Block a user