fix(ci): nothing a CI job starts outlives it, even when the runner kills the job #60

2026-10-06T01:57:29Z

ekklesia-bot commented

2026-10-06 01:57:29 +00:00

fix(ci): nothing a CI job starts outlives it, even when the runner kills the job

Cause of the leaked ekklesia-migration-111-1-migration-smoke-835 stack (PostgreSQL + SeaweedFS, two days on the runner):

Every CI script already removes its stack in an EXIT/INT/TERM trap.
A trap never runs when the runner kills the job container: a cancelled job, or the runner's own time limit, since Gitea 1.24 ignores timeout-minutes.
Reproduced: SIGKILL of smoke-infrastructure.sh's process group mid-run leaves exactly that pair of containers running, with their network and volumes.
The leaked name lacks the random suffix, so that run predates #52.

Three layers now

Layer	Covers
The script's own trap (unchanged)	every ordinary failure
A final `if: always()` step, `scripts/ci-sweep.sh --job`, in every job that starts stacks (`infrastructure-smoke`, `migration-smoke`, `image-build`, `deploy-smoke`)	a failed step, a script that died; everything this job started, at any age
A first step, `scripts/ci-sweep.sh --stale`, in the same jobs	whatever a killed job left. Removes CI stacks older than 3 h; no job runs that long

How stacks are found (`scripts/lib/ci-project.sh`)

Every CI Compose project is named ekklesia-ci-<run>-<attempt>-<job key>-<kind>-<unique>, and Compose labels its containers, networks and volumes with that name.
The deploy smoke labels everything it starts outside Compose with org.riverofglorymn.ekklesia.ci.project: browser, kiosk and driver containers, the kiosk volume, built images.
Images also record when CI built them. An image's own creation time can be its base's: BuildKit gives a metadata-only image its base's time, and so do reproducible builds. Before this, the stale sweep removed a brand-new image as "old". It now judges images only by that label.
EKKLESIA_CI_JOB_KEY is per matrix leg (deploy-smoke-${{ matrix.topology }}), so one topology's cleanup can't remove the other's running stack.

What the sweep never does

touch anything outside the ekklesia-ci- prefix, at any age;
remove a job container attached to a stack's network (it is disconnected);
run --job outside CI, where its prefix would match a developer's own stacks;
fail a job: it is best effort, and reports what it couldn't remove.

Also fixed in the sweep: Docker reports a network's creation time in a form date can't parse, so it is now read as RFC 3339.

Commits

Failing tests first.
- Static (tests/ci-scripts.test.ts): naming through the helper, labels on everything created outside Compose, and stale-before / own-cleanup-last in every stack job with a per-leg key.
- Behaviour against real Docker (scripts/test-ci-sweep.sh): what --job and --stale remove and what they leave alone, including another job's stack, a non-CI stack, a connected job container, and staging directories on the Docker host.
The fix: the helper, the sweep, the three scripts, the workflow (infrastructure-smoke also runs the sweep test before relying on it), and docs. security.md says plainly this complements, not replaces, a runner-side reaper: a killed stack is removed only when another sweeping job runs on that runner.

Verification

pnpm verify: passes.
scripts/test-ci-sweep.sh: every check passes.
Killed-job reproduction: the script runs in its own session, and that whole process group gets SIGKILL (verified: 0 processes left). 20 s later the stack is still running, so no trap ran. ci-sweep.sh --job then removed all of it: 2 containers, the network, and 2 volumes. Nothing is left. (A first attempt didn't actually kill the script, because setsid forked. Those numbers are not used here.)
Normal runs, no leftovers: scripts/smoke-infrastructure.sh passes, pnpm test:migrations:integration passes (306 tests), scripts/smoke-deploy.sh (same-origin) passes.

Same rule elsewhere

Signage's integration job: a Signage PR follows, re-pinned to this PR's head once it merges.
The module harness: ADR 0047 (#59) is amended.

🤖 Generated with Claude Code

## fix(ci): nothing a CI job starts outlives it, even when the runner kills the job **Cause of the leaked `ekklesia-migration-111-1-migration-smoke-835` stack** (PostgreSQL + SeaweedFS, two days on the runner): - Every CI script already removes its stack in an `EXIT`/`INT`/`TERM` trap. - A trap never runs when the runner kills the job container: a **cancelled job**, or the **runner's own time limit**, since Gitea 1.24 ignores `timeout-minutes`. - Reproduced: SIGKILL of `smoke-infrastructure.sh`'s process group mid-run leaves exactly that pair of containers running, with their network and volumes. - The leaked name lacks the random suffix, so that run predates #52. ### Three layers now | Layer | Covers | |---|---| | The script's own trap (unchanged) | every ordinary failure | | A final `if: always()` step, `scripts/ci-sweep.sh --job`, in **every job that starts stacks** (`infrastructure-smoke`, `migration-smoke`, `image-build`, `deploy-smoke`) | a failed step, a script that died; everything this job started, at any age | | A first step, `scripts/ci-sweep.sh --stale`, in the same jobs | whatever a **killed** job left. Removes CI stacks older than 3 h; no job runs that long | ### How stacks are found (`scripts/lib/ci-project.sh`) - Every CI Compose project is named `ekklesia-ci-<run>-<attempt>-<job key>-<kind>-<unique>`, and Compose labels its containers, networks and volumes with that name. - The deploy smoke labels everything it starts **outside** Compose with `org.riverofglorymn.ekklesia.ci.project`: browser, kiosk and driver containers, the kiosk volume, built images. - **Images also record when CI built them.** An image's own creation time can be its base's: BuildKit gives a metadata-only image its base's time, and so do reproducible builds. Before this, the stale sweep removed a brand-new image as "old". It now judges images only by that label. - `EKKLESIA_CI_JOB_KEY` is per matrix leg (`deploy-smoke-${{ matrix.topology }}`), so one topology's cleanup can't remove the other's running stack. ### What the sweep never does - touch anything outside the `ekklesia-ci-` prefix, at any age; - remove a job container attached to a stack's network (it is disconnected); - run `--job` outside CI, where its prefix would match a developer's own stacks; - fail a job: it is best effort, and reports what it couldn't remove. **Also fixed in the sweep:** Docker reports a network's creation time in a form `date` can't parse, so it is now read as RFC 3339. ### Commits 1. **Failing tests first.** - Static (`tests/ci-scripts.test.ts`): naming through the helper, labels on everything created outside Compose, and stale-before / own-cleanup-last in every stack job with a per-leg key. - Behaviour against real Docker (`scripts/test-ci-sweep.sh`): what `--job` and `--stale` remove and what they leave alone, including another job's stack, a non-CI stack, a connected job container, and staging directories on the Docker host. 2. **The fix:** the helper, the sweep, the three scripts, the workflow (`infrastructure-smoke` also runs the sweep test before relying on it), and docs. `security.md` says plainly this **complements, not replaces**, a runner-side reaper: a killed stack is removed only when another sweeping job runs on that runner. ### Verification - `pnpm verify`: passes. - `scripts/test-ci-sweep.sh`: every check passes. - **Killed-job reproduction:** the script runs in its own session, and that whole process group gets SIGKILL (verified: 0 processes left). 20 s later the stack is still running, so no trap ran. `ci-sweep.sh --job` then removed all of it: 2 containers, the network, and 2 volumes. Nothing is left. (A first attempt didn't actually kill the script, because `setsid` forked. Those numbers are not used here.) - Normal runs, no leftovers: `scripts/smoke-infrastructure.sh` passes, `pnpm test:migrations:integration` passes (306 tests), `scripts/smoke-deploy.sh` (same-origin) passes. ### Same rule elsewhere - **Signage's `integration` job:** a Signage PR follows, re-pinned to this PR's head once it merges. - **The module harness:** ADR 0047 (#59) is amended. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Zayin_Admin added 2 commits 2026-10-06 02:01:04 +00:00

test(ci): every CI stack is cleaned up, even when its job is killed 703d92436b

A migration-smoke stack (PostgreSQL and SeaweedFS) leaked on the shared
runner and ran for two days, until it was removed by hand. Every CI
script already tears its stack down in an EXIT/INT/TERM trap, but a
trap never runs when the runner kills the job container: a cancelled
job, or the runner's own time limit, since Gitea 1.24 ignores
timeout-minutes.

Static checks (tests/ci-scripts.test.ts):
- every stack script names its project through one helper: the reserved
  ekklesia-ci- prefix, run, attempt, job key, and a unique suffix;
- everything the deploy smoke creates outside Compose carries the CI
  label;
- every workflow job that starts stacks sweeps stale CI stacks first,
  and removes its own in a final if: always() step. Each matrix leg has
  its own key, so one cannot remove the other's running stack.

Behaviour against real Docker (scripts/test-ci-sweep.sh):
- --job removes this job's containers, network, volume, image and
  staging directory, and leaves other jobs' and non-CI stacks alone;
- --job disconnects, never removes, a job container on the stack's
  network, and refuses to run outside CI;
- --stale spares young stacks and removes old ones, never anything
  outside the prefix.

Fails today: there is no helper, no sweep, no label and no step.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(ci): nothing a CI job starts outlives it, even when the runner kills the job

CI / verify (pull_request) Successful in 57s

Details

CI / infrastructure-smoke (pull_request) Successful in 28s

Details

CI / migration-smoke (pull_request) Successful in 1m49s

Details

CI / image-build (pull_request) Successful in 19s

Details

CI / deploy-smoke (same-origin) (pull_request) Successful in 48s

Details

CI / deploy-smoke (split) (pull_request) Successful in 53s

Details

62951975da

A migration-smoke stack (PostgreSQL and SeaweedFS) ran on the shared
runner for two days. Every CI script removes its stack in an EXIT trap,
but a trap never runs when the runner kills the job container: a
cancelled job, or the runner's own time limit (Gitea 1.24 ignores
timeout-minutes). Reproduced: SIGKILL of the infrastructure smoke
mid-run leaves exactly that stack running.

Cleanup now has three layers:
- the script's own trap, unchanged;
- a final `if: always()` step in every job that starts stacks
  (infrastructure-smoke, migration-smoke, image-build, deploy-smoke)
  runs `scripts/ci-sweep.sh --job`, which removes everything that job
  started at any age;
- a first step in each of those jobs runs `scripts/ci-sweep.sh --stale`,
  which removes CI stacks older than three hours: what a killed job left
  before it could clean up.

How stacks are found is defined once, in scripts/lib/ci-project.sh:
- every CI Compose project is named `ekklesia-ci-<run>-<attempt>-<job
  key>-<kind>-<unique>`, and Compose labels its containers, networks and
  volumes with that name;
- the deploy smoke labels everything it starts outside Compose:
  browser, kiosk and driver containers, the kiosk volume, built images;
- images also record when CI built them. An image's own creation time
  can be its base's (BuildKit, metadata-only builds), and the sweep
  removed a brand-new image as "old" until it stopped trusting that.

The job key is per matrix leg, so one topology's cleanup cannot remove
the other's running stack. The sweep never touches anything outside the
ekklesia-ci- prefix, disconnects rather than removes a job container
attached to a stack's network, and refuses --job outside CI. Network
creation times are read as RFC 3339; Docker's default form was
unparseable.

scripts/test-ci-sweep.sh proves all of that against real Docker, and
CI runs it before relying on it. security.md and testing.md document
the rule, and that it complements, not replaces, a runner-side reaper.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Zayin_Admin force-pushed fix/ci-stack-cleanup from 00e544125b to 62951975da

2026-10-06 02:01:04 +00:00

Compare

Zayin_Admin approved these changes

2026-10-06 02:11:48 +00:00

Zayin_Admin merged commit 874deabea3 into main

2026-10-06 02:11:53 +00:00

Zayin_Admin deleted branch fix/ci-stack-cleanup

2026-10-06 02:11:53 +00:00

Zayin_Admin referenced this issue from a commit

2026-10-06 03:56:37 +00:00

chore(release): prepare 0.1.2