ekklesiaandClaude Opus 5.5
00e544125b
CI / verify (pull_request) Successful in 59s
CI / infrastructure-smoke (pull_request) Canceled after 0s
CI / migration-smoke (pull_request) Canceled after 0s
CI / image-build (pull_request) Canceled after 0s
CI / deploy-smoke (same-origin) (pull_request) Canceled after 0s
CI / deploy-smoke (split) (pull_request) Canceled after 0s
A migration-smoke stack (PostgreSQL and SeaweedFS) ran on the shared runner for two days. Every CI script removes its stack in an EXIT trap, but a trap never runs when the runner kills the job container: a cancelled job, or the runner's own time limit (Gitea 1.24 ignores timeout-minutes). Reproduced: SIGKILL of the infrastructure smoke mid-run leaves exactly that stack running. Cleanup now has three layers: - the script's own trap, unchanged; - a final `if: always()` step in every job that starts stacks (infrastructure-smoke, migration-smoke, image-build, deploy-smoke) runs `scripts/ci-sweep.sh --job`, which removes everything that job started at any age; - a first step in each of those jobs runs `scripts/ci-sweep.sh --stale`, which removes CI stacks older than three hours: what a killed job left before it could clean up. How stacks are found is defined once, in scripts/lib/ci-project.sh: - every CI Compose project is named `ekklesia-ci-<run>-<attempt>-<job key>-<kind>-<unique>`, and Compose labels its containers, networks and volumes with that name; - the deploy smoke labels everything it starts outside Compose: browser, kiosk and driver containers, the kiosk volume, built images; - images also record when CI built them. An image's own creation time can be its base's (BuildKit, metadata-only builds), and the sweep removed a brand-new image as "old" until it stopped trusting that. The job key is per matrix leg, so one topology's cleanup cannot remove the other's running stack. The sweep never touches anything outside the ekklesia-ci- prefix, disconnects rather than removes a job container attached to a stack's network, and refuses --job outside CI. Network creation times are read as RFC 3339; Docker's default form was unparseable. scripts/test-ci-sweep.sh proves all of that against real Docker, and CI runs it before relying on it. security.md and testing.md document the rule, and that it complements, not replaces, a runner-side reaper. Co-Authored-By: Claude Opus 5.5 <[email protected]>
41 lines
2.1 KiB
Bash
41 lines
2.1 KiB
Bash
# shellcheck shell=bash
|
|
# How CI names, labels, and later finds the disposable stacks it starts.
|
|
#
|
|
# Every Compose project a CI script starts is named
|
|
#
|
|
# ekklesia-ci-<run id>-<run attempt>-<job key>-<kind>-<pid><random>
|
|
#
|
|
# and Compose labels every container, network, and volume it creates with that name. What a script
|
|
# creates outside Compose (a browser container, a volume, an image) gets EKKLESIA_CI_LABEL with the
|
|
# same name. scripts/ci-sweep.sh finds stacks by those labels: this job's own, by the prefix up to
|
|
# the job key (in a final `if: always()` step), and any CI stack by age (before a job starts).
|
|
#
|
|
# The job key is EKKLESIA_CI_JOB_KEY, which the workflow sets per job, and per matrix leg where a job
|
|
# has one, so that one leg's cleanup cannot match another leg's running stack. Outside CI the
|
|
# identifiers are "local".
|
|
|
|
# shellcheck disable=SC2034 # read by the scripts that source this
|
|
EKKLESIA_CI_LABEL="org.riverofglorymn.ekklesia.ci.project"
|
|
# On images, when they were built, in Unix seconds. An image's own creation time is not that: BuildKit
|
|
# gives an image that only adds metadata its base's time, and a reproducible build a fixed one. A
|
|
# stale sweep that trusted it could remove an image another job is using right now.
|
|
EKKLESIA_CI_CREATED_LABEL="org.riverofglorymn.ekklesia.ci.created"
|
|
EKKLESIA_CI_PREFIX="ekklesia-ci-"
|
|
|
|
# ci_name_safe <text>: lower case, and only what a Compose project name may contain.
|
|
ci_name_safe() {
|
|
local text="${1//[^a-zA-Z0-9_-]/-}"
|
|
printf '%s' "${text,,}"
|
|
}
|
|
|
|
# ci_job_prefix: every project this job starts begins with exactly this.
|
|
ci_job_prefix() {
|
|
ci_name_safe "ekklesia-ci-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-0}-${EKKLESIA_CI_JOB_KEY:-${GITHUB_JOB:-local}}-"
|
|
}
|
|
|
|
# ci_project_name <kind>: a project name unique to this invocation. Not only per run: several runs,
|
|
# and several jobs of one run, share a runner's Docker daemon, and job containers often share a PID.
|
|
ci_project_name() {
|
|
ci_name_safe "ekklesia-ci-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-0}-${EKKLESIA_CI_JOB_KEY:-${GITHUB_JOB:-local}}-${1:?ci_project_name needs a kind}-$$-${RANDOM}${RANDOM}"
|
|
}
|