fix(ci): nothing a CI job starts outlives it, even when the runner kills the job

A migration-smoke stack (PostgreSQL and SeaweedFS) ran on the shared
runner for two days. Every CI script removes its stack in an EXIT trap,
but a trap never runs when the runner kills the job container: a
cancelled job, or the runner's own time limit (Gitea 1.24 ignores
timeout-minutes). Reproduced: SIGKILL of the infrastructure smoke
mid-run leaves exactly that stack running.

Cleanup now has three layers:
- the script's own trap, unchanged;
- a final `if: always()` step in every job that starts stacks
  (infrastructure-smoke, migration-smoke, image-build, deploy-smoke)
  runs `scripts/ci-sweep.sh --job`, which removes everything that job
  started at any age;
- a first step in each of those jobs runs `scripts/ci-sweep.sh --stale`,
  which removes CI stacks older than three hours: what a killed job left
  before it could clean up.

How stacks are found is defined once, in scripts/lib/ci-project.sh:
- every CI Compose project is named `ekklesia-ci-<run>-<attempt>-<job
  key>-<kind>-<unique>`, and Compose labels its containers, networks and
  volumes with that name;
- the deploy smoke labels everything it starts outside Compose:
  browser, kiosk and driver containers, the kiosk volume, built images;
- images also record when CI built them. An image's own creation time
  can be its base's (BuildKit, metadata-only builds), and the sweep
  removed a brand-new image as "old" until it stopped trusting that.

The job key is per matrix leg, so one topology's cleanup cannot remove
the other's running stack. The sweep never touches anything outside the
ekklesia-ci- prefix, disconnects rather than removes a job container
attached to a stack's network, and refuses --job outside CI. Network
creation times are read as RFC 3339; Docker's default form was
unparseable.

scripts/test-ci-sweep.sh proves all of that against real Docker, and
CI runs it before relying on it. security.md and testing.md document
the rule, and that it complements, not replaces, a runner-side reaper.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

scripts/ci-sweep.sh

@@ -0,0 +1,205 @@
+#!/usr/bin/env bash
+# Removes disposable CI stacks from the Docker daemon: containers, networks, volumes, images, and
+# staging directories on the Docker host.
+#
+#   bash scripts/ci-sweep.sh --stale [--max-age-seconds N]   any CI stack older than N (default 3 h)
+#   bash scripts/ci-sweep.sh --job                           this job's own stacks, at any age
+#
+# Every CI script already removes its own stack in an EXIT trap, but a trap never runs when the runner
+# kills the job container: a cancelled job, or the runner's own time limit (Gitea 1.24 ignores
+# `timeout-minutes`). So each workflow job that starts stacks ends with `--job` in an `if: always()`
+# step, and begins with `--stale`, which catches whatever a killed job left before it could clean up.
+#
+# What a stack is, and how it is found, is defined in scripts/lib/ci-project.sh: a Compose project
+# named with the `ekklesia-ci-` prefix (Compose labels everything it creates with that name), plus
+# anything labeled EKKLESIA_CI_LABEL. Nothing outside that prefix is ever touched, at any age.
+#
+# --stale's threshold is far above any job's run time, so a stack another job on this daemon is using
+# right now is never young enough to match. --job refuses to run outside CI: locally, its prefix would
+# match a developer's own stacks.
+#
+# Best effort by design: a resource that cannot be removed is reported, and the sweep goes on. Output
+# names resources only; nothing a stack holds is printed.
+
+set -Eeuo pipefail
+
+repository_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
+# shellcheck source=scripts/lib/ci-project.sh
+source "$repository_root/scripts/lib/ci-project.sh"
+
+compose_label="com.docker.compose.project"
+mode=""
+max_age_seconds="${EKKLESIA_CI_SWEEP_MAX_AGE_SECONDS:-10800}"
+
+usage() {
+  echo "usage: ci-sweep.sh --stale [--max-age-seconds N] | --job" >&2
+  exit 2
+}
+
+while (($# > 0)); do
+  case "$1" in
+  --stale | --job)
+    [[ -z "$mode" ]] || usage
+    mode="${1#--}"
+    ;;
+  --max-age-seconds)
+    [[ "${2:-}" =~ ^[0-9]+$ ]] || usage
+    max_age_seconds="$2"
+    shift
+    ;;
+  *) usage ;;
+  esac
+  shift
+done
+[[ -n "$mode" ]] || usage
+
+if [[ "$mode" == job ]]; then
+  [[ -n "${GITHUB_RUN_ID:-}" ]] || {
+    echo "ci-sweep: --job runs only in CI (GITHUB_RUN_ID is unset); locally it would match your own stacks." >&2
+    exit 2
+  }
+  prefix="$(ci_job_prefix)"
+  cutoff=""
+else
+  prefix="$EKKLESIA_CI_PREFIX"
+  cutoff="$(($(date +%s) - max_age_seconds))"
+fi
+
+if ! docker info >/dev/null 2>&1; then
+  echo "ci-sweep: no Docker daemon reachable; nothing to sweep."
+  exit 0
+fi
+
+removed=0
+warn() {
+  echo "ci-sweep: warning: $*" >&2
+}
+
+# selected <project> <created>: whether a resource belongs to the sweep.
+selected() {
+  local project="$1" created="$2" seconds
+  [[ -n "$project" && "$project" == "$prefix"* ]] || return 1
+  [[ -z "$cutoff" ]] && return 0
+  seconds="$(date -d "$created" +%s 2>/dev/null)" || return 1
+  ((seconds <= cutoff))
+}
+
+# --- Containers: Compose's own, and those a script started itself --------------------------------
+mapfile -t containers < <(
+  {
+    docker ps --all --quiet --no-trunc --filter "label=${compose_label}"
+    docker ps --all --quiet --no-trunc --filter "label=${EKKLESIA_CI_LABEL}"
+  } | sort -u
+)
+for id in "${containers[@]}"; do
+  [[ -n "$id" ]] || continue
+  record="$(docker inspect --format \
+    "{{index .Config.Labels \"${compose_label}\"}}|{{index .Config.Labels \"${EKKLESIA_CI_LABEL}\"}}|{{.Created}}|{{.Name}}" \
+    "$id" 2>/dev/null)" || continue
+  IFS='|' read -r compose_project ci_project created name <<<"$record"
+  project="${compose_project:-$ci_project}"
+  selected "$project" "$created" || continue
+  if docker rm --force --volumes "$id" >/dev/null 2>&1; then
+    echo "ci-sweep: removed container ${name#/} (${project})"
+    removed=$((removed + 1))
+  else
+    warn "could not remove container ${name#/}"
+  fi
+done
+
+# --- Networks: anything still attached (a job container) is disconnected, never removed ---------
+mapfile -t networks < <(docker network ls --quiet --no-trunc --filter "label=${compose_label}")
+for id in "${networks[@]}"; do
+  [[ -n "$id" ]] || continue
+  # A network's creation time is a time value, not a string: ask for RFC 3339, which `date` reads.
+  record="$(docker network inspect --format \
+    "{{index .Labels \"${compose_label}\"}}|{{.Created.Format \"2006-01-02T15:04:05Z07:00\"}}|{{.Name}}" \
+    "$id" 2>/dev/null)" || continue
+  IFS='|' read -r project created name <<<"$record"
+  selected "$project" "$created" || continue
+  for endpoint in $(docker network inspect --format '{{range $id, $_ := .Containers}}{{$id}} {{end}}' "$id" 2>/dev/null); do
+    docker network disconnect --force "$id" "$endpoint" >/dev/null 2>&1 ||
+      warn "could not disconnect a container from network ${name}"
+  done
+  if docker network rm "$id" >/dev/null 2>&1; then
+    echo "ci-sweep: removed network ${name} (${project})"
+    removed=$((removed + 1))
+  else
+    warn "could not remove network ${name}"
+  fi
+done
+
+# --- Volumes ---------------------------------------------------------------------------------------
+mapfile -t volumes < <(
+  {
+    docker volume ls --quiet --filter "label=${compose_label}"
+    docker volume ls --quiet --filter "label=${EKKLESIA_CI_LABEL}"
+  } | sort -u
+)
+for name in "${volumes[@]}"; do
+  [[ -n "$name" ]] || continue
+  record="$(docker volume inspect --format \
+    "{{index .Labels \"${compose_label}\"}}|{{index .Labels \"${EKKLESIA_CI_LABEL}\"}}|{{.CreatedAt}}" \
+    "$name" 2>/dev/null)" || continue
+  IFS='|' read -r compose_project ci_project created <<<"$record"
+  project="${compose_project:-$ci_project}"
+  selected "$project" "$created" || continue
+  if docker volume rm --force "$name" >/dev/null 2>&1; then
+    echo "ci-sweep: removed volume ${name} (${project})"
+    removed=$((removed + 1))
+  else
+    warn "could not remove volume ${name}"
+  fi
+done
+
+# --- Images built for a stack ------------------------------------------------------------------------
+mapfile -t images < <(docker image ls --quiet --no-trunc --filter "label=${EKKLESIA_CI_LABEL}" | sort -u)
+for id in "${images[@]}"; do
+  [[ -n "$id" ]] || continue
+  # Judged by when CI built it (EKKLESIA_CI_CREATED_LABEL), never by the image's own creation time,
+  # which BuildKit may have taken from its base. An image without that label is never swept by age.
+  record="$(docker image inspect --format \
+    "{{index .Config.Labels \"${EKKLESIA_CI_LABEL}\"}}|{{index .Config.Labels \"${EKKLESIA_CI_CREATED_LABEL}\"}}" \
+    "$id" 2>/dev/null)" || continue
+  IFS='|' read -r project built <<<"$record"
+  if [[ -n "$cutoff" ]]; then
+    [[ "$built" =~ ^[0-9]+$ ]] || continue
+    selected "$project" "@${built}" || continue
+  else
+    selected "$project" "" || continue
+  fi
+  if docker image rm --force "$id" >/dev/null 2>&1; then
+    echo "ci-sweep: removed image ${id#sha256:} (${project})"
+    removed=$((removed + 1))
+  else
+    warn "could not remove image ${id#sha256:}"
+  fi
+done
+
+# --- Staging directories on the Docker host ------------------------------------------------------
+# A deploy smoke run inside a job container stages its mounted files under the Docker host's /tmp,
+# named for its project, as root-owned files; they hold that run's disposable secrets and keys. They
+# are removed with the same pinned image that created them. If that image is not on this daemon, no
+# smoke has staged anything here.
+image="$(sed -n 's/^[[:space:]]*image: \(postgres:[^[:space:]]*\)$/\1/p' "$repository_root/deploy/compose.yaml" | head -1)"
+if [[ -n "$image" ]] && docker image inspect "$image" >/dev/null 2>&1; then
+  age=()
+  # `find -mmin +0` does not match something younger than a minute, so a threshold under a minute
+  # sweeps without an age filter, like the rest of this script.
+  if [[ -n "$cutoff" ]] && ((max_age_seconds >= 60)); then
+    age=(-mmin "+$((max_age_seconds / 60))")
+  fi
+  if staged="$(docker run --rm --entrypoint find -v /tmp:/host-tmp "$image" \
+    /host-tmp -mindepth 1 -maxdepth 1 -type d -name "${prefix}*" "${age[@]}" \
+    -print -exec rm -rf {} + 2>/dev/null)"; then
+    while IFS= read -r directory; do
+      [[ -n "$directory" ]] || continue
+      echo "ci-sweep: removed staging directory /tmp/${directory#/host-tmp/}"
+      removed=$((removed + 1))
+    done <<<"$staged"
+  else
+    warn "could not sweep staging directories on the Docker host"
+  fi
+fi
+
+echo "ci-sweep: ${mode} sweep removed ${removed} resource(s)."

scripts/lib/ci-project.sh

@@ -0,0 +1,40 @@
+# shellcheck shell=bash
+# How CI names, labels, and later finds the disposable stacks it starts.
+#
+# Every Compose project a CI script starts is named
+#
+#   ekklesia-ci-<run id>-<run attempt>-<job key>-<kind>-<pid><random>
+#
+# and Compose labels every container, network, and volume it creates with that name. What a script
+# creates outside Compose (a browser container, a volume, an image) gets EKKLESIA_CI_LABEL with the
+# same name. scripts/ci-sweep.sh finds stacks by those labels: this job's own, by the prefix up to
+# the job key (in a final `if: always()` step), and any CI stack by age (before a job starts).
+#
+# The job key is EKKLESIA_CI_JOB_KEY, which the workflow sets per job, and per matrix leg where a job
+# has one, so that one leg's cleanup cannot match another leg's running stack. Outside CI the
+# identifiers are "local".
+
+# shellcheck disable=SC2034 # read by the scripts that source this
+EKKLESIA_CI_LABEL="org.riverofglorymn.ekklesia.ci.project"
+# On images, when they were built, in Unix seconds. An image's own creation time is not that: BuildKit
+# gives an image that only adds metadata its base's time, and a reproducible build a fixed one. A
+# stale sweep that trusted it could remove an image another job is using right now.
+EKKLESIA_CI_CREATED_LABEL="org.riverofglorymn.ekklesia.ci.created"
+EKKLESIA_CI_PREFIX="ekklesia-ci-"
+
+# ci_name_safe <text>: lower case, and only what a Compose project name may contain.
+ci_name_safe() {
+  local text="${1//[^a-zA-Z0-9_-]/-}"
+  printf '%s' "${text,,}"
+}
+
+# ci_job_prefix: every project this job starts begins with exactly this.
+ci_job_prefix() {
+  ci_name_safe "ekklesia-ci-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-0}-${EKKLESIA_CI_JOB_KEY:-${GITHUB_JOB:-local}}-"
+}
+
+# ci_project_name <kind>: a project name unique to this invocation. Not only per run: several runs,
+# and several jobs of one run, share a runner's Docker daemon, and job containers often share a PID.
+ci_project_name() {
+  ci_name_safe "ekklesia-ci-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-0}-${EKKLESIA_CI_JOB_KEY:-${GITHUB_JOB:-local}}-${1:?ci_project_name needs a kind}-$$-${RANDOM}${RANDOM}"
+}

scripts/smoke-deploy.sh

@@ -39,6 +39,8 @@ cd "$repository_root"
 
 # shellcheck source=scripts/lib/compose-diagnostics.sh
 source "$repository_root/scripts/lib/compose-diagnostics.sh"
+# shellcheck source=scripts/lib/ci-project.sh
+source "$repository_root/scripts/lib/ci-project.sh"
 
 for command in docker openssl awk; do
   command -v "$command" >/dev/null || {
@@ -67,7 +69,10 @@ same-origin | split) ;;
 esac
 
 run_id="$(openssl rand -hex 4)"
-project="ekklesia-deploy-smoke-${run_id}"
+# Unique per invocation, and findable by run and job if this script is killed before its trap runs
+# (scripts/lib/ci-project.sh, scripts/ci-sweep.sh). Everything started outside Compose below carries
+# the same name in EKKLESIA_CI_LABEL.
+project="$(ci_project_name "deploy-${topology}")"
 work="$(mktemp -d)"
 image="${EKKLESIA_SMOKE_IMAGE:-}"
 built_image=""
@@ -186,7 +191,9 @@ fail() {
 if [[ -z "$image" && "$topology" == same-origin ]]; then
   step "Building the platform-only image"
   built_image="ekklesia-deploy-smoke:${run_id}"
-  DOCKER_BUILDKIT=1 docker build --quiet --file deploy/Dockerfile --tag "$built_image" . >/dev/null
+  DOCKER_BUILDKIT=1 docker build --quiet --file deploy/Dockerfile --tag "$built_image" \
+    --label "${EKKLESIA_CI_LABEL}=${project}" --label "${EKKLESIA_CI_CREATED_LABEL}=$(date +%s)" \
+    . >/dev/null
   image="$built_image"
 elif [[ -z "$image" ]]; then
   # TEST-ONLY: the repository's module template, installed as the module `example`, in a copy of the
@@ -206,6 +213,7 @@ elif [[ -z "$image" ]]; then
   sed -i 's|^  templates/module:$|  modules/example:|' "$context/pnpm-lock.yaml"
   built_image="ekklesia-deploy-smoke:${run_id}"
   DOCKER_BUILDKIT=1 docker build --quiet --file "$context/deploy/Dockerfile" --tag "$built_image" \
+    --label "${EKKLESIA_CI_LABEL}=${project}" --label "${EKKLESIA_CI_CREATED_LABEL}=$(date +%s)" \
     "$context" >/dev/null || fail "the test-module image did not build."
   rm -rf "$context"
   image="$built_image"
@@ -362,6 +370,7 @@ client() {
   local mode="$1"
   shift
   docker run --rm --read-only --cap-drop ALL --security-opt no-new-privileges \
+    --label "${EKKLESIA_CI_LABEL}=${project}" \
     --network "$edge_network" "$@" "${host_mappings[@]}" \
     -e NODE_EXTRA_CA_CERTS=/smoke/client-ca.pem -e SMOKE_HOSTNAME="$hostname" \
     -e SMOKE_DASHBOARD_HOSTNAME="$dashboard_hostname" \
@@ -401,7 +410,7 @@ if [[ "$topology" == split ]]; then
   browser_container="${project}-browser"
   # The image's entry script passes its arguments on unquoted, so none may contain a space: the
   # hostnames reach Chromium through /etc/hosts, which it reads, rather than a resolver rule.
-  docker run --detach --rm --name "$browser_container" \
+  docker run --detach --rm --name "$browser_container" --label "${EKKLESIA_CI_LABEL}=${project}" \
     --cap-drop ALL --security-opt no-new-privileges --tmpfs /tmp \
     --network "$edge_network" --ip "$browser_address" "${host_mappings[@]}" \
     "$browser_image" \
@@ -410,6 +419,7 @@ if [[ "$topology" == split ]]; then
   cp scripts/lib/deploy-smoke-browser.mjs "$work/browser.mjs"
   stage browser.mjs
   docker run --rm --read-only --cap-drop ALL --security-opt no-new-privileges \
+    --label "${EKKLESIA_CI_LABEL}=${project}" \
     --network "$edge_network" \
     -e SMOKE_CDP_ADDRESS="${browser_address}:9222" \
     -e SMOKE_HOSTNAME="$hostname" -e SMOKE_DASHBOARD_HOSTNAME="$dashboard_hostname" \
@@ -423,9 +433,9 @@ if [[ "$topology" == split ]]; then
     kiosk_container="${project}-kiosk"
     kiosk_volume="${project}-kiosk-profile"
     kiosk_address="10.249.${octet}.250"
-    docker volume create "$kiosk_volume" >/dev/null
+    docker volume create --label "${EKKLESIA_CI_LABEL}=${project}" "$kiosk_volume" >/dev/null
     start_kiosk() {
-      docker run --detach --rm --name "$kiosk_container" \
+      docker run --detach --rm --name "$kiosk_container" --label "${EKKLESIA_CI_LABEL}=${project}" \
         --cap-drop ALL --security-opt no-new-privileges --tmpfs /tmp \
         --network "$edge_network" --ip "$kiosk_address" "${host_mappings[@]}" \
         -v "${kiosk_volume}:/profile" \

scripts/smoke-infrastructure.sh

@@ -9,6 +9,8 @@ cd "$repository_root"
 source "$repository_root/scripts/lib/compose-job-network.sh"
 # shellcheck source=scripts/lib/compose-diagnostics.sh
 source "$repository_root/scripts/lib/compose-diagnostics.sh"
+# shellcheck source=scripts/lib/ci-project.sh
+source "$repository_root/scripts/lib/ci-project.sh"
 
 command -v docker >/dev/null || {
   echo "Docker is required for the infrastructure smoke test." >&2
@@ -30,11 +32,9 @@ if [[ "$curl_help" != *"--aws-sigv4"* ]]; then
   exit 1
 fi
 
-# Unique per invocation, not only per run: several runs can share a runner's Docker daemon at once,
-# and the shell's PID is often identical across job containers.
-raw_project="ekklesia-smoke-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-0}-$$-${RANDOM}${RANDOM}"
-project="${raw_project//[^a-zA-Z0-9_-]/-}"
-project="${project,,}"
+# Unique per invocation, and findable by run and job if this script is killed before its trap runs
+# (scripts/lib/ci-project.sh, scripts/ci-sweep.sh).
+project="$(ci_project_name infrastructure)"
 temporary_directory="$(mktemp -d)"
 payload_file="$temporary_directory/payload.txt"
 download_file="$temporary_directory/download.txt"

scripts/test-ci-sweep.sh

@@ -62,7 +62,8 @@ make_stack() {
     --network "${project}_default" -v "${project}_data:/data" --entrypoint sleep "$image" 600 >/dev/null
   docker run --detach --name "${project}-browser" --label "${ci_label}=${project}" \
     --entrypoint sleep "$image" 600 >/dev/null
-  printf 'FROM %s\nLABEL %s=%s\n' "$image" "$ci_label" "$project" |
+  printf 'FROM %s\nLABEL %s=%s %s=%s\n' "$image" "$ci_label" "$project" \
+    "org.riverofglorymn.ekklesia.ci.created" "$(date +%s)" |
     docker build --quiet --tag "${project}-image:test" - >/dev/null
   docker run --rm --entrypoint mkdir -v /tmp:/host-tmp "$image" -p "/host-tmp/${project}"
 }

scripts/test-migrations-integration.sh

@@ -9,6 +9,8 @@ cd "$repository_root"
 source "$repository_root/scripts/lib/compose-job-network.sh"
 # shellcheck source=scripts/lib/compose-diagnostics.sh
 source "$repository_root/scripts/lib/compose-diagnostics.sh"
+# shellcheck source=scripts/lib/ci-project.sh
+source "$repository_root/scripts/lib/ci-project.sh"
 
 command -v docker >/dev/null || {
   echo "Docker is required for the PostgreSQL migration integration suite." >&2
@@ -27,11 +29,9 @@ if ! compose_job_network_initialize; then
   exit 1
 fi
 
-# Unique per invocation, not only per run: several runs (and several jobs of one run) can share a
-# runner's Docker daemon at once, and the shell's PID is often identical across job containers.
-raw_project="ekklesia-migration-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-0}-${GITHUB_JOB:-job}-$$-${RANDOM}${RANDOM}"
-project="${raw_project//[^a-zA-Z0-9_-]/-}"
-project="${project,,}"
+# Unique per invocation, and findable by run and job if this script is killed before its trap runs
+# (scripts/lib/ci-project.sh, scripts/ci-sweep.sh).
+project="$(ci_project_name migration)"
 
 export POSTGRES_DB="ekklesia_migration_test"
 export POSTGRES_USER="ekklesia_migration_test"