Files
Ekklesia/scripts/test-migrations-integration.sh
T
ekklesiaandClaude Opus 5.5 62951975da
CI / verify (pull_request) Successful in 57s
CI / infrastructure-smoke (pull_request) Successful in 28s
CI / migration-smoke (pull_request) Successful in 1m49s
CI / image-build (pull_request) Successful in 19s
CI / deploy-smoke (same-origin) (pull_request) Successful in 48s
CI / deploy-smoke (split) (pull_request) Successful in 53s
fix(ci): nothing a CI job starts outlives it, even when the runner kills the job
A migration-smoke stack (PostgreSQL and SeaweedFS) ran on the shared
runner for two days. Every CI script removes its stack in an EXIT trap,
but a trap never runs when the runner kills the job container: a
cancelled job, or the runner's own time limit (Gitea 1.24 ignores
timeout-minutes). Reproduced: SIGKILL of the infrastructure smoke
mid-run leaves exactly that stack running.

Cleanup now has three layers:
- the script's own trap, unchanged;
- a final `if: always()` step in every job that starts stacks
  (infrastructure-smoke, migration-smoke, image-build, deploy-smoke)
  runs `scripts/ci-sweep.sh --job`, which removes everything that job
  started at any age;
- a first step in each of those jobs runs `scripts/ci-sweep.sh --stale`,
  which removes CI stacks older than three hours: what a killed job left
  before it could clean up.

How stacks are found is defined once, in scripts/lib/ci-project.sh:
- every CI Compose project is named `ekklesia-ci-<run>-<attempt>-<job
  key>-<kind>-<unique>`, and Compose labels its containers, networks and
  volumes with that name;
- the deploy smoke labels everything it starts outside Compose:
  browser, kiosk and driver containers, the kiosk volume, built images;
- images also record when CI built them. An image's own creation time
  can be its base's (BuildKit, metadata-only builds), and the sweep
  removed a brand-new image as "old" until it stopped trusting that.

The job key is per matrix leg, so one topology's cleanup cannot remove
the other's running stack. The sweep never touches anything outside the
ekklesia-ci- prefix, disconnects rather than removes a job container
attached to a stack's network, and refuses --job outside CI. Network
creation times are read as RFC 3339; Docker's default form was
unparseable.

scripts/test-ci-sweep.sh proves all of that against real Docker, and
CI runs it before relying on it. security.md and testing.md document
the rule, and that it complements, not replaces, a runner-side reaper.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-06 02:01:01 +00:00

152 lines
5.9 KiB
Bash
Executable File

#!/usr/bin/env bash
set -Eeuo pipefail
repository_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$repository_root"
# shellcheck source=scripts/lib/compose-job-network.sh
source "$repository_root/scripts/lib/compose-job-network.sh"
# shellcheck source=scripts/lib/compose-diagnostics.sh
source "$repository_root/scripts/lib/compose-diagnostics.sh"
# shellcheck source=scripts/lib/ci-project.sh
source "$repository_root/scripts/lib/ci-project.sh"
command -v docker >/dev/null || {
echo "Docker is required for the PostgreSQL migration integration suite." >&2
exit 1
}
command -v pnpm >/dev/null || {
echo "pnpm is required for the PostgreSQL migration integration suite." >&2
exit 1
}
if ! docker compose version >/dev/null 2>&1; then
echo "Docker Compose is required for the PostgreSQL migration integration suite." >&2
exit 1
fi
if ! compose_job_network_initialize; then
echo "The current Docker job container could not be identified." >&2
exit 1
fi
# Unique per invocation, and findable by run and job if this script is killed before its trap runs
# (scripts/lib/ci-project.sh, scripts/ci-sweep.sh).
project="$(ci_project_name migration)"
export POSTGRES_DB="ekklesia_migration_test"
export POSTGRES_USER="ekklesia_migration_test"
export POSTGRES_PASSWORD="temporary-$project-${RANDOM}-${RANDOM}"
# Both host ports are ephemeral: concurrent runs on one Docker daemon must never contend for a
# fixed one. `.env.example` pins the storage port for local development, so it is overridden here
# exactly like the database's. The bound ports are discovered below.
export POSTGRES_PORT="0"
export S3_PORT="0"
compose=(
docker compose
--project-name "$project"
--env-file .env.example
--file infrastructure/compose.yaml
)
cleanup() {
local exit_status=$?
trap - EXIT INT TERM
if ! compose_job_network_disconnect; then
echo "PostgreSQL migration test network cleanup failed." >&2
if ((exit_status == 0)); then
exit_status=1
fi
fi
if ! "${compose[@]}" down --volumes --remove-orphans --timeout 10 >/dev/null 2>&1; then
echo "PostgreSQL migration test infrastructure cleanup failed." >&2
if ((exit_status == 0)); then
exit_status=1
fi
fi
exit "$exit_status"
}
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
if ! "${compose[@]}" config --quiet >/dev/null 2>&1; then
echo "PostgreSQL migration test infrastructure configuration is invalid." >&2
exit 1
fi
# PostgreSQL and the S3-compatible object-storage service are both brought up from the repository's
# own pinned Compose file, so the live suites exercise exactly the services a developer runs locally
# and CI already knows how to provision. SeaweedFS is the intended deployment target (ADR 0004), and
# proving the storage contract against a mock instead would have hidden the provider quirks this
# suite exists to catch.
# Masked in any diagnostics below, along with anything else that looks like a credential.
COMPOSE_DIAGNOSTICS_SECRETS=(
"$POSTGRES_PASSWORD"
"$(grep -E '^S3_ACCESS_KEY=' .env.example | cut -d= -f2-)"
"$(grep -E '^S3_SECRET_KEY=' .env.example | cut -d= -f2-)"
)
if ! up_output="$(
"${compose[@]}" up --detach --wait --wait-timeout 180 postgres object-storage 2>&1
)"; then
echo "Disposable PostgreSQL and object storage did not become ready." >&2
compose_report_not_ready "$up_output" "${compose[@]}"
exit 1
fi
if [[ "$COMPOSE_JOB_NETWORK_RUNNING_IN_CONTAINER" == "true" ]]; then
if ! compose_job_network_connect postgres "${compose[@]}"; then
echo "The migration test job could not join the isolated PostgreSQL network." >&2
exit 1
fi
if ! compose_job_network_connect object-storage "${compose[@]}"; then
echo "The migration test job could not join the isolated object-storage network." >&2
exit 1
fi
database_host="postgres"
database_port="5432"
storage_endpoint="http://object-storage:8333"
else
if ! postgres_binding="$("${compose[@]}" port postgres 5432 2>/dev/null)"; then
echo "The disposable PostgreSQL loopback port could not be determined." >&2
exit 1
fi
if [[ ! "$postgres_binding" =~ ^127\.0\.0\.1:([0-9]+)$ ]]; then
echo "The disposable PostgreSQL port is not bound safely to loopback." >&2
exit 1
fi
database_host="127.0.0.1"
database_port="${BASH_REMATCH[1]}"
if ((database_port < 1 || database_port > 65535)); then
echo "The disposable PostgreSQL loopback port is invalid." >&2
exit 1
fi
if ! storage_binding="$("${compose[@]}" port object-storage 8333 2>/dev/null)"; then
echo "The disposable object-storage loopback port could not be determined." >&2
exit 1
fi
if [[ ! "$storage_binding" =~ ^127\.0\.0\.1:([0-9]+)$ ]]; then
echo "The disposable object-storage port is not bound safely to loopback." >&2
exit 1
fi
storage_port="${BASH_REMATCH[1]}"
if ((storage_port < 1 || storage_port > 65535)); then
echo "The disposable object-storage loopback port is invalid." >&2
exit 1
fi
storage_endpoint="http://127.0.0.1:$storage_port"
fi
# The TLS suite provisions its own disposable PostgreSQL rather than a Compose service, so it
# cannot call `compose_job_network_connect`. It receives the same topology decision this script
# already made, instead of detecting the topology a second time.
# The storage credentials come from `.env.example`, the same file `--env-file` already hands the
# Compose services, so the suite signs requests with exactly what the service was started with.
EKKLESIA_TEST_JOB_CONTAINER="$(compose_job_network_job_container)" \
EKKLESIA_MIGRATION_TEST_DATABASE_URL="postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@$database_host:$database_port/$POSTGRES_DB" \
EKKLESIA_TEST_STORAGE_ENDPOINT="$storage_endpoint" \
EKKLESIA_TEST_STORAGE_ACCESS_KEY="$(grep -E '^S3_ACCESS_KEY=' .env.example | cut -d= -f2-)" \
EKKLESIA_TEST_STORAGE_SECRET_KEY="$(grep -E '^S3_SECRET_KEY=' .env.example | cut -d= -f2-)" \
pnpm exec vitest run --config vitest.migrations.integration.config.ts