CI / verify (pull_request) Successful in 57s
CI / infrastructure-smoke (pull_request) Successful in 28s
CI / migration-smoke (pull_request) Successful in 1m49s
CI / image-build (pull_request) Successful in 19s
CI / deploy-smoke (same-origin) (pull_request) Successful in 48s
CI / deploy-smoke (split) (pull_request) Successful in 53s
A migration-smoke stack (PostgreSQL and SeaweedFS) ran on the shared runner for two days. Every CI script removes its stack in an EXIT trap, but a trap never runs when the runner kills the job container: a cancelled job, or the runner's own time limit (Gitea 1.24 ignores timeout-minutes). Reproduced: SIGKILL of the infrastructure smoke mid-run leaves exactly that stack running. Cleanup now has three layers: - the script's own trap, unchanged; - a final `if: always()` step in every job that starts stacks (infrastructure-smoke, migration-smoke, image-build, deploy-smoke) runs `scripts/ci-sweep.sh --job`, which removes everything that job started at any age; - a first step in each of those jobs runs `scripts/ci-sweep.sh --stale`, which removes CI stacks older than three hours: what a killed job left before it could clean up. How stacks are found is defined once, in scripts/lib/ci-project.sh: - every CI Compose project is named `ekklesia-ci-<run>-<attempt>-<job key>-<kind>-<unique>`, and Compose labels its containers, networks and volumes with that name; - the deploy smoke labels everything it starts outside Compose: browser, kiosk and driver containers, the kiosk volume, built images; - images also record when CI built them. An image's own creation time can be its base's (BuildKit, metadata-only builds), and the sweep removed a brand-new image as "old" until it stopped trusting that. The job key is per matrix leg, so one topology's cleanup cannot remove the other's running stack. The sweep never touches anything outside the ekklesia-ci- prefix, disconnects rather than removes a job container attached to a stack's network, and refuses --job outside CI. Network creation times are read as RFC 3339; Docker's default form was unparseable. scripts/test-ci-sweep.sh proves all of that against real Docker, and CI runs it before relying on it. security.md and testing.md document the rule, and that it complements, not replaces, a runner-side reaper. Co-Authored-By: Claude Opus 5.5 <[email protected]>
152 lines
5.9 KiB
Bash
Executable File
152 lines
5.9 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
|
|
set -Eeuo pipefail
|
|
|
|
repository_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
cd "$repository_root"
|
|
|
|
# shellcheck source=scripts/lib/compose-job-network.sh
|
|
source "$repository_root/scripts/lib/compose-job-network.sh"
|
|
# shellcheck source=scripts/lib/compose-diagnostics.sh
|
|
source "$repository_root/scripts/lib/compose-diagnostics.sh"
|
|
# shellcheck source=scripts/lib/ci-project.sh
|
|
source "$repository_root/scripts/lib/ci-project.sh"
|
|
|
|
command -v docker >/dev/null || {
|
|
echo "Docker is required for the PostgreSQL migration integration suite." >&2
|
|
exit 1
|
|
}
|
|
command -v pnpm >/dev/null || {
|
|
echo "pnpm is required for the PostgreSQL migration integration suite." >&2
|
|
exit 1
|
|
}
|
|
if ! docker compose version >/dev/null 2>&1; then
|
|
echo "Docker Compose is required for the PostgreSQL migration integration suite." >&2
|
|
exit 1
|
|
fi
|
|
if ! compose_job_network_initialize; then
|
|
echo "The current Docker job container could not be identified." >&2
|
|
exit 1
|
|
fi
|
|
|
|
# Unique per invocation, and findable by run and job if this script is killed before its trap runs
|
|
# (scripts/lib/ci-project.sh, scripts/ci-sweep.sh).
|
|
project="$(ci_project_name migration)"
|
|
|
|
export POSTGRES_DB="ekklesia_migration_test"
|
|
export POSTGRES_USER="ekklesia_migration_test"
|
|
export POSTGRES_PASSWORD="temporary-$project-${RANDOM}-${RANDOM}"
|
|
# Both host ports are ephemeral: concurrent runs on one Docker daemon must never contend for a
|
|
# fixed one. `.env.example` pins the storage port for local development, so it is overridden here
|
|
# exactly like the database's. The bound ports are discovered below.
|
|
export POSTGRES_PORT="0"
|
|
export S3_PORT="0"
|
|
|
|
compose=(
|
|
docker compose
|
|
--project-name "$project"
|
|
--env-file .env.example
|
|
--file infrastructure/compose.yaml
|
|
)
|
|
|
|
cleanup() {
|
|
local exit_status=$?
|
|
trap - EXIT INT TERM
|
|
|
|
if ! compose_job_network_disconnect; then
|
|
echo "PostgreSQL migration test network cleanup failed." >&2
|
|
if ((exit_status == 0)); then
|
|
exit_status=1
|
|
fi
|
|
fi
|
|
if ! "${compose[@]}" down --volumes --remove-orphans --timeout 10 >/dev/null 2>&1; then
|
|
echo "PostgreSQL migration test infrastructure cleanup failed." >&2
|
|
if ((exit_status == 0)); then
|
|
exit_status=1
|
|
fi
|
|
fi
|
|
exit "$exit_status"
|
|
}
|
|
trap cleanup EXIT
|
|
trap 'exit 130' INT
|
|
trap 'exit 143' TERM
|
|
|
|
if ! "${compose[@]}" config --quiet >/dev/null 2>&1; then
|
|
echo "PostgreSQL migration test infrastructure configuration is invalid." >&2
|
|
exit 1
|
|
fi
|
|
# PostgreSQL and the S3-compatible object-storage service are both brought up from the repository's
|
|
# own pinned Compose file, so the live suites exercise exactly the services a developer runs locally
|
|
# and CI already knows how to provision. SeaweedFS is the intended deployment target (ADR 0004), and
|
|
# proving the storage contract against a mock instead would have hidden the provider quirks this
|
|
# suite exists to catch.
|
|
# Masked in any diagnostics below, along with anything else that looks like a credential.
|
|
COMPOSE_DIAGNOSTICS_SECRETS=(
|
|
"$POSTGRES_PASSWORD"
|
|
"$(grep -E '^S3_ACCESS_KEY=' .env.example | cut -d= -f2-)"
|
|
"$(grep -E '^S3_SECRET_KEY=' .env.example | cut -d= -f2-)"
|
|
)
|
|
if ! up_output="$(
|
|
"${compose[@]}" up --detach --wait --wait-timeout 180 postgres object-storage 2>&1
|
|
)"; then
|
|
echo "Disposable PostgreSQL and object storage did not become ready." >&2
|
|
compose_report_not_ready "$up_output" "${compose[@]}"
|
|
exit 1
|
|
fi
|
|
|
|
if [[ "$COMPOSE_JOB_NETWORK_RUNNING_IN_CONTAINER" == "true" ]]; then
|
|
if ! compose_job_network_connect postgres "${compose[@]}"; then
|
|
echo "The migration test job could not join the isolated PostgreSQL network." >&2
|
|
exit 1
|
|
fi
|
|
if ! compose_job_network_connect object-storage "${compose[@]}"; then
|
|
echo "The migration test job could not join the isolated object-storage network." >&2
|
|
exit 1
|
|
fi
|
|
database_host="postgres"
|
|
database_port="5432"
|
|
storage_endpoint="http://object-storage:8333"
|
|
else
|
|
if ! postgres_binding="$("${compose[@]}" port postgres 5432 2>/dev/null)"; then
|
|
echo "The disposable PostgreSQL loopback port could not be determined." >&2
|
|
exit 1
|
|
fi
|
|
if [[ ! "$postgres_binding" =~ ^127\.0\.0\.1:([0-9]+)$ ]]; then
|
|
echo "The disposable PostgreSQL port is not bound safely to loopback." >&2
|
|
exit 1
|
|
fi
|
|
database_host="127.0.0.1"
|
|
database_port="${BASH_REMATCH[1]}"
|
|
if ((database_port < 1 || database_port > 65535)); then
|
|
echo "The disposable PostgreSQL loopback port is invalid." >&2
|
|
exit 1
|
|
fi
|
|
|
|
if ! storage_binding="$("${compose[@]}" port object-storage 8333 2>/dev/null)"; then
|
|
echo "The disposable object-storage loopback port could not be determined." >&2
|
|
exit 1
|
|
fi
|
|
if [[ ! "$storage_binding" =~ ^127\.0\.0\.1:([0-9]+)$ ]]; then
|
|
echo "The disposable object-storage port is not bound safely to loopback." >&2
|
|
exit 1
|
|
fi
|
|
storage_port="${BASH_REMATCH[1]}"
|
|
if ((storage_port < 1 || storage_port > 65535)); then
|
|
echo "The disposable object-storage loopback port is invalid." >&2
|
|
exit 1
|
|
fi
|
|
storage_endpoint="http://127.0.0.1:$storage_port"
|
|
fi
|
|
|
|
# The TLS suite provisions its own disposable PostgreSQL rather than a Compose service, so it
|
|
# cannot call `compose_job_network_connect`. It receives the same topology decision this script
|
|
# already made, instead of detecting the topology a second time.
|
|
# The storage credentials come from `.env.example`, the same file `--env-file` already hands the
|
|
# Compose services, so the suite signs requests with exactly what the service was started with.
|
|
EKKLESIA_TEST_JOB_CONTAINER="$(compose_job_network_job_container)" \
|
|
EKKLESIA_MIGRATION_TEST_DATABASE_URL="postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@$database_host:$database_port/$POSTGRES_DB" \
|
|
EKKLESIA_TEST_STORAGE_ENDPOINT="$storage_endpoint" \
|
|
EKKLESIA_TEST_STORAGE_ACCESS_KEY="$(grep -E '^S3_ACCESS_KEY=' .env.example | cut -d= -f2-)" \
|
|
EKKLESIA_TEST_STORAGE_SECRET_KEY="$(grep -E '^S3_SECRET_KEY=' .env.example | cut -d= -f2-)" \
|
|
pnpm exec vitest run --config vitest.migrations.integration.config.ts
|