fix(ci): isolate concurrent runs on a shared Docker daemon and explain readiness failures
CI / verify (pull_request) Successful in 50s
CI / infrastructure-smoke (pull_request) Successful in 14s
CI / migration-smoke (pull_request) Successful in 1m49s
CI / image-build (pull_request) Successful in 19s
CI / deploy-smoke (pull_request) Successful in 41s

PR #51's migration-smoke failed 0.7 s after the build: the suite pinned
PostgreSQL to an ephemeral port but inherited S3_PORT=8333 from
.env.example, so concurrent runs on one runner (Gitea ignores workflow
concurrency) contended for one host port. Compose's own error was
discarded, so the failure said only that the services "did not become
ready".

Isolation, from an audit of every script CI runs:
- test-migrations-integration.sh: S3_PORT=0, like POSTGRES_PORT=0.
- test-migrations-integration.sh and smoke-infrastructure.sh: project names
  gain a random suffix; run id plus PID alone is not unique when job
  containers share low PIDs.
- smoke-deploy.sh: the edge /24 is chosen from those no network on the
  daemon uses, and the first `up` chooses again when a concurrent run wins
  a race for the same subnet (Docker's address-pool overlap), up to five
  times.
- CI image-build: the image tag includes the run id and attempt, so a pull
  request and a push of the same commit cannot remove each other's image.
- Already isolated: every other published port (ephemeral), the TLS
  suite's container and network (random), smoke staging directories and
  image tags (random).

Diagnostics: scripts/lib/compose-diagnostics.sh prints the failed `up`
command's output, `docker compose ps`, and the last 40 log lines of each
service, all redacted (supplied literals, credential-looking NAME=value
pairs, URL user information, Authorization headers) with terminal control
sequences stripped. All three scripts use it at every readiness check; the
infrastructure smoke's unredacted full-log dump is replaced by it.

tests/ci-scripts.test.ts pins ephemeral ports, unique project names, the
subnet choice, the CI image tag, the diagnostics, and the redaction itself
(run through bash).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
ekklesia
2026-10-03 12:15:01 -05:00
co-authored by Claude Opus 5.5
parent c737ebb472
commit 278e66049f
6 changed files with 334 additions and 19 deletions
+17 -4
View File
@@ -7,6 +7,8 @@ cd "$repository_root"
# shellcheck source=scripts/lib/compose-job-network.sh
source "$repository_root/scripts/lib/compose-job-network.sh"
# shellcheck source=scripts/lib/compose-diagnostics.sh
source "$repository_root/scripts/lib/compose-diagnostics.sh"
command -v docker >/dev/null || {
echo "Docker is required for the infrastructure smoke test." >&2
@@ -28,7 +30,9 @@ if [[ "$curl_help" != *"--aws-sigv4"* ]]; then
exit 1
fi
raw_project="ekklesia-smoke-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-0}-$$"
# Unique per invocation, not only per run: several runs can share a runner's Docker daemon at once,
# and the shell's PID is often identical across job containers.
raw_project="ekklesia-smoke-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-0}-$$-${RANDOM}${RANDOM}"
project="${raw_project//[^a-zA-Z0-9_-]/-}"
project="${project,,}"
temporary_directory="$(mktemp -d)"
@@ -46,6 +50,8 @@ export S3_SECRET_KEY="ekklesia-smoke-secret-key"
export S3_BUCKET="ekklesia-smoke"
export S3_REGION="us-east-1"
export S3_PORT="${EKKLESIA_SMOKE_S3_PORT:-0}"
COMPOSE_DIAGNOSTICS_SECRETS=("$POSTGRES_PASSWORD" "$S3_ACCESS_KEY" "$S3_SECRET_KEY")
up_output=""
compose=(
docker compose
@@ -63,7 +69,7 @@ cleanup() {
local cleanup_output
trap - EXIT INT TERM
if ((exit_status != 0)); then
"${compose[@]}" logs --no-color >&2 || true
compose_report_not_ready "$up_output" "${compose[@]}" || true
fi
if ! compose_job_network_disconnect; then
echo "Infrastructure smoke network cleanup failed." >&2
@@ -88,8 +94,15 @@ trap 'exit 130' INT
trap 'exit 143' TERM
"${compose[@]}" config --quiet
"${compose[@]}" up --detach --wait --wait-timeout 120
"${compose[@]}" up --detach --wait --wait-timeout 120
# Started twice: the second `up` proves the project is idempotent. A failure's own output is kept for
# the cleanup's diagnostics, which also print every container and the tail of its logs.
for _ in 1 2; do
if ! up_output="$("${compose[@]}" up --detach --wait --wait-timeout 120 2>&1)"; then
echo "Disposable PostgreSQL and object storage did not become ready." >&2
exit 1
fi
done
up_output=""
database_result="$(
"${compose[@]}" exec -T \