fix(ci): isolate concurrent runs on a shared Docker daemon and explain readiness failures
CI / verify (pull_request) Successful in 50s
CI / infrastructure-smoke (pull_request) Successful in 14s
CI / migration-smoke (pull_request) Successful in 1m49s
CI / image-build (pull_request) Successful in 19s
CI / deploy-smoke (pull_request) Successful in 41s
CI / verify (pull_request) Successful in 50s
CI / infrastructure-smoke (pull_request) Successful in 14s
CI / migration-smoke (pull_request) Successful in 1m49s
CI / image-build (pull_request) Successful in 19s
CI / deploy-smoke (pull_request) Successful in 41s
PR #51's migration-smoke failed 0.7 s after the build: the suite pinned PostgreSQL to an ephemeral port but inherited S3_PORT=8333 from .env.example, so concurrent runs on one runner (Gitea ignores workflow concurrency) contended for one host port. Compose's own error was discarded, so the failure said only that the services "did not become ready". Isolation, from an audit of every script CI runs: - test-migrations-integration.sh: S3_PORT=0, like POSTGRES_PORT=0. - test-migrations-integration.sh and smoke-infrastructure.sh: project names gain a random suffix; run id plus PID alone is not unique when job containers share low PIDs. - smoke-deploy.sh: the edge /24 is chosen from those no network on the daemon uses, and the first `up` chooses again when a concurrent run wins a race for the same subnet (Docker's address-pool overlap), up to five times. - CI image-build: the image tag includes the run id and attempt, so a pull request and a push of the same commit cannot remove each other's image. - Already isolated: every other published port (ephemeral), the TLS suite's container and network (random), smoke staging directories and image tags (random). Diagnostics: scripts/lib/compose-diagnostics.sh prints the failed `up` command's output, `docker compose ps`, and the last 40 log lines of each service, all redacted (supplied literals, credential-looking NAME=value pairs, URL user information, Authorization headers) with terminal control sequences stripped. All three scripts use it at every readiness check; the infrastructure smoke's unredacted full-log dump is replaced by it. tests/ci-scripts.test.ts pins ephemeral ports, unique project names, the subnet choice, the CI image tag, the diagnostics, and the redaction itself (run through bash). Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
c737ebb472
commit
278e66049f
@@ -7,6 +7,8 @@ cd "$repository_root"
|
||||
|
||||
# shellcheck source=scripts/lib/compose-job-network.sh
|
||||
source "$repository_root/scripts/lib/compose-job-network.sh"
|
||||
# shellcheck source=scripts/lib/compose-diagnostics.sh
|
||||
source "$repository_root/scripts/lib/compose-diagnostics.sh"
|
||||
|
||||
command -v docker >/dev/null || {
|
||||
echo "Docker is required for the PostgreSQL migration integration suite." >&2
|
||||
@@ -25,14 +27,20 @@ if ! compose_job_network_initialize; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
raw_project="ekklesia-migration-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-0}-${GITHUB_JOB:-job}-$$"
|
||||
# Unique per invocation, not only per run: several runs (and several jobs of one run) can share a
|
||||
# runner's Docker daemon at once, and the shell's PID is often identical across job containers.
|
||||
raw_project="ekklesia-migration-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-0}-${GITHUB_JOB:-job}-$$-${RANDOM}${RANDOM}"
|
||||
project="${raw_project//[^a-zA-Z0-9_-]/-}"
|
||||
project="${project,,}"
|
||||
|
||||
export POSTGRES_DB="ekklesia_migration_test"
|
||||
export POSTGRES_USER="ekklesia_migration_test"
|
||||
export POSTGRES_PASSWORD="temporary-$project-${RANDOM}-${RANDOM}"
|
||||
# Both host ports are ephemeral: concurrent runs on one Docker daemon must never contend for a
|
||||
# fixed one. `.env.example` pins the storage port for local development, so it is overridden here
|
||||
# exactly like the database's. The bound ports are discovered below.
|
||||
export POSTGRES_PORT="0"
|
||||
export S3_PORT="0"
|
||||
|
||||
compose=(
|
||||
docker compose
|
||||
@@ -72,8 +80,17 @@ fi
|
||||
# and CI already knows how to provision. SeaweedFS is the intended deployment target (ADR 0004), and
|
||||
# proving the storage contract against a mock instead would have hidden the provider quirks this
|
||||
# suite exists to catch.
|
||||
if ! "${compose[@]}" up --detach --wait --wait-timeout 180 postgres object-storage >/dev/null 2>&1; then
|
||||
# Masked in any diagnostics below, along with anything else that looks like a credential.
|
||||
COMPOSE_DIAGNOSTICS_SECRETS=(
|
||||
"$POSTGRES_PASSWORD"
|
||||
"$(grep -E '^S3_ACCESS_KEY=' .env.example | cut -d= -f2-)"
|
||||
"$(grep -E '^S3_SECRET_KEY=' .env.example | cut -d= -f2-)"
|
||||
)
|
||||
if ! up_output="$(
|
||||
"${compose[@]}" up --detach --wait --wait-timeout 180 postgres object-storage 2>&1
|
||||
)"; then
|
||||
echo "Disposable PostgreSQL and object storage did not become ready." >&2
|
||||
compose_report_not_ready "$up_output" "${compose[@]}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
|
||||
Reference in New Issue
Block a user