fix(ci): isolate concurrent runs on a shared Docker daemon and explain readiness failures #52

2026-10-03T17:31:23Z

ekklesia-bot commented

2026-10-03 17:31:23 +00:00

fix(ci): isolate concurrent runs on a shared Docker daemon and explain readiness failures

Cause of PR #51's migration-smoke failure. scripts/test-migrations-integration.sh gave PostgreSQL an ephemeral host port (POSTGRES_PORT=0) but inherited S3_PORT=8333 from .env.example. Several runs were active on one runner's Docker daemon (Gitea 1.24 ignores concurrency), so they contended for host port 8333. Compose's own error was sent to /dev/null, so the job said only "did not become ready".

Isolation: audit of every script CI runs

Script / job	Shared resource	Change
`test-migrations-integration.sh`	host port 8333	`S3_PORT=0`; the bound port was already discovered with `compose port`
`test-migrations-integration.sh`, `smoke-infrastructure.sh`	Compose project name (run id + PID; job containers often share low PIDs)	random suffix
`smoke-deploy.sh`	edge subnet `10.249.<random>.0/24`, about a 1-in-200 collision for each pair of concurrent runs	skip subnets already on the daemon, and choose again on Docker's overlap error (up to 5 tries)
CI `image-build`	image tag `ekklesia-ci:$GITHUB_SHA` (a PR and a push of one commit share it; one run's cleanup removes the other's image)	tag adds `${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}`
Already isolated	every other published host port (ephemeral), the TLS suite's container and network (`pid` + random), smoke staging dirs and built image tags (random)	—

Not changed: the Dockerfile's BuildKit pnpm-store cache mount (id=ekklesia-pnpm-store) is shared between concurrent builds by design. The pnpm store is content-addressed and tolerates concurrent writers.

Readiness diagnostics

scripts/lib/compose-diagnostics.sh (compose_report_not_ready) prints:

the last 20 lines of the failed up;
docker compose ps --all;
the last 40 log lines of every service.

Everything passes through compose_diagnostics_redact, which masks:

literal values the script supplies (its generated passwords and keys);
credential-looking NAME=value / NAME: value pairs;
URL user information;
the rest of any Authorization header.

It also strips terminal control sequences. All three scripts use it at every readiness check. The infrastructure smoke's previous unredacted full-log dump on failure is replaced by it.

Example: a reproduced port collision now reports its cause.

Disposable PostgreSQL and object storage did not become ready.
--- docker compose up (last 20 lines)
...
Error response from daemon: driver failed programming external connectivity on endpoint ...-object-storage-1 (...): Bind for 127.0.0.1:8333 failed: port is already allocated
--- docker compose ps
...object-storage-1   ...   Created
...postgres-1         ...   Up Less than a second (health: starting)   127.0.0.1:32832->5432/tcp
--- docker compose logs postgres (last 40 lines)
...

Tests

tests/ci-scripts.test.ts covers:

ephemeral ports, and no fixed -p/PORT= anywhere in the CI scripts;
invocation-unique project names;
the subnet choice and retry;
the per-run CI image tag;
diagnostics wired into every readiness check, with no raw log printing;
the redaction itself, run through bash.

Verification (local)

pnpm ci:verify: passes.
pnpm test:migrations:integration: 306/306, with no S3_PORT override while a dev stack held 127.0.0.1:8333, which is exactly the collision that failed #51.
scripts/smoke-infrastructure.sh: passes.
scripts/smoke-deploy.sh: passes, run concurrently with the two above on one daemon.
Forced race: a probe network covering 10.249.0.0/16 made every subnet overlap. The smoke chose four times, then reported Docker's "networks have overlapping IPv4" with ps and logs, and left no containers or networks behind.

🤖 Generated with Claude Code

## fix(ci): isolate concurrent runs on a shared Docker daemon and explain readiness failures **Cause of PR #51's `migration-smoke` failure.** `scripts/test-migrations-integration.sh` gave PostgreSQL an ephemeral host port (`POSTGRES_PORT=0`) but inherited `S3_PORT=8333` from `.env.example`. Several runs were active on one runner's Docker daemon (Gitea 1.24 ignores `concurrency`), so they contended for host port 8333. Compose's own error was sent to `/dev/null`, so the job said only "did not become ready". ### Isolation: audit of every script CI runs | Script / job | Shared resource | Change | |---|---|---| | `test-migrations-integration.sh` | host port 8333 | `S3_PORT=0`; the bound port was already discovered with `compose port` | | `test-migrations-integration.sh`, `smoke-infrastructure.sh` | Compose project name (run id + PID; job containers often share low PIDs) | random suffix | | `smoke-deploy.sh` | edge subnet `10.249.<random>.0/24`, about a 1-in-200 collision for each pair of concurrent runs | skip subnets already on the daemon, and choose again on Docker's overlap error (up to 5 tries) | | CI `image-build` | image tag `ekklesia-ci:$GITHUB_SHA` (a PR and a push of one commit share it; one run's cleanup removes the other's image) | tag adds `${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}` | | Already isolated | every other published host port (ephemeral), the TLS suite's container and network (`pid` + random), smoke staging dirs and built image tags (random) | — | Not changed: the Dockerfile's BuildKit pnpm-store cache mount (`id=ekklesia-pnpm-store`) is shared between concurrent builds by design. The pnpm store is content-addressed and tolerates concurrent writers. ### Readiness diagnostics `scripts/lib/compose-diagnostics.sh` (`compose_report_not_ready`) prints: - the last 20 lines of the failed `up`; - `docker compose ps --all`; - the last 40 log lines of every service. Everything passes through `compose_diagnostics_redact`, which masks: - literal values the script supplies (its generated passwords and keys); - credential-looking `NAME=value` / `NAME: value` pairs; - URL user information; - the rest of any Authorization header. It also strips terminal control sequences. All three scripts use it at every readiness check. The infrastructure smoke's previous unredacted full-log dump on failure is replaced by it. **Example: a reproduced port collision now reports its cause.** ``` Disposable PostgreSQL and object storage did not become ready. --- docker compose up (last 20 lines) ... Error response from daemon: driver failed programming external connectivity on endpoint ...-object-storage-1 (...): Bind for 127.0.0.1:8333 failed: port is already allocated --- docker compose ps ...object-storage-1 ... Created ...postgres-1 ... Up Less than a second (health: starting) 127.0.0.1:32832->5432/tcp --- docker compose logs postgres (last 40 lines) ... ``` ### Tests `tests/ci-scripts.test.ts` covers: - ephemeral ports, and no fixed `-p`/`PORT=` anywhere in the CI scripts; - invocation-unique project names; - the subnet choice and retry; - the per-run CI image tag; - diagnostics wired into every readiness check, with no raw log printing; - the redaction itself, run through bash. ### Verification (local) - **`pnpm ci:verify`**: passes. - **`pnpm test:migrations:integration`**: 306/306, with **no** `S3_PORT` override while a dev stack held 127.0.0.1:8333, which is exactly the collision that failed #51. - **`scripts/smoke-infrastructure.sh`**: passes. - **`scripts/smoke-deploy.sh`**: passes, run concurrently with the two above on one daemon. - **Forced race:** a probe network covering `10.249.0.0/16` made every subnet overlap. The smoke chose four times, then reported Docker's "networks have overlapping IPv4" with ps and logs, and left no containers or networks behind. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

ekklesia-bot added 1 commit 2026-10-03 17:31:24 +00:00

fix(ci): isolate concurrent runs on a shared Docker daemon and explain readiness failures

CI / verify (pull_request) Successful in 50s

Details

CI / infrastructure-smoke (pull_request) Successful in 14s

Details

CI / migration-smoke (pull_request) Successful in 1m49s

Details

CI / image-build (pull_request) Successful in 19s

Details

CI / deploy-smoke (pull_request) Successful in 41s

Details

278e66049f

PR #51's migration-smoke failed 0.7 s after the build: the suite pinned
PostgreSQL to an ephemeral port but inherited S3_PORT=8333 from
.env.example, so concurrent runs on one runner (Gitea ignores workflow
concurrency) contended for one host port. Compose's own error was
discarded, so the failure said only that the services "did not become
ready".

Isolation, from an audit of every script CI runs:
- test-migrations-integration.sh: S3_PORT=0, like POSTGRES_PORT=0.
- test-migrations-integration.sh and smoke-infrastructure.sh: project names
  gain a random suffix; run id plus PID alone is not unique when job
  containers share low PIDs.
- smoke-deploy.sh: the edge /24 is chosen from those no network on the
  daemon uses, and the first `up` chooses again when a concurrent run wins
  a race for the same subnet (Docker's address-pool overlap), up to five
  times.
- CI image-build: the image tag includes the run id and attempt, so a pull
  request and a push of the same commit cannot remove each other's image.
- Already isolated: every other published port (ephemeral), the TLS
  suite's container and network (random), smoke staging directories and
  image tags (random).

Diagnostics: scripts/lib/compose-diagnostics.sh prints the failed `up`
command's output, `docker compose ps`, and the last 40 log lines of each
service, all redacted (supplied literals, credential-looking NAME=value
pairs, URL user information, Authorization headers) with terminal control
sequences stripped. All three scripts use it at every readiness check; the
infrastructure smoke's unredacted full-log dump is replaced by it.

tests/ci-scripts.test.ts pins ephemeral ports, unique project names, the
subnet choice, the CI image tag, the diagnostics, and the redaction itself
(run through bash).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

ekklesia-bot referenced this pull request

2026-10-03 17:31:41 +00:00

feat: support split Dashboard and API origins (ADR 0045) #54

Zayin_Admin approved these changes

2026-10-03 18:28:12 +00:00

Zayin_Admin merged commit b642d8fe30 into main

2026-10-03 18:28:18 +00:00

Zayin_Admin deleted branch fix/ci-concurrent-runs

2026-10-03 18:28:18 +00:00

ekklesia-bot referenced this pull request

2026-10-03 18:46:06 +00:00

Docs/adr 0045 split origins #51