Commit Graph
6 Commits
Author SHA1 Message Date
ekklesiaandClaude Opus 5.5 62951975da fix(ci): nothing a CI job starts outlives it, even when the runner kills the job
CI / verify (pull_request) Successful in 57s
CI / infrastructure-smoke (pull_request) Successful in 28s
CI / migration-smoke (pull_request) Successful in 1m49s
CI / image-build (pull_request) Successful in 19s
CI / deploy-smoke (same-origin) (pull_request) Successful in 48s
CI / deploy-smoke (split) (pull_request) Successful in 53s
A migration-smoke stack (PostgreSQL and SeaweedFS) ran on the shared
runner for two days. Every CI script removes its stack in an EXIT trap,
but a trap never runs when the runner kills the job container: a
cancelled job, or the runner's own time limit (Gitea 1.24 ignores
timeout-minutes). Reproduced: SIGKILL of the infrastructure smoke
mid-run leaves exactly that stack running.

Cleanup now has three layers:
- the script's own trap, unchanged;
- a final `if: always()` step in every job that starts stacks
  (infrastructure-smoke, migration-smoke, image-build, deploy-smoke)
  runs `scripts/ci-sweep.sh --job`, which removes everything that job
  started at any age;
- a first step in each of those jobs runs `scripts/ci-sweep.sh --stale`,
  which removes CI stacks older than three hours: what a killed job left
  before it could clean up.

How stacks are found is defined once, in scripts/lib/ci-project.sh:
- every CI Compose project is named `ekklesia-ci-<run>-<attempt>-<job
  key>-<kind>-<unique>`, and Compose labels its containers, networks and
  volumes with that name;
- the deploy smoke labels everything it starts outside Compose:
  browser, kiosk and driver containers, the kiosk volume, built images;
- images also record when CI built them. An image's own creation time
  can be its base's (BuildKit, metadata-only builds), and the sweep
  removed a brand-new image as "old" until it stopped trusting that.

The job key is per matrix leg, so one topology's cleanup cannot remove
the other's running stack. The sweep never touches anything outside the
ekklesia-ci- prefix, disconnects rather than removes a job container
attached to a stack's network, and refuses --job outside CI. Network
creation times are read as RFC 3339; Docker's default form was
unparseable.

scripts/test-ci-sweep.sh proves all of that against real Docker, and
CI runs it before relying on it. security.md and testing.md document
the rule, and that it complements, not replaces, a runner-side reaper.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-06 02:01:01 +00:00
ekklesiaandClaude Opus 5.5 54ac6ab83f test(deploy): Doctor must check the client-address configuration the API runs
Behind the reference proxy the API runs in trusted-proxy mode, but
compose.proxy.yaml gives those variables to `api` only. The documented
Doctor command runs as `migrate`, so it validated an empty
configuration and reported "Direct client-address mode" for an
installation that trusts its proxy. That is what the first production
installation saw.

Fails today:
- deploy.test.ts: the proxy overlay gives `migrate` no client-address
  environment;
- doctor.test.ts: the check does not say whose configuration it read;
- smoke-deploy.sh: Doctor through the proxy overlay does not report
  trusted-proxy mode (Docker).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-06 00:42:38 +00:00
ekklesiaandClaude Opus 5.5 ff8a3e8097 test(deploy): add opt-in display-device scenarios to the split smoke
CI / verify (pull_request) Successful in 53s
CI / infrastructure-smoke (pull_request) Successful in 13s
CI / migration-smoke (pull_request) Successful in 1m48s
CI / image-build (pull_request) Successful in 20s
CI / deploy-smoke (same-origin) (pull_request) Successful in 48s
CI / deploy-smoke (split) (pull_request) Successful in 55s
Reproduction harness for what the first production Signage deployment
found. With EKKLESIA_SMOKE_DISPLAY_DEVICE=1, the split smoke starts a
second headless Chromium (the "kiosk") whose profile directory is a
Docker volume. It pairs that kiosk through the real flow, closes it
cleanly, restarts it on the same profile, and then runs injected-failure
scenarios: two pages on one credential, a connectivity blip, one refused
viewer-session request, one refused realtime connection, a cross-origin
image-caching probe, a full realtime recycle, and revocation.

Every DevTools command and page-side request has its own deadline, and
each step has a budget. A step that overruns prints the exact call it
was waiting for. The proxy's "aborting with incomplete response" lines
are listed afterwards by UTC time and path only.

Off by default: several scenarios expect Signage and fail on today's
v0.1.1, by design.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 18:25:12 -05:00
ekklesiaandClaude Opus 5.5 f63f80f2f0 feat: support split Dashboard and API origins (ADR 0045)
CI / verify (pull_request) Successful in 49s
CI / infrastructure-smoke (pull_request) Successful in 13s
CI / migration-smoke (pull_request) Successful in 1m47s
CI / image-build (pull_request) Successful in 19s
CI / deploy-smoke (same-origin) (pull_request) Successful in 49s
CI / deploy-smoke (split) (pull_request) Successful in 1m4s
An installation can now serve the Dashboard and the API on sibling
hostnames (dash./api.) beside the default same-origin topology, from one
image and with configuration alone.

- The Dashboard learns its API origin at run time: the built page carries
  one empty <meta name="ekklesia-api-origin">, which the API fills with its
  identity base URL's origin at startup, before computing the page's ETag.
  The shell accepts only a canonical http(s) origin and otherwise fails
  closed without making a request. The build-time EKKLESIA_API_BASE_URL is
  gone from the bundle; the development server injects the same element.
- The display host stays on the API's origin, so its device cookies,
  requests, and offline cache are unchanged.
- The API trusts its own origin (trusted origins plus the identity base
  URL), as Better Auth already does, so the display page's mutations pass
  the origin check without operators listing it.
- Doctor validates EKKLESIA_DASHBOARD_ORIGIN exactly as the API's startup
  does (it previously skipped it), reports the topology without printing
  hostnames, and warns when split hostnames are not siblings.
- The module UI contract states the CORS header limit (safelisted headers
  plus Content-Type, safelisted response headers only).
- Reference deployment: Caddyfile.split.example (Dashboard hostname serves
  / and its script; API hostname serves /api/*, /display/*, and the
  display script; each refuses the other's paths) and
  EKKLESIA_DASHBOARD_HOSTNAME in the proxy overlay and env examples.
- The deploy smoke test runs either topology (CI runs both). The split run
  installs the module template as a test-only module, checks per-hostname
  routing and exact CORS, then drives a digest-pinned headless Chromium
  over the DevTools protocol, with no added dependency: sign-in on the
  Dashboard hostname, the session cookie host-only on the API hostname,
  the module UI imported cross-origin and its credentialed request
  succeeding, the display and its service worker on the API hostname, and
  sign-out.

ADR 0045 is marked Accepted.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 14:47:36 -05:00
ekklesiaandClaude Opus 5.5 278e66049f fix(ci): isolate concurrent runs on a shared Docker daemon and explain readiness failures
CI / verify (pull_request) Successful in 50s
CI / infrastructure-smoke (pull_request) Successful in 14s
CI / migration-smoke (pull_request) Successful in 1m49s
CI / image-build (pull_request) Successful in 19s
CI / deploy-smoke (pull_request) Successful in 41s
PR #51's migration-smoke failed 0.7 s after the build: the suite pinned
PostgreSQL to an ephemeral port but inherited S3_PORT=8333 from
.env.example, so concurrent runs on one runner (Gitea ignores workflow
concurrency) contended for one host port. Compose's own error was
discarded, so the failure said only that the services "did not become
ready".

Isolation, from an audit of every script CI runs:
- test-migrations-integration.sh: S3_PORT=0, like POSTGRES_PORT=0.
- test-migrations-integration.sh and smoke-infrastructure.sh: project names
  gain a random suffix; run id plus PID alone is not unique when job
  containers share low PIDs.
- smoke-deploy.sh: the edge /24 is chosen from those no network on the
  daemon uses, and the first `up` chooses again when a concurrent run wins
  a race for the same subnet (Docker's address-pool overlap), up to five
  times.
- CI image-build: the image tag includes the run id and attempt, so a pull
  request and a push of the same commit cannot remove each other's image.
- Already isolated: every other published port (ephemeral), the TLS
  suite's container and network (random), smoke staging directories and
  image tags (random).

Diagnostics: scripts/lib/compose-diagnostics.sh prints the failed `up`
command's output, `docker compose ps`, and the last 40 log lines of each
service, all redacted (supplied literals, credential-looking NAME=value
pairs, URL user information, Authorization headers) with terminal control
sequences stripped. All three scripts use it at every readiness check; the
infrastructure smoke's unredacted full-log dump is replaced by it.

tests/ci-scripts.test.ts pins ephemeral ports, unique project names, the
subnet choice, the CI image tag, the diagnostics, and the redaction itself
(run through bash).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 12:15:01 -05:00
ekklesiaandClaude Opus 5.5 167b4c4f14 feat(deploy): add the reference deployment Compose project
CI / verify (pull_request) Successful in 54s
CI / infrastructure-smoke (pull_request) Successful in 21s
CI / migration-smoke (pull_request) Successful in 1m48s
CI / image-build (pull_request) Successful in 27s
CI / deploy-smoke (pull_request) Successful in 42s
Implement ADR 0044's reference deployment as a base Compose file plus
overlays, so an installation that already runs a proxy can use the base
alone.

- compose.yaml: api, postgres, object-storage, and the operator-only
  migrate (profile ops, plan by default). No published port. api and
  migrate run init, read-only, tmpfs /tmp, cap_drop ALL,
  no-new-privileges. postgres is only on the internal data network.
- TLS everywhere from one installation-private CA: PostgreSQL TLS 1.3
  only, with pg_hba rejecting every non-TLS connection, and the API on
  verify-full via EKKLESIA_DATABASE_CA (PEM text). SeaweedFS serves S3
  over HTTPS (ADR 0030 requires https in production), trusted by the
  API through NODE_EXTRA_CA_CERTS. No Core change.
- compose.proxy.yaml + Caddyfile.example: Caddy pinned by digest, ACME
  HTTP-01 (DNS-01 documented), routes /, /api/*, /display/*, /assets/*
  to api and refuses everything else; the storage hostname preserves
  Host. The API trusts exactly the proxy's fixed /32 on an edge network
  whose dynamic range excludes the fixed addresses; Caddy overwrites
  X-Forwarded-For and strips caller-supplied forwarding headers.
- compose.tunnel.yaml: cloudflared pinned by digest, token from its own
  env file; Caddy honors CF-Connecting-IP only from its fixed address.
- Four per-service root-owned env files, examples with no values;
  scripts/init-tls.sh (idempotent, never overwrites a key) and
  scripts/gen-secrets.sh (prints only).
- CI: pnpm deploy:config in ci:verify for all three combinations, and a
  deploy-smoke job (scripts/smoke-deploy.sh) proving readiness, the
  Dashboard, signed storage delivery, the client-address chain, initial
  owner sign-in/out, verify-full database TLS, and Doctor through the
  proxy, against disposable services. tests/deploy.test.ts pins the
  structural guarantees.
- ADR 0044 (still Proposed) amended to match: API-to-storage HTTPS,
  DNS-01 needs a custom Caddy build and tunnel-only needs DNS-01, the
  tunnel's origin TLS settings live in the Cloudflare dashboard, port
  80 serves HTTP-01, and secrets are one env file per service.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-01 20:59:38 -05:00