fix(ci): nothing a CI job starts outlives it, even when the runner kills the job

A migration-smoke stack (PostgreSQL and SeaweedFS) ran on the shared
runner for two days. Every CI script removes its stack in an EXIT trap,
but a trap never runs when the runner kills the job container: a
cancelled job, or the runner's own time limit (Gitea 1.24 ignores
timeout-minutes). Reproduced: SIGKILL of the infrastructure smoke
mid-run leaves exactly that stack running.

Cleanup now has three layers:
- the script's own trap, unchanged;
- a final `if: always()` step in every job that starts stacks
  (infrastructure-smoke, migration-smoke, image-build, deploy-smoke)
  runs `scripts/ci-sweep.sh --job`, which removes everything that job
  started at any age;
- a first step in each of those jobs runs `scripts/ci-sweep.sh --stale`,
  which removes CI stacks older than three hours: what a killed job left
  before it could clean up.

How stacks are found is defined once, in scripts/lib/ci-project.sh:
- every CI Compose project is named `ekklesia-ci-<run>-<attempt>-<job
  key>-<kind>-<unique>`, and Compose labels its containers, networks and
  volumes with that name;
- the deploy smoke labels everything it starts outside Compose:
  browser, kiosk and driver containers, the kiosk volume, built images;
- images also record when CI built them. An image's own creation time
  can be its base's (BuildKit, metadata-only builds), and the sweep
  removed a brand-new image as "old" until it stopped trusting that.

The job key is per matrix leg, so one topology's cleanup cannot remove
the other's running stack. The sweep never touches anything outside the
ekklesia-ci- prefix, disconnects rather than removes a job container
attached to a stack's network, and refuses --job outside CI. Network
creation times are read as RFC 3339; Docker's default form was
unparseable.

scripts/test-ci-sweep.sh proves all of that against real Docker, and
CI runs it before relying on it. security.md and testing.md document
the rule, and that it complements, not replaces, a runner-side reaper.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

.github/workflows/ci.yml

@@ -63,6 +63,9 @@ jobs:
     needs: verify
     runs-on: ubuntu-latest
     timeout-minutes: 10
+    env:
+      # Names this job's stacks, so its final step removes exactly these (scripts/lib/ci-project.sh).
+      EKKLESIA_CI_JOB_KEY: infrastructure-smoke
     steps:
       - name: Check out repository
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
@@ -72,13 +75,29 @@ jobs:
           # in this workflow pushes, so the credential is never needed after checkout.
           persist-credentials: false
 
+      # A job the runner killed (cancelled, or past the runner's own time limit) never ran its
+      # cleanup. Anything a CI job left on this shared daemon more than three hours ago goes first.
+      - name: Sweep stale CI stacks from the runner
+        run: bash scripts/ci-sweep.sh --stale
+
+      - name: Prove the sweep removes exactly what it should, against real Docker resources
+        run: bash scripts/test-ci-sweep.sh
+
       - name: Exercise PostgreSQL and S3 infrastructure
         run: bash scripts/smoke-infrastructure.sh
 
+      # Whatever happened above, this job's own stacks do not outlive it.
+      - name: Remove this job's CI stacks
+        if: always()
+        run: bash scripts/ci-sweep.sh --job
+
   migration-smoke:
     needs: verify
     runs-on: ubuntu-latest
     timeout-minutes: 15
+    env:
+      # Names this job's stacks, so its final step removes exactly these (scripts/lib/ci-project.sh).
+      EKKLESIA_CI_JOB_KEY: migration-smoke
     steps:
       - name: Check out repository
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
@@ -88,6 +107,11 @@ jobs:
           # in this workflow pushes, so the credential is never needed after checkout.
           persist-credentials: false
 
+      # A job the runner killed (cancelled, or past the runner's own time limit) never ran its
+      # cleanup. Anything a CI job left on this shared daemon more than three hours ago goes first.
+      - name: Sweep stale CI stacks from the runner
+        run: bash scripts/ci-sweep.sh --stale
+
       - name: Set up Node.js
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
@@ -104,11 +128,18 @@ jobs:
       - name: Exercise transactional migrations against PostgreSQL
         run: pnpm test:migrations:integration
 
+      # Whatever happened above, this job's own stacks do not outlive it.
+      - name: Remove this job's CI stacks
+        if: always()
+        run: bash scripts/ci-sweep.sh --job
+
   image-build:
     needs: verify
     runs-on: ubuntu-latest
     timeout-minutes: 20
     env:
+      # Names this job's stacks, so its final step removes exactly these (scripts/lib/ci-project.sh).
+      EKKLESIA_CI_JOB_KEY: image-build
       # Cache mounts in deploy/Dockerfile need BuildKit; this selects it even where the Docker CLI
       # has no buildx plugin.
       DOCKER_BUILDKIT: "1"
@@ -121,22 +152,33 @@ jobs:
           # in this workflow pushes, so the credential is never needed after checkout.
           persist-credentials: false
 
+      # A job the runner killed (cancelled, or past the runner's own time limit) never ran its
+      # cleanup. Anything a CI job left on this shared daemon more than three hours ago goes first.
+      - name: Sweep stale CI stacks from the runner
+        run: bash scripts/ci-sweep.sh --stale
+
       # The platform-only image (ADR 0044): a checkout has no `modules/`, so this proves the image
       # builds, validates, and assembles with zero modules. Build only — no registry login, no
       # credentials, no push, no tag anywhere but this runner's local image store.
       # Tagged per run and attempt, not only per commit: a pull request and a push of the same commit
       # can build at once on one runner's Docker daemon, and neither may remove the other's image.
+      # Labeled like every CI stack, so this job's final step removes it even if the build is killed.
       - name: Build the production image without modules
-        run: >-
-          docker build
-          --file deploy/Dockerfile
-          --build-arg REVISION="$GITHUB_SHA"
-          --tag ekklesia-ci:"$GITHUB_SHA-${GITHUB_RUN_ID:-0}-${GITHUB_RUN_ATTEMPT:-1}"
+        shell: bash
+        run: |
+          source scripts/lib/ci-project.sh
+          docker build \
+            --file deploy/Dockerfile \
+            --build-arg REVISION="$GITHUB_SHA" \
+            --label "${EKKLESIA_CI_LABEL}=$(ci_project_name image)" \
+            --label "${EKKLESIA_CI_CREATED_LABEL}=$(date +%s)" \
+            --tag ekklesia-ci:"$GITHUB_SHA-${GITHUB_RUN_ID:-0}-${GITHUB_RUN_ATTEMPT:-1}" \
             .
 
-      - name: Remove the image from the runner
+      # Whatever happened above, this job's own stacks do not outlive it.
+      - name: Remove this job's CI stacks
         if: always()
-        run: docker image rm --force ekklesia-ci:"$GITHUB_SHA-${GITHUB_RUN_ID:-0}-${GITHUB_RUN_ATTEMPT:-1}" || true
+        run: bash scripts/ci-sweep.sh --job
 
   deploy-smoke:
     needs: verify
@@ -151,6 +193,8 @@ jobs:
           - same-origin
           - split
     env:
+      # Names this job's stacks, so its final step removes exactly these (scripts/lib/ci-project.sh).
+      EKKLESIA_CI_JOB_KEY: deploy-smoke-${{ matrix.topology }}
       DOCKER_BUILDKIT: "1"
       EKKLESIA_SMOKE_TOPOLOGY: ${{ matrix.topology }}
     steps:
@@ -162,6 +206,11 @@ jobs:
           # in this workflow pushes, so the credential is never needed after checkout.
           persist-credentials: false
 
+      # A job the runner killed (cancelled, or past the runner's own time limit) never ran its
+      # cleanup. Anything a CI job left on this shared daemon more than three hours ago goes first.
+      - name: Sweep stale CI stacks from the runner
+        run: bash scripts/ci-sweep.sh --stale
+
       # The reference deployment end to end (ADR 0044): a platform-only image, base + proxy with
       # Caddy's internal CA (test-only), an explicit migration apply against this job's own fresh
       # PostgreSQL, then the API, Dashboard, storage delivery, client-address chain, and verify-full
@@ -169,3 +218,8 @@ jobs:
       # it; nothing is pushed, published, or logged in to.
       - name: Smoke-test the reference deployment (${{ matrix.topology }})
         run: bash scripts/smoke-deploy.sh
+
+      # Whatever happened above, this job's own stacks do not outlive it.
+      - name: Remove this job's CI stacks
+        if: always()
+        run: bash scripts/ci-sweep.sh --job

core/src/diagnostics/doctor.ts

@@ -905,17 +905,25 @@ function databaseTransportCheck(
   }
 }
 
+/** Doctor reads only its own process's environment, so it says so: run as the reference
+ * deployment's `migrate`, it is given the API's values by the proxy overlay, but run any other way
+ * it may be checking a configuration the API is not running. */
+const CLIENT_ADDRESS_SCOPE =
+  "Checked: the environment Doctor runs with. In the reference deployment `migrate` is given the API's client-address values; a running API's own value is shown by `docker compose exec api printenv EKKLESIA_CLIENT_ADDRESS_MODE`.";
+
 function clientAddressCheck(input: ClientAddressConfigInput | undefined): DoctorCheck {
   try {
     const config = createClientAddressConfig(input, TRUSTED_CLIENT_ADDRESS_HEADER);
     if (config.mode === "direct") {
       return freezeCheck("identity.client-address", "pass", "Direct client-address mode", [
         "Trusts only the actual immediate socket peer for rate limiting; no forwarded header is trusted.",
+        CLIENT_ADDRESS_SCOPE,
       ]);
     }
     return freezeCheck("identity.client-address", "pass", "Trusted-proxy client-address mode", [
       `Trusts header "${config.header}" only from a request whose immediate socket peer matches one of ${config.trustedProxies.length} configured trusted ${plural(config.trustedProxies.length, "proxy")}.`,
       "The reverse proxy must overwrite (never append to) this header, and the origin must not be reachable except through it.",
+      CLIENT_ADDRESS_SCOPE,
     ]);
   } catch (error) {
     const code =
@@ -926,7 +934,7 @@ function clientAddressCheck(input: ClientAddressConfigInput | undefined): Doctor
       "identity.client-address",
       "failure",
       "Client-address configuration invalid",
-      [`[${code}] Configured client-address values failed validation.`],
+      [`[${code}] Configured client-address values failed validation.`, CLIENT_ADDRESS_SCOPE],
     );
   }
 }

core/tests/doctor-report.test.ts

@@ -117,6 +117,7 @@ Identity
     Identity runtime configuration (signing key, base URL, trusted origins) is not set.
   ✓ Direct client-address mode
     Trusts only the actual immediate socket peer for rate limiting; no forwarded header is trusted.
+    Checked: the environment Doctor runs with. In the reference deployment \`migrate\` is given the API's client-address values; a running API's own value is shown by \`docker compose exec api printenv EKKLESIA_CLIENT_ADDRESS_MODE\`.
 
 Object storage
   ○ Object storage not configured

core/tests/doctor.test.ts

@@ -958,6 +958,21 @@ describe("Ekklesia Doctor diagnostics", () => {
     expect(report.result).toBe("ready-with-warnings");
   });
 
+  it("says whose client-address configuration it checked: this process's, not a running API's", async () => {
+    for (const clientAddress of [
+      undefined,
+      { header: "x-forwarded-for", mode: "trusted-proxy" as const, trustedProxies: ["10.0.0.1"] },
+      { header: "x-forwarded-for", mode: "trusted-proxy" as const, trustedProxies: ["bad"] },
+    ]) {
+      const report = await diagnoseEkklesia(
+        { ...options(), ...(clientAddress ? { clientAddress } : {}) },
+        dependencies(),
+      );
+      const details = check(report, "identity.client-address").details;
+      expect(details.some((detail) => detail.includes("environment Doctor runs with"))).toBe(true);
+    }
+  });
+
   it("passes client-address configuration with an explicitly configured trusted-proxy mode", async () => {
     const report = await diagnoseEkklesia(
       {

deploy/Caddyfile.example

@@ -70,18 +70,57 @@
 
 # Browsers fetch signed object URLs here (S3_DELIVERY_ENDPOINT, ADR 0031).
 {$EKKLESIA_STORAGE_HOSTNAME} {
+	# Delivery-layer CORS belongs to the deployment, not to the object store: this block decides it,
+	# so it stays the same with any S3 backend behind it. Only the origins whose pages read objects
+	# with fetch() get CORS: the Ekklesia origin, which serves the Dashboard and the displays. An <img> needs none. Every object
+	# URL is still a signed, expiring capability; CORS only decides which pages may read the bytes.
+	@cors_origin header Origin https://{$EKKLESIA_HOSTNAME}
+
+	# A preflight is answered here and never reaches storage. Only GET and HEAD are ever allowed, and
+	# no request header beyond the CORS-safelisted ones.
+	@preflight method OPTIONS
+	handle @preflight {
+		header @cors_origin {
+			Access-Control-Allow-Origin {http.request.header.Origin}
+			Access-Control-Allow-Methods "GET, HEAD"
+			Access-Control-Max-Age 600
+		}
+		header Vary Origin
+		respond 204
+	}
+
+	handle {
+		# Set when the response is written, after the upstream's own CORS headers were removed below.
+		header @cors_origin >Access-Control-Allow-Origin {http.request.header.Origin}
+		header >Vary Origin
+
 		reverse_proxy https://object-storage:8443 {
-		# An S3 signature covers the Host header: pass the original through unchanged, or every signed
-		# URL fails verification.
+			# An S3 signature covers the Host header: pass the original through unchanged, or every
+			# signed URL fails verification.
 			header_up Host {host}
 			header_up -CF-Connecting-IP
 			header_up -X-Real-IP
 			header_up -Forwarded
 			header_up -True-Client-IP
+			# The object store's own CORS answer (SeaweedFS reflects any origin, with every method)
+			# is not the deployment's policy.
+			header_down -Access-Control-Allow-Origin
+			header_down -Access-Control-Allow-Methods
+			header_down -Access-Control-Allow-Headers
+			header_down -Access-Control-Expose-Headers
+			header_down -Access-Control-Allow-Credentials
+			header_down -Access-Control-Max-Age
+			header_down -Vary
 			transport http {
 				# object-storage presents a certificate from the installation-private CA.
 				tls_trust_pool file /etc/caddy/ekklesia-ca.crt
 				tls_server_name object-storage
+				# HTTP/1.1 only. Over HTTP/2, SeaweedFS answers a conditional request with a 304 that
+				# declares a Content-Length it never sends; Caddy then aborts the browser's stream,
+				# which a browser reports as ERR_HTTP2_PROTOCOL_ERROR on every revalidation. Its
+				# HTTP/1.1 304 is clean. Browsers still reach this proxy over HTTP/2 and HTTP/3.
+				versions 1.1
+			}
 		}
 	}
 }

deploy/Caddyfile.split.example

@@ -92,18 +92,61 @@
 
 # Browsers fetch signed object URLs here (S3_DELIVERY_ENDPOINT, ADR 0031).
 {$EKKLESIA_STORAGE_HOSTNAME} {
+	# Delivery-layer CORS belongs to the deployment, not to the object store: this block decides it,
+	# so it stays the same with any S3 backend behind it. Only the origins whose pages read objects
+	# with fetch() get CORS: the API origin, where displays run, and the Dashboard origin,
+	# where a module's preview may cache images too. An <img> needs none. Every object
+	# URL is still a signed, expiring capability; CORS only decides which pages may read the bytes.
+	@cors_origin {
+		header Origin https://{$EKKLESIA_HOSTNAME}
+		header Origin https://{$EKKLESIA_DASHBOARD_HOSTNAME}
+	}
+
+	# A preflight is answered here and never reaches storage. Only GET and HEAD are ever allowed, and
+	# no request header beyond the CORS-safelisted ones.
+	@preflight method OPTIONS
+	handle @preflight {
+		header @cors_origin {
+			Access-Control-Allow-Origin {http.request.header.Origin}
+			Access-Control-Allow-Methods "GET, HEAD"
+			Access-Control-Max-Age 600
+		}
+		header Vary Origin
+		respond 204
+	}
+
+	handle {
+		# Set when the response is written, after the upstream's own CORS headers were removed below.
+		header @cors_origin >Access-Control-Allow-Origin {http.request.header.Origin}
+		header >Vary Origin
+
 		reverse_proxy https://object-storage:8443 {
-		# An S3 signature covers the Host header: pass the original through unchanged, or every signed
-		# URL fails verification.
+			# An S3 signature covers the Host header: pass the original through unchanged, or every
+			# signed URL fails verification.
 			header_up Host {host}
 			header_up -CF-Connecting-IP
 			header_up -X-Real-IP
 			header_up -Forwarded
 			header_up -True-Client-IP
+			# The object store's own CORS answer (SeaweedFS reflects any origin, with every method)
+			# is not the deployment's policy.
+			header_down -Access-Control-Allow-Origin
+			header_down -Access-Control-Allow-Methods
+			header_down -Access-Control-Allow-Headers
+			header_down -Access-Control-Expose-Headers
+			header_down -Access-Control-Allow-Credentials
+			header_down -Access-Control-Max-Age
+			header_down -Vary
 			transport http {
 				# object-storage presents a certificate from the installation-private CA.
 				tls_trust_pool file /etc/caddy/ekklesia-ca.crt
 				tls_server_name object-storage
+				# HTTP/1.1 only. Over HTTP/2, SeaweedFS answers a conditional request with a 304 that
+				# declares a Content-Length it never sends; Caddy then aborts the browser's stream,
+				# which a browser reports as ERR_HTTP2_PROTOCOL_ERROR on every revalidation. Its
+				# HTTP/1.1 304 is clean. Browsers still reach this proxy over HTTP/2 and HTTP/3.
+				versions 1.1
+			}
 		}
 	}
 }

deploy/README.md

@@ -178,6 +178,29 @@ the topology and warns when the two hostnames are not siblings.
 The Dashboard learns the API's origin at run time: the API writes its identity base URL into the page
 it serves (`<meta name="ekklesia-api-origin">`). Nothing about the topology is built into the image.
 
+### Storage delivery: CORS is the deployment's
+
+Browsers fetch objects directly from the storage hostname through signed, expiring URLs (ADR 0031).
+**Which pages may read those bytes with `fetch()` is decided by the delivery layer, the proxy, and
+never by the object store.** That policy therefore stays the same with SeaweedFS, R2, or any other
+S3 backend behind it, and an installation that changes backends changes nothing about CORS.
+
+The reference proxy's storage block:
+
+- grants CORS only to the origins whose pages read objects with `fetch()`: the Ekklesia origin in the
+  same-origin topology; the API origin (displays) and the Dashboard origin (a module's preview) in
+  the split topology. An `<img>` needs no CORS and is unaffected;
+- allows `GET` and `HEAD` only, and no request header beyond the CORS-safelisted ones, always with
+  `Vary: Origin`;
+- answers preflight `OPTIONS` requests itself, so they never reach storage;
+- removes the object store's own CORS headers. SeaweedFS's default reflects any origin and
+  advertises `PUT`, `POST`, and `DELETE`.
+
+The proxy reaches storage over **HTTP/1.1 only**. Over HTTP/2, SeaweedFS answers a conditional
+request with a `304` that declares a `Content-Length` it never sends. The proxy then resets the
+browser's stream, and every reload of a cached image fails with `ERR_HTTP2_PROTOCOL_ERROR`.
+Browsers still reach the proxy itself over HTTP/2 and HTTP/3.
+
 ### Behind an existing reverse proxy
 
 Run the base alone and attach `api` and `object-storage` to your proxy's network with an overlay of
@@ -213,6 +236,8 @@ Then make your proxy do what `Caddyfile.example` does:
 - forward the storage hostname to `https://object-storage:8443`, verifying it against
   `/etc/ekklesia/tls/ca.crt` with server name `object-storage`, and **pass the original `Host`
   through unchanged**, or every signed URL fails (ADR 0031).
+- reach `object-storage` over **HTTP/1.1** only, and own the storage hostname's CORS as described in
+  [Storage delivery](#storage-delivery-cors-is-the-deployments).
 
 The proxy's address must be fixed and must never be handed to anything else. With Docker, give its
 network an explicit subnet, limit `ip_range` to part of it, and fix the proxy's address outside that
@@ -288,7 +313,10 @@ deployment:
   platform-only image and a disposable PostgreSQL and SeaweedFS that exist only for that run. It
   follows the first boot above and checks the following through the proxy:
   - readiness and the Dashboard;
-  - a signed object URL fetched through the storage hostname;
+  - a signed object URL fetched through the storage hostname, and the same image loaded twice over
+    HTTP/2, the second time as a conditional request that must be a clean `304`;
+  - storage CORS: exactly the allowed origins, `GET` and `HEAD` only, preflights answered by the
+    proxy, and none of the object store's own CORS headers;
   - the client-address chain, both from an ordinary peer and from the tunnel connector's address;
   - initial-owner sign-in and sign-out;
   - that the API's database connection is TLS 1.3, and that it is refused by a foreign CA and by a
@@ -440,6 +468,14 @@ for a development checkout, not for the image.
 `migrate` has `profiles: ["ops"]`, so `up` never starts it. It never restarts, has no health check,
 and its default command is the read-only plan. **Nothing applies migrations automatically.**
 
+Run operator commands with the same Compose files as the stack. `COMPOSE_FILE` in `deploy/.env`
+(set in [`env/compose.env.example`](env/compose.env.example)) does that for every plain
+`docker compose` command below. It matters for Doctor in particular: Doctor checks the
+client-address configuration of the process it runs in. The proxy overlay gives `migrate` the
+API's trusted-proxy settings, so Doctor reports what the API runs. Without the overlay, Doctor
+reports direct mode for an API that trusts its proxy. The running API's own value is shown by
+`docker compose exec api printenv EKKLESIA_CLIENT_ADDRESS_MODE`.
+
 | Task | Command (from `deploy/`) |
 | --- | --- |
 | Plan migrations (read-only) | `docker compose run --rm migrate` |

deploy/compose.proxy.yaml

@@ -13,6 +13,13 @@
 # its lower half; the proxy (and the tunnel connector, if used) take addresses from the upper half,
 # which Docker never hands to another container.
 
+# Exactly one trusted peer: this proxy. Set from the same variable that fixes the proxy's address,
+# so the two cannot drift apart. These override any value in ekklesia.env.
+x-ekklesia-client-address: &ekklesia-client-address
+  EKKLESIA_CLIENT_ADDRESS_MODE: trusted-proxy
+  EKKLESIA_CLIENT_ADDRESS_HEADER: x-forwarded-for
+  EKKLESIA_CLIENT_ADDRESS_TRUSTED_PROXIES: ${EKKLESIA_PROXY_ADDRESS:-10.249.250.254}/32
+
 services:
   proxy:
     image: caddy:2.11.4-alpine@sha256:6aeddd44c3078b0f9a35206472a11420648a79c184603ef95957d0a20044cb2b
@@ -59,12 +66,12 @@ services:
   api:
     networks:
       edge: {}
-    environment:
-      # Exactly one trusted peer: this proxy. Set here from the same variable that fixes the proxy's
-      # address, so the two cannot drift apart. These override any value in ekklesia.env.
-      EKKLESIA_CLIENT_ADDRESS_MODE: trusted-proxy
-      EKKLESIA_CLIENT_ADDRESS_HEADER: x-forwarded-for
-      EKKLESIA_CLIENT_ADDRESS_TRUSTED_PROXIES: ${EKKLESIA_PROXY_ADDRESS:-10.249.250.254}/32
+    environment: *ekklesia-client-address
+
+  # Operator commands (migrations, Doctor) run as `migrate`. It gets the API's client-address
+  # configuration too, so that Doctor checks what the API actually runs rather than an empty one.
+  migrate:
+    environment: *ekklesia-client-address
 
   object-storage:
     networks:

deploy/env/compose.env.example

@@ -2,6 +2,12 @@
 # Compose reads automatically from this directory. Nothing here is a secret; secrets live in the
 # per-service files under EKKLESIA_ENV_DIR.
 
+# Which Compose files make up this installation, so that every plain `docker compose …` command
+# run from this directory, operator commands through `migrate` included, sees the same services and
+# settings as the running stack. Add compose.tunnel.yaml if you use the tunnel connector. Without
+# the proxy overlay here, Doctor checks a `migrate` that lacks the API's client-address settings.
+COMPOSE_FILE=compose.yaml:compose.proxy.yaml
+
 # The image to run, by registry, name, and tag (or digest). Required.
 EKKLESIA_IMAGE=
 

docs/development/security.md

@@ -746,6 +746,33 @@ The declarations are kept deliberately: they are already correct for a supported
 for GitHub Actions, and keeping them means the upgrade does not require re-deriving the intent. They
 are documented as ignored, not removed.
 
+### Disposable stacks never outlive their job
+
+Every CI script removes its own Compose stack in an `EXIT` trap, but a trap never runs when the runner
+kills the job container: a cancelled job, or the runner's own time limit. One such stack ran on the
+shared runner for two days. So stack cleanup has three layers:
+
+- **The script's own trap**, for every ordinary failure.
+- **A final `if: always()` step** in every job that starts stacks: `scripts/ci-sweep.sh --job` removes
+  everything that job started, at any age, even after a failed step.
+- **A sweep before every such job starts**: `scripts/ci-sweep.sh --stale` removes anything a CI job
+  left more than three hours ago, which catches whatever a killed job could not clean up itself.
+
+A stack is found by name and label, never by guesswork (`scripts/lib/ci-project.sh`):
+
+- every CI Compose project is named `ekklesia-ci-<run>-<attempt>-<job key>-…`, and Compose labels
+  its containers, networks, and volumes with that name;
+- anything a script starts outside Compose carries `org.riverofglorymn.ekklesia.ci.project`;
+- images also carry the time CI built them, because an image's own creation time can be its base's.
+
+The job key is per matrix leg, so one leg's cleanup cannot match another's running stack. Nothing
+outside the `ekklesia-ci-` prefix is ever removed. A job container attached to a stack's network is
+disconnected, never removed.
+
+**This is not a substitute for a runner-side reaper.** A stack killed with its job is removed only
+when another job that sweeps runs on the same runner. Bounded execution and teardown are still the
+runner host's to enforce, as above.
+
 ### Runner topology is not verified
 
 **The runner topology is not verified in this repository and is not documented here as though it

docs/development/testing.md

@@ -159,6 +159,12 @@ also receives a safely named disposable database. Direct host runs use a Docker-
 port bound only to `127.0.0.1`; a containerized Gitea job joins the isolated Compose network and
 uses the PostgreSQL service name.
 
+In CI, a stack whose job was killed before its own cleanup ran is still removed: by that job's final
+`if: always()` step, or by the stale sweep before a later job starts (`scripts/ci-sweep.sh`, see
+[Disposable stacks never outlive their job](security.md#disposable-stacks-never-outlive-their-job)).
+`bash scripts/test-ci-sweep.sh` proves what the sweep removes and what it leaves alone, against real
+Docker resources.
+
 The live suite proves fresh read-only planning, initial and incremental application, genuine
 no-op behavior, exact applied checksums and history, edited or corrupt history rejection,
 whole-batch DDL rollback, Drizzle breakpoint execution, real advisory-lock concurrency and

scripts/ci-sweep.sh

@@ -0,0 +1,205 @@
+#!/usr/bin/env bash
+# Removes disposable CI stacks from the Docker daemon: containers, networks, volumes, images, and
+# staging directories on the Docker host.
+#
+#   bash scripts/ci-sweep.sh --stale [--max-age-seconds N]   any CI stack older than N (default 3 h)
+#   bash scripts/ci-sweep.sh --job                           this job's own stacks, at any age
+#
+# Every CI script already removes its own stack in an EXIT trap, but a trap never runs when the runner
+# kills the job container: a cancelled job, or the runner's own time limit (Gitea 1.24 ignores
+# `timeout-minutes`). So each workflow job that starts stacks ends with `--job` in an `if: always()`
+# step, and begins with `--stale`, which catches whatever a killed job left before it could clean up.
+#
+# What a stack is, and how it is found, is defined in scripts/lib/ci-project.sh: a Compose project
+# named with the `ekklesia-ci-` prefix (Compose labels everything it creates with that name), plus
+# anything labeled EKKLESIA_CI_LABEL. Nothing outside that prefix is ever touched, at any age.
+#
+# --stale's threshold is far above any job's run time, so a stack another job on this daemon is using
+# right now is never young enough to match. --job refuses to run outside CI: locally, its prefix would
+# match a developer's own stacks.
+#
+# Best effort by design: a resource that cannot be removed is reported, and the sweep goes on. Output
+# names resources only; nothing a stack holds is printed.
+
+set -Eeuo pipefail
+
+repository_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
+# shellcheck source=scripts/lib/ci-project.sh
+source "$repository_root/scripts/lib/ci-project.sh"
+
+compose_label="com.docker.compose.project"
+mode=""
+max_age_seconds="${EKKLESIA_CI_SWEEP_MAX_AGE_SECONDS:-10800}"
+
+usage() {
+  echo "usage: ci-sweep.sh --stale [--max-age-seconds N] | --job" >&2
+  exit 2
+}
+
+while (($# > 0)); do
+  case "$1" in
+  --stale | --job)
+    [[ -z "$mode" ]] || usage
+    mode="${1#--}"
+    ;;
+  --max-age-seconds)
+    [[ "${2:-}" =~ ^[0-9]+$ ]] || usage
+    max_age_seconds="$2"
+    shift
+    ;;
+  *) usage ;;
+  esac
+  shift
+done
+[[ -n "$mode" ]] || usage
+
+if [[ "$mode" == job ]]; then
+  [[ -n "${GITHUB_RUN_ID:-}" ]] || {
+    echo "ci-sweep: --job runs only in CI (GITHUB_RUN_ID is unset); locally it would match your own stacks." >&2
+    exit 2
+  }
+  prefix="$(ci_job_prefix)"
+  cutoff=""
+else
+  prefix="$EKKLESIA_CI_PREFIX"
+  cutoff="$(($(date +%s) - max_age_seconds))"
+fi
+
+if ! docker info >/dev/null 2>&1; then
+  echo "ci-sweep: no Docker daemon reachable; nothing to sweep."
+  exit 0
+fi
+
+removed=0
+warn() {
+  echo "ci-sweep: warning: $*" >&2
+}
+
+# selected <project> <created>: whether a resource belongs to the sweep.
+selected() {
+  local project="$1" created="$2" seconds
+  [[ -n "$project" && "$project" == "$prefix"* ]] || return 1
+  [[ -z "$cutoff" ]] && return 0
+  seconds="$(date -d "$created" +%s 2>/dev/null)" || return 1
+  ((seconds <= cutoff))
+}
+
+# --- Containers: Compose's own, and those a script started itself --------------------------------
+mapfile -t containers < <(
+  {
+    docker ps --all --quiet --no-trunc --filter "label=${compose_label}"
+    docker ps --all --quiet --no-trunc --filter "label=${EKKLESIA_CI_LABEL}"
+  } | sort -u
+)
+for id in "${containers[@]}"; do
+  [[ -n "$id" ]] || continue
+  record="$(docker inspect --format \
+    "{{index .Config.Labels \"${compose_label}\"}}|{{index .Config.Labels \"${EKKLESIA_CI_LABEL}\"}}|{{.Created}}|{{.Name}}" \
+    "$id" 2>/dev/null)" || continue
+  IFS='|' read -r compose_project ci_project created name <<<"$record"
+  project="${compose_project:-$ci_project}"
+  selected "$project" "$created" || continue
+  if docker rm --force --volumes "$id" >/dev/null 2>&1; then
+    echo "ci-sweep: removed container ${name#/} (${project})"
+    removed=$((removed + 1))
+  else
+    warn "could not remove container ${name#/}"
+  fi
+done
+
+# --- Networks: anything still attached (a job container) is disconnected, never removed ---------
+mapfile -t networks < <(docker network ls --quiet --no-trunc --filter "label=${compose_label}")
+for id in "${networks[@]}"; do
+  [[ -n "$id" ]] || continue
+  # A network's creation time is a time value, not a string: ask for RFC 3339, which `date` reads.
+  record="$(docker network inspect --format \
+    "{{index .Labels \"${compose_label}\"}}|{{.Created.Format \"2006-01-02T15:04:05Z07:00\"}}|{{.Name}}" \
+    "$id" 2>/dev/null)" || continue
+  IFS='|' read -r project created name <<<"$record"
+  selected "$project" "$created" || continue
+  for endpoint in $(docker network inspect --format '{{range $id, $_ := .Containers}}{{$id}} {{end}}' "$id" 2>/dev/null); do
+    docker network disconnect --force "$id" "$endpoint" >/dev/null 2>&1 ||
+      warn "could not disconnect a container from network ${name}"
+  done
+  if docker network rm "$id" >/dev/null 2>&1; then
+    echo "ci-sweep: removed network ${name} (${project})"
+    removed=$((removed + 1))
+  else
+    warn "could not remove network ${name}"
+  fi
+done
+
+# --- Volumes ---------------------------------------------------------------------------------------
+mapfile -t volumes < <(
+  {
+    docker volume ls --quiet --filter "label=${compose_label}"
+    docker volume ls --quiet --filter "label=${EKKLESIA_CI_LABEL}"
+  } | sort -u
+)
+for name in "${volumes[@]}"; do
+  [[ -n "$name" ]] || continue
+  record="$(docker volume inspect --format \
+    "{{index .Labels \"${compose_label}\"}}|{{index .Labels \"${EKKLESIA_CI_LABEL}\"}}|{{.CreatedAt}}" \
+    "$name" 2>/dev/null)" || continue
+  IFS='|' read -r compose_project ci_project created <<<"$record"
+  project="${compose_project:-$ci_project}"
+  selected "$project" "$created" || continue
+  if docker volume rm --force "$name" >/dev/null 2>&1; then
+    echo "ci-sweep: removed volume ${name} (${project})"
+    removed=$((removed + 1))
+  else
+    warn "could not remove volume ${name}"
+  fi
+done
+
+# --- Images built for a stack ------------------------------------------------------------------------
+mapfile -t images < <(docker image ls --quiet --no-trunc --filter "label=${EKKLESIA_CI_LABEL}" | sort -u)
+for id in "${images[@]}"; do
+  [[ -n "$id" ]] || continue
+  # Judged by when CI built it (EKKLESIA_CI_CREATED_LABEL), never by the image's own creation time,
+  # which BuildKit may have taken from its base. An image without that label is never swept by age.
+  record="$(docker image inspect --format \
+    "{{index .Config.Labels \"${EKKLESIA_CI_LABEL}\"}}|{{index .Config.Labels \"${EKKLESIA_CI_CREATED_LABEL}\"}}" \
+    "$id" 2>/dev/null)" || continue
+  IFS='|' read -r project built <<<"$record"
+  if [[ -n "$cutoff" ]]; then
+    [[ "$built" =~ ^[0-9]+$ ]] || continue
+    selected "$project" "@${built}" || continue
+  else
+    selected "$project" "" || continue
+  fi
+  if docker image rm --force "$id" >/dev/null 2>&1; then
+    echo "ci-sweep: removed image ${id#sha256:} (${project})"
+    removed=$((removed + 1))
+  else
+    warn "could not remove image ${id#sha256:}"
+  fi
+done
+
+# --- Staging directories on the Docker host ------------------------------------------------------
+# A deploy smoke run inside a job container stages its mounted files under the Docker host's /tmp,
+# named for its project, as root-owned files; they hold that run's disposable secrets and keys. They
+# are removed with the same pinned image that created them. If that image is not on this daemon, no
+# smoke has staged anything here.
+image="$(sed -n 's/^[[:space:]]*image: \(postgres:[^[:space:]]*\)$/\1/p' "$repository_root/deploy/compose.yaml" | head -1)"
+if [[ -n "$image" ]] && docker image inspect "$image" >/dev/null 2>&1; then
+  age=()
+  # `find -mmin +0` does not match something younger than a minute, so a threshold under a minute
+  # sweeps without an age filter, like the rest of this script.
+  if [[ -n "$cutoff" ]] && ((max_age_seconds >= 60)); then
+    age=(-mmin "+$((max_age_seconds / 60))")
+  fi
+  if staged="$(docker run --rm --entrypoint find -v /tmp:/host-tmp "$image" \
+    /host-tmp -mindepth 1 -maxdepth 1 -type d -name "${prefix}*" "${age[@]}" \
+    -print -exec rm -rf {} + 2>/dev/null)"; then
+    while IFS= read -r directory; do
+      [[ -n "$directory" ]] || continue
+      echo "ci-sweep: removed staging directory /tmp/${directory#/host-tmp/}"
+      removed=$((removed + 1))
+    done <<<"$staged"
+  else
+    warn "could not sweep staging directories on the Docker host"
+  fi
+fi
+
+echo "ci-sweep: ${mode} sweep removed ${removed} resource(s)."

scripts/lib/ci-project.sh

@@ -0,0 +1,40 @@
+# shellcheck shell=bash
+# How CI names, labels, and later finds the disposable stacks it starts.
+#
+# Every Compose project a CI script starts is named
+#
+#   ekklesia-ci-<run id>-<run attempt>-<job key>-<kind>-<pid><random>
+#
+# and Compose labels every container, network, and volume it creates with that name. What a script
+# creates outside Compose (a browser container, a volume, an image) gets EKKLESIA_CI_LABEL with the
+# same name. scripts/ci-sweep.sh finds stacks by those labels: this job's own, by the prefix up to
+# the job key (in a final `if: always()` step), and any CI stack by age (before a job starts).
+#
+# The job key is EKKLESIA_CI_JOB_KEY, which the workflow sets per job, and per matrix leg where a job
+# has one, so that one leg's cleanup cannot match another leg's running stack. Outside CI the
+# identifiers are "local".
+
+# shellcheck disable=SC2034 # read by the scripts that source this
+EKKLESIA_CI_LABEL="org.riverofglorymn.ekklesia.ci.project"
+# On images, when they were built, in Unix seconds. An image's own creation time is not that: BuildKit
+# gives an image that only adds metadata its base's time, and a reproducible build a fixed one. A
+# stale sweep that trusted it could remove an image another job is using right now.
+EKKLESIA_CI_CREATED_LABEL="org.riverofglorymn.ekklesia.ci.created"
+EKKLESIA_CI_PREFIX="ekklesia-ci-"
+
+# ci_name_safe <text>: lower case, and only what a Compose project name may contain.
+ci_name_safe() {
+  local text="${1//[^a-zA-Z0-9_-]/-}"
+  printf '%s' "${text,,}"
+}
+
+# ci_job_prefix: every project this job starts begins with exactly this.
+ci_job_prefix() {
+  ci_name_safe "ekklesia-ci-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-0}-${EKKLESIA_CI_JOB_KEY:-${GITHUB_JOB:-local}}-"
+}
+
+# ci_project_name <kind>: a project name unique to this invocation. Not only per run: several runs,
+# and several jobs of one run, share a runner's Docker daemon, and job containers often share a PID.
+ci_project_name() {
+  ci_name_safe "ekklesia-ci-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-0}-${EKKLESIA_CI_JOB_KEY:-${GITHUB_JOB:-local}}-${1:?ci_project_name needs a kind}-$$-${RANDOM}${RANDOM}"
+}

scripts/lib/deploy-smoke-client.mjs

@@ -14,6 +14,8 @@
 // Output is one line per check; any failed check exits non-zero. Nothing printed carries a
 // credential, cookie, signed URL, or response body beyond the fixed shapes asserted here.
 
+import { connect as connectHttp2 } from "node:http2";
+
 const mode = process.argv[2];
 const apiOrigin = `https://${required("SMOKE_HOSTNAME")}`;
 const dashboardOrigin = process.env.SMOKE_DASHBOARD_HOSTNAME
@@ -192,6 +194,46 @@ async function splitCorsChecks() {
   );
 }
 
+/**
+ * One GET over a real HTTP/2 connection. Node's fetch speaks HTTP/1.1, and the failure this guards
+ * against exists only on HTTP/2: a reset stream, which a browser reports as ERR_HTTP2_PROTOCOL_ERROR.
+ */
+function http2Get(url, headers = {}) {
+  return new Promise((resolve) => {
+    const session = connectHttp2(url.origin);
+    const finish = (result) => {
+      clearTimeout(timer);
+      session.close();
+      resolve(result);
+    };
+    const timer = setTimeout(() => finish({ error: "timeout" }), 10_000);
+    session.on("error", () => finish({ error: "session" }));
+    const stream = session.request({
+      ":method": "GET",
+      ":path": `${url.pathname}${url.search}`,
+      ...headers,
+    });
+    let status;
+    let responseHeaders = {};
+    let bytes = 0;
+    stream.on("response", (received) => {
+      status = received[":status"];
+      responseHeaders = received;
+    });
+    stream.on("data", (chunk) => {
+      bytes += chunk.length;
+    });
+    stream.on("error", (error) => finish({ error: error.code ?? "stream" }));
+    stream.on("end", () => finish({ bytes, headers: responseHeaders, status }));
+    stream.end();
+  });
+}
+
+/** The CORS headers of a response, by name: what a browser would act on. */
+function corsOf(response) {
+  return [...response.headers.keys()].filter((name) => name.startsWith("access-control-"));
+}
+
 async function storageDeliveryCheck() {
   // Core's own provider, from the image, exactly as the API composes it: it reaches object-storage
   // over HTTPS on the private network and signs read URLs for the storage hostname.
@@ -238,6 +280,79 @@ async function storageDeliveryCheck() {
       "a delivered object is Cache-Control: private",
     );
     await provider.delete(bucket, "smoke/object.txt");
+
+    // The same image, twice, over HTTP/2: a browser's second load of a signed URL it has cached is a
+    // conditional request, and must come back as a clean 304, not a reset stream.
+    const png = Buffer.from(
+      "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg==",
+      "base64",
+    );
+    await provider.put(bucket, "smoke/image.png", png, {
+      contentLength: png.byteLength,
+      contentType: "image/png",
+    });
+    const image = new URL(
+      (await provider.createReadUrl(bucket, "smoke/image.png", { expiresInSeconds: 300 })).url,
+    );
+    const first = await http2Get(image);
+    check(
+      first.status === 200 && first.bytes === png.byteLength && Boolean(first.headers?.etag),
+      "an image loads over HTTP/2 through the proxy, with a validator (200, ETag)",
+    );
+    const revalidated = await http2Get(image, { "if-none-match": first.headers?.etag ?? "" });
+    check(
+      revalidated.status === 304 && revalidated.error === undefined,
+      `the same image loaded again over HTTP/2 revalidates cleanly (${revalidated.status ?? revalidated.error})`,
+    );
+    const sinceModified = await http2Get(image, {
+      "if-modified-since": first.headers?.["last-modified"] ?? "",
+    });
+    check(
+      sinceModified.status === 304 && sinceModified.error === undefined,
+      `a revalidation by date over HTTP/2 is a clean 304 too (${sinceModified.status ?? sinceModified.error})`,
+    );
+
+    // Delivery-layer CORS is the deployment's own, set by the proxy: exactly the origins whose pages
+    // read objects with fetch(), GET and HEAD only, and nothing of the object store's own answer.
+    const fromApi = await fetch(image, { headers: { origin: apiOrigin } });
+    check(
+      fromApi.headers.get("access-control-allow-origin") === apiOrigin &&
+        (fromApi.headers.get("vary") ?? "").toLowerCase().includes("origin") &&
+        corsOf(fromApi).length === 1,
+      "a read from the API origin is allowed for exactly that origin, with Vary: Origin and nothing else",
+    );
+    if (split) {
+      const fromDashboard = await fetch(image, { headers: { origin: dashboardOrigin } });
+      check(
+        fromDashboard.headers.get("access-control-allow-origin") === dashboardOrigin,
+        "a read from the Dashboard origin is allowed for exactly that origin",
+      );
+    }
+    const fromElsewhere = await fetch(image, { headers: { origin: untrustedOrigin } });
+    check(
+      fromElsewhere.status === 200 && corsOf(fromElsewhere).length === 0,
+      "a read from any other origin gets no CORS header at all",
+    );
+    const preflight = await fetch(image, {
+      headers: { "access-control-request-method": "GET", origin: apiOrigin },
+      method: "OPTIONS",
+    });
+    check(
+      preflight.status === 204 &&
+        preflight.headers.get("access-control-allow-origin") === apiOrigin &&
+        preflight.headers.get("access-control-allow-methods") === "GET, HEAD" &&
+        preflight.headers.get("x-amz-request-id") === null,
+      "the proxy answers a preflight itself (204, GET and HEAD only), never the object store",
+    );
+    const foreignPreflight = await fetch(image, {
+      headers: { "access-control-request-method": "GET", origin: untrustedOrigin },
+      method: "OPTIONS",
+    });
+    check(
+      foreignPreflight.status === 204 && corsOf(foreignPreflight).length === 0,
+      "a preflight from any other origin is answered without permission",
+    );
+    await provider.delete(bucket, "smoke/image.png");
   } finally {
     await provider.close();
   }

scripts/lib/deploy-smoke-display.mjs

@@ -0,0 +1,979 @@
+// Real-browser display-device scenarios for scripts/smoke-deploy.sh in the split topology (ADR 0036,
+// ADR 0045). Runs inside a disposable container on the deployment's `edge` network and drives two
+// headless Chromiums over the Chrome DevTools Protocol, with Node.js's own fetch and WebSocket and no
+// other dependency: the smoke run's browser, where the owner signs in, and a "kiosk" browser whose
+// profile directory is a Docker volume that outlives the kiosk container.
+//
+//   node deploy-smoke-display.mjs <step>
+//
+//   pair               seed content, pair the kiosk, then close its browser cleanly
+//   restart            same profile, new browser process: still paired and live?
+//   two-pages          two pages, one device credential: does either flap?
+//   connectivity-blip  the browser reports offline then online: is a live display remounted?
+//   session-blip       Signage's first viewer-session request fails once: does the display recover?
+//   stream-failure     the first realtime connection is refused: is realtime re-established?
+//   caching-probe      can a display on the API origin store storage-origin images in Cache Storage?
+//   stream-recycle     one page past the five-minute realtime recycle, with a timestamped timeline
+//   revoke             revoke the paired kiosk while its display is open
+//
+// Every DevTools command and every page-side request has its own deadline, and a step has an overall
+// budget. A call that never answers fails the step with a line naming that exact call, because a call
+// that can hang the driver may be a request that can hang a real display.
+//
+// Output is one line per check or observation; nothing printed carries a credential, cookie value,
+// signed URL, query string, or response body.
+
+function required(name) {
+  const value = process.env[name];
+  if (!value) {
+    throw new Error(`${name} is required.`);
+  }
+  return value;
+}
+
+/** A positive whole number of milliseconds from the environment, or the default. */
+function milliseconds(name, fallback) {
+  const raw = process.env[name];
+  const value = raw === undefined || raw === "" ? fallback : Number(raw);
+  if (!Number.isSafeInteger(value) || value <= 0) {
+    throw new Error(`${name} must be a positive whole number of milliseconds.`);
+  }
+  return value;
+}
+
+const mode = process.argv[2];
+const apiOrigin = `https://${required("SMOKE_HOSTNAME")}`;
+const dashboardOrigin = `https://${required("SMOKE_DASHBOARD_HOSTNAME")}`;
+const ownerAddress = required("SMOKE_CDP_ADDRESS");
+const kioskAddress = required("SMOKE_KIOSK_CDP_ADDRESS");
+const watchMs = milliseconds("SMOKE_WATCH_MS", 120_000);
+const recycleWatchMs = milliseconds("SMOKE_RECYCLE_WATCH_MS", 330_000);
+const stepBudgetMs = milliseconds("SMOKE_STEP_BUDGET_MS", 180_000);
+/** How long one DevTools command may go unanswered. */
+const callTimeoutMs = milliseconds("SMOKE_CALL_TIMEOUT_MS", 10_000);
+/** How long one page-side HTTP request may go unanswered; shorter than the call deadline, so a hung
+ * request is reported as the request rather than as the evaluation that awaited it. */
+const requestTimeoutMs = Math.max(1_000, callTimeoutMs - 2_000);
+const waitTimeoutMs = 30_000;
+const deviceCookie = "ekklesia_device_session";
+/** Far above what three cached slides need, which is one object URL each. */
+const MAX_LIVE_OBJECT_URLS = 50;
+const stepStartedAt = Date.now();
+
+let failures = 0;
+function check(condition, description) {
+  console.log(`${condition ? "ok  " : "FAIL"} ${description}`);
+  if (!condition) {
+    failures += 1;
+  }
+}
+function note(description) {
+  console.log(`     ${description}`);
+}
+const clock = (ms = Date.now()) => new Date(ms).toISOString().slice(11, 19);
+const since = (from) => `+${Math.round((Date.now() - from) / 1000)}s`;
+function progress(description) {
+  console.log(`···· ${clock()} ${since(stepStartedAt)} ${description}`);
+}
+const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
+/** A URL's path only: a signed delivery URL's query is a credential. */
+const pathOf = (url) => {
+  try {
+    return new URL(url).pathname;
+  } catch {
+    return "(unparseable)";
+  }
+};
+
+// --- Hang guards ---------------------------------------------------------------------------------
+
+class HangError extends Error {
+  constructor(label, ms) {
+    super(`HANG ${label}: no answer within ${ms / 1000}s`);
+    this.name = "HangError";
+  }
+}
+
+/** Everything awaited right now, by name, so a step that overruns its budget says what it was
+ * waiting for. */
+const inFlight = new Map();
+let nextOperation = 1;
+
+function guard(label, promise, ms = callTimeoutMs) {
+  const id = nextOperation;
+  nextOperation += 1;
+  inFlight.set(id, { at: Date.now(), label });
+  let timer;
+  return Promise.race([
+    promise,
+    new Promise((_, reject) => {
+      timer = setTimeout(() => reject(new HangError(label, ms)), ms);
+    }),
+  ]).finally(() => {
+    clearTimeout(timer);
+    inFlight.delete(id);
+  });
+}
+
+function describeInFlight() {
+  const pending = [...inFlight.values()].map(
+    (operation) => `${operation.label} (${Math.round((Date.now() - operation.at) / 1000)}s)`,
+  );
+  return pending.length === 0 ? "nothing — the step itself overran" : pending.join("; ");
+}
+
+setTimeout(() => {
+  console.log(
+    `FAIL ${mode}: step budget of ${stepBudgetMs / 1000}s exceeded; in flight: ${describeInFlight()}`,
+  );
+  process.exit(2);
+}, stepBudgetMs).unref();
+
+// --- DevTools ------------------------------------------------------------------------------------
+
+/** The same minimal DevTools client as deploy-smoke-browser.mjs, with a deadline on every command. */
+async function connect(devtools, name) {
+  const deadline = Date.now() + 30_000;
+  let version;
+  while (version === undefined) {
+    try {
+      const response = await fetch(`http://${devtools}/json/version`, {
+        signal: AbortSignal.timeout(2_000),
+      });
+      version = await response.json();
+    } catch {
+      if (Date.now() > deadline) {
+        throw new HangError(`the ${name} browser's DevTools endpoint`, 30_000);
+      }
+      await sleep(500);
+    }
+  }
+  const socketUrl = version.webSocketDebuggerUrl.replace(/^ws:\/\/[^/]+/u, `ws://${devtools}`);
+  const socket = new WebSocket(socketUrl);
+  await guard(
+    `the ${name} browser's DevTools socket`,
+    new Promise((resolve, reject) => {
+      socket.addEventListener("open", resolve, { once: true });
+      socket.addEventListener("error", () => reject(new Error("DevTools connection failed.")), {
+        once: true,
+      });
+    }),
+  );
+  let nextId = 1;
+  const pending = new Map();
+  const listeners = [];
+  socket.addEventListener("message", (event) => {
+    const message = JSON.parse(String(event.data));
+    if (message.id !== undefined) {
+      const waiter = pending.get(message.id);
+      pending.delete(message.id);
+      if (message.error) {
+        waiter?.reject(new Error(`DevTools ${waiter.method} failed.`));
+      } else {
+        waiter?.resolve(message.result);
+      }
+      return;
+    }
+    for (const listener of listeners) {
+      listener(message);
+    }
+  });
+  return {
+    close: () => socket.close(),
+    on: (listener) => listeners.push(listener),
+    send(method, params = {}, sessionId = undefined, where = name) {
+      const id = nextId;
+      nextId += 1;
+      socket.send(JSON.stringify({ id, method, params, ...(sessionId ? { sessionId } : {}) }));
+      return guard(
+        `DevTools ${method} on ${where}`,
+        new Promise((resolve, reject) => pending.set(id, { method, reject, resolve })),
+      );
+    },
+  };
+}
+
+/** Finds a test id anywhere in the document, including inside module artifacts' open shadow roots
+ * (ADR 0040). */
+const deepFind = `((testId) => {
+  const walk = (root) => {
+    const found = root.querySelector('[data-testid="' + testId + '"]');
+    if (found) return found;
+    for (const element of root.querySelectorAll("*")) {
+      if (element.shadowRoot) {
+        const inner = walk(element.shadowRoot);
+        if (inner) return inner;
+      }
+    }
+    return null;
+  };
+  return walk(document);
+})`;
+const find = (testId) => `${deepFind}(${JSON.stringify(testId)})`;
+
+/** Installed before any page script: the browser's own connectivity events and the display service
+ * worker's fallback reports, which are what the display host remounts on (ADR 0043). */
+const instrumentation = `(() => {
+  const events = [];
+  // Object URLs created and revoked in this document. Each live one pins its blob in the browser's
+  // blob storage until it is revoked or the document goes away.
+  const blobs = { created: 0, revoked: 0 };
+  Object.defineProperty(window, "__displayRepro", { value: { blobs, events } });
+  const create = URL.createObjectURL.bind(URL);
+  const revoke = URL.revokeObjectURL.bind(URL);
+  URL.createObjectURL = (object) => { blobs.created += 1; return create(object); };
+  URL.revokeObjectURL = (url) => { blobs.revoked += 1; return revoke(url); };
+  const record = (kind) => events.push({ at: Date.now(), kind });
+  addEventListener("online", () => record("online"));
+  addEventListener("offline", () => record("offline"));
+  navigator.serviceWorker?.addEventListener("message", (event) => {
+    const status = event.data && event.data.ekklesiaDisplayOffline;
+    if (status) record(status.usingFallback ? "sw-fallback:" + (status.kind || "") : "sw-network");
+  });
+})()`;
+
+/** One page target with its own flat session, and what it does on the network. */
+async function openPage(cdp, url, label) {
+  const { targetId } = await cdp.send("Target.createTarget", { url: "about:blank" });
+  const { sessionId } = await cdp.send("Target.attachToTarget", { flatten: true, targetId });
+  const send = (method, params) => cdp.send(method, params, sessionId, `page "${label}"`);
+  const requests = new Map();
+  const page = {
+    label,
+    /** Every realtime connection: when it opened, its status, and when and how it ended. */
+    streams: [],
+    /** Signage viewer-session requests: one per module mount. */
+    sessionRequests: [],
+    /** Heartbeat statuses, by status code. */
+    heartbeats: new Map(),
+    /** Channel-state fetches and how they went. */
+    stateFetches: [],
+    /** Non-2xx responses and failed requests, by path only. */
+    problems: [],
+  };
+  cdp.on((message) => {
+    if (message.sessionId !== sessionId) {
+      return;
+    }
+    const params = message.params ?? {};
+    if (message.method === "Network.requestWillBeSent") {
+      const path = pathOf(params.request.url);
+      const entry = { at: Date.now(), path, type: params.type };
+      requests.set(params.requestId, entry);
+      if (params.type === "EventSource") {
+        entry.stream = {
+          openedAt: Date.now(),
+          path,
+          status: undefined,
+          endedAt: undefined,
+          how: undefined,
+        };
+        page.streams.push(entry.stream);
+      } else if (path.endsWith("/viewers/session")) {
+        page.sessionRequests.push(Date.now());
+      } else if (path.endsWith("/display-state") && !path.includes("module-realtime")) {
+        entry.state = { at: Date.now(), outcome: "pending" };
+        page.stateFetches.push(entry.state);
+      }
+    } else if (message.method === "Network.responseReceived") {
+      const entry = requests.get(params.requestId);
+      const { status } = params.response;
+      if (entry?.stream) {
+        entry.stream.status = status;
+      }
+      if (entry?.state) {
+        entry.state.outcome = String(status);
+      }
+      if (entry?.path.endsWith("/heartbeat")) {
+        page.heartbeats.set(status, (page.heartbeats.get(status) ?? 0) + 1);
+      }
+      if (status >= 400 || status === 0) {
+        page.problems.push(`${clock()} ${status} ${pathOf(params.response.url)}`);
+      }
+    } else if (message.method === "Network.loadingFinished") {
+      const entry = requests.get(params.requestId);
+      if (entry?.stream) {
+        entry.stream.endedAt = Date.now();
+        entry.stream.how = "finished";
+      }
+    } else if (message.method === "Network.loadingFailed") {
+      const entry = requests.get(params.requestId);
+      if (entry?.stream) {
+        entry.stream.endedAt = Date.now();
+        entry.stream.how = params.canceled ? "canceled" : `failed ${params.errorText}`;
+      }
+      if (entry?.state) {
+        entry.state.outcome = params.canceled ? "canceled" : `failed ${params.errorText}`;
+      }
+      if (entry && !params.canceled) {
+        page.problems.push(`${clock()} failed ${params.errorText} ${entry.path}`);
+      }
+    }
+  });
+  await send("Network.enable");
+  await send("Page.enable");
+  await send("Runtime.enable");
+  await send("Page.addScriptToEvaluateOnNewDocument", { source: instrumentation });
+  await send("Emulation.setDeviceMetricsOverride", {
+    deviceScaleFactor: 1,
+    height: 1080,
+    mobile: false,
+    width: 1920,
+  });
+  page.send = send;
+  /** Runs an expression in the page. `what` names it if it never comes back. */
+  page.evaluate = async (expression, what = "an evaluation") => {
+    const result = await guard(
+      `page "${label}": ${what}`,
+      send("Runtime.evaluate", { awaitPromise: true, expression, returnByValue: true }),
+    );
+    if (result.exceptionDetails) {
+      throw new Error(`A page expression threw (${what}).`);
+    }
+    return result.result.value;
+  };
+  page.waitFor = async (expression, description, ms = waitTimeoutMs) => {
+    const deadline = Date.now() + ms;
+    while (Date.now() < deadline) {
+      if (
+        await page
+          .evaluate(`Boolean(${expression})`, `waiting for ${description}`)
+          .catch((error) => {
+            if (error instanceof HangError) throw error;
+            return false;
+          })
+      ) {
+        return true;
+      }
+      await sleep(250);
+    }
+    note(`timed out waiting for ${description}`);
+    return false;
+  };
+  /** A credentialed request from this page, bounded in the page itself, so a request that never
+   * answers is reported as that request. */
+  page.request = async (method, url, body) => {
+    const result = await page.evaluate(
+      `fetch(${JSON.stringify(url)}, {
+        credentials: "include",
+        method: ${JSON.stringify(method)},
+        signal: AbortSignal.timeout(${requestTimeoutMs}),
+        ${body === undefined ? "" : `headers: { "content-type": "application/json" }, body: ${JSON.stringify(JSON.stringify(body))},`}
+      }).then(
+        async (response) => ({ status: response.status, body: await response.json().catch(() => null) }),
+        (error) => ({ status: error && error.name === "TimeoutError" ? "timeout" : "network-error" }),
+      )`,
+      `${method} ${pathOf(url)}`,
+    );
+    if (result.status === "timeout") {
+      throw new HangError(`HTTP ${method} ${pathOf(url)} from page "${label}"`, requestTimeoutMs);
+    }
+    return result;
+  };
+  page.close = () => cdp.send("Target.closeTarget", { targetId });
+  page.navigate = (to) => send("Page.navigate", { url: to });
+  if (url) {
+    await page.navigate(url);
+  }
+  return page;
+}
+
+/** The owner, signed in on the Dashboard hostname of the smoke run's own browser. */
+async function ownerPage() {
+  const cdp = await connect(ownerAddress, "owner");
+  const page = await openPage(cdp, `${dashboardOrigin}/`, "owner");
+  await page.waitFor(`${find("login-email")} || ${find("host-shell")}`, "the Dashboard");
+  if (!(await page.evaluate(`Boolean(${find("host-shell")})`, "checking the session"))) {
+    await page.evaluate(
+      `(() => {
+        const setValue = (element, value) => {
+          Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, "value").set.call(element, value);
+          element.dispatchEvent(new Event("input", { bubbles: true }));
+        };
+        setValue(${find("login-email")}, ${JSON.stringify(required("SMOKE_OWNER_EMAIL"))});
+        setValue(${find("login-password")}, ${JSON.stringify(required("SMOKE_OWNER_PASSWORD"))});
+        ${find("login-email")}.form.requestSubmit();
+      })()`,
+      "submitting the sign-in form",
+    );
+    check(await page.waitFor(find("host-shell"), "the signed-in Dashboard"), "the owner signs in");
+  }
+  page.api = (method, path, body) => page.request(method, `${apiOrigin}${path}`, body);
+  return { cdp, page };
+}
+
+/** Closes the owner's tab as well as the connection: the owner browser is shared by every step, and a
+ * tab left behind by each one accumulates until the Dashboard no longer renders in time. */
+async function finishOwner(owner) {
+  await owner.page.close().catch(() => undefined);
+  owner.cdp.close();
+}
+
+async function displayModuleId(owner) {
+  const navigation = await owner.api("GET", "/api/v1/dashboard/navigation");
+  const entries = [
+    ...navigation.body.ungrouped,
+    ...navigation.body.groups.flatMap((group) => group.modules),
+  ];
+  const withDisplay = entries.filter((entry) => entry.module.display !== null);
+  const signage = withDisplay.find((entry) => entry.module.id === "signage");
+  return (signage ?? withDisplay[0])?.module.id;
+}
+
+/** Three slides of plain colour, on a five-second rotation, so a display has real content to persist
+ * and continue from, as the production screen did. Signage only. */
+async function seedSignage(owner) {
+  const base = "/api/v1/modules/signage";
+  const images = [];
+  for (const [index, colour] of ["#b03030", "#3060b0", "#30a050"].entries()) {
+    const uploaded = await owner.evaluate(
+      `(async () => {
+        const canvas = document.createElement("canvas");
+        canvas.width = 640;
+        canvas.height = 360;
+        const context = canvas.getContext("2d");
+        context.fillStyle = ${JSON.stringify(colour)};
+        context.fillRect(0, 0, 640, 360);
+        const blob = await new Promise((resolve) => canvas.toBlob(resolve, "image/png"));
+        const response = await fetch(${JSON.stringify(`${apiOrigin}${base}/images?filename=slide-${index}.png&displayName=Slide%20${index}`)}, {
+          body: blob, credentials: "include", headers: { "content-type": "image/png" }, method: "POST",
+          signal: AbortSignal.timeout(${requestTimeoutMs}),
+        });
+        return { status: response.status, body: await response.json().catch(() => null) };
+      })()`,
+      `uploading slide ${index}`,
+    );
+    images.push(uploaded.body?.image?.id);
+  }
+  check(
+    images.every((id) => typeof id === "string"),
+    "three slides are uploaded to Signage",
+  );
+  const timeline = await owner.api("POST", `${base}/timelines`, { name: "Display repro" });
+  const timelineId = timeline.body?.timeline?.id ?? timeline.body?.id;
+  for (const itemId of images) {
+    await owner.api("POST", `${base}/timelines/${timelineId}/slots`, {
+      durationOverrideMs: 5_000,
+      itemId,
+      providerId: "signage.native-images",
+    });
+  }
+  const groups = await owner.api("GET", `${base}/channel-groups`);
+  const groupId = groups.body?.groups?.[0]?.id;
+  const activated = await owner.api("POST", `${base}/playback/${groupId}/activate`, {
+    timelineId,
+  });
+  check(activated.status === 200, "a three-slide timeline is playing on the default channel group");
+}
+
+async function signageViewers(owner) {
+  const listed = await owner.api("GET", "/api/v1/modules/signage/viewers");
+  return listed.body?.viewers ?? [];
+}
+
+async function coreDevices(owner) {
+  const listed = await owner.api("GET", "/api/v1/devices");
+  return listed.body?.devices ?? [];
+}
+
+/** The display's own view of itself: which host state is showing, and Signage's offline flag. */
+function displayState(page) {
+  return page.evaluate(
+    `(() => {
+      const stage = ${find("signage-stage")};
+      return {
+        pairing: Boolean(${find("display-pairing")}),
+        hostLoading: Boolean(${find("display-host-loading")}) || Boolean(${find("display-loading")}),
+        hostError: Boolean(${find("display-host-error")}) || Boolean(${find("display-error")}),
+        hostOffline: Boolean(${find("display-offline")}),
+        mounted: (${find("display-module-container")}?.childElementCount ?? 0) > 0,
+        signageOffline: stage ? stage.dataset.offline === "true" : null,
+      };
+    })()`,
+    "reading the display state",
+  );
+}
+
+async function blobCounts(page) {
+  return page.evaluate(
+    "window.__displayRepro ? window.__displayRepro.blobs : { created: 0, revoked: 0 }",
+    "reading object-URL counts",
+  );
+}
+
+async function connectivityEvents(page) {
+  return page.evaluate(
+    "window.__displayRepro ? window.__displayRepro.events : []",
+    "reading connectivity events",
+  );
+}
+
+async function describeDeviceCookie(cdp) {
+  const { cookies } = await cdp.send("Storage.getCookies");
+  const cookie = cookies.find((candidate) => candidate.name === deviceCookie);
+  if (!cookie) {
+    note(`${deviceCookie}: absent from the profile`);
+    return undefined;
+  }
+  const days = cookie.session
+    ? "session"
+    : `${((cookie.expires * 1000 - Date.now()) / 86_400_000).toFixed(1)} days`;
+  note(
+    `${deviceCookie}: domain=${cookie.domain} session=${cookie.session} lifetime=${days} httpOnly=${cookie.httpOnly} secure=${cookie.secure} sameSite=${cookie.sameSite}`,
+  );
+  return cookie;
+}
+
+const liveExpression = `(() => {
+  const stage = ${find("signage-stage")};
+  const mounted = (${find("display-module-container")}?.childElementCount ?? 0) > 0;
+  return mounted && !${find("display-pairing")} && (stage === null || stage.dataset.offline === "false");
+})()`;
+
+function waitLive(page, description, ms = 45_000) {
+  return page.waitFor(liveExpression, description, ms);
+}
+
+/** A page's network story, one line each, by path only. */
+function summarize(page, from) {
+  const at = (ms) => (ms === undefined ? "-" : `${clock(ms)} (${Math.round((ms - from) / 1000)}s)`);
+  note(`${page.label}: viewer sessions opened ${page.sessionRequests.length} (= module mounts)`);
+  note(
+    `${page.label}: heartbeats ${[...page.heartbeats].map(([status, count]) => `${status}×${count}`).join(" ") || "none"}`,
+  );
+  const states = page.stateFetches.reduce((tally, fetch) => {
+    tally[fetch.outcome] = (tally[fetch.outcome] ?? 0) + 1;
+    return tally;
+  }, {});
+  note(`${page.label}: channel-state fetches ${JSON.stringify(states)}`);
+  for (const stream of page.streams) {
+    const lived =
+      stream.endedAt === undefined
+        ? "open"
+        : `${Math.round((stream.endedAt - stream.openedAt) / 1000)}s`;
+    note(
+      `${page.label}: stream opened ${at(stream.openedAt)} status ${stream.status ?? "-"} ended ${at(stream.endedAt)} ${stream.how ?? ""} lived ${lived}`,
+    );
+  }
+  note(`${page.label}: problems ${page.problems.slice(-12).join(" | ") || "none"}`);
+}
+
+/** Samples display state on an interval for a fixed time, printing progress, and returns a tally. */
+async function watch(pages, ms, { onSample, viewerOwner } = {}) {
+  const from = Date.now();
+  const tally = new Map(
+    pages.map((page) => [
+      page.label,
+      { loading: 0, offline: 0, pairing: 0, samples: 0, transitions: [] },
+    ]),
+  );
+  const last = new Map();
+  let nextProgress = from + 15_000;
+  const roster = [];
+  while (Date.now() - from < ms) {
+    for (const page of pages) {
+      const state = await displayState(page);
+      const t = tally.get(page.label);
+      t.samples += 1;
+      if (state.signageOffline || state.hostOffline) t.offline += 1;
+      if (state.pairing) t.pairing += 1;
+      if (state.hostLoading) t.loading += 1;
+      const key = JSON.stringify(state);
+      if (last.get(page.label) !== key) {
+        last.set(page.label, key);
+        t.transitions.push(`${clock()} ${since(from)} ${key}`);
+      }
+      if (onSample && (await onSample(page, state))) {
+        return { from, roster, stopped: true, tally };
+      }
+    }
+    if (Date.now() >= nextProgress) {
+      nextProgress = Date.now() + 15_000;
+      if (viewerOwner) {
+        const viewers = await signageViewers(viewerOwner);
+        roster.push(
+          `${since(from)} ${viewers.map((v) => `${v.name}:${v.online ? "Online" : "Offline"}`).join(",")}`,
+        );
+      }
+      const lines = [];
+      for (const page of pages) {
+        const t = tally.get(page.label);
+        const blobs = await blobCounts(page);
+        lines.push(
+          `${page.label}: ${t.samples} samples, offline ${t.offline}, mounts ${page.sessionRequests.length}, streams ${page.streams.length}, live object URLs ${blobs.created - blobs.revoked}`,
+        );
+      }
+      progress(lines.join(" · "));
+    }
+    await sleep(2_000);
+  }
+  return { from, roster, stopped: false, tally };
+}
+
+function printTally(result) {
+  for (const [label, t] of result.tally) {
+    note(
+      `${label}: ${t.samples} samples, offline in ${t.offline}, pairing in ${t.pairing}, loading in ${t.loading}`,
+    );
+    for (const transition of t.transitions.slice(0, 30)) {
+      note(`  ${label} ${transition}`);
+    }
+  }
+  if (result.roster.length > 0) {
+    note(`roster over time: ${result.roster.join(" ; ")}`);
+  }
+}
+
+// --- Steps ---------------------------------------------------------------------------------------
+
+async function run() {
+  if (mode === "pair") {
+    const owner = await ownerPage();
+    const moduleId = await displayModuleId(owner.page);
+    check(moduleId !== undefined, `a module publishes a display (${moduleId})`);
+    if (moduleId === "signage") {
+      await seedSignage(owner.page);
+    }
+    const kiosk = await connect(kioskAddress, "kiosk");
+    const display = await openPage(kiosk, `${apiOrigin}/display/${moduleId}`, "kiosk");
+    check(
+      await display.waitFor(find("display-pairing-code"), "a pairing code"),
+      "an unpaired kiosk shows a pairing code",
+    );
+    const code = (
+      await display.evaluate(`${find("display-pairing-code")}.textContent`, "reading the code")
+    ).replace(/\D/gu, "");
+    const approved = await owner.page.api("POST", "/api/v1/device-pairing/approvals", {
+      code,
+      deviceName: "Repro kiosk",
+    });
+    check(approved.status === 200, "the owner approves the code");
+    check(
+      await waitLive(display, "the paired display to go live"),
+      "the paired kiosk mounts the display and is live",
+    );
+    await describeDeviceCookie(kiosk);
+    progress("letting heartbeats, a persisted snapshot, and media prefetch happen (20s)");
+    await sleep(20_000);
+    for (const viewer of await signageViewers(owner.page)) {
+      note(
+        `signage viewer "${viewer.name}" online=${viewer.online} offlineReady=${viewer.offlineReady} cachedAssets=${viewer.cachedAssetCount} error=${viewer.lastError?.code ?? "none"}`,
+      );
+    }
+    summarize(display, stepStartedAt);
+    // A clean exit: the browser shuts down normally, exactly as a systemd stop of a well-behaved kiosk.
+    await kiosk.send("Browser.close").catch(() => undefined);
+    kiosk.close();
+    await finishOwner(owner);
+  } else if (mode === "restart") {
+    const owner = await ownerPage();
+    const moduleId = await displayModuleId(owner.page);
+    const kiosk = await connect(kioskAddress, "kiosk");
+    const cookie = await describeDeviceCookie(kiosk);
+    check(cookie !== undefined, "the device credential survived a clean browser exit");
+    const display = await openPage(kiosk, `${apiOrigin}/display/${moduleId}`, "kiosk");
+    const live = await waitLive(display, "the restarted display to go live");
+    const state = await displayState(display);
+    check(
+      live && !state.pairing,
+      `after a clean restart the kiosk is still paired and live (${JSON.stringify(state)})`,
+    );
+    summarize(display, stepStartedAt);
+    await display.close();
+    kiosk.close();
+    await finishOwner(owner);
+  } else if (mode === "two-pages") {
+    const owner = await ownerPage();
+    const moduleId = await displayModuleId(owner.page);
+    const kiosk = await connect(kioskAddress, "kiosk");
+    const url = `${apiOrigin}/display/${moduleId}`;
+    const first = await openPage(kiosk, url, "first");
+    check(await waitLive(first, "the first page"), "the first page is live");
+    const second = await openPage(kiosk, url, "second");
+    check(await waitLive(second, "the second page"), "the second page is live");
+    const result = await watch([first, second], watchMs, { viewerOwner: owner.page });
+    printTally(result);
+    for (const page of [first, second]) {
+      summarize(page, result.from);
+      note(`${page.label}: connectivity events ${JSON.stringify(await connectivityEvents(page))}`);
+    }
+    check(
+      [...result.tally.values()].every((t) => t.offline === 0),
+      `neither page showed an offline state while both were open (${watchMs / 1000}s)`,
+    );
+    check(
+      [first, second].every((page) => page.sessionRequests.length === 1),
+      "neither page was remounted while both were open",
+    );
+    await first.close();
+    await second.close();
+    kiosk.close();
+    await finishOwner(owner);
+  } else if (mode === "connectivity-blip") {
+    // The browser's own offline/online events, as a Wi-Fi reassociation or a link flap produces. The
+    // display host re-resolves its descriptor on "online" (ADR 0043); this measures what that costs.
+    const owner = await ownerPage();
+    const moduleId = await displayModuleId(owner.page);
+    const kiosk = await connect(kioskAddress, "kiosk");
+    const page = await openPage(kiosk, `${apiOrigin}/display/${moduleId}`, "blip");
+    check(await waitLive(page, "the display"), "the display is live");
+    const mountsBefore = page.sessionRequests.length;
+    await page.evaluate(
+      `dispatchEvent(new Event("offline")); setTimeout(() => dispatchEvent(new Event("online")), 1000)`,
+      "dispatching offline then online",
+    );
+    let firstOfflineAt;
+    let lastOfflineAt;
+    const result = await watch([page], 30_000, {
+      onSample: (_page, state) => {
+        if (state.signageOffline || state.hostOffline || state.hostLoading || !state.mounted) {
+          firstOfflineAt ??= Date.now();
+          lastOfflineAt = Date.now();
+        }
+        return false;
+      },
+    });
+    printTally(result);
+    const remounts = page.sessionRequests.length - mountsBefore;
+    note(`remounts caused by one offline/online pair: ${remounts}`);
+    note(
+      firstOfflineAt === undefined
+        ? "nothing visible changed"
+        : `visible disruption from ${clock(firstOfflineAt)} to ${clock(lastOfflineAt)} (sampled every 2s)`,
+    );
+    summarize(page, result.from);
+    check(remounts === 0, "a connectivity blip does not remount a live display");
+    check(
+      firstOfflineAt === undefined,
+      "a connectivity blip does not show an offline state on a live display",
+    );
+    await page.close();
+    kiosk.close();
+    await finishOwner(owner);
+  } else if (mode === "session-blip") {
+    // One transient failure of Signage's viewer-session request at start-up, as a kiosk booting
+    // before its network settles, or an API restart, would produce. Everything else reaches the
+    // server.
+    const owner = await ownerPage();
+    const moduleId = await displayModuleId(owner.page);
+    const kiosk = await connect(kioskAddress, "kiosk");
+    const page = await openPage(kiosk, undefined, "session-blip");
+    let blocked = 0;
+    kiosk.on((message) => {
+      if (message.method === "Fetch.requestPaused") {
+        const { requestId, request } = message.params;
+        if (blocked === 0 && pathOf(request.url).endsWith("/viewers/session")) {
+          blocked += 1;
+          page
+            .send("Fetch.failRequest", { errorReason: "ConnectionRefused", requestId })
+            .catch((error) => note(`interception reply failed: ${error.message}`));
+        } else {
+          page
+            .send("Fetch.continueRequest", { requestId })
+            .catch((error) => note(`interception reply failed: ${error.message}`));
+        }
+      }
+    });
+    await page.send("Fetch.enable", { patterns: [{ urlPattern: "*/api/v1/modules/*" }] });
+    await page.navigate(`${apiOrigin}/display/${moduleId}`);
+    await page.waitFor(
+      `(${find("display-module-container")}?.childElementCount ?? 0) > 0`,
+      "the module to mount",
+    );
+    note(`viewer-session requests failed on purpose: ${blocked}`);
+    const recovered = await page.waitFor(
+      `${find("signage-stage")}?.dataset.offline === "false"`,
+      "recovery from one failed viewer-session request",
+      90_000,
+    );
+    summarize(page, stepStartedAt);
+    check(
+      recovered,
+      `one failed viewer-session request does not leave the display offline (${JSON.stringify(await displayState(page))})`,
+    );
+    await page.close();
+    kiosk.close();
+    await finishOwner(owner);
+  } else if (mode === "stream-failure") {
+    // The first realtime connection is refused with a 503, as an API mid-restart or a proxy with no
+    // upstream answers. A non-200 ends an EventSource for good; the question is whether anything
+    // re-establishes realtime without a reload.
+    const owner = await ownerPage();
+    const moduleId = await displayModuleId(owner.page);
+    const kiosk = await connect(kioskAddress, "kiosk");
+    const page = await openPage(kiosk, undefined, "stream-failure");
+    let refused = 0;
+    kiosk.on((message) => {
+      if (message.method === "Fetch.requestPaused") {
+        const { requestId } = message.params;
+        if (refused === 0) {
+          refused += 1;
+          page
+            .send("Fetch.fulfillRequest", {
+              body: Buffer.from('{"code":"REALTIME_UNAVAILABLE"}').toString("base64"),
+              requestId,
+              responseCode: 503,
+              responseHeaders: [
+                { name: "content-type", value: "application/json" },
+                { name: "cache-control", value: "no-store" },
+              ],
+            })
+            .catch((error) => note(`interception reply failed: ${error.message}`));
+        } else {
+          page
+            .send("Fetch.continueRequest", { requestId })
+            .catch((error) => note(`interception reply failed: ${error.message}`));
+        }
+      }
+    });
+    await page.send("Fetch.enable", { patterns: [{ urlPattern: "*/api/v1/module-realtime/*" }] });
+    await page.navigate(`${apiOrigin}/display/${moduleId}`);
+    await page.waitFor(
+      `(${find("display-module-container")}?.childElementCount ?? 0) > 0`,
+      "the module to mount",
+    );
+    note(`realtime connections refused on purpose: ${refused}`);
+    const result = await watch([page], 90_000, {
+      onSample: () => page.streams.some((stream) => stream.status === 200),
+    });
+    printTally(result);
+    summarize(page, result.from);
+    check(
+      page.streams.some((stream) => stream.status === 200),
+      "after one refused realtime connection, realtime is re-established without a reload",
+    );
+    await page.close();
+    kiosk.close();
+    await finishOwner(owner);
+  } else if (mode === "caching-probe") {
+    // Whether a display on the API origin can store an image from the storage origin in Cache
+    // Storage, which is what Signage's offline media relies on. Reports only statuses, response
+    // types, and whether CORS headers were present — never a URL.
+    const owner = await ownerPage();
+    const moduleId = await displayModuleId(owner.page);
+    const groups = await owner.page.api("GET", "/api/v1/modules/signage/channel-groups");
+    const channelId = groups.body?.groups?.[0]?.channels?.[0]?.id;
+    const kiosk = await connect(kioskAddress, "kiosk");
+    const page = await openPage(kiosk, `${apiOrigin}/display/${moduleId}`, "probe");
+    check(await waitLive(page, "the display"), "the display is live");
+    progress("letting the display prefetch its slides (15s)");
+    await sleep(15_000);
+    const probe = await page.evaluate(
+      `(async () => {
+        const result = {};
+        const state = await fetch(${JSON.stringify(`${apiOrigin}/api/v1/modules/signage/channels/${channelId}/display-state`)}, { credentials: "include", signal: AbortSignal.timeout(${requestTimeoutMs}) }).then((r) => r.json());
+        const url = (state.selections || []).map((s) => s.media && s.media.display && s.media.display.url).find(Boolean);
+        result.selections = (state.selections || []).length;
+        result.haveUrl = Boolean(url);
+        if (!url) return result;
+        result.sameOrigin = new URL(url).origin === location.origin;
+        try {
+          const response = await fetch(url, { signal: AbortSignal.timeout(${requestTimeoutMs}) });
+          result.cors = { status: response.status, type: response.type, allowOrigin: response.headers.get("access-control-allow-origin") === null ? "absent" : (response.headers.get("access-control-allow-origin") === location.origin || response.headers.get("access-control-allow-origin") === "*" ? "matches" : "other") };
+          try {
+            const cache = await caches.open("display-repro-probe");
+            await cache.put("/display-repro-probe/cors", response.clone());
+            result.cors.cachePut = "stored";
+          } catch (error) { result.cors.cachePut = "refused: " + error.name; }
+        } catch (error) { result.cors = { error: error.name }; }
+        try {
+          const response = await fetch(url, { mode: "no-cors", signal: AbortSignal.timeout(${requestTimeoutMs}) });
+          result.noCors = { status: response.status, type: response.type, ok: response.ok };
+          try {
+            const cache = await caches.open("display-repro-probe");
+            await cache.put("/display-repro-probe/opaque", response.clone());
+            result.noCors.cachePut = "stored";
+          } catch (error) { result.noCors.cachePut = "refused: " + error.name; }
+        } catch (error) { result.noCors = { error: error.name }; }
+        result.imgPlain = await new Promise((resolve) => { const i = new Image(); i.onload = () => resolve("loaded"); i.onerror = () => resolve("error"); i.src = url; });
+        result.imgCors = await new Promise((resolve) => { const i = new Image(); i.crossOrigin = "anonymous"; i.onload = () => resolve("loaded"); i.onerror = () => resolve("error"); i.src = url + (url.includes("?") ? "&" : "?") + "cors-probe=1"; });
+        await caches.delete("display-repro-probe");
+        result.signageCachedEntries = await caches.open("ekklesia-signage-media-v1").then((c) => c.keys()).then((k) => k.length).catch(() => "unavailable");
+        return result;
+      })()`,
+      "probing cross-origin image caching",
+    );
+    note(`caching probe: ${JSON.stringify(probe)}`);
+    summarize(page, stepStartedAt);
+    check(probe.signageCachedEntries > 0, "Signage has stored slide images in Cache Storage");
+    await page.close();
+    kiosk.close();
+    await finishOwner(owner);
+  } else if (mode === "stream-recycle") {
+    // One page, past the five-minute realtime connection age, with every stream's open and end
+    // time in UTC so they can be laid against the proxy's own log.
+    const owner = await ownerPage();
+    const moduleId = await displayModuleId(owner.page);
+    const kiosk = await connect(kioskAddress, "kiosk");
+    const page = await openPage(kiosk, `${apiOrigin}/display/${moduleId}`, "recycle");
+    check(await waitLive(page, "the display"), "the display is live");
+    const result = await watch([page], recycleWatchMs, { viewerOwner: owner.page });
+    printTally(result);
+    summarize(page, result.from);
+    note(`connectivity events ${JSON.stringify(await connectivityEvents(page))}`);
+    const blobs = await blobCounts(page);
+    const exhausted = page.stateFetches.filter((fetch) =>
+      fetch.outcome.includes("INSUFFICIENT_RESOURCES"),
+    );
+    note(
+      `object URLs: created ${blobs.created}, revoked ${blobs.revoked}, live ${blobs.created - blobs.revoked}; channel-state fetches refused for insufficient resources: ${exhausted.length}`,
+    );
+    check(
+      blobs.created - blobs.revoked <= MAX_LIVE_OBJECT_URLS,
+      `live object URLs stay bounded (at most ${MAX_LIVE_OBJECT_URLS}; ${blobs.created - blobs.revoked} live)`,
+    );
+    check(
+      result.tally.get("recycle").offline === 0,
+      `the display stayed live for the whole ${recycleWatchMs / 1000}s watch${recycleWatchMs > 300_000 ? ", across a realtime recycle" : ""}`,
+    );
+    check(page.sessionRequests.length === 1, "the display was never remounted during the watch");
+    await page.close();
+    kiosk.close();
+    await finishOwner(owner);
+  } else if (mode === "revoke") {
+    const owner = await ownerPage();
+    const moduleId = await displayModuleId(owner.page);
+    const kiosk = await connect(kioskAddress, "kiosk");
+    const page = await openPage(kiosk, `${apiOrigin}/display/${moduleId}`, "revoked");
+    check(
+      await waitLive(page, "the display before revocation"),
+      "the display is live before revocation",
+    );
+    const devices = (await coreDevices(owner.page)).filter((device) => !device.revoked);
+    for (const device of devices) {
+      await owner.page.api("POST", `/api/v1/devices/${device.id}/revoke`);
+    }
+    note(`revoked ${devices.length} device(s) at ${clock()}; watching the open page`);
+    const result = await watch([page], watchMs, { onSample: (_page, state) => state.pairing });
+    printTally(result);
+    summarize(page, result.from);
+    check(
+      result.stopped,
+      `a revoked display returns to the pairing screen without a reload (within ${watchMs / 1000}s)`,
+    );
+    await page.close();
+    kiosk.close();
+    await finishOwner(owner);
+  } else {
+    throw new Error(
+      "usage: deploy-smoke-display.mjs pair|restart|two-pages|connectivity-blip|session-blip|stream-failure|caching-probe|stream-recycle|revoke",
+    );
+  }
+}
+
+try {
+  await run();
+} catch (error) {
+  if (error instanceof HangError) {
+    console.log(`FAIL ${mode}: ${error.message}; also in flight: ${describeInFlight()}`);
+  } else {
+    console.log(`FAIL ${mode}: ${error instanceof Error ? error.message : "an unexpected error"}`);
+  }
+  process.exit(2);
+}
+
+if (failures > 0) {
+  console.error(`${failures} check(s) failed`);
+  process.exit(1);
+}
+process.exit(0);

scripts/smoke-deploy.sh

@@ -15,6 +15,12 @@
 # With EKKLESIA_SMOKE_IMAGE, the given image is used as is in either topology; a split run then
 # exercises whichever of its modules publishes UI and a display.
 #
+# Opt-in, split topology only: EKKLESIA_SMOKE_DISPLAY_DEVICE=1 adds the display-device scenarios in
+# scripts/lib/deploy-smoke-display.mjs (pairing a kiosk browser with a persistent profile, a clean
+# restart, two pages, injected network failures, revocation), each with its own time budget.
+# EKKLESIA_SMOKE_DISPLAY_SCENARIOS selects a subset; EKKLESIA_SMOKE_WATCH_MS and
+# EKKLESIA_SMOKE_RECYCLE_WATCH_MS change how long they watch. Several scenarios expect Signage.
+#
 # TEST-ONLY: the proxy runs with Caddy's internal CA (`local_certs`) instead of ACME, because a test
 # run has no public hostname. Never use that option in an installation.
 #
@@ -33,6 +39,8 @@ cd "$repository_root"
 
 # shellcheck source=scripts/lib/compose-diagnostics.sh
 source "$repository_root/scripts/lib/compose-diagnostics.sh"
+# shellcheck source=scripts/lib/ci-project.sh
+source "$repository_root/scripts/lib/ci-project.sh"
 
 for command in docker openssl awk; do
   command -v "$command" >/dev/null || {
@@ -61,7 +69,10 @@ same-origin | split) ;;
 esac
 
 run_id="$(openssl rand -hex 4)"
-project="ekklesia-deploy-smoke-${run_id}"
+# Unique per invocation, and findable by run and job if this script is killed before its trap runs
+# (scripts/lib/ci-project.sh, scripts/ci-sweep.sh). Everything started outside Compose below carries
+# the same name in EKKLESIA_CI_LABEL.
+project="$(ci_project_name "deploy-${topology}")"
 work="$(mktemp -d)"
 image="${EKKLESIA_SMOKE_IMAGE:-}"
 built_image=""
@@ -141,6 +152,12 @@ cleanup() {
   if [[ -n "$browser_container" ]]; then
     docker rm --force "$browser_container" >/dev/null 2>&1 || true
   fi
+  if [[ -n "${kiosk_container:-}" ]]; then
+    docker ps --all --quiet --filter "name=${project}-display-" | xargs -r docker rm --force \
+      >/dev/null 2>&1 || true
+    docker rm --force "$kiosk_container" >/dev/null 2>&1 || true
+    docker volume rm --force "$kiosk_volume" >/dev/null 2>&1 || true
+  fi
   if [[ -f "$work/compose.env" ]]; then
     "${compose[@]}" --profile ops down --volumes --remove-orphans --timeout 10 >/dev/null 2>&1 || status=1
   fi
@@ -174,7 +191,9 @@ fail() {
 if [[ -z "$image" && "$topology" == same-origin ]]; then
   step "Building the platform-only image"
   built_image="ekklesia-deploy-smoke:${run_id}"
-  DOCKER_BUILDKIT=1 docker build --quiet --file deploy/Dockerfile --tag "$built_image" . >/dev/null
+  DOCKER_BUILDKIT=1 docker build --quiet --file deploy/Dockerfile --tag "$built_image" \
+    --label "${EKKLESIA_CI_LABEL}=${project}" --label "${EKKLESIA_CI_CREATED_LABEL}=$(date +%s)" \
+    . >/dev/null
   image="$built_image"
 elif [[ -z "$image" ]]; then
   # TEST-ONLY: the repository's module template, installed as the module `example`, in a copy of the
@@ -194,6 +213,7 @@ elif [[ -z "$image" ]]; then
   sed -i 's|^  templates/module:$|  modules/example:|' "$context/pnpm-lock.yaml"
   built_image="ekklesia-deploy-smoke:${run_id}"
   DOCKER_BUILDKIT=1 docker build --quiet --file "$context/deploy/Dockerfile" --tag "$built_image" \
+    --label "${EKKLESIA_CI_LABEL}=${project}" --label "${EKKLESIA_CI_CREATED_LABEL}=$(date +%s)" \
     "$context" >/dev/null || fail "the test-module image did not build."
   rm -rf "$context"
   image="$built_image"
@@ -350,6 +370,7 @@ client() {
   local mode="$1"
   shift
   docker run --rm --read-only --cap-drop ALL --security-opt no-new-privileges \
+    --label "${EKKLESIA_CI_LABEL}=${project}" \
     --network "$edge_network" "$@" "${host_mappings[@]}" \
     -e NODE_EXTRA_CA_CERTS=/smoke/client-ca.pem -e SMOKE_HOSTNAME="$hostname" \
     -e SMOKE_DASHBOARD_HOSTNAME="$dashboard_hostname" \
@@ -389,7 +410,7 @@ if [[ "$topology" == split ]]; then
   browser_container="${project}-browser"
   # The image's entry script passes its arguments on unquoted, so none may contain a space: the
   # hostnames reach Chromium through /etc/hosts, which it reads, rather than a resolver rule.
-  docker run --detach --rm --name "$browser_container" \
+  docker run --detach --rm --name "$browser_container" --label "${EKKLESIA_CI_LABEL}=${project}" \
     --cap-drop ALL --security-opt no-new-privileges --tmpfs /tmp \
     --network "$edge_network" --ip "$browser_address" "${host_mappings[@]}" \
     "$browser_image" \
@@ -398,12 +419,88 @@ if [[ "$topology" == split ]]; then
   cp scripts/lib/deploy-smoke-browser.mjs "$work/browser.mjs"
   stage browser.mjs
   docker run --rm --read-only --cap-drop ALL --security-opt no-new-privileges \
+    --label "${EKKLESIA_CI_LABEL}=${project}" \
     --network "$edge_network" \
     -e SMOKE_CDP_ADDRESS="${browser_address}:9222" \
     -e SMOKE_HOSTNAME="$hostname" -e SMOKE_DASHBOARD_HOSTNAME="$dashboard_hostname" \
     -e SMOKE_OWNER_EMAIL="$owner_email" -e SMOKE_OWNER_PASSWORD="$owner_password" \
     -v "$daemon_work/browser.mjs:/smoke/browser.mjs:ro" \
     --entrypoint node "$image" /smoke/browser.mjs || fail "browser checks failed."
+
+  if [[ -n "${EKKLESIA_SMOKE_DISPLAY_DEVICE:-}" ]]; then
+    # A kiosk: a second browser whose profile directory is a volume, so it outlives the container
+    # exactly as a kiosk's --user-data-dir outlives a browser restart.
+    kiosk_container="${project}-kiosk"
+    kiosk_volume="${project}-kiosk-profile"
+    kiosk_address="10.249.${octet}.250"
+    docker volume create --label "${EKKLESIA_CI_LABEL}=${project}" "$kiosk_volume" >/dev/null
+    start_kiosk() {
+      docker run --detach --rm --name "$kiosk_container" --label "${EKKLESIA_CI_LABEL}=${project}" \
+        --cap-drop ALL --security-opt no-new-privileges --tmpfs /tmp \
+        --network "$edge_network" --ip "$kiosk_address" "${host_mappings[@]}" \
+        -v "${kiosk_volume}:/profile" \
+        "$browser_image" \
+        "--ignore-certificate-errors-spki-list=${spki}" \
+        --user-data-dir=/profile >/dev/null || fail "the kiosk browser did not start."
+    }
+    wait_kiosk_exit() {
+      for _ in $(seq 1 60); do
+        docker inspect "$kiosk_container" >/dev/null 2>&1 || return 0
+        sleep 0.5
+      done
+      fail "the kiosk browser did not exit after a clean close."
+    }
+    cp scripts/lib/deploy-smoke-display.mjs "$work/display.mjs"
+    stage display.mjs
+    # display_driver <budget seconds> <step>: one scenario with a hard budget. The driver enforces it
+    # itself and names whatever call was still waiting; `timeout` is the backstop if it cannot.
+    display_driver() {
+      local budget="$1" step_name="$2" driver="${project}-display-${2}"
+      timeout --signal=TERM "$((budget + 30))" \
+        docker run --rm --name "$driver" --read-only --cap-drop ALL --security-opt no-new-privileges \
+        --network "$edge_network" \
+        -e SMOKE_CDP_ADDRESS="${browser_address}:9222" \
+        -e SMOKE_KIOSK_CDP_ADDRESS="${kiosk_address}:9222" \
+        -e SMOKE_STEP_BUDGET_MS="$((budget * 1000))" \
+        -e SMOKE_WATCH_MS="${EKKLESIA_SMOKE_WATCH_MS:-120000}" \
+        -e SMOKE_RECYCLE_WATCH_MS="${EKKLESIA_SMOKE_RECYCLE_WATCH_MS:-330000}" \
+        -e SMOKE_HOSTNAME="$hostname" -e SMOKE_DASHBOARD_HOSTNAME="$dashboard_hostname" \
+        -e SMOKE_OWNER_EMAIL="$owner_email" -e SMOKE_OWNER_PASSWORD="$owner_password" \
+        -v "$daemon_work/display.mjs:/smoke/display.mjs:ro" \
+        --entrypoint node "$image" /smoke/display.mjs "$step_name"
+      local status=$?
+      if ((status == 124)); then
+        echo "FAIL ${step_name}: the driver did not stop within its budget plus 30s; removed it"
+        docker rm --force "$driver" >/dev/null 2>&1 || true
+      fi
+      return "$status"
+    }
+    display_status=0
+    step "Pairing a kiosk display, then closing its browser cleanly"
+    start_kiosk
+    display_driver 180 pair || display_status=1
+    wait_kiosk_exit
+    step "Restarting the kiosk browser on the same profile"
+    start_kiosk
+    display_driver 120 restart || display_status=1
+    for scenario in ${EKKLESIA_SMOKE_DISPLAY_SCENARIOS:-two-pages connectivity-blip session-blip stream-failure caching-probe stream-recycle revoke}; do
+      case "$scenario" in
+      two-pages) budget=240 ;;
+      stream-recycle) budget=420 ;;
+      revoke) budget=210 ;;
+      *) budget=150 ;;
+      esac
+      step "Display scenario: ${scenario} (budget ${budget}s)"
+      display_driver "$budget" "$scenario" || display_status=1
+    done
+    step "Proxy: streams aborted with an incomplete response (UTC time, path only)"
+    # Captured, never printed: only each line's time and path (no query, so no signature) leave it.
+    proxy_logs="$("${compose[@]}" logs --no-color proxy 2>/dev/null)" || proxy_logs=""
+    grep -o '{.*"aborting with incomplete response".*}' <<<"$proxy_logs" |
+      sed -E 's/.*"ts":([0-9]+)[^,]*,.*"uri":"([^"?]*)[^"]*".*/\1 \2/' |
+      while read -r ts uri; do echo "     $(date -u -d "@${ts}" +%T) ${uri}"; done || true
+    ((display_status == 0)) || fail "display-device checks failed."
+  fi
 fi
 
 step "Proving the API's database connection is certificate-verified TLS"
@@ -468,12 +565,16 @@ grep -q 'TLS required with certificate and hostname verification (explicit priva
   <<<"$doctor_output" || fail "Doctor did not report verify-full with the private CA."
 grep -q '✓ Object storage reachable' <<<"$doctor_output" ||
   fail "Doctor could not reach object storage over HTTPS."
+# Doctor runs as `migrate`; it must check the client-address configuration the API runs with, which
+# behind the proxy is trusted-proxy mode, not the direct mode an unconfigured process reports.
+grep -q 'Trusted-proxy client-address mode' <<<"$doctor_output" ||
+  fail "Doctor did not check the API's trusted-proxy client-address configuration."
 for value in "$postgres_password" "$s3_secret_key" "$identity_secret"; do
   if grep -qF -- "$value" <<<"$doctor_output"; then
     fail "Doctor output contains a secret."
   fi
 done
-echo "ok   Doctor: READY, verify-full with the private CA, storage reachable, no secret printed"
+echo "ok   Doctor: READY, verify-full with the private CA, storage reachable, the API's client-address mode, no secret printed"
 
 api_logs="$("${compose[@]}" logs --no-color api 2>&1)"
 for value in "$postgres_password" "$s3_secret_key" "$identity_secret" "$owner_password"; do

scripts/smoke-infrastructure.sh

@@ -9,6 +9,8 @@ cd "$repository_root"
 source "$repository_root/scripts/lib/compose-job-network.sh"
 # shellcheck source=scripts/lib/compose-diagnostics.sh
 source "$repository_root/scripts/lib/compose-diagnostics.sh"
+# shellcheck source=scripts/lib/ci-project.sh
+source "$repository_root/scripts/lib/ci-project.sh"
 
 command -v docker >/dev/null || {
   echo "Docker is required for the infrastructure smoke test." >&2
@@ -30,11 +32,9 @@ if [[ "$curl_help" != *"--aws-sigv4"* ]]; then
   exit 1
 fi
 
-# Unique per invocation, not only per run: several runs can share a runner's Docker daemon at once,
-# and the shell's PID is often identical across job containers.
-raw_project="ekklesia-smoke-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-0}-$$-${RANDOM}${RANDOM}"
-project="${raw_project//[^a-zA-Z0-9_-]/-}"
-project="${project,,}"
+# Unique per invocation, and findable by run and job if this script is killed before its trap runs
+# (scripts/lib/ci-project.sh, scripts/ci-sweep.sh).
+project="$(ci_project_name infrastructure)"
 temporary_directory="$(mktemp -d)"
 payload_file="$temporary_directory/payload.txt"
 download_file="$temporary_directory/download.txt"

scripts/test-ci-sweep.sh

@@ -0,0 +1,129 @@
+#!/usr/bin/env bash
+# Proves scripts/ci-sweep.sh against real Docker resources: what it removes, and above all what it
+# leaves alone. CI runs it before trusting the sweep with a shared runner's daemon.
+#
+#   bash scripts/test-ci-sweep.sh
+#
+# Every resource it creates is named and labeled for this run alone, under fake run identifiers, so
+# it can never touch a real stack; and it removes all of them on exit, whatever happens.
+
+set -Eeuo pipefail
+
+repository_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$repository_root"
+
+image="$(sed -n 's/^[[:space:]]*image: \(postgres:[^[:space:]]*\)$/\1/p' deploy/compose.yaml | head -1)"
+[[ -n "$image" ]] || {
+  echo "test-ci-sweep: no pinned PostgreSQL image in deploy/compose.yaml." >&2
+  exit 1
+}
+docker image inspect "$image" >/dev/null 2>&1 || docker pull --quiet "$image" >/dev/null
+
+nonce="$(openssl rand -hex 4)"
+compose_label="com.docker.compose.project"
+ci_label="org.riverofglorymn.ekklesia.ci.project"
+failures=0
+created=()
+
+check() {
+  if eval "$1"; then
+    echo "ok   $2"
+  else
+    echo "FAIL $2"
+    failures=$((failures + 1))
+  fi
+}
+
+# project_exists <project>: any container, network, volume, or image still carries its label.
+project_exists() {
+  local project="$1"
+  [[ -n "$(
+    docker ps --all --quiet --filter "label=${compose_label}=${project}"
+    docker ps --all --quiet --filter "label=${ci_label}=${project}"
+    docker network ls --quiet --filter "label=${compose_label}=${project}"
+    docker volume ls --quiet --filter "label=${compose_label}=${project}"
+    docker image ls --quiet --filter "label=${ci_label}=${project}"
+  )" ]]
+}
+
+staging_exists() {
+  docker run --rm --entrypoint test -v /tmp:/host-tmp "$image" -d "/host-tmp/$1"
+}
+
+# make_stack <project>: what a CI script leaves behind when it is killed: a Compose-labeled network,
+# volume, and running container; a CI-labeled container and image; and a staging directory on the
+# Docker host.
+make_stack() {
+  local project="$1"
+  created+=("$project")
+  docker network create --label "${compose_label}=${project}" "${project}_default" >/dev/null
+  docker volume create --label "${compose_label}=${project}" "${project}_data" >/dev/null
+  docker run --detach --name "${project}-db" --label "${compose_label}=${project}" \
+    --network "${project}_default" -v "${project}_data:/data" --entrypoint sleep "$image" 600 >/dev/null
+  docker run --detach --name "${project}-browser" --label "${ci_label}=${project}" \
+    --entrypoint sleep "$image" 600 >/dev/null
+  printf 'FROM %s\nLABEL %s=%s %s=%s\n' "$image" "$ci_label" "$project" \
+    "org.riverofglorymn.ekklesia.ci.created" "$(date +%s)" |
+    docker build --quiet --tag "${project}-image:test" - >/dev/null
+  docker run --rm --entrypoint mkdir -v /tmp:/host-tmp "$image" -p "/host-tmp/${project}"
+}
+
+job_container="test-ci-sweep-job-${nonce}"
+cleanup() {
+  local status=$?
+  trap - EXIT INT TERM
+  docker rm --force "$job_container" >/dev/null 2>&1 || true
+  for project in "${created[@]}"; do
+    docker ps --all --quiet --filter "label=${compose_label}=${project}" | xargs -r docker rm --force >/dev/null 2>&1 || true
+    docker ps --all --quiet --filter "label=${ci_label}=${project}" | xargs -r docker rm --force >/dev/null 2>&1 || true
+    docker network rm "${project}_default" >/dev/null 2>&1 || true
+    docker volume rm --force "${project}_data" >/dev/null 2>&1 || true
+    docker image rm --force "${project}-image:test" >/dev/null 2>&1 || true
+    docker run --rm --entrypoint rm -v /tmp:/host-tmp "$image" -rf "/host-tmp/${project}" >/dev/null 2>&1 || true
+  done
+  exit "$status"
+}
+trap cleanup EXIT INT TERM
+
+# shellcheck source=scripts/lib/ci-project.sh
+source "$repository_root/scripts/lib/ci-project.sh"
+
+# Fake run identifiers: nothing here can match a real run's stacks.
+export GITHUB_RUN_ID="sweeptest${nonce}"
+export GITHUB_RUN_ATTEMPT="1"
+mine="$(EKKLESIA_CI_JOB_KEY=alpha ci_project_name mine)"
+sibling="$(EKKLESIA_CI_JOB_KEY=beta ci_project_name sibling)"
+foreign="ekklesia-sweep-control-${nonce}"
+for project in "$mine" "$sibling" "$foreign"; do
+  make_stack "$project"
+done
+# A job container connected to this job's stack network, exactly as compose-job-network.sh does.
+docker run --detach --name "$job_container" --entrypoint sleep "$image" 600 >/dev/null
+docker network connect "${mine}_default" "$job_container"
+
+echo "--- --job removes this job's stacks, at any age, and nothing else"
+EKKLESIA_CI_JOB_KEY=alpha bash scripts/ci-sweep.sh --job
+check '! project_exists "$mine"' "this job's containers, network, volume, and image are gone"
+check '! staging_exists "$mine"' "this job's staging directory on the Docker host is gone"
+check 'project_exists "$sibling"' "another job's stack in the same run is left alone"
+check 'project_exists "$foreign"' "a stack outside the CI prefix is left alone"
+check 'docker inspect "$job_container" >/dev/null 2>&1' \
+  "a job container connected to the stack's network is disconnected, never removed"
+
+echo "--- --job refuses to run outside CI, where it could remove a developer's own stacks"
+check '! env -u GITHUB_RUN_ID bash scripts/ci-sweep.sh --job >/dev/null 2>&1' \
+  "--job without GITHUB_RUN_ID fails"
+
+echo "--- --stale removes only CI stacks older than its threshold"
+bash scripts/ci-sweep.sh --stale
+check 'project_exists "$sibling"' "a young CI stack survives the default threshold"
+bash scripts/ci-sweep.sh --stale --max-age-seconds 0
+check '! project_exists "$sibling"' "a CI stack older than the threshold is removed"
+check '! staging_exists "$sibling"' "its staging directory too"
+check 'project_exists "$foreign"' "a stack outside the CI prefix is never removed, at any age"
+
+if ((failures > 0)); then
+  echo "test-ci-sweep: ${failures} check(s) failed." >&2
+  exit 1
+fi
+echo "CI sweep checks passed."

scripts/test-migrations-integration.sh

@@ -9,6 +9,8 @@ cd "$repository_root"
 source "$repository_root/scripts/lib/compose-job-network.sh"
 # shellcheck source=scripts/lib/compose-diagnostics.sh
 source "$repository_root/scripts/lib/compose-diagnostics.sh"
+# shellcheck source=scripts/lib/ci-project.sh
+source "$repository_root/scripts/lib/ci-project.sh"
 
 command -v docker >/dev/null || {
   echo "Docker is required for the PostgreSQL migration integration suite." >&2
@@ -27,11 +29,9 @@ if ! compose_job_network_initialize; then
   exit 1
 fi
 
-# Unique per invocation, not only per run: several runs (and several jobs of one run) can share a
-# runner's Docker daemon at once, and the shell's PID is often identical across job containers.
-raw_project="ekklesia-migration-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-0}-${GITHUB_JOB:-job}-$$-${RANDOM}${RANDOM}"
-project="${raw_project//[^a-zA-Z0-9_-]/-}"
-project="${project,,}"
+# Unique per invocation, and findable by run and job if this script is killed before its trap runs
+# (scripts/lib/ci-project.sh, scripts/ci-sweep.sh).
+project="$(ci_project_name migration)"
 
 export POSTGRES_DB="ekklesia_migration_test"
 export POSTGRES_USER="ekklesia_migration_test"

tests/ci-scripts.test.ts

@@ -4,6 +4,7 @@ import path from "node:path";
 import { fileURLToPath } from "node:url";
 import { promisify } from "node:util";
 import { describe, expect, it } from "vitest";
+import { parse } from "yaml";
 
 /**
  * The scripts CI runs against a shared Docker daemon. Several runs, and several jobs of one run, can
@@ -59,21 +60,68 @@ describe("CI scripts on a shared Docker daemon", () => {
     }
   });
 
-  it("give every Compose project a name unique to the invocation, not only the run", async () => {
-    expect(await source("scripts/test-migrations-integration.sh")).toMatch(
-      /raw_project="ekklesia-migration-[^"\n]*\$\$-\$\{RANDOM\}\$\{RANDOM\}"/u,
+  it("name every stack through one helper: the reserved CI prefix, the job's key, and a unique suffix", async () => {
+    const helper = await source("scripts/lib/ci-project.sh");
+    // Run, attempt, job key, kind, then a per-invocation suffix: unique even when job containers
+    // share a PID, and findable later by run and job alone (scripts/ci-sweep.sh --job).
+    expect(helper).toContain(
+      "ekklesia-ci-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-0}-${EKKLESIA_CI_JOB_KEY:-${GITHUB_JOB:-local}}-",
     );
-    expect(await source("scripts/smoke-infrastructure.sh")).toMatch(
-      /raw_project="ekklesia-smoke-[^"\n]*\$\$-\$\{RANDOM\}\$\{RANDOM\}"/u,
-    );
-    const deploy = await source("scripts/smoke-deploy.sh");
-    expect(deploy).toContain('run_id="$(openssl rand -hex 4)"');
-    expect(deploy).toContain('project="ekklesia-deploy-smoke-${run_id}"');
+    expect(helper).toContain("$$-${RANDOM}${RANDOM}");
     for (const relativePath of ciScripts) {
-      expect(await source(relativePath), relativePath).toContain('--project-name "$project"');
+      const text = await source(relativePath);
+      expect(text, relativePath).toContain('source "$repository_root/scripts/lib/ci-project.sh"');
+      expect(text, relativePath).toMatch(/^project="\$\(ci_project_name [a-z"${}_-]+\)"$/mu);
+      expect(text, relativePath).not.toContain("raw_project=");
+      expect(text, relativePath).toContain('--project-name "$project"');
     }
   });
 
+  it("label everything the deploy smoke creates outside Compose, so a sweep can find it", async () => {
+    // Continuation lines joined, so each command is checked whole.
+    const deploy = (await source("scripts/smoke-deploy.sh")).replace(/\\\n\s*/gu, " ");
+    const creating = deploy
+      .split("\n")
+      .filter((line) => /\bdocker (?:run --detach|volume create|build)\b/u.test(line));
+    expect(creating.length).toBeGreaterThanOrEqual(4);
+    for (const line of creating) {
+      expect(line).toContain('--label "${EKKLESIA_CI_LABEL}=${project}"');
+    }
+  });
+
+  it("sweep stale stacks before, and this job's own stacks after, every job that starts any", async () => {
+    const workflow = parse(await source(".github/workflows/ci.yml")) as {
+      jobs: Record<
+        string,
+        { env?: Record<string, string>; steps: { if?: string; run?: string; name?: string }[] }
+      >;
+    };
+    const startsStacks =
+      /smoke-infrastructure\.sh|test:migrations:integration|smoke-deploy\.sh|docker build/u;
+    const jobs = Object.entries(workflow.jobs).filter(([, job]) =>
+      job.steps.some((step) => startsStacks.test(step.run ?? "")),
+    );
+    expect(jobs.map(([id]) => id).sort()).toEqual(
+      ["deploy-smoke", "image-build", "infrastructure-smoke", "migration-smoke"].sort(),
+    );
+    for (const [id, job] of jobs) {
+      expect(job.env?.EKKLESIA_CI_JOB_KEY, id).toBeTruthy();
+      const runs = job.steps.map((step) => step.run ?? "");
+      const firstStack = runs.findIndex((run) => startsStacks.test(run));
+      const stale = runs.findIndex((run) => run.includes("bash scripts/ci-sweep.sh --stale"));
+      const own = job.steps.findIndex(
+        (step) =>
+          step.if === "always()" && (step.run ?? "").includes("bash scripts/ci-sweep.sh --job"),
+      );
+      expect(stale, `${id}: stale sweep`).toBeGreaterThanOrEqual(0);
+      expect(stale, `${id}: stale sweep runs first`).toBeLessThan(firstStack);
+      expect(own, `${id}: its own cleanup, always`).toBeGreaterThan(firstStack);
+      expect(own, `${id}: cleanup is the last step`).toBe(job.steps.length - 1);
+    }
+    // One matrix leg's cleanup must never remove the other leg's running stack.
+    expect(workflow.jobs["deploy-smoke"]?.env?.EKKLESIA_CI_JOB_KEY).toContain("matrix.topology");
+  });
+
   it("choose the deploy smoke's edge subnet from those not in use, and choose again on a race", async () => {
     const deploy = await source("scripts/smoke-deploy.sh");
     expect(deploy).not.toMatch(/^octet=\$\(\(RANDOM/mu);

tests/deploy.test.ts

@@ -153,6 +153,11 @@ describe("reference deployment compose files", () => {
       EKKLESIA_CLIENT_ADDRESS_MODE: "trusted-proxy",
       EKKLESIA_CLIENT_ADDRESS_TRUSTED_PROXIES: `${proxyAddress}/32`,
     });
+    // Doctor runs as `migrate`. Given anything else, it validates a configuration the API is not
+    // running, and reports "Direct" for an installation that trusts its proxy.
+    expect(proxyOverlay.services?.migrate?.environment).toEqual(
+      proxyOverlay.services?.api?.environment,
+    );
     expect(proxyOverlay.networks?.edge?.ipam?.config).toEqual([
       {
         ip_range: "${EKKLESIA_EDGE_DYNAMIC_RANGE:-10.249.250.0/25}",