11 Commits
Author SHA1 Message Date
The_River fc504eee0a Merge pull request 'chore(release): prepare 0.1.2' (#62) from chore/release-0.1.2 into main
CI / verify (push) Successful in 56s
CI / infrastructure-smoke (push) Successful in 38s
CI / migration-smoke (push) Successful in 1m51s
CI / image-build (push) Successful in 7s
CI / deploy-smoke (same-origin) (push) Successful in 41s
CI / deploy-smoke (split) (push) Successful in 52s
Reviewed-on: https://git.riverofglorymn.org/River_of_Glory_Church/Ekklesia/pulls/62
Reviewed-by: The_River <[email protected]>
2026-10-06 04:08:58 +00:00
ekklesiaandClaude Opus 5.5 5abc8be167 chore(release): prepare 0.1.2
CI / verify (pull_request) Successful in 55s
CI / infrastructure-smoke (pull_request) Successful in 36s
CI / migration-smoke (pull_request) Successful in 1m51s
CI / image-build (pull_request) Successful in 27s
CI / deploy-smoke (same-origin) (pull_request) Successful in 42s
CI / deploy-smoke (split) (pull_request) Successful in 1m11s
Every workspace package declares 0.1.2, as the 0.1.1 release did for
itself, and the deploy README's build example names it. The tag
v0.1.2 created on 12dca3e carries package.json 0.1.1, which the
instance's assemble.sh refuses ("its tag and package.json disagree"),
and the running platform would report 0.1.1.

0.1.2 carries what the first production deployment found:
- #57: Doctor checks the client-address configuration the API runs;
- #58: storage over HTTP/1.1 (repeat image loads no longer fail with
  ERR_HTTP2_PROTOCOL_ERROR), and storage CORS owned by the proxy;
- #60: CI stacks never outlive their job;
- #56: display-device smoke scenarios;
- #59, #61: ADR 0047 proposed, and the CI documentation corrected for
  Gitea 1.27.2.

templates/module keeps its own version.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 22:56:35 -05:00
The_River 12dca3e8a4 Merge pull request 'docs: the server runs Gitea 1.27.2, not 1.24.2; say what it actually honours' (#61) from docs/gitea-1.27-ci-facts into main
CI / verify (push) Successful in 51s
CI / infrastructure-smoke (push) Successful in 34s
CI / migration-smoke (push) Successful in 1m50s
CI / image-build (push) Successful in 7s
CI / deploy-smoke (same-origin) (push) Successful in 45s
CI / deploy-smoke (split) (push) Successful in 45s
Reviewed-on: https://git.riverofglorymn.org/River_of_Glory_Church/Ekklesia/pulls/61
Reviewed-by: The_River <[email protected]>
2026-10-06 03:50:31 +00:00
ekklesiaandClaude Opus 5.5 0ec1755279 docs: the server runs Gitea 1.27.2, not 1.24.2; say what it actually honours
CI / verify (pull_request) Successful in 54s
CI / infrastructure-smoke (pull_request) Successful in 37s
CI / migration-smoke (pull_request) Successful in 1m50s
CI / image-build (pull_request) Successful in 18s
CI / deploy-smoke (same-origin) (pull_request) Successful in 42s
CI / deploy-smoke (split) (pull_request) Successful in 1m1s
The CI documentation, the workflow header, the README, ADR 0047 and the
sweep's comment all said the server was Gitea 1.24.2, which ignores
`permissions`, `concurrency` and `timeout-minutes`. The server reports
1.27.2 (GET /api/v1/version).

Each claim is now stated from what is verified on this server:
- `concurrency` is honoured, observed: Ekklesia runs 133 and 135 were
  cancelled when a newer run started for the same pull request;
- `permissions` is supported by 1.27 with Gitea's own scopes, but the
  token's actual authority is not visible through the API and not yet
  proven, so nothing claims the job token is read-only;
- `timeout-minutes` is no longer listed as unsupported, but is not
  verified, and teardown still relies on the stack sweep and a reaper.

Requirement 1 for executing untrusted contributions (upgrade to the
1.26 line or later) is marked done; items 2 and 3, configuring and
proving token authority, remain. ADR 0047 (still Proposed) restates
its "docker run, not a reusable workflow" reasons for 1.27, where
private sources are supported. The reasons still hold without the 1.24
argument.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 22:11:00 -05:00
The_River 874deabea3 Merge pull request 'fix(ci): nothing a CI job starts outlives it, even when the runner kills the job' (#60) from fix/ci-stack-cleanup into main
CI / verify (push) Successful in 55s
CI / infrastructure-smoke (push) Successful in 29s
CI / migration-smoke (push) Successful in 1m49s
CI / image-build (push) Successful in 7s
CI / deploy-smoke (same-origin) (push) Successful in 48s
CI / deploy-smoke (split) (push) Successful in 50s
Reviewed-on: https://git.riverofglorymn.org/River_of_Glory_Church/Ekklesia/pulls/60
Reviewed-by: The_River <[email protected]>
2026-10-06 02:11:53 +00:00
ekklesiaandClaude Opus 5.5 62951975da fix(ci): nothing a CI job starts outlives it, even when the runner kills the job
CI / verify (pull_request) Successful in 57s
CI / infrastructure-smoke (pull_request) Successful in 28s
CI / migration-smoke (pull_request) Successful in 1m49s
CI / image-build (pull_request) Successful in 19s
CI / deploy-smoke (same-origin) (pull_request) Successful in 48s
CI / deploy-smoke (split) (pull_request) Successful in 53s
A migration-smoke stack (PostgreSQL and SeaweedFS) ran on the shared
runner for two days. Every CI script removes its stack in an EXIT trap,
but a trap never runs when the runner kills the job container: a
cancelled job, or the runner's own time limit (Gitea 1.24 ignores
timeout-minutes). Reproduced: SIGKILL of the infrastructure smoke
mid-run leaves exactly that stack running.

Cleanup now has three layers:
- the script's own trap, unchanged;
- a final `if: always()` step in every job that starts stacks
  (infrastructure-smoke, migration-smoke, image-build, deploy-smoke)
  runs `scripts/ci-sweep.sh --job`, which removes everything that job
  started at any age;
- a first step in each of those jobs runs `scripts/ci-sweep.sh --stale`,
  which removes CI stacks older than three hours: what a killed job left
  before it could clean up.

How stacks are found is defined once, in scripts/lib/ci-project.sh:
- every CI Compose project is named `ekklesia-ci-<run>-<attempt>-<job
  key>-<kind>-<unique>`, and Compose labels its containers, networks and
  volumes with that name;
- the deploy smoke labels everything it starts outside Compose:
  browser, kiosk and driver containers, the kiosk volume, built images;
- images also record when CI built them. An image's own creation time
  can be its base's (BuildKit, metadata-only builds), and the sweep
  removed a brand-new image as "old" until it stopped trusting that.

The job key is per matrix leg, so one topology's cleanup cannot remove
the other's running stack. The sweep never touches anything outside the
ekklesia-ci- prefix, disconnects rather than removes a job container
attached to a stack's network, and refuses --job outside CI. Network
creation times are read as RFC 3339; Docker's default form was
unparseable.

scripts/test-ci-sweep.sh proves all of that against real Docker, and
CI runs it before relying on it. security.md and testing.md document
the rule, and that it complements, not replaces, a runner-side reaper.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-06 02:01:01 +00:00
ekklesiaandClaude Opus 5.5 703d92436b test(ci): every CI stack is cleaned up, even when its job is killed
A migration-smoke stack (PostgreSQL and SeaweedFS) leaked on the shared
runner and ran for two days, until it was removed by hand. Every CI
script already tears its stack down in an EXIT/INT/TERM trap, but a
trap never runs when the runner kills the job container: a cancelled
job, or the runner's own time limit, since Gitea 1.24 ignores
timeout-minutes.

Static checks (tests/ci-scripts.test.ts):
- every stack script names its project through one helper: the reserved
  ekklesia-ci- prefix, run, attempt, job key, and a unique suffix;
- everything the deploy smoke creates outside Compose carries the CI
  label;
- every workflow job that starts stacks sweeps stale CI stacks first,
  and removes its own in a final if: always() step. Each matrix leg has
  its own key, so one cannot remove the other's running stack.

Behaviour against real Docker (scripts/test-ci-sweep.sh):
- --job removes this job's containers, network, volume, image and
  staging directory, and leaves other jobs' and non-CI stacks alone;
- --job disconnects, never removes, a job container on the stack's
  network, and refuses to run outside CI;
- --stale spares young stacks and removes old ones, never anything
  outside the prefix.

Fails today: there is no helper, no sweep, no label and no step.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-06 02:01:01 +00:00
The_River 43911a088d Merge pull request 'docs: propose ADR 0047, a published harness every module runs in CI' (#59) from docs/adr-0047-module-harness into main
CI / verify (push) Successful in 58s
CI / infrastructure-smoke (push) Successful in 13s
CI / migration-smoke (push) Successful in 1m50s
CI / image-build (push) Successful in 6s
CI / deploy-smoke (same-origin) (push) Successful in 40s
CI / deploy-smoke (split) (push) Successful in 49s
Reviewed-on: https://git.riverofglorymn.org/River_of_Glory_Church/Ekklesia/pulls/59
Reviewed-by: The_River <[email protected]>
2026-10-06 02:00:39 +00:00
ekklesiaandClaude Opus 5.5 3a2f5d68f6 docs: ADR 0047, nothing the harness starts outlives its job
CI / verify (pull_request) Successful in 58s
CI / infrastructure-smoke (pull_request) Successful in 21s
CI / migration-smoke (pull_request) Successful in 1m51s
CI / image-build (pull_request) Successful in 19s
CI / deploy-smoke (same-origin) (pull_request) Successful in 43s
CI / deploy-smoke (split) (pull_request) Successful in 57s
A job the runner kills never runs its own cleanup trap, and the
platform's CI leaked a whole migration stack that way. The harness
follows the platform's three-layer rule:
- everything it starts is named and labeled as a CI stack;
- it cleans up in a trap;
- the template workflow sweeps stale CI stacks before running it, and
  removes its own in a final if: always() step.

The sweep ships in the harness image.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 20:54:38 -05:00
ekklesiaandClaude Opus 5.5 3827841205 docs: propose ADR 0047, a published harness every module runs in CI
CI / verify (pull_request) Successful in 53s
CI / infrastructure-smoke (pull_request) Successful in 21s
CI / migration-smoke (pull_request) Successful in 1m49s
CI / image-build (pull_request) Successful in 19s
CI / deploy-smoke (same-origin) (pull_request) Successful in 49s
CI / deploy-smoke (split) (pull_request) Canceled after 18s
Signage reached production with three defects that appear only inside
Ekklesia in a real browser, and no CI that ran it there. The ADR
proposes:
- a versioned harness image, ekklesia-module-harness:<platform version>,
  built and pushed by the platform's release CI;
- what the harness does: assemble a module into that platform, validate
  and build it, migrate a disposable database, start the split reference
  deployment, and mount every declared UI surface in headless Chromium;
- what fails it: console errors, unexpected 4xx/5xx, unhandled
  rejections, a display soak, and generic no-dead-ends injections;
- testing at the lowest and newest platform versions of a module's
  range;
- extra smoke steps in ekklesia.harness.json, beside the manifest
  rather than in it;
- a later data-access and egress check against declarations;
- a short template workflow that calls the harness by digest with one
  `docker run`. Not a reusable workflow: thinly supported on Gitea
  1.24, and private sources only arrive in 1.26.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-06 01:50:06 +00:00
The_River 2152eb036a Merge pull request 'fix(deploy): reach storage over HTTP/1.1, and own storage CORS in the proxy' (#58) from fix/storage-delivery-http1-cors into main
CI / verify (push) Successful in 53s
CI / infrastructure-smoke (push) Successful in 20s
CI / migration-smoke (push) Successful in 1m47s
CI / image-build (push) Successful in 7s
CI / deploy-smoke (same-origin) (push) Successful in 48s
CI / deploy-smoke (split) (push) Successful in 52s
Reviewed-on: https://git.riverofglorymn.org/River_of_Glory_Church/Ekklesia/pulls/58
Reviewed-by: The_River <[email protected]>
2026-10-06 01:39:39 +00:00
26 changed files with 862 additions and 84 deletions
+68 -19
View File
@@ -1,18 +1,13 @@
name: CI name: CI
# IMPORTANT — the currently inventoried server is Gitea 1.24.2, which *ignores* several keys used # The server runs Gitea 1.27.2 (docs/development/security.md, "What this server honours"):
# below. Per https://docs.gitea.com/1.24/usage/actions/comparison, its unsupported workflow syntax
# includes `permissions` / `jobs.<job_id>.permissions`, `concurrency`, and
# `jobs.<job_id>.timeout-minutes`. They are declared here deliberately, so the workflow is already
# correct on a supported release and on GitHub Actions, but on this server they are inert:
# #
# * `permissions:` does NOT restrict the in-job `GITEA_TOKEN`. Job-token permission controls, # * `concurrency:` is honoured: a newer push to a pull request cancels the run it supersedes, so a
# repository/owner defaults, and server-side maximum-permission clamping arrive in the 1.26 # cancelled run's own cleanup is never guaranteed (see the CI stack sweep in each stack job).
# feature line (https://docs.gitea.com/1.26/usage/actions/token-permissions). Until the server # * `permissions:` is supported by 1.27 with Gitea's own scopes, but what the job token can do on
# is upgraded and configured, the token's authority is whatever the server grants — not what # this server is not yet proven, so nothing here relies on it being read-only.
# this file requests. # * `timeout-minutes:` is no longer listed as unsupported, but is not verified here either;
# * `concurrency:` and `timeout-minutes:` are NOT bounded-execution guarantees here. Cancellation, # teardown relies on the stack sweep and the runner's own reaper.
# timeout, and teardown must be enforced by the trusted runner provisioner/reaper.
# #
# `persist-credentials: false` below is unaffected by any of this and is enforced by the action # `persist-credentials: false` below is unaffected by any of this and is enforced by the action
# itself. See docs/development/security.md. # itself. See docs/development/security.md.
@@ -63,6 +58,9 @@ jobs:
needs: verify needs: verify
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 10 timeout-minutes: 10
env:
# Names this job's stacks, so its final step removes exactly these (scripts/lib/ci-project.sh).
EKKLESIA_CI_JOB_KEY: infrastructure-smoke
steps: steps:
- name: Check out repository - name: Check out repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
@@ -72,13 +70,29 @@ jobs:
# in this workflow pushes, so the credential is never needed after checkout. # in this workflow pushes, so the credential is never needed after checkout.
persist-credentials: false persist-credentials: false
# A job the runner killed (cancelled, or past the runner's own time limit) never ran its
# cleanup. Anything a CI job left on this shared daemon more than three hours ago goes first.
- name: Sweep stale CI stacks from the runner
run: bash scripts/ci-sweep.sh --stale
- name: Prove the sweep removes exactly what it should, against real Docker resources
run: bash scripts/test-ci-sweep.sh
- name: Exercise PostgreSQL and S3 infrastructure - name: Exercise PostgreSQL and S3 infrastructure
run: bash scripts/smoke-infrastructure.sh run: bash scripts/smoke-infrastructure.sh
# Whatever happened above, this job's own stacks do not outlive it.
- name: Remove this job's CI stacks
if: always()
run: bash scripts/ci-sweep.sh --job
migration-smoke: migration-smoke:
needs: verify needs: verify
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 15 timeout-minutes: 15
env:
# Names this job's stacks, so its final step removes exactly these (scripts/lib/ci-project.sh).
EKKLESIA_CI_JOB_KEY: migration-smoke
steps: steps:
- name: Check out repository - name: Check out repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
@@ -88,6 +102,11 @@ jobs:
# in this workflow pushes, so the credential is never needed after checkout. # in this workflow pushes, so the credential is never needed after checkout.
persist-credentials: false persist-credentials: false
# A job the runner killed (cancelled, or past the runner's own time limit) never ran its
# cleanup. Anything a CI job left on this shared daemon more than three hours ago goes first.
- name: Sweep stale CI stacks from the runner
run: bash scripts/ci-sweep.sh --stale
- name: Set up Node.js - name: Set up Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0 uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with: with:
@@ -104,11 +123,18 @@ jobs:
- name: Exercise transactional migrations against PostgreSQL - name: Exercise transactional migrations against PostgreSQL
run: pnpm test:migrations:integration run: pnpm test:migrations:integration
# Whatever happened above, this job's own stacks do not outlive it.
- name: Remove this job's CI stacks
if: always()
run: bash scripts/ci-sweep.sh --job
image-build: image-build:
needs: verify needs: verify
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 20 timeout-minutes: 20
env: env:
# Names this job's stacks, so its final step removes exactly these (scripts/lib/ci-project.sh).
EKKLESIA_CI_JOB_KEY: image-build
# Cache mounts in deploy/Dockerfile need BuildKit; this selects it even where the Docker CLI # Cache mounts in deploy/Dockerfile need BuildKit; this selects it even where the Docker CLI
# has no buildx plugin. # has no buildx plugin.
DOCKER_BUILDKIT: "1" DOCKER_BUILDKIT: "1"
@@ -121,22 +147,33 @@ jobs:
# in this workflow pushes, so the credential is never needed after checkout. # in this workflow pushes, so the credential is never needed after checkout.
persist-credentials: false persist-credentials: false
# A job the runner killed (cancelled, or past the runner's own time limit) never ran its
# cleanup. Anything a CI job left on this shared daemon more than three hours ago goes first.
- name: Sweep stale CI stacks from the runner
run: bash scripts/ci-sweep.sh --stale
# The platform-only image (ADR 0044): a checkout has no `modules/`, so this proves the image # The platform-only image (ADR 0044): a checkout has no `modules/`, so this proves the image
# builds, validates, and assembles with zero modules. Build only — no registry login, no # builds, validates, and assembles with zero modules. Build only — no registry login, no
# credentials, no push, no tag anywhere but this runner's local image store. # credentials, no push, no tag anywhere but this runner's local image store.
# Tagged per run and attempt, not only per commit: a pull request and a push of the same commit # Tagged per run and attempt, not only per commit: a pull request and a push of the same commit
# can build at once on one runner's Docker daemon, and neither may remove the other's image. # can build at once on one runner's Docker daemon, and neither may remove the other's image.
# Labeled like every CI stack, so this job's final step removes it even if the build is killed.
- name: Build the production image without modules - name: Build the production image without modules
run: >- shell: bash
docker build run: |
--file deploy/Dockerfile source scripts/lib/ci-project.sh
--build-arg REVISION="$GITHUB_SHA" docker build \
--tag ekklesia-ci:"$GITHUB_SHA-${GITHUB_RUN_ID:-0}-${GITHUB_RUN_ATTEMPT:-1}" --file deploy/Dockerfile \
--build-arg REVISION="$GITHUB_SHA" \
--label "${EKKLESIA_CI_LABEL}=$(ci_project_name image)" \
--label "${EKKLESIA_CI_CREATED_LABEL}=$(date +%s)" \
--tag ekklesia-ci:"$GITHUB_SHA-${GITHUB_RUN_ID:-0}-${GITHUB_RUN_ATTEMPT:-1}" \
. .
- name: Remove the image from the runner # Whatever happened above, this job's own stacks do not outlive it.
- name: Remove this job's CI stacks
if: always() if: always()
run: docker image rm --force ekklesia-ci:"$GITHUB_SHA-${GITHUB_RUN_ID:-0}-${GITHUB_RUN_ATTEMPT:-1}" || true run: bash scripts/ci-sweep.sh --job
deploy-smoke: deploy-smoke:
needs: verify needs: verify
@@ -151,6 +188,8 @@ jobs:
- same-origin - same-origin
- split - split
env: env:
# Names this job's stacks, so its final step removes exactly these (scripts/lib/ci-project.sh).
EKKLESIA_CI_JOB_KEY: deploy-smoke-${{ matrix.topology }}
DOCKER_BUILDKIT: "1" DOCKER_BUILDKIT: "1"
EKKLESIA_SMOKE_TOPOLOGY: ${{ matrix.topology }} EKKLESIA_SMOKE_TOPOLOGY: ${{ matrix.topology }}
steps: steps:
@@ -162,6 +201,11 @@ jobs:
# in this workflow pushes, so the credential is never needed after checkout. # in this workflow pushes, so the credential is never needed after checkout.
persist-credentials: false persist-credentials: false
# A job the runner killed (cancelled, or past the runner's own time limit) never ran its
# cleanup. Anything a CI job left on this shared daemon more than three hours ago goes first.
- name: Sweep stale CI stacks from the runner
run: bash scripts/ci-sweep.sh --stale
# The reference deployment end to end (ADR 0044): a platform-only image, base + proxy with # The reference deployment end to end (ADR 0044): a platform-only image, base + proxy with
# Caddy's internal CA (test-only), an explicit migration apply against this job's own fresh # Caddy's internal CA (test-only), an explicit migration apply against this job's own fresh
# PostgreSQL, then the API, Dashboard, storage delivery, client-address chain, and verify-full # PostgreSQL, then the API, Dashboard, storage delivery, client-address chain, and verify-full
@@ -169,3 +213,8 @@ jobs:
# it; nothing is pushed, published, or logged in to. # it; nothing is pushed, published, or logged in to.
- name: Smoke-test the reference deployment (${{ matrix.topology }}) - name: Smoke-test the reference deployment (${{ matrix.topology }})
run: bash scripts/smoke-deploy.sh run: bash scripts/smoke-deploy.sh
# Whatever happened above, this job's own stacks do not outlive it.
- name: Remove this job's CI stacks
if: always()
run: bash scripts/ci-sweep.sh --job
+4 -3
View File
@@ -218,9 +218,10 @@ undeployed state, and the launch checklist in a single place. **CI is accepted a
trusted-code-only boundary**: the self-hosted Gitea runner executes only code controlled and trusted-code-only boundary**: the self-hosted Gitea runner executes only code controlled and
reviewed by trusted repository collaborators, and unknown or untrusted pull requests must not reviewed by trusted repository collaborators, and unknown or untrusted pull requests must not
execute on it automatically. What remains is operational and cannot be proven from this repository — execute on it automatically. What remains is operational and cannot be proven from this repository —
upgrading Gitea past 1.24.2 (which *ignores* the workflow's `permissions`, `concurrency`, and configuring and proving job-token authority server-side (the server now runs Gitea 1.27.2, which
`timeout-minutes` declarations), clamping job-token authority server-side, enforcing timeout and honours `concurrency` and documents `permissions`, but neither token limits nor `timeout-minutes` are
teardown in a trusted provisioner/reaper, and proving disposable runner isolation. Those are verified on it yet), enforcing timeout and teardown in a trusted provisioner/reaper, and proving
disposable runner isolation. Those are
prerequisites for **executing untrusted contributions**, alongside the deployment checklist that prerequisites for **executing untrusted contributions**, alongside the deployment checklist that
gates a **public launch**; moving future public contribution workflows to a service with isolated gates a **public launch**; moving future public contribution workflows to a service with isolated
hosted runners is an acceptable alternative. **Neither set of requirements blocks Phase 6** — Surface hosted runners is an acceptable alternative. **Neither set of requirements blocks Phase 6** — Surface
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@ekklesia/api", "name": "@ekklesia/api",
"version": "0.1.1", "version": "0.1.2",
"private": true, "private": true,
"type": "module", "type": "module",
"files": [ "files": [
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@ekklesia/core", "name": "@ekklesia/core",
"version": "0.1.1", "version": "0.1.2",
"private": true, "private": true,
"type": "module", "type": "module",
"files": [ "files": [
+2 -2
View File
@@ -384,10 +384,10 @@ finds its files:
```sh ```sh
docker build \ docker build \
--file deploy/Dockerfile \ --file deploy/Dockerfile \
--build-arg VERSION=0.1.1 \ --build-arg VERSION=0.1.2 \
--build-arg REVISION="$(git rev-parse HEAD)" \ --build-arg REVISION="$(git rev-parse HEAD)" \
--build-arg SOURCE=https://git.example.org/your-org/ekklesia \ --build-arg SOURCE=https://git.example.org/your-org/ekklesia \
--tag ekklesia:0.1.1 \ --tag ekklesia:0.1.2 \
. .
``` ```
@@ -0,0 +1,256 @@
# 0047: Every module proves itself inside Ekklesia, with one harness the platform publishes
- Status: Proposed
- Date: 2026-10-05
- Supersedes: none
- Related: [ADR 0016](0016-module-database-access.md),
[ADR 0021](0021-modules-as-hostable-products.md),
[ADR 0028](0028-module-operational-lifecycle.md),
[ADR 0030](0030-host-provided-module-object-storage.md),
[ADR 0033](0033-display-kiosk-host-contract.md),
[ADR 0036](0036-managed-device-identity-and-pairing.md),
[ADR 0041](0041-module-browser-contributions.md),
[ADR 0043](0043-display-offline-code-fallback.md),
[ADR 0044](0044-self-hosted-reference-deployment.md),
[ADR 0045](0045-split-dashboard-and-api-origins.md)
## Context
Modules are independently versioned products (ADR 0021). They live in their own repositories, and
none of them can be built, run, or rendered without Ekklesia. The first production module, Signage,
reached a church with three defects that only show when it runs **inside** Ekklesia, in a real
browser:
- every heartbeat its display sent was refused with a 400, because the display and the server
disagreed about `null`;
- a display that failed one request at start-up never recovered without a reload;
- its media cache leaked one blob per slide per refresh, until the browser refused the display's own
requests.
Signage had unit tests, browser tests against a stubbed host, and no CI. Every test passed, because
each half was tested against its own idea of the other, and nothing ever ran the module in the
platform it ships into.
The platform's own deployment smoke (ADR 0044, ADR 0045) does run a module in a real browser, but
only the repository's template, and only far enough to mount it. The instance repository runs the
smoke against its real modules, but only at release time, after a module has already been tagged.
Each module repository that wants more than that has to assemble the platform, build an image,
start a deployment, and drive a browser itself. Signage is about to do exactly that by hand. The
second module, Classroom, would copy it, and the copies would drift from the platform they test.
## Decision
### The platform publishes a versioned module harness
Each platform release publishes one OCI image, `ekklesia-module-harness:<platform version>`. It is
built from the platform at that release tag by the platform's own release CI, and pushed to the
organization's registry. A version is never overwritten. Modules reference it **by digest**.
The harness contains the platform source at that version: its `deploy/` directory, its Dockerfile,
its lockfile, and its smoke scripts. It also contains the pinned Node.js and pnpm, the browser
driver, and the pinned headless Chromium image reference. Given a module checkout, it:
1. **assembles** the module into the platform at `modules/<id>`, exactly as an instance does (ADR
0044), and installs with `--frozen-lockfile`;
2. runs the platform's own gates against that set: `pnpm boundaries`, `modules:validate`,
`migrations:validate`, and the build;
3. **builds** the deployment image from the assembled workspace with the platform's Dockerfile;
4. starts the **reference deployment** in the split topology (ADR 0045), with a disposable
PostgreSQL and object store, Caddy's internal CA, and freshly generated secrets;
5. plans, then **explicitly applies**, migrations against that fresh database. This is the only
database the harness ever migrates;
6. creates the initial owner, grants the module's declared permissions to a test person, and
**mounts every UI surface the manifest declares**, in headless Chromium. That covers:
- the primary UI in the Dashboard;
- the display, paired as a device with no human session (ADR 0036);
- each browser contribution, whenever the module that hosts it is present (ADR 0041);
7. runs a short **display soak**, and the **no-dead-ends** injections below;
8. tears everything down: containers, networks, volumes, and the work directory.
It **fails** on any of the following:
- a `console.error` or uncaught exception on any page;
- an unhandled promise rejection;
- any 4xx or 5xx response that the harness did not itself expect. Expected ones are listed in the
harness, for example a display's descriptor before it is paired;
- a page that never mounts;
- a failed or refused request (CORS, blocked, network error);
- a soak leak.
Each failure line names the surface, the request path (never its query), and the step.
### The display soak and the no-dead-ends checks are generic
These checks come from what Signage got wrong, and they apply to any module with a display.
- **Soak**, two minutes by default. The display stays mounted with no remount and no offline
indicator. Its realtime streams stay healthy. Live object URLs stay bounded. JavaScript heap
growth, measured over DevTools, stays within a fixed ratio.
- **No dead ends.** One at a time, the harness refuses the first request to each module route the
display calls at start-up, and the first realtime connection. Within a bounded time and with no
reload, the display must make that request again and succeed. A display has nobody to press
Retry.
Module-specific behaviour, such as Signage's heartbeat being recorded, is not guessed by the
harness. A module declares it (below).
### A module is tested against both ends of its compatibility range
A module's CI runs the harness twice: against the **lowest** and the **newest** published platform
version within its manifest's `compatibility.platform` range. Testing every version in between adds
cost and finds little. Testing only the newest misses the floor the manifest promises to work on.
The two versions, and their harness digests, are pinned in the module's repository. The harness
refuses to run if:
- a pinned version lies outside the module's range;
- the lowest pinned version is not the range's floor.
Moving to a newer platform is a deliberate, reviewed pin change, as an instance's is (ADR 0044). It
is never a moving tag.
A frozen install of the module assembled into each of those two platforms needs a lockfile per
pinned version. The module keeps them (`ci/lock/<platform version>.yaml`), and the harness provides
the one command that regenerates them inside its own pinned Node.js. A stale lockfile fails the
frozen install, which is the signal to regenerate it in a reviewed change.
### Modules declare extra smoke steps, beside their manifest rather than in it
A module may add steps of its own in `ekklesia.harness.json`, next to `ekklesia.module.json`. Steps
are declarative first:
- open a surface;
- wait for a test id;
- call one of the module's routes as a named role, expecting a status;
- soak for N minutes;
- inject a failure into a named route;
- check a value in the module's own table, through its own route.
For what a declaration cannot express, a step may name a driver script in the module's repository.
The harness runs that script with the same DevTools client and the same failure rules.
**This is deliberately not a manifest field.** Manifest versions are strict, hard cutovers that every
host validates (ADR 0033). A test step is not a runtime contract, and a host must never read one.
Putting steps in the manifest would force a manifest version bump on every module for something
that has no meaning outside CI.
### Later: the harness checks what a module actually touches against what it declares
In a later phase, the harness records what the module does while it runs:
- the tables and schemas it reads and writes;
- the buckets it uses;
- its outbound network connections, with the egress network blocked and every attempt logged.
It compares them with what the manifest declares (ADR 0016, ADR 0030). A module that touches what it
did not declare fails. This is the module-trust check. The egress half needs a manifest declaration
of outbound destinations that does not exist yet, and is a separate decision. This ADR fixes only
the harness's shape, so that the recording phases can be added without changing how modules call
it.
### The template's workflow stays short, and calls the harness by digest
`templates/module` ships a workflow that:
- checks out the module with `persist-credentials: false` and pinned actions;
- runs the module's own unit tests, typecheck, and lint;
- sweeps stale CI stacks from the runner, then runs the harness, once per pinned platform version,
with one `docker run` of the harness image **by digest**, and finally, always, removes this job's
own stacks. The module directory is mounted read-only, and the runner's Docker is used exactly
as the platform's smoke uses it.
It references no secret, needs no registry credential (the harness image is public within the
organization), and uses no `pull_request_target`. The trusted-code-only rule for self-hosted runners
applies to module repositories exactly as it does to the platform (see `docs/development/security.md`).
**Why `docker run`, and not a reusable workflow or a composite action.** The server runs Gitea
1.27.2. (This proposal first assumed 1.24.2; the reasons below hold on the actual version.)
- Its runner can call a reusable workflow from another repository by cloning that repository, and
since 1.26 that includes private repositories (go-gitea/gitea#32562). That support is still thinly
documented, and has known problems in practice.
- A reusable workflow would make the harness's behaviour depend on how this server and runner
version expand it. An image runs the same everywhere it runs.
- Composite actions from another repository work through absolute `uses:` URLs. That would still
wrap the same `docker run`, adding a second pinned reference and nothing else.
An image digest is the one reference that is equally immutable, equally reviewable, and works
unchanged on any Gitea release and on GitHub Actions. A composite action can be added later as a
convenience over the same image, without changing what a module pins.
### Nothing the harness starts outlives the job that started it
The harness starts Docker stacks on a shared runner, and a runner can kill a job outright: a
cancelled or superseded job (`concurrency` cancels superseded runs on this server), or the runner's
own time limit. A script's own cleanup trap does not run then. The platform's CI leaked a whole
migration stack that way, and it ran until someone removed it by hand. So the harness follows the
platform's rule (`scripts/ci-sweep.sh`, `docs/development/security.md`):
- everything it starts is named and labeled as a CI stack of its run, attempt, and job key;
- it removes its own stacks in a trap;
- the template workflow runs the harness's **stale sweep** before the harness starts, and its
**job sweep** in a final `if: always()` step, so a killed run's stacks are removed by that run's
last step, or failing that by the next job on the runner.
The sweep ships inside the harness image, so a module never copies it.
### Publishing follows the instance's release pattern
The harness image is pushed only by a `publish` job that runs on `v*` tags, after the platform's CI
has built and tested that exact image (checked by image ID). That job:
- uses a write-only package token, the only secret in the platform's workflow;
- refuses to overwrite an existing version;
- prints the digest.
Pull requests and branch builds build the harness and run it against the template, but never push.
The registry is reached exactly as the instance's publish job reaches it.
### Adoption
- **Signage** uses an interim CI job until the first harness is published. That job assembles
Signage into a pinned platform commit and runs the platform's display-device smoke scenarios.
Signage switches to the harness as soon as one is published.
- **Classroom** follows.
- **The template** carries the workflow from the first harness release on, so every new module has
it from its first commit.
## Consequences
- A module cannot reach a release without having run inside Ekklesia, in a real browser, at both
ends of the platform range it claims.
- The defects that reached the first production Signage deployment fail a module's CI instead:
refused requests, dead ends, leaks, and console errors.
- The platform owns one harness, versioned with itself. Modules do not copy deployment scripts that
then drift.
- Every module's CI needs Docker on its runner, and takes minutes rather than seconds. The harness
runs as separate jobs from the unit tests, so a quick failure still reports quickly.
- The platform's release process gains one published artifact, one credential, and one more failure
mode at release time.
- A module now carries a lockfile for each pinned platform version. The harness makes regenerating
them one command, and a stale one fails visibly.
## Alternatives
- **Each module keeps its own copy of the smoke scripts.** Rejected. That is today's situation in
embryo, and it drifts.
- **A reusable workflow in the platform repository.** Rejected for now, for the reasons above: thin
support in practice, and a harness whose behaviour depends on whichever runner and server version
happens to expand it.
- **Publish the harness as an npm package.** Rejected. The harness needs Docker, a browser, and the
platform's deployment files anyway, so a package would ship a fraction of it and assume the rest.
- **Test only the newest platform version.** Rejected. A module's range floor would go untested
until an installation that has not upgraded meets it.
- **Put smoke steps in the manifest.** Rejected. Manifest versions are hard cutovers with runtime
meaning, and test steps have neither.
- **Run the harness in the platform's CI against every known module.** Rejected. The platform would
need every module's source and would fail on modules it does not own. The direction is module
tests platform, not the reverse.
## Deferred
- The data-access and egress recording phases, and the manifest declaration egress needs.
- A composite action over the harness image.
- Running the harness against a module's declared dependencies at their own pinned versions, for
contributions hosted by another module.
- Visual regression of mounted surfaces.
+1
View File
@@ -55,3 +55,4 @@ ADRs capture durable choices that affect more than one package or phase.
- [0044: Ekklesia ships one self-hosted reference deployment, composed from pinned sources](0044-self-hosted-reference-deployment.md) — Accepted - [0044: Ekklesia ships one self-hosted reference deployment, composed from pinned sources](0044-self-hosted-reference-deployment.md) — Accepted
- [0045: The Dashboard and the API may live on separate origins](0045-split-dashboard-and-api-origins.md) — Accepted - [0045: The Dashboard and the API may live on separate origins](0045-split-dashboard-and-api-origins.md) — Accepted
- [0046: Ekklesia's branding is never forced on an installation](0046-installation-owned-branding.md) — Proposed - [0046: Ekklesia's branding is never forced on an installation](0046-installation-owned-branding.md) — Proposed
- [0047: Every module proves itself inside Ekklesia, with one harness the platform publishes](0047-module-ci-harness.md) — Proposed
+54 -22
View File
@@ -693,8 +693,10 @@ public Dashboard login must be assessed with the actual edge/proxy controls befo
## Continuous integration boundary ## Continuous integration boundary
CI runs on self-hosted Gitea Actions runners. The inventoried server is **Gitea 1.24.2**, and that CI runs on self-hosted Gitea Actions runners. The server reports **Gitea 1.27.2** (`GET
version determines which parts of the workflow are real guarantees and which are not. /api/v1/version`, checked 2026-10-06; this document previously said 1.24.2, which was stale). The
server's actual version, not the documentation of an earlier one, determines which parts of the
workflow are real guarantees and which are not.
### The current operating rule: trusted code only ### The current operating rule: trusted code only
@@ -727,24 +729,53 @@ so a later step in the same job cannot read or reuse it from there.
It does **not** remove the short-lived `GITEA_TOKEN` from the job environment, and it does **not** It does **not** remove the short-lived `GITEA_TOKEN` from the job environment, and it does **not**
reduce that token's server-side authority. A step that wants the token can still read it. reduce that token's server-side authority. A step that wants the token can still read it.
### What Gitea 1.24 ignores — so the branch cannot claim it ### What this server honours, and what is not yet verified
Per the official comparison for this version Each item separates what Gitea documents for 1.27
(<https://docs.gitea.com/1.24/usage/actions/comparison>), Gitea 1.24's unsupported workflow syntax (<https://docs.gitea.com/1.27/usage/actions/comparison>) from what has been observed on this server:
includes `permissions` and `jobs.<job_id>.permissions`, `concurrency`, and
`jobs.<job_id>.timeout-minutes`. On this server those declarations are inert. Therefore:
- **The workflow's `permissions: contents: read` does not restrict the job token.** This branch - **`concurrency` is honoured.** Gitea supports it since 1.26 (go-gitea/gitea#32751), and on this
cannot claim the `GITEA_TOKEN` is read-only, because nothing in workflow YAML makes it so on server a newer push to a pull request cancels the run it supersedes. Observed: Ekklesia run 133 was
1.24. Job-token permission controls, repository/owner defaults, and server-side cancelled when run 134 started for the same pull request, and run 135 when run 137 did. A
maximum-permission clamping arrive in the 1.26 feature line superseded run's own cleanup is therefore never guaranteed to run, which is one reason for the
(<https://docs.gitea.com/1.26/usage/actions/token-permissions>). stack sweep below.
- **`concurrency` and `timeout-minutes` are not bounded-execution guarantees here.** Cancellation, - **`permissions` is supported by 1.27, with Gitea's own scopes, but is not verified here.** The
timeout, and teardown must be enforced by a trusted provisioner and reaper outside the workflow. server-side token-permission settings are not visible through the API, and nothing has yet proven
what the job token can and cannot do on this server. Until that proof exists (requirement 3
below), this repository still does not claim the `GITEA_TOKEN` is read-only.
- **`timeout-minutes` is no longer listed as unsupported for 1.27, but is not verified here.**
Teardown does not rely on it: the stack sweep and a trusted reaper do.
- **Still ignored:** `jobs.<job_id>.environment`, and any `runs-on` other than a single label.
The declarations are kept deliberately: they are already correct for a supported Gitea release and The declarations stay in the workflow either way: they state the intent, and are correct on GitHub
for GitHub Actions, and keeping them means the upgrade does not require re-deriving the intent. They Actions as well.
are documented as ignored, not removed.
### Disposable stacks never outlive their job
Every CI script removes its own Compose stack in an `EXIT` trap, but a trap never runs when the runner
kills the job container: a cancelled job, or the runner's own time limit. One such stack ran on the
shared runner for two days. So stack cleanup has three layers:
- **The script's own trap**, for every ordinary failure.
- **A final `if: always()` step** in every job that starts stacks: `scripts/ci-sweep.sh --job` removes
everything that job started, at any age, even after a failed step.
- **A sweep before every such job starts**: `scripts/ci-sweep.sh --stale` removes anything a CI job
left more than three hours ago, which catches whatever a killed job could not clean up itself.
A stack is found by name and label, never by guesswork (`scripts/lib/ci-project.sh`):
- every CI Compose project is named `ekklesia-ci-<run>-<attempt>-<job key>-…`, and Compose labels
its containers, networks, and volumes with that name;
- anything a script starts outside Compose carries `org.riverofglorymn.ekklesia.ci.project`;
- images also carry the time CI built them, because an image's own creation time can be its base's.
The job key is per matrix leg, so one leg's cleanup cannot match another's running stack. Nothing
outside the `ekklesia-ci-` prefix is ever removed. A job container attached to a stack's network is
disconnected, never removed.
**This is not a substitute for a runner-side reaper.** A stack killed with its job is removed only
when another job that sweeps runs on the same runner. Bounded execution and teardown are still the
runner host's to enforce, as above.
### Runner topology is not verified ### Runner topology is not verified
@@ -766,16 +797,17 @@ they are not conditions on merging trusted, reviewed work.
None of them can be satisfied by editing this repository; each requires access to the Gitea server None of them can be satisfied by editing this repository; each requires access to the Gitea server
and runner hosts, and each must be proven there. and runner hosts, and each must be proven there.
1. Upgrade Gitea to a reviewed supported release with job-token permission controls — at least the 1. ~~Upgrade Gitea to a reviewed supported release with job-token permission controls — at least
1.26 feature line. Select the actual release during infrastructure work, after backup and the 1.26 feature line.~~ **Done:** the server runs 1.27.2. Items 2 and 3 now configure and prove
compatibility review; do not pin a version from this document. what it makes possible.
2. Configure repository and owner defaults, and the server-side maximum permission, so workflow 2. Configure repository and owner defaults, and the server-side maximum permission, so workflow
code cannot request more than read-only repository access. code cannot request more than read-only repository access.
3. Prove against the real server that the job's `GITEA_TOKEN` cannot push code, create or update 3. Prove against the real server that the job's `GITEA_TOKEN` cannot push code, create or update
releases, packages, issues, or pull requests, change Actions configuration, or reach another releases, packages, issues, or pull requests, change Actions configuration, or reach another
private repository. private repository.
4. Enforce job timeout, cancellation, concurrency, and teardown in the trusted 4. Enforce job timeout and teardown in the trusted provisioner/reaper. `concurrency` is honoured
provisioner/reaper rather than relying on the ignored workflow keys above. by the server; `timeout-minutes` is not yet verified, and the in-workflow stack sweep only
complements a reaper.
5. Complete a disposable-guest runner isolation proof: one throwaway guest per job, torn down by a 5. Complete a disposable-guest runner isolation proof: one throwaway guest per job, torn down by a
trusted reaper outside workflow YAML, holding no deployment credential, registration token, or trusted reaper outside workflow YAML, holding no deployment credential, registration token, or
reusable key. reusable key.
+6
View File
@@ -159,6 +159,12 @@ also receives a safely named disposable database. Direct host runs use a Docker-
port bound only to `127.0.0.1`; a containerized Gitea job joins the isolated Compose network and port bound only to `127.0.0.1`; a containerized Gitea job joins the isolated Compose network and
uses the PostgreSQL service name. uses the PostgreSQL service name.
In CI, a stack whose job was killed before its own cleanup ran is still removed: by that job's final
`if: always()` step, or by the stale sweep before a later job starts (`scripts/ci-sweep.sh`, see
[Disposable stacks never outlive their job](security.md#disposable-stacks-never-outlive-their-job)).
`bash scripts/test-ci-sweep.sh` proves what the sweep removes and what it leaves alone, against real
Docker resources.
The live suite proves fresh read-only planning, initial and incremental application, genuine The live suite proves fresh read-only planning, initial and incremental application, genuine
no-op behavior, exact applied checksums and history, edited or corrupt history rejection, no-op behavior, exact applied checksums and history, edited or corrupt history rejection,
whole-batch DDL rollback, Drizzle breakpoint execution, real advisory-lock concurrency and whole-batch DDL rollback, Drizzle breakpoint execution, real advisory-lock concurrency and
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "ekklesia", "name": "ekklesia",
"version": "0.1.1", "version": "0.1.2",
"private": true, "private": true,
"type": "module", "type": "module",
"description": "Open, modular church technology platform", "description": "Open, modular church technology platform",
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@ekklesia/contracts", "name": "@ekklesia/contracts",
"version": "0.1.1", "version": "0.1.2",
"type": "module", "type": "module",
"files": [ "files": [
"dist" "dist"
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@ekklesia/design-tokens", "name": "@ekklesia/design-tokens",
"version": "0.1.1", "version": "0.1.2",
"type": "module", "type": "module",
"files": [ "files": [
"dist" "dist"
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@ekklesia/module-sdk", "name": "@ekklesia/module-sdk",
"version": "0.1.1", "version": "0.1.2",
"type": "module", "type": "module",
"files": [ "files": [
"dist" "dist"
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@ekklesia/module-ui-build", "name": "@ekklesia/module-ui-build",
"version": "0.1.1", "version": "0.1.2",
"type": "module", "type": "module",
"files": [ "files": [
"bin", "bin",
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@ekklesia/module-ui-host", "name": "@ekklesia/module-ui-host",
"version": "0.1.1", "version": "0.1.2",
"type": "module", "type": "module",
"files": [ "files": [
"dist" "dist"
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@ekklesia/module-ui-sdk", "name": "@ekklesia/module-ui-sdk",
"version": "0.1.1", "version": "0.1.2",
"type": "module", "type": "module",
"files": [ "files": [
"dist" "dist"
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@ekklesia/typescript-config", "name": "@ekklesia/typescript-config",
"version": "0.1.1", "version": "0.1.2",
"private": true, "private": true,
"files": [ "files": [
"base.json", "base.json",
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@ekklesia/ui", "name": "@ekklesia/ui",
"version": "0.1.1", "version": "0.1.2",
"type": "module", "type": "module",
"files": [ "files": [
"dist" "dist"
+206
View File
@@ -0,0 +1,206 @@
#!/usr/bin/env bash
# Removes disposable CI stacks from the Docker daemon: containers, networks, volumes, images, and
# staging directories on the Docker host.
#
# bash scripts/ci-sweep.sh --stale [--max-age-seconds N] any CI stack older than N (default 3 h)
# bash scripts/ci-sweep.sh --job this job's own stacks, at any age
#
# Every CI script already removes its own stack in an EXIT trap, but a trap never runs when the runner
# kills the job container: a cancelled or superseded job (the server honours `concurrency`), or
# the runner's own time limit. So each workflow job that starts stacks ends with `--job` in an
# `if: always()` step, and begins with `--stale`, which catches whatever a killed job left before
# it could clean up.
#
# What a stack is, and how it is found, is defined in scripts/lib/ci-project.sh: a Compose project
# named with the `ekklesia-ci-` prefix (Compose labels everything it creates with that name), plus
# anything labeled EKKLESIA_CI_LABEL. Nothing outside that prefix is ever touched, at any age.
#
# --stale's threshold is far above any job's run time, so a stack another job on this daemon is using
# right now is never young enough to match. --job refuses to run outside CI: locally, its prefix would
# match a developer's own stacks.
#
# Best effort by design: a resource that cannot be removed is reported, and the sweep goes on. Output
# names resources only; nothing a stack holds is printed.
set -Eeuo pipefail
repository_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
# shellcheck source=scripts/lib/ci-project.sh
source "$repository_root/scripts/lib/ci-project.sh"
compose_label="com.docker.compose.project"
mode=""
max_age_seconds="${EKKLESIA_CI_SWEEP_MAX_AGE_SECONDS:-10800}"
usage() {
echo "usage: ci-sweep.sh --stale [--max-age-seconds N] | --job" >&2
exit 2
}
while (($# > 0)); do
case "$1" in
--stale | --job)
[[ -z "$mode" ]] || usage
mode="${1#--}"
;;
--max-age-seconds)
[[ "${2:-}" =~ ^[0-9]+$ ]] || usage
max_age_seconds="$2"
shift
;;
*) usage ;;
esac
shift
done
[[ -n "$mode" ]] || usage
if [[ "$mode" == job ]]; then
[[ -n "${GITHUB_RUN_ID:-}" ]] || {
echo "ci-sweep: --job runs only in CI (GITHUB_RUN_ID is unset); locally it would match your own stacks." >&2
exit 2
}
prefix="$(ci_job_prefix)"
cutoff=""
else
prefix="$EKKLESIA_CI_PREFIX"
cutoff="$(($(date +%s) - max_age_seconds))"
fi
if ! docker info >/dev/null 2>&1; then
echo "ci-sweep: no Docker daemon reachable; nothing to sweep."
exit 0
fi
removed=0
warn() {
echo "ci-sweep: warning: $*" >&2
}
# selected <project> <created>: whether a resource belongs to the sweep.
selected() {
local project="$1" created="$2" seconds
[[ -n "$project" && "$project" == "$prefix"* ]] || return 1
[[ -z "$cutoff" ]] && return 0
seconds="$(date -d "$created" +%s 2>/dev/null)" || return 1
((seconds <= cutoff))
}
# --- Containers: Compose's own, and those a script started itself --------------------------------
mapfile -t containers < <(
{
docker ps --all --quiet --no-trunc --filter "label=${compose_label}"
docker ps --all --quiet --no-trunc --filter "label=${EKKLESIA_CI_LABEL}"
} | sort -u
)
for id in "${containers[@]}"; do
[[ -n "$id" ]] || continue
record="$(docker inspect --format \
"{{index .Config.Labels \"${compose_label}\"}}|{{index .Config.Labels \"${EKKLESIA_CI_LABEL}\"}}|{{.Created}}|{{.Name}}" \
"$id" 2>/dev/null)" || continue
IFS='|' read -r compose_project ci_project created name <<<"$record"
project="${compose_project:-$ci_project}"
selected "$project" "$created" || continue
if docker rm --force --volumes "$id" >/dev/null 2>&1; then
echo "ci-sweep: removed container ${name#/} (${project})"
removed=$((removed + 1))
else
warn "could not remove container ${name#/}"
fi
done
# --- Networks: anything still attached (a job container) is disconnected, never removed ---------
mapfile -t networks < <(docker network ls --quiet --no-trunc --filter "label=${compose_label}")
for id in "${networks[@]}"; do
[[ -n "$id" ]] || continue
# A network's creation time is a time value, not a string: ask for RFC 3339, which `date` reads.
record="$(docker network inspect --format \
"{{index .Labels \"${compose_label}\"}}|{{.Created.Format \"2006-01-02T15:04:05Z07:00\"}}|{{.Name}}" \
"$id" 2>/dev/null)" || continue
IFS='|' read -r project created name <<<"$record"
selected "$project" "$created" || continue
for endpoint in $(docker network inspect --format '{{range $id, $_ := .Containers}}{{$id}} {{end}}' "$id" 2>/dev/null); do
docker network disconnect --force "$id" "$endpoint" >/dev/null 2>&1 ||
warn "could not disconnect a container from network ${name}"
done
if docker network rm "$id" >/dev/null 2>&1; then
echo "ci-sweep: removed network ${name} (${project})"
removed=$((removed + 1))
else
warn "could not remove network ${name}"
fi
done
# --- Volumes ---------------------------------------------------------------------------------------
mapfile -t volumes < <(
{
docker volume ls --quiet --filter "label=${compose_label}"
docker volume ls --quiet --filter "label=${EKKLESIA_CI_LABEL}"
} | sort -u
)
for name in "${volumes[@]}"; do
[[ -n "$name" ]] || continue
record="$(docker volume inspect --format \
"{{index .Labels \"${compose_label}\"}}|{{index .Labels \"${EKKLESIA_CI_LABEL}\"}}|{{.CreatedAt}}" \
"$name" 2>/dev/null)" || continue
IFS='|' read -r compose_project ci_project created <<<"$record"
project="${compose_project:-$ci_project}"
selected "$project" "$created" || continue
if docker volume rm --force "$name" >/dev/null 2>&1; then
echo "ci-sweep: removed volume ${name} (${project})"
removed=$((removed + 1))
else
warn "could not remove volume ${name}"
fi
done
# --- Images built for a stack ------------------------------------------------------------------------
mapfile -t images < <(docker image ls --quiet --no-trunc --filter "label=${EKKLESIA_CI_LABEL}" | sort -u)
for id in "${images[@]}"; do
[[ -n "$id" ]] || continue
# Judged by when CI built it (EKKLESIA_CI_CREATED_LABEL), never by the image's own creation time,
# which BuildKit may have taken from its base. An image without that label is never swept by age.
record="$(docker image inspect --format \
"{{index .Config.Labels \"${EKKLESIA_CI_LABEL}\"}}|{{index .Config.Labels \"${EKKLESIA_CI_CREATED_LABEL}\"}}" \
"$id" 2>/dev/null)" || continue
IFS='|' read -r project built <<<"$record"
if [[ -n "$cutoff" ]]; then
[[ "$built" =~ ^[0-9]+$ ]] || continue
selected "$project" "@${built}" || continue
else
selected "$project" "" || continue
fi
if docker image rm --force "$id" >/dev/null 2>&1; then
echo "ci-sweep: removed image ${id#sha256:} (${project})"
removed=$((removed + 1))
else
warn "could not remove image ${id#sha256:}"
fi
done
# --- Staging directories on the Docker host ------------------------------------------------------
# A deploy smoke run inside a job container stages its mounted files under the Docker host's /tmp,
# named for its project, as root-owned files; they hold that run's disposable secrets and keys. They
# are removed with the same pinned image that created them. If that image is not on this daemon, no
# smoke has staged anything here.
image="$(sed -n 's/^[[:space:]]*image: \(postgres:[^[:space:]]*\)$/\1/p' "$repository_root/deploy/compose.yaml" | head -1)"
if [[ -n "$image" ]] && docker image inspect "$image" >/dev/null 2>&1; then
age=()
# `find -mmin +0` does not match something younger than a minute, so a threshold under a minute
# sweeps without an age filter, like the rest of this script.
if [[ -n "$cutoff" ]] && ((max_age_seconds >= 60)); then
age=(-mmin "+$((max_age_seconds / 60))")
fi
if staged="$(docker run --rm --entrypoint find -v /tmp:/host-tmp "$image" \
/host-tmp -mindepth 1 -maxdepth 1 -type d -name "${prefix}*" "${age[@]}" \
-print -exec rm -rf {} + 2>/dev/null)"; then
while IFS= read -r directory; do
[[ -n "$directory" ]] || continue
echo "ci-sweep: removed staging directory /tmp/${directory#/host-tmp/}"
removed=$((removed + 1))
done <<<"$staged"
else
warn "could not sweep staging directories on the Docker host"
fi
fi
echo "ci-sweep: ${mode} sweep removed ${removed} resource(s)."
+40
View File
@@ -0,0 +1,40 @@
# shellcheck shell=bash
# How CI names, labels, and later finds the disposable stacks it starts.
#
# Every Compose project a CI script starts is named
#
# ekklesia-ci-<run id>-<run attempt>-<job key>-<kind>-<pid><random>
#
# and Compose labels every container, network, and volume it creates with that name. What a script
# creates outside Compose (a browser container, a volume, an image) gets EKKLESIA_CI_LABEL with the
# same name. scripts/ci-sweep.sh finds stacks by those labels: this job's own, by the prefix up to
# the job key (in a final `if: always()` step), and any CI stack by age (before a job starts).
#
# The job key is EKKLESIA_CI_JOB_KEY, which the workflow sets per job, and per matrix leg where a job
# has one, so that one leg's cleanup cannot match another leg's running stack. Outside CI the
# identifiers are "local".
# shellcheck disable=SC2034 # read by the scripts that source this
EKKLESIA_CI_LABEL="org.riverofglorymn.ekklesia.ci.project"
# On images, when they were built, in Unix seconds. An image's own creation time is not that: BuildKit
# gives an image that only adds metadata its base's time, and a reproducible build a fixed one. A
# stale sweep that trusted it could remove an image another job is using right now.
EKKLESIA_CI_CREATED_LABEL="org.riverofglorymn.ekklesia.ci.created"
EKKLESIA_CI_PREFIX="ekklesia-ci-"
# ci_name_safe <text>: lower case, and only what a Compose project name may contain.
ci_name_safe() {
local text="${1//[^a-zA-Z0-9_-]/-}"
printf '%s' "${text,,}"
}
# ci_job_prefix: every project this job starts begins with exactly this.
ci_job_prefix() {
ci_name_safe "ekklesia-ci-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-0}-${EKKLESIA_CI_JOB_KEY:-${GITHUB_JOB:-local}}-"
}
# ci_project_name <kind>: a project name unique to this invocation. Not only per run: several runs,
# and several jobs of one run, share a runner's Docker daemon, and job containers often share a PID.
ci_project_name() {
ci_name_safe "ekklesia-ci-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-0}-${EKKLESIA_CI_JOB_KEY:-${GITHUB_JOB:-local}}-${1:?ci_project_name needs a kind}-$$-${RANDOM}${RANDOM}"
}
+15 -5
View File
@@ -39,6 +39,8 @@ cd "$repository_root"
# shellcheck source=scripts/lib/compose-diagnostics.sh # shellcheck source=scripts/lib/compose-diagnostics.sh
source "$repository_root/scripts/lib/compose-diagnostics.sh" source "$repository_root/scripts/lib/compose-diagnostics.sh"
# shellcheck source=scripts/lib/ci-project.sh
source "$repository_root/scripts/lib/ci-project.sh"
for command in docker openssl awk; do for command in docker openssl awk; do
command -v "$command" >/dev/null || { command -v "$command" >/dev/null || {
@@ -67,7 +69,10 @@ same-origin | split) ;;
esac esac
run_id="$(openssl rand -hex 4)" run_id="$(openssl rand -hex 4)"
project="ekklesia-deploy-smoke-${run_id}" # Unique per invocation, and findable by run and job if this script is killed before its trap runs
# (scripts/lib/ci-project.sh, scripts/ci-sweep.sh). Everything started outside Compose below carries
# the same name in EKKLESIA_CI_LABEL.
project="$(ci_project_name "deploy-${topology}")"
work="$(mktemp -d)" work="$(mktemp -d)"
image="${EKKLESIA_SMOKE_IMAGE:-}" image="${EKKLESIA_SMOKE_IMAGE:-}"
built_image="" built_image=""
@@ -186,7 +191,9 @@ fail() {
if [[ -z "$image" && "$topology" == same-origin ]]; then if [[ -z "$image" && "$topology" == same-origin ]]; then
step "Building the platform-only image" step "Building the platform-only image"
built_image="ekklesia-deploy-smoke:${run_id}" built_image="ekklesia-deploy-smoke:${run_id}"
DOCKER_BUILDKIT=1 docker build --quiet --file deploy/Dockerfile --tag "$built_image" . >/dev/null DOCKER_BUILDKIT=1 docker build --quiet --file deploy/Dockerfile --tag "$built_image" \
--label "${EKKLESIA_CI_LABEL}=${project}" --label "${EKKLESIA_CI_CREATED_LABEL}=$(date +%s)" \
. >/dev/null
image="$built_image" image="$built_image"
elif [[ -z "$image" ]]; then elif [[ -z "$image" ]]; then
# TEST-ONLY: the repository's module template, installed as the module `example`, in a copy of the # TEST-ONLY: the repository's module template, installed as the module `example`, in a copy of the
@@ -206,6 +213,7 @@ elif [[ -z "$image" ]]; then
sed -i 's|^ templates/module:$| modules/example:|' "$context/pnpm-lock.yaml" sed -i 's|^ templates/module:$| modules/example:|' "$context/pnpm-lock.yaml"
built_image="ekklesia-deploy-smoke:${run_id}" built_image="ekklesia-deploy-smoke:${run_id}"
DOCKER_BUILDKIT=1 docker build --quiet --file "$context/deploy/Dockerfile" --tag "$built_image" \ DOCKER_BUILDKIT=1 docker build --quiet --file "$context/deploy/Dockerfile" --tag "$built_image" \
--label "${EKKLESIA_CI_LABEL}=${project}" --label "${EKKLESIA_CI_CREATED_LABEL}=$(date +%s)" \
"$context" >/dev/null || fail "the test-module image did not build." "$context" >/dev/null || fail "the test-module image did not build."
rm -rf "$context" rm -rf "$context"
image="$built_image" image="$built_image"
@@ -362,6 +370,7 @@ client() {
local mode="$1" local mode="$1"
shift shift
docker run --rm --read-only --cap-drop ALL --security-opt no-new-privileges \ docker run --rm --read-only --cap-drop ALL --security-opt no-new-privileges \
--label "${EKKLESIA_CI_LABEL}=${project}" \
--network "$edge_network" "$@" "${host_mappings[@]}" \ --network "$edge_network" "$@" "${host_mappings[@]}" \
-e NODE_EXTRA_CA_CERTS=/smoke/client-ca.pem -e SMOKE_HOSTNAME="$hostname" \ -e NODE_EXTRA_CA_CERTS=/smoke/client-ca.pem -e SMOKE_HOSTNAME="$hostname" \
-e SMOKE_DASHBOARD_HOSTNAME="$dashboard_hostname" \ -e SMOKE_DASHBOARD_HOSTNAME="$dashboard_hostname" \
@@ -401,7 +410,7 @@ if [[ "$topology" == split ]]; then
browser_container="${project}-browser" browser_container="${project}-browser"
# The image's entry script passes its arguments on unquoted, so none may contain a space: the # The image's entry script passes its arguments on unquoted, so none may contain a space: the
# hostnames reach Chromium through /etc/hosts, which it reads, rather than a resolver rule. # hostnames reach Chromium through /etc/hosts, which it reads, rather than a resolver rule.
docker run --detach --rm --name "$browser_container" \ docker run --detach --rm --name "$browser_container" --label "${EKKLESIA_CI_LABEL}=${project}" \
--cap-drop ALL --security-opt no-new-privileges --tmpfs /tmp \ --cap-drop ALL --security-opt no-new-privileges --tmpfs /tmp \
--network "$edge_network" --ip "$browser_address" "${host_mappings[@]}" \ --network "$edge_network" --ip "$browser_address" "${host_mappings[@]}" \
"$browser_image" \ "$browser_image" \
@@ -410,6 +419,7 @@ if [[ "$topology" == split ]]; then
cp scripts/lib/deploy-smoke-browser.mjs "$work/browser.mjs" cp scripts/lib/deploy-smoke-browser.mjs "$work/browser.mjs"
stage browser.mjs stage browser.mjs
docker run --rm --read-only --cap-drop ALL --security-opt no-new-privileges \ docker run --rm --read-only --cap-drop ALL --security-opt no-new-privileges \
--label "${EKKLESIA_CI_LABEL}=${project}" \
--network "$edge_network" \ --network "$edge_network" \
-e SMOKE_CDP_ADDRESS="${browser_address}:9222" \ -e SMOKE_CDP_ADDRESS="${browser_address}:9222" \
-e SMOKE_HOSTNAME="$hostname" -e SMOKE_DASHBOARD_HOSTNAME="$dashboard_hostname" \ -e SMOKE_HOSTNAME="$hostname" -e SMOKE_DASHBOARD_HOSTNAME="$dashboard_hostname" \
@@ -423,9 +433,9 @@ if [[ "$topology" == split ]]; then
kiosk_container="${project}-kiosk" kiosk_container="${project}-kiosk"
kiosk_volume="${project}-kiosk-profile" kiosk_volume="${project}-kiosk-profile"
kiosk_address="10.249.${octet}.250" kiosk_address="10.249.${octet}.250"
docker volume create "$kiosk_volume" >/dev/null docker volume create --label "${EKKLESIA_CI_LABEL}=${project}" "$kiosk_volume" >/dev/null
start_kiosk() { start_kiosk() {
docker run --detach --rm --name "$kiosk_container" \ docker run --detach --rm --name "$kiosk_container" --label "${EKKLESIA_CI_LABEL}=${project}" \
--cap-drop ALL --security-opt no-new-privileges --tmpfs /tmp \ --cap-drop ALL --security-opt no-new-privileges --tmpfs /tmp \
--network "$edge_network" --ip "$kiosk_address" "${host_mappings[@]}" \ --network "$edge_network" --ip "$kiosk_address" "${host_mappings[@]}" \
-v "${kiosk_volume}:/profile" \ -v "${kiosk_volume}:/profile" \
+5 -5
View File
@@ -9,6 +9,8 @@ cd "$repository_root"
source "$repository_root/scripts/lib/compose-job-network.sh" source "$repository_root/scripts/lib/compose-job-network.sh"
# shellcheck source=scripts/lib/compose-diagnostics.sh # shellcheck source=scripts/lib/compose-diagnostics.sh
source "$repository_root/scripts/lib/compose-diagnostics.sh" source "$repository_root/scripts/lib/compose-diagnostics.sh"
# shellcheck source=scripts/lib/ci-project.sh
source "$repository_root/scripts/lib/ci-project.sh"
command -v docker >/dev/null || { command -v docker >/dev/null || {
echo "Docker is required for the infrastructure smoke test." >&2 echo "Docker is required for the infrastructure smoke test." >&2
@@ -30,11 +32,9 @@ if [[ "$curl_help" != *"--aws-sigv4"* ]]; then
exit 1 exit 1
fi fi
# Unique per invocation, not only per run: several runs can share a runner's Docker daemon at once, # Unique per invocation, and findable by run and job if this script is killed before its trap runs
# and the shell's PID is often identical across job containers. # (scripts/lib/ci-project.sh, scripts/ci-sweep.sh).
raw_project="ekklesia-smoke-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-0}-$$-${RANDOM}${RANDOM}" project="$(ci_project_name infrastructure)"
project="${raw_project//[^a-zA-Z0-9_-]/-}"
project="${project,,}"
temporary_directory="$(mktemp -d)" temporary_directory="$(mktemp -d)"
payload_file="$temporary_directory/payload.txt" payload_file="$temporary_directory/payload.txt"
download_file="$temporary_directory/download.txt" download_file="$temporary_directory/download.txt"
+129
View File
@@ -0,0 +1,129 @@
#!/usr/bin/env bash
# Proves scripts/ci-sweep.sh against real Docker resources: what it removes, and above all what it
# leaves alone. CI runs it before trusting the sweep with a shared runner's daemon.
#
# bash scripts/test-ci-sweep.sh
#
# Every resource it creates is named and labeled for this run alone, under fake run identifiers, so
# it can never touch a real stack; and it removes all of them on exit, whatever happens.
set -Eeuo pipefail
repository_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$repository_root"
image="$(sed -n 's/^[[:space:]]*image: \(postgres:[^[:space:]]*\)$/\1/p' deploy/compose.yaml | head -1)"
[[ -n "$image" ]] || {
echo "test-ci-sweep: no pinned PostgreSQL image in deploy/compose.yaml." >&2
exit 1
}
docker image inspect "$image" >/dev/null 2>&1 || docker pull --quiet "$image" >/dev/null
nonce="$(openssl rand -hex 4)"
compose_label="com.docker.compose.project"
ci_label="org.riverofglorymn.ekklesia.ci.project"
failures=0
created=()
check() {
if eval "$1"; then
echo "ok $2"
else
echo "FAIL $2"
failures=$((failures + 1))
fi
}
# project_exists <project>: any container, network, volume, or image still carries its label.
project_exists() {
local project="$1"
[[ -n "$(
docker ps --all --quiet --filter "label=${compose_label}=${project}"
docker ps --all --quiet --filter "label=${ci_label}=${project}"
docker network ls --quiet --filter "label=${compose_label}=${project}"
docker volume ls --quiet --filter "label=${compose_label}=${project}"
docker image ls --quiet --filter "label=${ci_label}=${project}"
)" ]]
}
staging_exists() {
docker run --rm --entrypoint test -v /tmp:/host-tmp "$image" -d "/host-tmp/$1"
}
# make_stack <project>: what a CI script leaves behind when it is killed: a Compose-labeled network,
# volume, and running container; a CI-labeled container and image; and a staging directory on the
# Docker host.
make_stack() {
local project="$1"
created+=("$project")
docker network create --label "${compose_label}=${project}" "${project}_default" >/dev/null
docker volume create --label "${compose_label}=${project}" "${project}_data" >/dev/null
docker run --detach --name "${project}-db" --label "${compose_label}=${project}" \
--network "${project}_default" -v "${project}_data:/data" --entrypoint sleep "$image" 600 >/dev/null
docker run --detach --name "${project}-browser" --label "${ci_label}=${project}" \
--entrypoint sleep "$image" 600 >/dev/null
printf 'FROM %s\nLABEL %s=%s %s=%s\n' "$image" "$ci_label" "$project" \
"org.riverofglorymn.ekklesia.ci.created" "$(date +%s)" |
docker build --quiet --tag "${project}-image:test" - >/dev/null
docker run --rm --entrypoint mkdir -v /tmp:/host-tmp "$image" -p "/host-tmp/${project}"
}
job_container="test-ci-sweep-job-${nonce}"
cleanup() {
local status=$?
trap - EXIT INT TERM
docker rm --force "$job_container" >/dev/null 2>&1 || true
for project in "${created[@]}"; do
docker ps --all --quiet --filter "label=${compose_label}=${project}" | xargs -r docker rm --force >/dev/null 2>&1 || true
docker ps --all --quiet --filter "label=${ci_label}=${project}" | xargs -r docker rm --force >/dev/null 2>&1 || true
docker network rm "${project}_default" >/dev/null 2>&1 || true
docker volume rm --force "${project}_data" >/dev/null 2>&1 || true
docker image rm --force "${project}-image:test" >/dev/null 2>&1 || true
docker run --rm --entrypoint rm -v /tmp:/host-tmp "$image" -rf "/host-tmp/${project}" >/dev/null 2>&1 || true
done
exit "$status"
}
trap cleanup EXIT INT TERM
# shellcheck source=scripts/lib/ci-project.sh
source "$repository_root/scripts/lib/ci-project.sh"
# Fake run identifiers: nothing here can match a real run's stacks.
export GITHUB_RUN_ID="sweeptest${nonce}"
export GITHUB_RUN_ATTEMPT="1"
mine="$(EKKLESIA_CI_JOB_KEY=alpha ci_project_name mine)"
sibling="$(EKKLESIA_CI_JOB_KEY=beta ci_project_name sibling)"
foreign="ekklesia-sweep-control-${nonce}"
for project in "$mine" "$sibling" "$foreign"; do
make_stack "$project"
done
# A job container connected to this job's stack network, exactly as compose-job-network.sh does.
docker run --detach --name "$job_container" --entrypoint sleep "$image" 600 >/dev/null
docker network connect "${mine}_default" "$job_container"
echo "--- --job removes this job's stacks, at any age, and nothing else"
EKKLESIA_CI_JOB_KEY=alpha bash scripts/ci-sweep.sh --job
check '! project_exists "$mine"' "this job's containers, network, volume, and image are gone"
check '! staging_exists "$mine"' "this job's staging directory on the Docker host is gone"
check 'project_exists "$sibling"' "another job's stack in the same run is left alone"
check 'project_exists "$foreign"' "a stack outside the CI prefix is left alone"
check 'docker inspect "$job_container" >/dev/null 2>&1' \
"a job container connected to the stack's network is disconnected, never removed"
echo "--- --job refuses to run outside CI, where it could remove a developer's own stacks"
check '! env -u GITHUB_RUN_ID bash scripts/ci-sweep.sh --job >/dev/null 2>&1' \
"--job without GITHUB_RUN_ID fails"
echo "--- --stale removes only CI stacks older than its threshold"
bash scripts/ci-sweep.sh --stale
check 'project_exists "$sibling"' "a young CI stack survives the default threshold"
bash scripts/ci-sweep.sh --stale --max-age-seconds 0
check '! project_exists "$sibling"' "a CI stack older than the threshold is removed"
check '! staging_exists "$sibling"' "its staging directory too"
check 'project_exists "$foreign"' "a stack outside the CI prefix is never removed, at any age"
if ((failures > 0)); then
echo "test-ci-sweep: ${failures} check(s) failed." >&2
exit 1
fi
echo "CI sweep checks passed."
+5 -5
View File
@@ -9,6 +9,8 @@ cd "$repository_root"
source "$repository_root/scripts/lib/compose-job-network.sh" source "$repository_root/scripts/lib/compose-job-network.sh"
# shellcheck source=scripts/lib/compose-diagnostics.sh # shellcheck source=scripts/lib/compose-diagnostics.sh
source "$repository_root/scripts/lib/compose-diagnostics.sh" source "$repository_root/scripts/lib/compose-diagnostics.sh"
# shellcheck source=scripts/lib/ci-project.sh
source "$repository_root/scripts/lib/ci-project.sh"
command -v docker >/dev/null || { command -v docker >/dev/null || {
echo "Docker is required for the PostgreSQL migration integration suite." >&2 echo "Docker is required for the PostgreSQL migration integration suite." >&2
@@ -27,11 +29,9 @@ if ! compose_job_network_initialize; then
exit 1 exit 1
fi fi
# Unique per invocation, not only per run: several runs (and several jobs of one run) can share a # Unique per invocation, and findable by run and job if this script is killed before its trap runs
# runner's Docker daemon at once, and the shell's PID is often identical across job containers. # (scripts/lib/ci-project.sh, scripts/ci-sweep.sh).
raw_project="ekklesia-migration-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-0}-${GITHUB_JOB:-job}-$$-${RANDOM}${RANDOM}" project="$(ci_project_name migration)"
project="${raw_project//[^a-zA-Z0-9_-]/-}"
project="${project,,}"
export POSTGRES_DB="ekklesia_migration_test" export POSTGRES_DB="ekklesia_migration_test"
export POSTGRES_USER="ekklesia_migration_test" export POSTGRES_USER="ekklesia_migration_test"
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@ekklesia/shell", "name": "@ekklesia/shell",
"version": "0.1.1", "version": "0.1.2",
"private": true, "private": true,
"type": "module", "type": "module",
"files": [ "files": [
+58 -10
View File
@@ -4,6 +4,7 @@ import path from "node:path";
import { fileURLToPath } from "node:url"; import { fileURLToPath } from "node:url";
import { promisify } from "node:util"; import { promisify } from "node:util";
import { describe, expect, it } from "vitest"; import { describe, expect, it } from "vitest";
import { parse } from "yaml";
/** /**
* The scripts CI runs against a shared Docker daemon. Several runs, and several jobs of one run, can * The scripts CI runs against a shared Docker daemon. Several runs, and several jobs of one run, can
@@ -59,21 +60,68 @@ describe("CI scripts on a shared Docker daemon", () => {
} }
}); });
it("give every Compose project a name unique to the invocation, not only the run", async () => { it("name every stack through one helper: the reserved CI prefix, the job's key, and a unique suffix", async () => {
expect(await source("scripts/test-migrations-integration.sh")).toMatch( const helper = await source("scripts/lib/ci-project.sh");
/raw_project="ekklesia-migration-[^"\n]*\$\$-\$\{RANDOM\}\$\{RANDOM\}"/u, // Run, attempt, job key, kind, then a per-invocation suffix: unique even when job containers
// share a PID, and findable later by run and job alone (scripts/ci-sweep.sh --job).
expect(helper).toContain(
"ekklesia-ci-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-0}-${EKKLESIA_CI_JOB_KEY:-${GITHUB_JOB:-local}}-",
); );
expect(await source("scripts/smoke-infrastructure.sh")).toMatch( expect(helper).toContain("$$-${RANDOM}${RANDOM}");
/raw_project="ekklesia-smoke-[^"\n]*\$\$-\$\{RANDOM\}\$\{RANDOM\}"/u,
);
const deploy = await source("scripts/smoke-deploy.sh");
expect(deploy).toContain('run_id="$(openssl rand -hex 4)"');
expect(deploy).toContain('project="ekklesia-deploy-smoke-${run_id}"');
for (const relativePath of ciScripts) { for (const relativePath of ciScripts) {
expect(await source(relativePath), relativePath).toContain('--project-name "$project"'); const text = await source(relativePath);
expect(text, relativePath).toContain('source "$repository_root/scripts/lib/ci-project.sh"');
expect(text, relativePath).toMatch(/^project="\$\(ci_project_name [a-z"${}_-]+\)"$/mu);
expect(text, relativePath).not.toContain("raw_project=");
expect(text, relativePath).toContain('--project-name "$project"');
} }
}); });
it("label everything the deploy smoke creates outside Compose, so a sweep can find it", async () => {
// Continuation lines joined, so each command is checked whole.
const deploy = (await source("scripts/smoke-deploy.sh")).replace(/\\\n\s*/gu, " ");
const creating = deploy
.split("\n")
.filter((line) => /\bdocker (?:run --detach|volume create|build)\b/u.test(line));
expect(creating.length).toBeGreaterThanOrEqual(4);
for (const line of creating) {
expect(line).toContain('--label "${EKKLESIA_CI_LABEL}=${project}"');
}
});
it("sweep stale stacks before, and this job's own stacks after, every job that starts any", async () => {
const workflow = parse(await source(".github/workflows/ci.yml")) as {
jobs: Record<
string,
{ env?: Record<string, string>; steps: { if?: string; run?: string; name?: string }[] }
>;
};
const startsStacks =
/smoke-infrastructure\.sh|test:migrations:integration|smoke-deploy\.sh|docker build/u;
const jobs = Object.entries(workflow.jobs).filter(([, job]) =>
job.steps.some((step) => startsStacks.test(step.run ?? "")),
);
expect(jobs.map(([id]) => id).sort()).toEqual(
["deploy-smoke", "image-build", "infrastructure-smoke", "migration-smoke"].sort(),
);
for (const [id, job] of jobs) {
expect(job.env?.EKKLESIA_CI_JOB_KEY, id).toBeTruthy();
const runs = job.steps.map((step) => step.run ?? "");
const firstStack = runs.findIndex((run) => startsStacks.test(run));
const stale = runs.findIndex((run) => run.includes("bash scripts/ci-sweep.sh --stale"));
const own = job.steps.findIndex(
(step) =>
step.if === "always()" && (step.run ?? "").includes("bash scripts/ci-sweep.sh --job"),
);
expect(stale, `${id}: stale sweep`).toBeGreaterThanOrEqual(0);
expect(stale, `${id}: stale sweep runs first`).toBeLessThan(firstStack);
expect(own, `${id}: its own cleanup, always`).toBeGreaterThan(firstStack);
expect(own, `${id}: cleanup is the last step`).toBe(job.steps.length - 1);
}
// One matrix leg's cleanup must never remove the other leg's running stack.
expect(workflow.jobs["deploy-smoke"]?.env?.EKKLESIA_CI_JOB_KEY).toContain("matrix.topology");
});
it("choose the deploy smoke's edge subnet from those not in use, and choose again on a race", async () => { it("choose the deploy smoke's edge subnet from those not in use, and choose again on a race", async () => {
const deploy = await source("scripts/smoke-deploy.sh"); const deploy = await source("scripts/smoke-deploy.sh");
expect(deploy).not.toMatch(/^octet=\$\(\(RANDOM/mu); expect(deploy).not.toMatch(/^octet=\$\(\(RANDOM/mu);