A migration-smoke stack (PostgreSQL and SeaweedFS) leaked on the shared
runner and ran for two days, until it was removed by hand. Every CI
script already tears its stack down in an EXIT/INT/TERM trap, but a
trap never runs when the runner kills the job container: a cancelled
job, or the runner's own time limit, since Gitea 1.24 ignores
timeout-minutes.
Static checks (tests/ci-scripts.test.ts):
- every stack script names its project through one helper: the reserved
ekklesia-ci- prefix, run, attempt, job key, and a unique suffix;
- everything the deploy smoke creates outside Compose carries the CI
label;
- every workflow job that starts stacks sweeps stale CI stacks first,
and removes its own in a final if: always() step. Each matrix leg has
its own key, so one cannot remove the other's running stack.
Behaviour against real Docker (scripts/test-ci-sweep.sh):
- --job removes this job's containers, network, volume, image and
staging directory, and leaves other jobs' and non-CI stacks alone;
- --job disconnects, never removes, a job container on the stack's
network, and refuses to run outside CI;
- --stale spares young stacks and removes old ones, never anything
outside the prefix.
Fails today: there is no helper, no sweep, no label and no step.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A job the runner kills never runs its own cleanup trap, and the
platform's CI leaked a whole migration stack that way. The harness
follows the platform's three-layer rule:
- everything it starts is named and labeled as a CI stack;
- it cleans up in a trap;
- the template workflow sweeps stale CI stacks before running it, and
removes its own in a final if: always() step.
The sweep ships in the harness image.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signage reached production with three defects that appear only inside
Ekklesia in a real browser, and no CI that ran it there. The ADR
proposes:
- a versioned harness image, ekklesia-module-harness:<platform version>,
built and pushed by the platform's release CI;
- what the harness does: assemble a module into that platform, validate
and build it, migrate a disposable database, start the split reference
deployment, and mount every declared UI surface in headless Chromium;
- what fails it: console errors, unexpected 4xx/5xx, unhandled
rejections, a display soak, and generic no-dead-ends injections;
- testing at the lowest and newest platform versions of a module's
range;
- extra smoke steps in ekklesia.harness.json, beside the manifest
rather than in it;
- a later data-access and egress check against declarations;
- a short template workflow that calls the harness by digest with one
`docker run`. Not a reusable workflow: thinly supported on Gitea
1.24, and private sources only arrive in 1.26.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Every reload of a cached files. image failed in the browser with
ERR_HTTP2_PROTOCOL_ERROR. A browser revalidates a cached signed URL
with If-None-Match. Caddy reached SeaweedFS over HTTP/2, and there
SeaweedFS answers with a 304 declaring content-type: application/xml
and content-length: 327 while sending no body. Caddy then logged
"aborting with incomplete response ... unexpected EOF" and reset the
stream. Over HTTP/1.1 SeaweedFS's 304 is clean. Measured layer by layer:
DevTools, curl through Caddy, and curl direct over HTTP/1.1 and HTTP/2.
The storage upstream is now HTTP/1.1 only (`versions 1.1`). Browsers
still reach the proxy over HTTP/2 and HTTP/3.
Delivery-layer CORS is the deployment's, so it no longer depends on
which S3 backend is behind it. SeaweedFS's default reflected any origin
and advertised PUT, POST and DELETE. Now the proxy:
- grants CORS only to the origins that read objects with fetch(): the
Ekklesia origin, or, when split, the API origin (displays) and the
Dashboard origin (a module's preview caches images through fetch());
- allows GET and HEAD only, with Vary: Origin;
- answers preflights itself, so they never reach storage;
- strips the object store's own CORS headers.
The deploy README documents the policy, its reasons, and what a
self-run proxy must do.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Two checks on storage delivery through the proxy, in both topologies.
The same signed image, twice, over a real HTTP/2 connection (node:http2;
Node's fetch speaks HTTP/1.1, where the failure does not exist). The
second load is conditional, as a browser's revalidation is, and must be
a clean 304. Today Caddy reaches SeaweedFS over HTTP/2, SeaweedFS's 304
declares Content-Length: 327 and sends no body, and Caddy resets the
stream. A browser reports that as ERR_HTTP2_PROTOCOL_ERROR on every
reload of a cached files. image.
CORS on delivery is the deployment's own:
- exactly the API origin (and the Dashboard origin when split) is
allowed, with Vary: Origin and nothing else;
- any other origin gets no CORS header;
- a preflight is answered by the proxy (204, GET and HEAD only) and
never reaches the object store.
Today SeaweedFS reflects every origin with PUT, POST and DELETE, and
answers the preflight itself.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Doctor checks the client-address configuration of the process it runs
in, and operators run it as `migrate`. The proxy overlay set
trusted-proxy mode on `api` only, so Doctor reported "Direct
client-address mode" for an installation that trusts its proxy.
- compose.proxy.yaml defines the three variables once, as a YAML
anchor, and applies it to both `api` and `migrate`. Rendered with
`docker compose config`, `api` is unchanged and `migrate` now
matches it, keeping NODE_EXTRA_CA_CERTS from the base.
- The example deploy/.env sets
COMPOSE_FILE=compose.yaml:compose.proxy.yaml, as the instance already
does. Plain `docker compose run --rm migrate ...` in the README then
runs with the overlay; it used to read compose.yaml alone.
- Every client-address check in Doctor now says it checked the
environment Doctor runs with, and how to read the running API's own
value (`docker compose exec api printenv
EKKLESIA_CLIENT_ADDRESS_MODE`, confirmed to work in the image).
- The deploy README explains why operator commands need the same files
as the stack.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Behind the reference proxy the API runs in trusted-proxy mode, but
compose.proxy.yaml gives those variables to `api` only. The documented
Doctor command runs as `migrate`, so it validated an empty
configuration and reported "Direct client-address mode" for an
installation that trusts its proxy. That is what the first production
installation saw.
Fails today:
- deploy.test.ts: the proxy overlay gives `migrate` no client-address
environment;
- doctor.test.ts: the check does not say whose configuration it read;
- smoke-deploy.sh: Doctor through the proxy overlay does not report
trusted-proxy mode (Docker).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The recycle step's checks spoke of "the realtime recycle" even when
run with a watch shorter than the five-minute connection age, as module
CI does. They now say what they check: live for the whole watch (plus
"across a realtime recycle" when the watch is long enough), and never
remounted.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Reproduction harness for what the first production Signage deployment
found. With EKKLESIA_SMOKE_DISPLAY_DEVICE=1, the split smoke starts a
second headless Chromium (the "kiosk") whose profile directory is a
Docker volume. It pairs that kiosk through the real flow, closes it
cleanly, restarts it on the same profile, and then runs injected-failure
scenarios: two pages on one credential, a connectivity blip, one refused
viewer-session request, one refused realtime connection, a cross-origin
image-caching probe, a full realtime recycle, and revocation.
Every DevTools command and page-side request has its own deadline, and
each step has a budget. A step that overruns prints the exact call it
was waiting for. The proxy's "aborting with incomplete response" lines
are listed afterwards by UTC time and path only.
Off by default: several scenarios expect Signage and fail on today's
v0.1.1, by design.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
An installation can now serve the Dashboard and the API on sibling
hostnames (dash./api.) beside the default same-origin topology, from one
image and with configuration alone.
- The Dashboard learns its API origin at run time: the built page carries
one empty <meta name="ekklesia-api-origin">, which the API fills with its
identity base URL's origin at startup, before computing the page's ETag.
The shell accepts only a canonical http(s) origin and otherwise fails
closed without making a request. The build-time EKKLESIA_API_BASE_URL is
gone from the bundle; the development server injects the same element.
- The display host stays on the API's origin, so its device cookies,
requests, and offline cache are unchanged.
- The API trusts its own origin (trusted origins plus the identity base
URL), as Better Auth already does, so the display page's mutations pass
the origin check without operators listing it.
- Doctor validates EKKLESIA_DASHBOARD_ORIGIN exactly as the API's startup
does (it previously skipped it), reports the topology without printing
hostnames, and warns when split hostnames are not siblings.
- The module UI contract states the CORS header limit (safelisted headers
plus Content-Type, safelisted response headers only).
- Reference deployment: Caddyfile.split.example (Dashboard hostname serves
/ and its script; API hostname serves /api/*, /display/*, and the
display script; each refuses the other's paths) and
EKKLESIA_DASHBOARD_HOSTNAME in the proxy overlay and env examples.
- The deploy smoke test runs either topology (CI runs both). The split run
installs the module template as a test-only module, checks per-hostname
routing and exact CORS, then drives a digest-pinned headless Chromium
over the DevTools protocol, with no added dependency: sign-in on the
Dashboard hostname, the session cookie host-only on the API hostname,
the module UI imported cross-origin and its credentialed request
succeeding, the display and its service worker on the API hostname, and
sign-out.
ADR 0045 is marked Accepted.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Records a core principle: Ekklesia's branding is never forced on an
installation. Anything a member, visitor, or volunteer can see (titles,
sign-in and auth screens, emails, display and offline screens, error
messages, app names and icons, attribution) is configurable and removable
entirely. With no name configured, wording is neutral, so the default
names nothing and the rule is testable. Internal identifiers, operator
tooling, and developer diagnostics are exempt. Modules follow the same
rule.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Records how an installation runs the Dashboard and the API on sibling
hostnames (dash./api.) beside the default same-origin topology, from one
image: the API origin reaches the Dashboard at run time in a meta element,
the display host stays on the API's origin (device cookies and the offline
code cache depend on it), the API trusts its own origin as Better Auth
already does, cookies stay host-only, CORS stays narrow and becomes part of
the module UI contract, the reference proxy gains per-hostname routing, and
the deploy smoke test proves both topologies, the split one in a real
browser.
Amends ADR 0034 and extends ADR 0044.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
PR #51's migration-smoke failed 0.7 s after the build: the suite pinned
PostgreSQL to an ephemeral port but inherited S3_PORT=8333 from
.env.example, so concurrent runs on one runner (Gitea ignores workflow
concurrency) contended for one host port. Compose's own error was
discarded, so the failure said only that the services "did not become
ready".
Isolation, from an audit of every script CI runs:
- test-migrations-integration.sh: S3_PORT=0, like POSTGRES_PORT=0.
- test-migrations-integration.sh and smoke-infrastructure.sh: project names
gain a random suffix; run id plus PID alone is not unique when job
containers share low PIDs.
- smoke-deploy.sh: the edge /24 is chosen from those no network on the
daemon uses, and the first `up` chooses again when a concurrent run wins
a race for the same subnet (Docker's address-pool overlap), up to five
times.
- CI image-build: the image tag includes the run id and attempt, so a pull
request and a push of the same commit cannot remove each other's image.
- Already isolated: every other published port (ephemeral), the TLS
suite's container and network (random), smoke staging directories and
image tags (random).
Diagnostics: scripts/lib/compose-diagnostics.sh prints the failed `up`
command's output, `docker compose ps`, and the last 40 log lines of each
service, all redacted (supplied literals, credential-looking NAME=value
pairs, URL user information, Authorization headers) with terminal control
sequences stripped. All three scripts use it at every readiness check; the
infrastructure smoke's unredacted full-log dump is replaced by it.
tests/ci-scripts.test.ts pins ephemeral ports, unique project names, the
subnet choice, the CI image tag, the diagnostics, and the redaction itself
(run through bash).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Bump the root and every platform workspace package from 0.1.0 to 0.1.1, so
the platform reports the version the v0.1.1 tag will carry. Module discovery
checks compatibility.platform against the root version and
compatibility.moduleSdk against @ekklesia/module-sdk; both stay inside every
existing ^0.1.0 range, so no module needs a new range. Everything since v0.1.0
is additive (health endpoints, Dashboard serving, production dependency
declarations, the deployment image and Compose project).
templates/module keeps 0.1.0: it is a module, versioned on its own, and its
manifest, source, and tests carry its own product version.
Mark ADR 0044 Accepted. Slices 1-4 are implemented and proven by CI's image
build and deploy smoke; slice 6 (scheduled backups and the restore
procedure) remains its open item.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Implement ADR 0044's reference deployment as a base Compose file plus
overlays, so an installation that already runs a proxy can use the base
alone.
- compose.yaml: api, postgres, object-storage, and the operator-only
migrate (profile ops, plan by default). No published port. api and
migrate run init, read-only, tmpfs /tmp, cap_drop ALL,
no-new-privileges. postgres is only on the internal data network.
- TLS everywhere from one installation-private CA: PostgreSQL TLS 1.3
only, with pg_hba rejecting every non-TLS connection, and the API on
verify-full via EKKLESIA_DATABASE_CA (PEM text). SeaweedFS serves S3
over HTTPS (ADR 0030 requires https in production), trusted by the
API through NODE_EXTRA_CA_CERTS. No Core change.
- compose.proxy.yaml + Caddyfile.example: Caddy pinned by digest, ACME
HTTP-01 (DNS-01 documented), routes /, /api/*, /display/*, /assets/*
to api and refuses everything else; the storage hostname preserves
Host. The API trusts exactly the proxy's fixed /32 on an edge network
whose dynamic range excludes the fixed addresses; Caddy overwrites
X-Forwarded-For and strips caller-supplied forwarding headers.
- compose.tunnel.yaml: cloudflared pinned by digest, token from its own
env file; Caddy honors CF-Connecting-IP only from its fixed address.
- Four per-service root-owned env files, examples with no values;
scripts/init-tls.sh (idempotent, never overwrites a key) and
scripts/gen-secrets.sh (prints only).
- CI: pnpm deploy:config in ci:verify for all three combinations, and a
deploy-smoke job (scripts/smoke-deploy.sh) proving readiness, the
Dashboard, signed storage delivery, the client-address chain, initial
owner sign-in/out, verify-full database TLS, and Doctor through the
proxy, against disposable services. tests/deploy.test.ts pins the
structural guarantees.
- ADR 0044 (still Proposed) amended to match: API-to-storage HTTPS,
DNS-01 needs a custom Caddy build and tunnel-only needs DNS-01, the
tunnel's origin TLS settings live in the Cloudflare dashboard, port
80 serves HTTP-01, and secrets are one env file per service.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A module that imports a package at run time without declaring it a
production dependency now fails the image build instead of the API's
startup, since the runtime receives production dependencies only.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Add deploy/Dockerfile and .dockerignore (ADR 0044): one image for the
api service and every operator-run CLI, built from an already-assembled
workspace (the platform plus whatever modules/<id> the context holds).
- node:24.18.0-trixie-slim pinned by index digest; pnpm via Corepack at
the root packageManager version.
- Build stage: frozen install, modules:validate, migrations:validate,
build. Runtime: production-only node_modules plus every dist/, core
migrations, and each module directory, in the repository layout the
API and CLIs resolve paths from; root-owned, run as node.
- NODE_ENV=production and EKKLESIA_API_HOST=0.0.0.0 inside the image
only; the platform default stays 127.0.0.1. Configuration comes only
from the environment; no .env enters the context or any layer.
- HEALTHCHECK on /api/v1/health/ready with Node's fetch; OCI labels for
source, revision, and version.
- CI builds the image without modules: build only, no credentials, no
push.
- deploy/README.md documents the image, the read-only production
posture (verified), init: true, and the exact CLI commands.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
@ekklesia/module-sdk imports drizzle-orm at run time but declared it
only as a peer and a development dependency. A workspace package is
linked into its consumers as a directory, so its imports resolve from
its own node_modules, where a consumer never satisfies a peer; a
production install omits development dependencies. In the production
image this made every module with a schema fail: the API exited at
startup and Doctor reported NOT READY.
- Move drizzle-orm to module-sdk's dependencies, keeping the peer.
- Move react to @ekklesia/ui's dependencies, keeping the peer. The new
guard flags it the same way; it is bundled at build time today, so
it was not a live failure. Both lockfile entries keep the identical
resolution, so nothing new is installed and the shell bundles still
contain one React.
- Add the boundary rule runtime-imports-are-production-dependencies:
code a package ships (src/, bin/, excluding tests and .d.ts) must
declare every runtime import in dependencies or optionalDependencies.
Type-only imports are exempt; .tsx/.jsx implies react (automatic
JSX runtime). Fixtures cover peer-only, dev-only, JSX, undeclared,
and the accepted cases.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Serve surfaces/shell's built Dashboard from apps/api on the same origin
as /api (ADR 0044), exactly as the display host is served: a closed set
of files read once at startup, never a static directory, never a path
built from request input.
- The shell exports @ekklesia/shell/dashboard-assets (the exact built
index.html and script URLs), emits the script at
assets/ekklesia-dashboard/app.js, and index.html names that absolute
path so dev and production serve the same URLs.
- GET / and GET /assets/ekklesia-dashboard/app.js answer from a frozen
snapshot: no-cache with a strong ETag and 304 on match. Source maps
are never served; every other path is the ordinary sanitized 404.
- The page sets nosniff, X-Frame-Options: DENY, Content-Security-Policy:
frame-ancestors 'none' (that directive only), and Referrer-Policy:
no-referrer, on 200 and 304 alike.
- A missing or empty shell build fails startup with a fixed
DashboardHostAssetError before any module code is imported.
- The If-None-Match comparison moves to conditional-request.ts, shared
with the display host routes.
- The emailed-link token stripping moves from browser/main.tsx into
consumeAuthLanding (location and history injected) so it is tested;
behavior is unchanged.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Add GET /api/v1/health/live and GET /api/v1/health/ready per ADR 0044.
- live returns a fixed 200 {"status":"live"} whenever the process serves.
- ready returns 200 {"status":"ready"} only when module startup has
completed (and shutdown has not begun) and a database ping succeeds
within 2s; otherwise a fixed 503 {"code":"NOT_READY"}.
- The ping runs on the shared pool as one simple-protocol message with
a transaction-local statement_timeout, so PostgreSQL cancels a stalled
statement; the pool discards the connection on failure.
- At most one ping is ever unsettled: probes share its outcome and, once
its deadline passes, answer 503 immediately until it settles.
- Both routes set Cache-Control: no-store, accept GET only (405 with
Allow: GET otherwise, HEAD included), never resolve a principal, and
are mounted in app.ts beside, never inside, /api/v1/modules/.
- createApiApp gains a required `health` option; startApi supplies it
and clears module readiness when shutdown begins.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Add ADR 0044 (Status: Proposed), the single self-hosted reference
deployment composed from pinned sources, and list it in the ADR index.
Correct CLAUDE.md, which described manifest v6; the contracts accept
only manifestVersion 7 (MODULE_MANIFEST_VERSION in
packages/contracts/src/module-manifest.ts).
No code changes.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The fetch listener called `respondWith` for every request and re-issued the
unrecognized ones itself. That routes the whole page through the worker,
re-sends request bodies from a different context, and makes the worker's
presence observable on requests it has no interest in.
Classify synchronously and respond only for the shell and artifact URLs this
worker actually owns. Not responding is what leaves the browser doing exactly
what it would have done without a worker at all — which is the property the
allowlist was written to preserve.
Co-Authored-By: Claude Opus 5 <[email protected]>
Bring the living documentation in line with what the five passes actually
shipped, and correct the claims they superseded rather than leaving them to
mislead: the module UI context is six members, not five; module styling is
compiled CSS, not inline objects only; the strict manifest is version 7.
Also close a silent failure the new contribution mechanism introduces. A
contribution whose target capability nobody provides — or whose provider moved
outside its declared range — is not published, and the manifest is valid and
the artifact exists, so an operator sees a screen that renders nothing with no
error anywhere. `pnpm modules:validate` now lists each declared contribution
with its capability and range and says whether a compatible provider is
installed, and Doctor's discovery check reports the same per module. Both
derive from the publication registry the API serves from, so neither can
disagree with what a running platform would actually publish.
Co-Authored-By: Claude Opus 5 <[email protected]>
Joey