feat: one generic image, with installed modules in installation-owned storage (ADR 0048)
CI / verify (pull_request) Failing after 47s
CI / infrastructure-smoke (pull_request) Skipped
CI / migration-smoke (pull_request) Skipped
CI / image-build (pull_request) Skipped
CI / deploy-smoke (same-origin) (pull_request) Skipped
CI / deploy-smoke (split) (pull_request) Skipped

The production image now contains zero feature modules and is the same for every
installation. Installed modules live in module storage that one Core resolver
selects from EKKLESIA_MODULES_DIR: the checkout's modules/ when unset, an
absolute path that must exist when set (/var/lib/ekklesia/modules in the image).
A relative, blank, missing, or unreadable configured directory fails closed with
a sanitized error that never contains the value.

- Host configuration reads EKKLESIA_MODULES_DIR. The API, migration plan, apply
  and validate, module validation, module enablement, and Doctor all resolve it
  the same way; the test-only API override still takes precedence.
- Doctor receives Core's migrations directory explicitly instead of deriving it
  from the modules directory's parent.
- The migration runner retains and reports the applied history of a module that
  is not installed, and plans and applies Core's and present modules' migrations
  around it. Core's history and every installed module's history stay strict: a
  missing or changed applied migration, or a downgrade, still fails. No down
  migrations, no data deletion.
- deploy/Dockerfile copies no module and installs nothing for the module
  template; .dockerignore excludes modules/ and the build refuses a context that
  contains it; the runtime has an empty root-owned /var/lib/ekklesia/modules and
  sets EKKLESIA_MODULES_DIR.
- deploy/compose.yaml bind-mounts EKKLESIA_MODULES_HOST_DIR read-only into api
  and migrate, with create_host_path disabled.
- scripts/check-image.sh proves the image holds no modules and that a populated
  modules/ directory leaves its runtime file system identical; CI runs it.
- The split deploy smoke now uses the same generic image: it boots with no
  modules, installs a self-contained test module packaged from the template
  (test-only), migrates and restarts, removes it (Core runs, planning reports the
  retained history, data survives), and restores it with its data.

Docs: deploy README (module storage, install/update/remove/restore, the runtime
modules target), .env.example, API configuration, module development guide,
testing, workflow, security, and the architecture overview.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
ekklesia
2026-10-07 08:38:14 -05:00
co-authored by Claude Opus 5.5
parent 4d28a82937
commit e36b70d8d4
50 changed files with 2275 additions and 224 deletions
+155 -56
View File
@@ -64,6 +64,7 @@ TLS is everywhere, without any exception in Core:
| `/etc/ekklesia/object-storage.env` | The S3 key pair | `object-storage` |
| `/etc/ekklesia/tunnel.env` | `TUNNEL_TOKEN` | `tunnel` |
| `/etc/ekklesia/tls/` | The CA and server certificates and keys | mounted file by file |
| `/var/lib/ekklesia/modules/` | Installed modules ([Installed modules](#installed-modules)); not a secret | `api`, `migrate`, read-only |
The secret files are owned by root with mode 600, one per service, so no container receives another
service's credentials. None is committed, and `deploy/.env` is ignored by Git. `EKKLESIA_ENV_DIR`
@@ -94,10 +95,19 @@ and `EKKLESIA_TLS_DIR` in `deploy/.env` move the directories.
`EKKLESIA_IDENTITY_BASE_URL` and `EKKLESIA_IDENTITY_TRUSTED_ORIGINS` are `https://` and your
Ekklesia hostname. `S3_DELIVERY_ENDPOINT` is `https://` and your storage hostname.
5. **Configure the proxy:** copy `Caddyfile.example` to `Caddyfile`, and set `EKKLESIA_HOSTNAME`,
5. **Create module storage**, empty, even if you install no module yet:
```sh
sudo install -d -o root -g root -m 755 /var/lib/ekklesia/modules
```
`api` and `migrate` refuse to start without it: Compose never creates it for you, so a mistyped
path cannot quietly become an empty directory. `EKKLESIA_MODULES_HOST_DIR` in `deploy/.env`
moves it.
6. **Configure the proxy:** copy `Caddyfile.example` to `Caddyfile`, and set `EKKLESIA_HOSTNAME`,
`EKKLESIA_STORAGE_HOSTNAME`, and `EKKLESIA_ACME_EMAIL` in `deploy/.env`. Both hostnames must
resolve to this host with ports 80 and 443 reachable for HTTP-01, or see [DNS-01](#dns-01).
6. **Start the data services, then migrate explicitly:**
7. **Start the data services, then migrate explicitly:**
```sh
docker compose -f compose.yaml -f compose.proxy.yaml up -d --wait postgres object-storage
@@ -107,8 +117,8 @@ and `EKKLESIA_TLS_DIR` in `deploy/.env` move the directories.
Nothing ever applies migrations automatically: the API never does, and `migrate` runs only when
you run it.
7. **Start everything:** `docker compose -f compose.yaml -f compose.proxy.yaml up -d --wait`.
8. **Check it:** `curl https://<hostname>/api/v1/health/ready` answers `{"status":"ready"}`, and
8. **Start everything:** `docker compose -f compose.yaml -f compose.proxy.yaml up -d --wait`.
9. **Check it:** `curl https://<hostname>/api/v1/health/ready` answers `{"status":"ready"}`, and
`docker compose run --rm migrate node core/dist/cli/doctor.js` reports `READY` (with warnings
for anything optional you left unconfigured, such as email).
@@ -123,25 +133,93 @@ is created already verified, so they can sign in before any email relay exists
([ADR 0035](../docs/architecture/decisions/0035-initial-platform-owner-bootstrap.md)). To supply the
password from a secret manager instead, pipe it on standard input and use `-T` instead of `-it`.
### Installed modules
The image contains no modules
([ADR 0048](../docs/architecture/decisions/0048-generic-image-and-installed-module-storage.md)).
Installed modules live on the host, in module storage: `/var/lib/ekklesia/modules`, or
`EKKLESIA_MODULES_HOST_DIR`. Compose mounts it **read-only** at `/var/lib/ekklesia/modules` in `api`
and `migrate`, which is where `EKKLESIA_MODULES_DIR` points. Ekklesia never writes there.
Installing, updating, or removing a module never needs a new image, only a restart.
**What belongs there.** One directory per module, each a self-contained installed module: its
`package.json`, `ekklesia.module.json`, built artifacts, migrations, and the production
dependencies its server code imports, in its own `node_modules`
([Module development](../docs/module-development/README.md#an-installed-module-is-self-contained)).
Copying a module's `dist/` alone is not enough. Files are owned by root and readable by everyone
(directories 755, files 644), so the API's unprivileged user can read them and never change them.
**Nothing else belongs there.** Every non-hidden directory must be a valid module, or the API does
not start. That includes `lost+found`, so if module storage is a file system of its own, mount it
one level up and point `EKKLESIA_MODULES_HOST_DIR` at a directory inside it. Hidden entries are
ignored, which is what makes staging possible.
**Changes take effect only at a restart.** A running API never notices module storage changing, and
never loads or unloads a module while it runs. Change module storage only while the API is stopped:
it reads browser artifacts from disk as it serves them, so files changed under a running API would
serve one version's browser code beside another's server code.
To **install or update** a module, with `<id>` its module id:
```sh
docker compose stop api
# Stage under a hidden name, which discovery ignores, then rename it into place in one step.
sudo cp -R ./<id> /var/lib/ekklesia/modules/.incoming-<id>
sudo chown -R root:root /var/lib/ekklesia/modules/.incoming-<id>
sudo chmod -R a+rX,go-w /var/lib/ekklesia/modules/.incoming-<id>
sudo mv /var/lib/ekklesia/modules/<id> /var/lib/ekklesia/modules/.previous-<id> # update only
sudo mv /var/lib/ekklesia/modules/.incoming-<id> /var/lib/ekklesia/modules/<id>
docker compose run --rm migrate node core/dist/cli/validate-modules.js
docker compose run --rm migrate # plan: review it
docker compose run --rm migrate node core/dist/cli/apply-migrations.js # only when the plan is right
docker compose up -d --wait api
```
Back up before an update that brings migrations: migrations are forward-only. A newly installed
module is enabled unless an administrator disabled it before; set its configuration and grants as
its documentation says. Keep `.previous-<id>` until the update has proven itself, then delete it.
To **remove** a module, stop the API, delete (or move away) its directory, and start the API again.
Remove first any module that *requires* it, or the API refuses to start. Removing a module's code
deletes nothing else. Its tables and rows, its buckets and objects, its enablement, configuration,
Dashboard placement, and grants all stay. Its applied migration history stays too, and the
migration plan and Doctor report it as `retained` history for a module that is not installed. That
is information, not a problem, and it blocks no other migration. To stop a module without removing
it, disable it instead (`set-module-enabled.js --disable`, [Operator commands](#operator-commands)).
There is no purge: deleting a module's data is a separate operation that does not exist yet.
To **restore** a removed module, install it again as above, at the same version or a newer one. It
finds its data, its settings, and its history where it left them. Its history is strict again: a
package that no longer contains a migration that was applied, or contains one with different
contents, is refused by the migration plan, so an older version cannot be installed over newer data.
Module storage is installation state: back it up with the database and the buckets.
### Upgrading
1. **Pull or build the new image**, and change `EKKLESIA_IMAGE` in `deploy/.env`. Keep the old tag.
Installed modules stay where they are, in module storage.
2. **Back up** ([Backups](#backups)).
3. **Plan, review, then apply:** `docker compose run --rm migrate` with the new image, then
3. **Check the installed modules against the new platform:**
`docker compose run --rm migrate node core/dist/cli/validate-modules.js`. A module whose
`compatibility.platform` range excludes the new version fails here, and would stop the API from
starting: update it first, or stay on the old image.
4. **Plan, review, then apply:** `docker compose run --rm migrate` with the new image, then
`docker compose run --rm migrate node core/dist/cli/apply-migrations.js`.
4. **Restart on the new image:** `docker compose -f compose.yaml -f compose.proxy.yaml up -d --wait`.
5. **Restart on the new image:** `docker compose -f compose.yaml -f compose.proxy.yaml up -d --wait`.
### Rolling back
Set `EKKLESIA_IMAGE` back to the previous tag and run `up -d --wait` again. That is enough only when
the upgrade applied no migrations (the plan in step 3 showed none pending). Migrations are
the upgrade applied no migrations (the plan in step 4 showed none pending). Migrations are
forward-only. If any were applied, restore the backup taken in step 2 before starting the previous
image, because it was not built for the newer schema.
image, because it was not built for the newer schema. Module storage is not part of the image, so a
rollback does not change installed modules.
### Backups
Slice 6 adds the scheduled backup job and the restore procedure (ADR 0044, Backups). Until then,
take a `pg_dump` and a copy of every bucket before each upgrade by hand.
take a `pg_dump`, a copy of every bucket, and a copy of module storage before each upgrade by hand.
### Split Dashboard and API origins
@@ -309,9 +387,13 @@ deployment:
- `pnpm deploy:config` validates the base, base + proxy, and base + proxy + tunnel with the example
env files. CI runs it in `pnpm ci:verify`.
- `pnpm deploy:smoke` (`scripts/smoke-deploy.sh`) runs the whole deployment end to end, against a
platform-only image and a disposable PostgreSQL and SeaweedFS that exist only for that run. It
follows the first boot above and checks the following through the proxy:
- `bash scripts/check-image.sh` builds the image and proves it is generic: no `/opt/ekklesia/modules`,
no module template, no `ekklesia.module.json` outside installed dependencies, an empty root-owned
`/var/lib/ekklesia/modules`, and `EKKLESIA_MODULES_DIR` set. It then builds again from a checkout
with a populated `modules/` directory and requires an identical runtime file system. CI runs it.
- `pnpm deploy:smoke` (`scripts/smoke-deploy.sh`) runs the whole deployment end to end, against the
generic image, empty module storage, and a disposable PostgreSQL and SeaweedFS that exist only for
that run. It follows the first boot above and checks the following through the proxy:
- readiness and the Dashboard;
- a signed object URL fetched through the storage hostname, and the same image loaded twice over
HTTP/2, the second time as a conditional request that must be a clean `304`;
@@ -324,12 +406,23 @@ deployment:
- Doctor.
`EKKLESIA_SMOKE_TOPOLOGY=split` runs the [split topology](#split-dashboard-and-api-origins)
instead: an image with the repository's module template installed as a test-only module, every
hostname's served and refused paths, exact CORS for the Dashboard's origin and none for others,
and then a real headless Chromium (pinned by digest, driven over the DevTools protocol) that signs
in on the Dashboard's hostname, checks the session cookie lives only on the API's, mounts the
module's UI from the API's origin and sees its own credentialed requests succeed, mounts its display
on the API's hostname with its service worker, and signs out.
instead, on the **same generic image**, and also proves [Installed modules](#installed-modules)
end to end:
- the image boots with no module;
- the repository's module template, packaged as a self-contained test module
(`scripts/lib/test-module-package.Dockerfile`, test-only and not a package format), is installed
into module storage, its migration is planned and explicitly applied, the API restarts, and the
module reads data it owns;
- every hostname's served and refused paths, exact CORS for the Dashboard's origin and none for
others;
- a real headless Chromium (pinned by digest, driven over the DevTools protocol) signs in on the
Dashboard's hostname, checks the session cookie lives only on the API's, mounts the module's UI
from the API's origin and sees its own credentialed requests succeed, mounts its display on the
API's hostname with its service worker, and signs out;
- the module is removed and the API restarted: Core still runs, the module's routes are gone,
migration planning and applying succeed and report its retained history, and its data is intact;
- the module is restored and the API restarted: its history validates strictly and it returns its
previous data.
CI runs both topologies on every change.
- **Test-only:** the smoke test runs Caddy with its internal CA (`local_certs`), because a test run
@@ -350,40 +443,37 @@ is no second image and no second build.
### What it builds
The build context is an **already-assembled workspace**: this repository, plus zero or more module
packages already present in `modules/<id>`, plus a `pnpm-lock.yaml` describing exactly that set.
The Dockerfile clones nothing and chooses no modules. It builds what it is given. With no
`modules/` directory, it builds the platform alone; CI builds that case on every change.
The build context is **this repository**, and the image is the same for every installation
([ADR 0048](../docs/architecture/decisions/0048-generic-image-and-installed-module-storage.md)). It
contains **zero feature modules**: installed modules live in module storage outside it
([Installed modules](#installed-modules)). No input is specific to an installation: no module list,
no lockfile of its own, no branding, no configuration.
Assembling the context (checking out the platform at a release tag, each module at a pinned tag or
commit, and the instance's own lockfile) belongs to the installation's instance composition, not to
this file.
> **Being replaced.** [ADR 0048](../docs/architecture/decisions/0048-generic-image-and-installed-module-storage.md)
> supersedes this composition model. The image will contain no modules and ignore `modules/` in its
> build context. Installed modules will live in installation-owned storage
> (`EKKLESIA_MODULES_DIR`, `/var/lib/ekklesia/modules` in the image), mounted read-only. Until that
> lands, the image behaves exactly as described here. Do not start a new per-installation assembly.
A `modules/` directory in the checkout never reaches the image. `.dockerignore` keeps it out of the
build context, and the build refuses a context that still contains one. The module template is a
workspace package, so it is built and validated with everything else, but neither its files nor its
dependencies reach the runtime image. `scripts/check-image.sh` proves all of this on every change
([Testing this deployment](#testing-this-deployment)).
| Stage | Does |
| --- | --- |
| `base` | `node:24.18.0-trixie-slim`, pinned by its multi-architecture index digest. Debian, so glibc, because module dependencies such as sharp ship glibc prebuilt binaries. Every stage starts here. |
| `toolchain` | Enables pnpm through Corepack, at exactly the root `packageManager` version. |
| `manifests` | Collects the lockfile, workspace configuration, `.npmrc`, and every workspace `package.json` at its repository path. |
| `build` | `pnpm install --frozen-lockfile`, then `pnpm boundaries`, `pnpm modules:validate`, `pnpm migrations:validate`, and `pnpm build`, against exactly the modules in the context. Never uses host `node_modules`: `.dockerignore` excludes them. |
| `prod-deps` | The same lockfile with `--prod`: production dependencies only. |
| `runtime` | Production dependencies plus built outputs, owned by root and run as the unprivileged `node` user (uid 1000). No pnpm, no compiler, no curl. |
| `manifests` | Collects the lockfile, workspace configuration, `.npmrc`, and the `package.json` of every platform package (not the module template's) at its repository path. |
| `build` | Refuses a context containing `modules/`, then `pnpm install --frozen-lockfile`, `pnpm boundaries`, `pnpm modules:validate`, `pnpm migrations:validate`, and `pnpm build`. Never uses host `node_modules`: `.dockerignore` excludes them. |
| `prod-deps` | The same lockfile with `--prod`, for the platform packages only: production dependencies only. |
| `runtime` | Production dependencies plus built outputs, owned by root and run as the unprivileged `node` user (uid 1000), and an empty, root-owned `/var/lib/ekklesia/modules`. No pnpm, no compiler, no curl. |
The runtime keeps the repository layout at `/opt/ekklesia`, because that layout is how the platform
finds its files:
- the API takes the repository root as three directories above `apps/api/dist/`, and reads
`package.json`, `packages/module-sdk/package.json`, and `modules/` beneath it;
`package.json` and `packages/module-sdk/package.json` beneath it;
- every CLI takes its **working directory** as the root and also reads Core's migrations in
`core/drizzle/`, so commands must run from `/opt/ekklesia`, the image's `WORKDIR`;
- the Dashboard and display host files are read from `surfaces/shell/dist/browser/`;
- each module directory is copied whole, minus `node_modules`, because each manifest names its
own entrypoints, UI artifacts, and migrations directory.
- installed modules are read from `EKKLESIA_MODULES_DIR`, `/var/lib/ekklesia/modules`, never from
under `/opt/ekklesia`. With nothing mounted there, the image runs with no modules.
### Building
@@ -405,30 +495,37 @@ set `DOCKER_BUILDKIT=1` if your Docker CLI has no buildx plugin.
No secret is a build argument, and none enters any layer. `.env` files are excluded from the build
context entirely, and no registry credentials are needed.
**A module missing a production dependency fails the build, not the API.** The runtime receives
production dependencies only, so a module that imports a package at run time but declares it only
as a development or peer dependency would build and then fail at startup. `pnpm boundaries` runs
in the build stage and rejects that module first
(`runtime-imports-are-production-dependencies`, see
[Repository boundaries](../docs/architecture/repository-boundaries.md)). The same applies to every
other boundary rule: a module the platform's own CI would reject cannot be built into an image.
### The runtime an installed module targets
**A platform-only image does not exercise native module dependencies.** With no modules in the
context, nothing like sharp is installed, so the CI image build proves nothing about them. An
instance build that includes such a module must smoke-test the built image itself: start it,
confirm `/api/v1/health/ready` answers `200`, and exercise the module's native code path (for
example, process one image).
An installed module runs on the image's Node.js, inside the API process, from its own directory
([ADR 0048](../docs/architecture/decisions/0048-generic-image-and-installed-module-storage.md)):
- **Node.js 24.18.0 on Debian 13, so glibc**, for the architecture the image was built for. A native
dependency such as sharp must be built or downloaded for exactly that.
- **Its own dependencies only.** Node resolves a module's imports from the module's directory, and
the platform's `node_modules` is never on that path. A module whose server code imports something
its directory does not contain stops the API from starting, with a sanitized
`MODULE_RUNTIME_IMPORT_FAILED`.
- **The platform's exact Drizzle version, for now.** Core runs a module's queries through its own
Drizzle instance over tables the module defined with its own copy, so the two copies must be
interchangeable. That is a constraint of the current implementation, not a permanent contract.
The repository's boundary checks (`pnpm boundaries`) prove only the source modules in a workspace.
An installed module's conformance is its own CI's job, so check a module before installing it:
`validate-modules.js` ([Operator commands](#operator-commands)) validates what can be checked on an
installed directory, and the API's startup refuses anything it cannot load.
### Configuration
Configuration comes **only from the container environment**, never from a file inside the image.
The image sets exactly three variables itself:
The image sets exactly four variables itself:
| Variable | Image value | Why |
| --- | --- | --- |
| `NODE_ENV` | `production` | Stated rather than relied on. Anything other than `development` already validates as production. |
| `EKKLESIA_API_HOST` | `0.0.0.0` | Reachable from other containers. Outside a container the platform default stays `127.0.0.1`. |
| `EKKLESIA_API_PORT` | `3000` | The platform default, made explicit; `EXPOSE 3000`. |
| `EKKLESIA_MODULES_DIR` | `/var/lib/ekklesia/modules` | Where installed modules are read from (ADR 0048). Outside a container, unset means the checkout's own `modules/`. Compose states the same value beside the mount. |
Everything else is the ordinary host configuration in
[API § Required configuration](../docs/development/api.md#required-configuration), supplied by the
@@ -449,8 +546,8 @@ Never publish the API's port to the host or the Internet. Only the proxy publish
- **Read-only:** the image was verified under exactly these flags. The API reached `healthy`, and
the migration and Doctor commands ran. Nothing wrote to `/tmp`, which stays mounted as headroom
for module code that needs scratch space. The installation, `/home/node` included, is not
writable, so anything that needs a writable path other than `/tmp` is a defect to report, not to
work around.
writable, and neither is module storage, which is mounted read-only. Anything that needs a
writable path other than `/tmp` is a defect to report, not to work around.
### Health
@@ -488,14 +585,16 @@ reports direct mode for an API that trusts its proxy. The running API's own valu
| Apply migrations | `docker compose run --rm migrate node core/dist/cli/apply-migrations.js` |
| Doctor (read-only report) | `docker compose run --rm migrate node core/dist/cli/doctor.js` |
| Create the initial owner (interactive) | `docker compose run --rm -it migrate node core/dist/cli/create-initial-owner.js --email owner@example.org` |
| Validate the installed modules (read-only) | `docker compose run --rm migrate node core/dist/cli/validate-modules.js` |
| Enable or disable a module | `docker compose run --rm migrate node core/dist/cli/set-module-enabled.js --enable --module <id>` (or `--disable`) |
The plan, apply, Doctor, and initial-owner commands are exercised by the smoke test; module
enablement is not.
Every command reads installed modules from the same `EKKLESIA_MODULES_DIR` the API does. The plan,
apply, Doctor, and initial-owner commands are exercised by the smoke test, and so are installing,
removing, and restoring a module; module validation and enablement are not.
### Size
The platform-only image is about 378 MB: the Debian slim Node base (about 230 MB), production
The image is about 378 MB, and it contains no module: the Debian slim Node base (about 230 MB), production
dependencies (about 141 MB), and built outputs (about 6 MB). A rebuild that reuses a warm pnpm store
cache can come out about 21 MB larger. When esbuild's install script runs, it hard-links its
`bin/esbuild` to the native binary it ships beside it; when pnpm restores esbuild from the store's