Ekklesia/deploy/README.md

# Deployment

This directory holds Ekklesia's reference self-hosted deployment
([ADR 0044](../docs/architecture/decisions/0044-self-hosted-reference-deployment.md)): one Linux
host, one Docker Compose project, and the production container image it runs. Nothing here is
specific to one organization.

| File | What it is |
| --- | --- |
| [`compose.yaml`](compose.yaml) | The base: `api`, `postgres`, `object-storage`, and the operator-only `migrate`. Publishes no port. |
| [`compose.proxy.yaml`](compose.proxy.yaml) | Overlay: Caddy as the TLS proxy, the only service with published ports. |
| [`compose.tunnel.yaml`](compose.tunnel.yaml) | Overlay on top of the proxy: a Cloudflare Tunnel connector. |
| [`Caddyfile.example`](Caddyfile.example) | The proxy's configuration for the default same-origin topology; copy to `Caddyfile`. |
| [`Caddyfile.split.example`](Caddyfile.split.example) | The proxy's configuration when the Dashboard and the API have separate hostnames ([Split Dashboard and API origins](#split-dashboard-and-api-origins)); copy to `Caddyfile` instead. |
| [`postgres/pg_hba.conf`](postgres/pg_hba.conf) | PostgreSQL client authentication: TLS only, SCRAM passwords. |
| [`env/`](env/) | Commented examples of every env file. No values. |
| [`scripts/init-tls.sh`](scripts/init-tls.sh) | Creates the installation-private CA and the `postgres` and `object-storage` server certificates. |
| [`scripts/gen-secrets.sh`](scripts/gen-secrets.sh) | Prints freshly generated secrets to paste into the env files. Writes nothing. |
| [`Dockerfile`](Dockerfile) | The one production image, for the API and every operator command ([The production image](#the-production-image)). |

## The reference deployment

### Shape

```sh
cd deploy
# Base and proxy: the reference.
docker compose -f compose.yaml -f compose.proxy.yaml up -d
# Base, proxy, and Cloudflare Tunnel.
docker compose -f compose.yaml -f compose.proxy.yaml -f compose.tunnel.yaml up -d
# Base only, behind a reverse proxy the installation already runs.
docker compose -f compose.yaml -f compose.myproxy.yaml up -d
```

Pass the same `-f` files to every later `up`, `down`, `ps`, and `logs`. `migrate` commands need only
the base: `docker compose run --rm migrate …` from this directory uses `compose.yaml` by default.

| Network | Kind | Members |
| --- | --- | --- |
| `data` | internal | `postgres`, `api`, `migrate` |
| `storage` | internal | `object-storage`, `api`, `migrate` |
| `egress` | bridge | `api`, `migrate` (outbound only, for example an SMTP relay) |
| `edge` | bridge, fixed subnet (proxy overlay) | `proxy` (fixed address), `api`, `object-storage`, `tunnel` (fixed address) |

PostgreSQL is only ever on `data`, so no proxy and no tunnel can reach it. `internal` networks have
no route off the host. Object storage joins `edge` only when the proxy overlay attaches it.

TLS is everywhere, without any exception in Core:

- **Browsers to the proxy:** publicly trusted ACME certificates.
- **API to PostgreSQL:** `verify-full` against the installation-private CA, passed as PEM text in
  `EKKLESIA_DATABASE_CA`. PostgreSQL accepts only TLS 1.3 network connections.
- **API (and proxy) to object storage:** HTTPS on port 8443 with a certificate from the same CA.
  Production storage must be https (ADR 0030). Core's S3 client uses Node's default trust store, so
  Compose adds the CA with `NODE_EXTRA_CA_CERTS`.

### Where configuration and secrets live

| File (on the host) | Holds | Read by |
| --- | --- | --- |
| `deploy/.env` (from [`env/compose.env.example`](env/compose.env.example)) | Non-secret settings: the image, hostnames, paths, the edge subnet | Compose itself |
| `/etc/ekklesia/ekklesia.env` | Everything the API needs, including `DATABASE_URL` and the S3 keys | `api`, `migrate` |
| `/etc/ekklesia/postgres.env` | `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB` | `postgres` |
| `/etc/ekklesia/object-storage.env` | The S3 key pair | `object-storage` |
| `/etc/ekklesia/tunnel.env` | `TUNNEL_TOKEN` | `tunnel` |
| `/etc/ekklesia/tls/` | The CA and server certificates and keys | mounted file by file |
| `/var/lib/ekklesia/modules/` | Installed modules ([Installed modules](#installed-modules)); not a secret | `api`, `migrate`, read-only |

The secret files are owned by root with mode 600, one per service, so no container receives another
service's credentials. None is committed, and `deploy/.env` is ignored by Git. `EKKLESIA_ENV_DIR`
and `EKKLESIA_TLS_DIR` in `deploy/.env` move the directories.

### First boot

1. **Get the image.** Build it ([Building](#building)) or pull it, and set `EKKLESIA_IMAGE` in
   `deploy/.env`.
2. **Create the TLS material**, as root:

   ```sh
   sudo sh deploy/scripts/init-tls.sh /etc/ekklesia/tls
   ```

   It creates the CA once and a certificate for each of `postgres` and `object-storage`, gives each
   key to its server's uid (PostgreSQL 70, SeaweedFS 1000) with mode 600, and prints the exact
   `EKKLESIA_DATABASE_CA="…"` line. It never overwrites an existing key or certificate and never
   prints a private key. `ca.key` is needed only to issue certificates: keep a copy offline.
3. **Generate secrets** with `sh deploy/scripts/gen-secrets.sh`, and paste each line into the file
   named above it.
4. **Write the env files** from [`env/`](env/), then install each one owned by root with mode 600:

   ```sh
   sudo install -d -m 755 /etc/ekklesia
   sudo install -o root -g root -m 600 ekklesia.env /etc/ekklesia/ekklesia.env
   ```

   `EKKLESIA_IDENTITY_BASE_URL` and `EKKLESIA_IDENTITY_TRUSTED_ORIGINS` are `https://` and your
   Ekklesia hostname. `S3_DELIVERY_ENDPOINT` is `https://` and your storage hostname.
5. **Create module storage**, empty, even if you install no module yet:

   ```sh
   sudo install -d -o root -g root -m 755 /var/lib/ekklesia/modules
   ```

   `api` and `migrate` refuse to start without it: Compose never creates it for you, so a mistyped
   path cannot quietly become an empty directory. `EKKLESIA_MODULES_HOST_DIR` in `deploy/.env`
   moves it.
6. **Configure the proxy:** copy `Caddyfile.example` to `Caddyfile`, and set `EKKLESIA_HOSTNAME`,
   `EKKLESIA_STORAGE_HOSTNAME`, and `EKKLESIA_ACME_EMAIL` in `deploy/.env`. Both hostnames must
   resolve to this host with ports 80 and 443 reachable for HTTP-01, or see [DNS-01](#dns-01).
7. **Start the data services, then migrate explicitly:**

   ```sh
   docker compose -f compose.yaml -f compose.proxy.yaml up -d --wait postgres object-storage
   docker compose run --rm migrate                                   # plan: read-only
   docker compose run --rm migrate node core/dist/cli/apply-migrations.js
   ```

   Nothing ever applies migrations automatically: the API never does, and `migrate` runs only when
   you run it.
8. **Start everything:** `docker compose -f compose.yaml -f compose.proxy.yaml up -d --wait`.
9. **Check it:** `curl https://<hostname>/api/v1/health/ready` answers `{"status":"ready"}`, and
   `docker compose run --rm migrate node core/dist/cli/doctor.js` reports `READY` (with warnings
   for anything optional you left unconfigured, such as email).

### The initial owner

```sh
docker compose run --rm -it migrate node core/dist/cli/create-initial-owner.js --email [email protected]
```

It prompts for the password with hidden input and never accepts it on the command line. The owner
is created already verified, so they can sign in before any email relay exists
([ADR 0035](../docs/architecture/decisions/0035-initial-platform-owner-bootstrap.md)). To supply the
password from a secret manager instead, pipe it on standard input and use `-T` instead of `-it`.

### Installed modules

The image contains no modules
([ADR 0048](../docs/architecture/decisions/0048-generic-image-and-installed-module-storage.md)).
Installed modules live on the host, in module storage: `/var/lib/ekklesia/modules`, or
`EKKLESIA_MODULES_HOST_DIR`. Compose mounts it **read-only** at `/var/lib/ekklesia/modules` in `api`
and `migrate`, which is where `EKKLESIA_MODULES_DIR` points. Ekklesia never writes there.
Installing, updating, or removing a module never needs a new image, only a restart.

**What belongs there.** One directory per module, each extracted from that module's **Ekklesia
Module Package**
([ADR 0049](../docs/architecture/decisions/0049-module-package-v1.md)): one archive,
`<id>-<version>-<target>.tar.gz`, and the SHA-256 published beside it. A package holds the module's
generated `package.json`, its manifest, `ekklesia.package.json`, bundled backend code, built browser
artifacts, migrations, and `THIRD_PARTY_NOTICES`
([Module development](../docs/module-development/README.md#an-installed-module-is-self-contained)).
Copying a module's `dist/` alone is not enough. Files are owned by root and readable by everyone
(directories 755, files 644), so the API's unprivileged user can read them and never change them.
The digest proves the archive is the one you were given; it does not prove who published it.

**Nothing else belongs there.** Every non-hidden directory must be a valid module, or the API does
not start. That includes `lost+found`, so if module storage is a file system of its own, mount it
one level up and point `EKKLESIA_MODULES_HOST_DIR` at a directory inside it. Hidden entries are
ignored, which is what makes staging possible.

**Changes take effect only at a restart.** A running API never notices module storage changing, and
never loads or unloads a module while it runs. Change module storage only while the API is stopped:
it reads browser artifacts from disk as it serves them, so files changed under a running API would
serve one version's browser code beside another's server code.

To **install or update** a module, with `<id>` its module id and the package and its `.sha256` in
the current directory, first **verify** it with the image's own verifier. It checks the archive's
digest, its safety (no links, special files, or escaping paths), every file's digest, the package's
identity, target, and Node.js range against this platform, runs the platform's module and migration
validation, and loads the backend once in a short-lived process with no environment. Only then does
it write the verified module directory, to a new directory outside module storage:

The backend probe executes the package's server-side code; the verifier container is its isolation
boundary. Keep the general-purpose `/tmp` mount `noexec`, and give only this one-shot verifier a
separate executable scratch tmpfs. The mount below is ephemeral, bounded at 1100 MiB (just above the
package format's 1 GiB expanded limit), `nodev`, `nosuid`, and writable only by the image's
unprivileged `node` user. `EKKLESIA_MODULE_VERIFIER_SCRATCH_DIR` tells the CLI where that mount is;
the verifier creates a unique child for each run and removes it on success or failure.

```sh
mkdir -m 777 staging        # writable by the image's unprivileged user; nothing else is
docker run --rm --network none --read-only \
  --tmpfs /tmp:rw,nosuid,nodev,noexec,size=64m \
  --tmpfs /run/ekklesia-verifier:rw,nosuid,nodev,exec,size=1100m,mode=0700,uid=1000,gid=1000 \
  --cap-drop ALL --security-opt no-new-privileges \
  -e EKKLESIA_MODULE_VERIFIER_SCRATCH_DIR=/run/ekklesia-verifier \
  -v "$PWD:/package:ro" -v "$PWD/staging:/staging" \
  --entrypoint node "$EKKLESIA_IMAGE" core/dist/cli/verify-module-package.js \
  /package/<id>-<version>-any.tar.gz --sha256-file /package/<id>-<version>-any.tar.gz.sha256 \
  --extract-to /staging/<id>
```

Then install the verified directory:

```sh
docker compose stop api
# Stage under a hidden name, which discovery ignores, then rename it into place in one step.
sudo cp -R staging/<id> /var/lib/ekklesia/modules/.incoming-<id>
sudo chown -R root:root /var/lib/ekklesia/modules/.incoming-<id>
sudo chmod -R a+rX,go-w /var/lib/ekklesia/modules/.incoming-<id>
sudo mv /var/lib/ekklesia/modules/<id> /var/lib/ekklesia/modules/.previous-<id>   # update only
sudo mv /var/lib/ekklesia/modules/.incoming-<id> /var/lib/ekklesia/modules/<id>
docker compose run --rm migrate node core/dist/cli/validate-modules.js
docker compose run --rm migrate                                         # plan: review it
docker compose run --rm migrate node core/dist/cli/apply-migrations.js  # only when the plan is right
docker compose up -d --wait api
```

Back up before an update that brings migrations: migrations are forward-only. A newly installed
module is enabled unless an administrator disabled it before; set its configuration and grants as
its documentation says. Keep `.previous-<id>` until the update has proven itself, then delete it.

To **remove** a module, stop the API, delete (or move away) its directory, and start the API again.
Remove first any module that *requires* it, or the API refuses to start. Removing a module's code
deletes nothing else. Its tables and rows, its buckets and objects, its enablement, configuration,
Dashboard placement, and grants all stay. Its applied migration history stays too, and the
migration plan and Doctor report it as `retained` history for a module that is not installed. That
is information, not a problem, and it blocks no other migration. To stop a module without removing
it, disable it instead (`set-module-enabled.js --disable`, [Operator commands](#operator-commands)).
There is no purge: deleting a module's data is a separate operation that does not exist yet.

To **restore** a removed module, verify and install its package again as above, at the same version
or a newer one. It
finds its data, its settings, and its history where it left them. Its history is strict again: a
package that no longer contains a migration that was applied, or contains one with different
contents, is refused by the migration plan, so an older version cannot be installed over newer data.

Module storage is installation state: back it up with the database and the buckets.

### Upgrading

1. **Pull or build the new image**, and change `EKKLESIA_IMAGE` in `deploy/.env`. Keep the old tag.
   Installed modules stay where they are, in module storage.
2. **Back up** ([Backups](#backups)).
3. **Check the installed modules against the new platform:**
   `docker compose run --rm migrate node core/dist/cli/validate-modules.js`. A module whose
   `compatibility.platform` range excludes the new version fails here, and would stop the API from
   starting: update it first, or stay on the old image.
4. **Plan, review, then apply:** `docker compose run --rm migrate` with the new image, then
   `docker compose run --rm migrate node core/dist/cli/apply-migrations.js`.
5. **Restart on the new image:** `docker compose -f compose.yaml -f compose.proxy.yaml up -d --wait`.

### Rolling back

Set `EKKLESIA_IMAGE` back to the previous tag and run `up -d --wait` again. That is enough only when
the upgrade applied no migrations (the plan in step 4 showed none pending). Migrations are
forward-only. If any were applied, restore the backup taken in step 2 before starting the previous
image, because it was not built for the newer schema. Module storage is not part of the image, so a
rollback does not change installed modules.

### Backups

Slice 6 adds the scheduled backup job and the restore procedure (ADR 0044, Backups). Until then,
take a `pg_dump`, a copy of every bucket, and a copy of module storage before each upgrade by hand.

### Split Dashboard and API origins

By default one hostname serves the Dashboard (`/`), the API (`/api/*`), and displays (`/display/*`).
An installation may instead give the Dashboard its own hostname, for example `dash.example.org`
beside `api.example.org` ([ADR 0045](../docs/architecture/decisions/0045-split-dashboard-and-api-origins.md)).
The same image serves both; only configuration differs.

| Hostname | Serves | Refuses (`404`) |
| --- | --- | --- |
| Dashboard (`EKKLESIA_DASHBOARD_HOSTNAME`) | `/` and `/assets/ekklesia-dashboard/*` | everything else, including `/api/*` |
| API (`EKKLESIA_HOSTNAME`) | `/api/*`, `/display/*`, `/assets/ekklesia-display/*` | everything else, including `/` |
| Storage (`EKKLESIA_STORAGE_HOSTNAME`) | signed object URLs, unchanged | |

Displays live on the API's hostname (`https://api.example.org/display/<module>`), because their
device cookies are bound to it and their offline cache keeps only same-origin responses. That
hostname is what every screen and paired device depends on: choose it once.

To run split:

1. Copy `Caddyfile.split.example` to `Caddyfile` instead of `Caddyfile.example`.
2. In `deploy/.env`, set `EKKLESIA_HOSTNAME` to the API's hostname and `EKKLESIA_DASHBOARD_HOSTNAME`
   to the Dashboard's.
3. In `ekklesia.env`, set `EKKLESIA_IDENTITY_BASE_URL` to `https://` and the API's hostname,
   `EKKLESIA_DASHBOARD_ORIGIN` to `https://` and the Dashboard's, and
   `EKKLESIA_IDENTITY_TRUSTED_ORIGINS` to the Dashboard's origin. The API always trusts its own
   origin, so it need not be listed.

**Both hostnames must belong to one registrable domain** (`dash.example.org` and `api.example.org`,
not `dash.example.org` and `api.example.net`). The session cookie stays host-only on the API's
hostname; browsers send it from the Dashboard only between origins of the same site. Doctor reports
the topology and warns when the two hostnames are not siblings.

The Dashboard learns the API's origin at run time: the API writes its identity base URL into the page
it serves (`<meta name="ekklesia-api-origin">`). Nothing about the topology is built into the image.

### Storage delivery: CORS is the deployment's

Browsers fetch objects directly from the storage hostname through signed, expiring URLs (ADR 0031).
**Which pages may read those bytes with `fetch()` is decided by the delivery layer, the proxy, and
never by the object store.** That policy therefore stays the same with SeaweedFS, R2, or any other
S3 backend behind it, and an installation that changes backends changes nothing about CORS.

The reference proxy's storage block:

- grants CORS only to the origins whose pages read objects with `fetch()`: the Ekklesia origin in the
  same-origin topology; the API origin (displays) and the Dashboard origin (a module's preview) in
  the split topology. An `<img>` needs no CORS and is unaffected;
- allows `GET` and `HEAD` only, and no request header beyond the CORS-safelisted ones, always with
  `Vary: Origin`;
- answers preflight `OPTIONS` requests itself, so they never reach storage;
- removes the object store's own CORS headers. SeaweedFS's default reflects any origin and
  advertises `PUT`, `POST`, and `DELETE`.

The proxy reaches storage over **HTTP/1.1 only**. Over HTTP/2, SeaweedFS answers a conditional
request with a `304` that declares a `Content-Length` it never sends. The proxy then resets the
browser's stream, and every reload of a cached image fails with `ERR_HTTP2_PROTOCOL_ERROR`.
Browsers still reach the proxy itself over HTTP/2 and HTTP/3.

### Behind an existing reverse proxy

Run the base alone and attach `api` and `object-storage` to your proxy's network with an overlay of
your own, for example `compose.myproxy.yaml`:

```yaml
services:
  api:
    networks:
      proxy: {}
    environment:
      # Exactly one trusted peer: your proxy, at an address that cannot change.
      EKKLESIA_CLIENT_ADDRESS_MODE: trusted-proxy
      EKKLESIA_CLIENT_ADDRESS_HEADER: x-forwarded-for
      EKKLESIA_CLIENT_ADDRESS_TRUSTED_PROXIES: 10.0.100.254/32
  object-storage:
    networks:
      proxy: {}

networks:
  proxy:
    external: true
    name: your-proxy-network
```

Then make your proxy do what `Caddyfile.example` does:

- forward `/`, `/api/*`, `/display/*`, and `/assets/*` on the Ekklesia hostname to `api:3000`;
- **overwrite** `X-Forwarded-For` with the one client address it resolved, and strip
  `CF-Connecting-IP`, `X-Real-IP`, `Forwarded`, and `True-Client-IP`;
- stream responses on `/api/v1/module-realtime/*` without buffering (Server-Sent Events);
- set no request-body limit below 32 MiB; the API enforces its own ceilings;
- forward the storage hostname to `https://object-storage:8443`, verifying it against
  `/etc/ekklesia/tls/ca.crt` with server name `object-storage`, and **pass the original `Host`
  through unchanged**, or every signed URL fails (ADR 0031).
- reach `object-storage` over **HTTP/1.1** only, and own the storage hostname's CORS as described in
  [Storage delivery](#storage-delivery-cors-is-the-deployments).

The proxy's address must be fixed and must never be handed to anything else. With Docker, give its
network an explicit subnet, limit `ip_range` to part of it, and fix the proxy's address outside that
range, as `compose.proxy.yaml` does.

### Fixed addresses on `edge`

`compose.proxy.yaml` gives `edge` the subnet `10.249.250.0/24` and lets Docker assign dynamic
addresses only from its lower half (`10.249.250.0/25`). The proxy (`.254`) and the tunnel connector
(`.253`) take fixed addresses from the top of the upper half, which Docker never assigns to anything
else. That is what makes "only the proxy" and "only the connector" expressible as `/32` trust
entries. If the subnet collides with one on your host, change `EKKLESIA_EDGE_SUBNET`,
`EKKLESIA_EDGE_DYNAMIC_RANGE`, `EKKLESIA_PROXY_ADDRESS`, and `EKKLESIA_TUNNEL_ADDRESS` together,
keeping the fixed addresses at the top and outside the dynamic range.

### Cloudflare Tunnel

`compose.tunnel.yaml` runs `cloudflared` with a token, so the tunnel is managed in the Cloudflare
dashboard. No Cloudflare API is called from here.

1. In Cloudflare Zero Trust, under **Networks → Tunnels**, create a Cloudflared tunnel. Copy its
   token into `/etc/ekklesia/tunnel.env` as `TUNNEL_TOKEN`.
2. Add a **public hostname** (shown as a *published application route* in newer dashboards) for
   the Ekklesia hostname. Service type **HTTPS**, URL `proxy:443`.
3. Under that hostname's **TLS** settings, set **Origin Server Name** to the same Ekklesia hostname,
   and leave **No TLS Verify** off. The connector then verifies the proxy's certificate for that name.
4. Add a public hostname for the storage hostname, the same way, with its own origin server name.
   In the [split topology](#split-dashboard-and-api-origins), add one for the Dashboard's hostname
   too: three in all, each with its own origin server name.
5. Start with all three files. The proxy honors `CF-Connecting-IP` only from the connector's fixed
   address. From any other peer it is stripped.

The proxy's certificate must still be publicly trusted, and an HTTP-01 challenge cannot reach the
proxy through a tunnel, because Cloudflare terminates TLS at its edge. A tunnel-only installation
therefore uses [DNS-01](#dns-01). An installation that also serves local displays should give the
same hostnames a LAN path to the proxy (split-horizon DNS), so displays do not depend on the
Internet link.

### DNS-01

For an installation off the public Internet, LAN-only, or tunnel-only. DNS-01 needs a DNS provider
module, which the stock Caddy image does not contain:

1. Build a Caddy image with the module (for example
   `xcaddy build --with github.com/caddy-dns/cloudflare`) and pin it by digest.
2. Override the proxy's `image:` in an overlay of your own. Give it `CLOUDFLARE_API_TOKEN` (a token
   scoped to DNS edits for the zone) through a root-owned env file.
3. Enable the commented `acme_dns` line in your `Caddyfile`.

### Launch checklist

The ten items in [Security § launch checklist](../docs/development/security.md) mapped to this
deployment:

| # | Item | Where it is met |
| --- | --- | --- |
| 1 | The proxy forwards the Dashboard and `/api` on the intended origin | `Caddyfile.example` routes `/`, `/api/*`, `/display/*`, `/assets/*` on one hostname to `api`; the API serves the Dashboard itself. In the split topology, `Caddyfile.split.example` gives each hostname exactly its own paths and the exact trusted-origin CORS of ADR 0045 covers the rest. |
| 2 | The origin port is unreachable from the Internet | The base publishes no port; only `proxy` publishes 80 and 443. |
| 3 | The origin port is unreachable from untrusted networks | `api` is reachable only on Docker networks; `EKKLESIA_PUBLISH_ADDRESS` limits the proxy to one interface. Firewall the host itself. |
| 4 | The proxy overwrites the trusted client-IP header | `header_up X-Forwarded-For {client_ip}` in `Caddyfile.example`. |
| 5 | Rotating forwarding headers cannot rotate the rate-limit bucket | The same, plus stripping `CF-Connecting-IP`, `X-Real-IP`, `Forwarded`, and `True-Client-IP`; the API overwrites its internal header. Proven by the smoke test. |
| 6 | The proxy peer is the only trusted proxy | `compose.proxy.yaml` sets `EKKLESIA_CLIENT_ADDRESS_TRUSTED_PROXIES` to the proxy's fixed `/32`. |
| 7 | Proxy logs record no cookies, credentials, bodies, or forwarding chains | Caddy access logs are off in `Caddyfile.example`. If you enable them, exclude request headers. |
| 8 | PostgreSQL only from the API, over verified TLS | `postgres` is only on internal `data`; `pg_hba.conf` rejects non-TLS; `verify-full` with `EKKLESIA_DATABASE_CA`. Proven by the smoke test. |
| 9 | The deployed login and logout flow keeps its contract | Unchanged by the proxy; the smoke test signs in and out through it. |
| 10 | A supported way to create the first production identity | `create-initial-owner.js` through `migrate` ([The initial owner](#the-initial-owner)). Exercised by the smoke test. |

### Testing this deployment

- `pnpm deploy:config` validates the base, base + proxy, and base + proxy + tunnel with the example
  env files. CI runs it in `pnpm ci:verify`.
- `bash scripts/check-image.sh` builds the image and proves it is generic: no `/opt/ekklesia/modules`,
  no module template, no `ekklesia.module.json` outside installed dependencies, an empty root-owned
  `/var/lib/ekklesia/modules`, and `EKKLESIA_MODULES_DIR` set. It then builds again from a checkout
  with a populated `modules/` directory and requires an identical runtime file system. CI runs it.
- `pnpm deploy:smoke` (`scripts/smoke-deploy.sh`) runs the whole deployment end to end, against the
  generic image, empty module storage, and a disposable PostgreSQL and SeaweedFS that exist only for
  that run. It follows the first boot above and checks the following through the proxy:
  - readiness and the Dashboard;
  - a signed object URL fetched through the storage hostname, and the same image loaded twice over
    HTTP/2, the second time as a conditional request that must be a clean `304`;
  - storage CORS: exactly the allowed origins, `GET` and `HEAD` only, preflights answered by the
    proxy, and none of the object store's own CORS headers;
  - the client-address chain, both from an ordinary peer and from the tunnel connector's address;
  - initial-owner sign-in and sign-out;
  - that the API's database connection is TLS 1.3, and that it is refused by a foreign CA and by a
    wrong host name;
  - Doctor.

  `EKKLESIA_SMOKE_TOPOLOGY=split` runs the [split topology](#split-dashboard-and-api-origins)
  instead, on the **same generic image**, and also proves [Installed modules](#installed-modules)
  end to end:
  - the image boots with no module;
  - the repository's module template, packaged as an Ekklesia Module Package v1 by the canonical
    packager, is verified by the image itself and installed into module storage, its migration is
    planned and explicitly applied, the API restarts, and the module reads data it owns;
  - every hostname's served and refused paths, exact CORS for the Dashboard's origin and none for
    others;
  - a real headless Chromium (pinned by digest, driven over the DevTools protocol) signs in on the
    Dashboard's hostname, checks the session cookie lives only on the API's, mounts the module's UI
    from the API's origin and sees its own credentialed requests succeed, mounts its display on the
    API's hostname with its service worker, and signs out;
  - the module is removed and the API restarted: Core still runs, the module's routes are gone,
    migration planning and applying succeed and report its retained history, and its data is intact;
  - the exact same package is verified and installed again and the API restarted: its history
    validates strictly and it returns its previous data.

  CI runs both topologies on every change.
- **Test-only:** the smoke test runs Caddy with its internal CA (`local_certs`), because a test run
  has no public hostname. Never use it in an installation; `Caddyfile.example` does not contain it.

The smoke test uses no module, so it does not exercise realtime streams or large uploads. Through
this same proxy configuration, a scratch module confirmed both:

- a Server-Sent Event crossed the proxy in milliseconds;
- a 32 MiB streamed upload passed with and without `Content-Length`;
- one byte more was refused by the API's own `413 REQUEST_BODY_TOO_LARGE`.

## The production image

[`Dockerfile`](Dockerfile) builds **one image** for two jobs: the long-running `api` service, and
every operator-run command (migrations, Doctor, initial-owner creation, module enablement). There
is no second image and no second build.

### What it builds

The build context is **this repository**, and the image is the same for every installation
([ADR 0048](../docs/architecture/decisions/0048-generic-image-and-installed-module-storage.md)). It
contains **zero feature modules**: installed modules live in module storage outside it
([Installed modules](#installed-modules)). No input is specific to an installation: no module list,
no lockfile of its own, no branding, no configuration.

A `modules/` directory in the checkout never reaches the image. `.dockerignore` keeps it out of the
build context, and the build refuses a context that still contains one. The module template is a
workspace package, so it is built and validated with everything else, but neither its files nor its
dependencies reach the runtime image. `scripts/check-image.sh` proves all of this on every change
([Testing this deployment](#testing-this-deployment)).

| Stage | Does |
| --- | --- |
| `base` | `node:24.18.0-trixie-slim`, pinned by its multi-architecture index digest. Debian, so glibc, because module dependencies such as sharp ship glibc prebuilt binaries. Every stage starts here. |
| `toolchain` | Enables pnpm through Corepack, at exactly the root `packageManager` version. |
| `manifests` | Collects the lockfile, workspace configuration, `.npmrc`, and the `package.json` of every platform package (not the module template's) at its repository path. |
| `build` | Refuses a context containing `modules/`, then `pnpm install --frozen-lockfile`, `pnpm boundaries`, `pnpm modules:validate`, `pnpm migrations:validate`, and `pnpm build`. Never uses host `node_modules`: `.dockerignore` excludes them. |
| `prod-deps` | The same lockfile with `--prod`, for the platform packages only: production dependencies only. |
| `runtime` | Production dependencies plus built outputs, owned by root and run as the unprivileged `node` user (uid 1000), and an empty, root-owned `/var/lib/ekklesia/modules`. No pnpm, no compiler, no curl. |

The runtime keeps the repository layout at `/opt/ekklesia`, because that layout is how the platform
finds its files:

- the API takes the repository root as three directories above `apps/api/dist/`, and reads
  `package.json` and `packages/module-sdk/package.json` beneath it;
- every CLI takes its **working directory** as the root and also reads Core's migrations in
  `core/drizzle/`, so commands must run from `/opt/ekklesia`, the image's `WORKDIR`;
- the Dashboard and display host files are read from `surfaces/shell/dist/browser/`;
- installed modules are read from `EKKLESIA_MODULES_DIR`, `/var/lib/ekklesia/modules`, never from
  under `/opt/ekklesia`. With nothing mounted there, the image runs with no modules.

### Building

```sh
docker build \
  --file deploy/Dockerfile \
  --build-arg VERSION=0.2.4 \
  --build-arg REVISION="$(git rev-parse HEAD)" \
  --build-arg SOURCE=https://git.example.org/your-org/ekklesia \
  --tag ekklesia:0.2.4 \
  .
```

The three build arguments become the OCI labels `org.opencontainers.image.version`, `.revision`,
and `.source`. They have no defaults: the repository cannot know an installation's Git host. The
image also records its base image name and digest. BuildKit is required (Docker's default builder);
set `DOCKER_BUILDKIT=1` if your Docker CLI has no buildx plugin.

No secret is a build argument, and none enters any layer. `.env` files are excluded from the build
context entirely, and no registry credentials are needed.

### The runtime an installed module targets

An installed module runs on the image's Node.js, inside the API process, from its own directory
([ADR 0048](../docs/architecture/decisions/0048-generic-image-and-installed-module-storage.md)):

- **Node.js 24.18.0 on Debian 13, so glibc**, for the architecture the image was built for. A native
  dependency such as sharp must be built or downloaded for exactly that.
- **Its own dependencies only.** Node resolves a module's imports from the module's directory, and
  the platform's `node_modules` is never on that path. A module whose server code imports something
  its directory does not contain stops the API from starting, with a sanitized
  `MODULE_RUNTIME_IMPORT_FAILED`.
- **The platform's exact Drizzle version, for now.** Core runs a module's queries through its own
  Drizzle instance over tables the module defined with its own copy, so the two copies must be
  interchangeable. That is a constraint of the current implementation, not a permanent contract.

The repository's boundary checks (`pnpm boundaries`) prove only the source modules in a workspace.
An installed module's conformance is its own CI's job, so check a module before installing it:
`validate-modules.js` ([Operator commands](#operator-commands)) validates what can be checked on an
installed directory, and the API's startup refuses anything it cannot load.

### Configuration

Configuration comes **only from the container environment**, never from a file inside the image.
The image sets exactly four variables itself:

| Variable | Image value | Why |
| --- | --- | --- |
| `NODE_ENV` | `production` | Stated rather than relied on. Anything other than `development` already validates as production. |
| `EKKLESIA_API_HOST` | `0.0.0.0` | Reachable from other containers. Outside a container the platform default stays `127.0.0.1`. |
| `EKKLESIA_API_PORT` | `3000` | The platform default, made explicit; `EXPOSE 3000`. |
| `EKKLESIA_MODULES_DIR` | `/var/lib/ekklesia/modules` | Where installed modules are read from (ADR 0048). Outside a container, unset means the checkout's own `modules/`. Compose states the same value beside the mount. |

Everything else is the ordinary host configuration in
[API § Required configuration](../docs/development/api.md#required-configuration), supplied by the
reference deployment's `ekklesia.env`. `EKKLESIA_DATABASE_CA` is the CA's PEM text itself, on one
double-quoted line with `\n` escapes. Compose expands those escapes; plain `docker run --env-file`
does not, which is why every command here goes through Compose.

Never publish the API's port to the host or the Internet. Only the proxy publishes ports.

### Process, signals, and the read-only posture

`compose.yaml` runs `api` and `migrate` with `init: true`, `read_only: true`, `tmpfs: /tmp`,
`cap_drop: [ALL]`, and `no-new-privileges`. That is the supported production posture:

- **`init: true`** runs a minimal init as PID 1 that forwards signals and reaps children, so
  `docker stop` reaches the API's own SIGTERM handler promptly. The API then drains in-flight
  requests, stops modules in reverse order, and closes its pool.
- **Read-only:** the image was verified under exactly these flags. The API reached `healthy`, and
  the migration and Doctor commands ran. Nothing wrote to `/tmp`, which stays mounted as headroom
  for module code that needs scratch space. The installation, `/home/node` included, is not
  writable, and neither is module storage, which is mounted read-only. Anything that needs a
  writable path other than `/tmp` is a defect to report, not to work around.

### Health

The image's `HEALTHCHECK` calls `GET /api/v1/health/ready` on `127.0.0.1` with Node's built-in
`fetch`, every 30 seconds after a 30-second start period, with three retries. Readiness means the
database answered within 2 seconds and the instance is not shutting down; see
[API § Health](../docs/development/api.md#health). Because it probes `127.0.0.1`, keep
`EKKLESIA_API_HOST` at `0.0.0.0` (or any address that includes loopback) inside the container. The
API trusts only the proxy's address for client-address headers, so the loopback probe is an
ordinary direct peer.

An external monitor can use the same endpoint through the proxy. Use `/api/v1/health/live` to ask
only whether the process is serving.

### Operator commands

The same image runs every CLI, directly with `node core/dist/cli/<command>.js`, through the
`migrate` service. The `pnpm` scripts of the same names rebuild first and read `.env`, so they are
for a development checkout, not for the image.

`migrate` has `profiles: ["ops"]`, so `up` never starts it. It never restarts, has no health check,
and its default command is the read-only plan. **Nothing applies migrations automatically.**

Run operator commands with the same Compose files as the stack. `COMPOSE_FILE` in `deploy/.env`
(set in [`env/compose.env.example`](env/compose.env.example)) does that for every plain
`docker compose` command below. It matters for Doctor in particular: Doctor checks the
client-address configuration of the process it runs in. The proxy overlay gives `migrate` the
API's trusted-proxy settings, so Doctor reports what the API runs. Without the overlay, Doctor
reports direct mode for an API that trusts its proxy. The running API's own value is shown by
`docker compose exec api printenv EKKLESIA_CLIENT_ADDRESS_MODE`.

| Task | Command (from `deploy/`) |
| --- | --- |
| Plan migrations (read-only) | `docker compose run --rm migrate` |
| Apply migrations | `docker compose run --rm migrate node core/dist/cli/apply-migrations.js` |
| Doctor (read-only report) | `docker compose run --rm migrate node core/dist/cli/doctor.js` |
| Create the initial owner (interactive) | `docker compose run --rm -it migrate node core/dist/cli/create-initial-owner.js --email owner@example.org` |
| Validate the installed modules (read-only) | `docker compose run --rm migrate node core/dist/cli/validate-modules.js` |
| Verify a module package (read-only unless extracting) | `node core/dist/cli/verify-module-package.js <archive> --sha256-file <archive>.sha256 [--extract-to <new directory>]`, in a `docker run` of the image ([Installed modules](#installed-modules)) |
| Enable or disable a module | `docker compose run --rm migrate node core/dist/cli/set-module-enabled.js --enable --module <id>` (or `--disable`) |

Every command reads installed modules from the same `EKKLESIA_MODULES_DIR` the API does. The plan,
apply, Doctor, and initial-owner commands are exercised by the smoke test, and so are installing,
removing, and restoring a module; module validation and enablement are not.

### Size

The image is about 378 MB, and it contains no module: the Debian slim Node base (about 230 MB), production
dependencies (about 141 MB), and built outputs (about 6 MB). A rebuild that reuses a warm pnpm store
cache can come out about 21 MB larger. When esbuild's install script runs, it hard-links its
`bin/esbuild` to the native binary it ships beside it; when pnpm restores esbuild from the store's
cached post-install state instead, that file is a second full copy. Both are esbuild, one of the
unused tools below.

About 57 MB of the production dependencies are development tools that no runtime code imports:
`vitest`, `vite`, `drizzle-kit`, `tsx`, `esbuild`, and their native bindings. They arrive as
optional peer dependencies of Better Auth (and `esbuild` through the module UI builder), which pnpm
resolves because the workspace already has them. Excluding them means changing dependency
resolution and the lockfile, which is separate work.