# Deployment

This directory holds Ekklesia's reference self-hosted deployment
([ADR 0044](../docs/architecture/decisions/0044-self-hosted-reference-deployment.md)): one Linux
host, one Docker Compose project, and the production container image it runs. Nothing here is
specific to one organization.

| File | What it is |
| --- | --- |
| [`compose.yaml`](compose.yaml) | The base: `api`, `postgres`, `object-storage`, and the operator-only `migrate`. Publishes no port. |
| [`compose.proxy.yaml`](compose.proxy.yaml) | Overlay: Caddy as the TLS proxy, the only service with published ports. |
| [`compose.tunnel.yaml`](compose.tunnel.yaml) | Overlay on top of the proxy: a Cloudflare Tunnel connector. |
| [`Caddyfile.example`](Caddyfile.example) | The proxy's configuration; copy to `Caddyfile`. |
| [`postgres/pg_hba.conf`](postgres/pg_hba.conf) | PostgreSQL client authentication: TLS only, SCRAM passwords. |
| [`env/`](env/) | Commented examples of every env file. No values. |
| [`scripts/init-tls.sh`](scripts/init-tls.sh) | Creates the installation-private CA and the `postgres` and `object-storage` server certificates. |
| [`scripts/gen-secrets.sh`](scripts/gen-secrets.sh) | Prints freshly generated secrets to paste into the env files. Writes nothing. |
| [`Dockerfile`](Dockerfile) | The one production image, for the API and every operator command ([The production image](#the-production-image)). |

## The reference deployment

### Shape

```sh
cd deploy
# Base and proxy: the reference.
docker compose -f compose.yaml -f compose.proxy.yaml up -d
# Base, proxy, and Cloudflare Tunnel.
docker compose -f compose.yaml -f compose.proxy.yaml -f compose.tunnel.yaml up -d
# Base only, behind a reverse proxy the installation already runs.
docker compose -f compose.yaml -f compose.myproxy.yaml up -d
```

Pass the same `-f` files to every later `up`, `down`, `ps`, and `logs`. `migrate` commands need only
the base: `docker compose run --rm migrate …` from this directory uses `compose.yaml` by default.

| Network | Kind | Members |
| --- | --- | --- |
| `data` | internal | `postgres`, `api`, `migrate` |
| `storage` | internal | `object-storage`, `api`, `migrate` |
| `egress` | bridge | `api`, `migrate` (outbound only, for example an SMTP relay) |
| `edge` | bridge, fixed subnet (proxy overlay) | `proxy` (fixed address), `api`, `object-storage`, `tunnel` (fixed address) |

PostgreSQL is only ever on `data`, so no proxy and no tunnel can reach it. `internal` networks have
no route off the host. Object storage joins `edge` only when the proxy overlay attaches it.

TLS is everywhere, without any exception in Core:

- **Browsers to the proxy:** publicly trusted ACME certificates.
- **API to PostgreSQL:** `verify-full` against the installation-private CA, passed as PEM text in
  `EKKLESIA_DATABASE_CA`. PostgreSQL accepts only TLS 1.3 network connections.
- **API (and proxy) to object storage:** HTTPS on port 8443 with a certificate from the same CA.
  Production storage must be https (ADR 0030). Core's S3 client uses Node's default trust store, so
  Compose adds the CA with `NODE_EXTRA_CA_CERTS`.

### Where configuration and secrets live

| File (on the host) | Holds | Read by |
| --- | --- | --- |
| `deploy/.env` (from [`env/compose.env.example`](env/compose.env.example)) | Non-secret settings: the image, hostnames, paths, the edge subnet | Compose itself |
| `/etc/ekklesia/ekklesia.env` | Everything the API needs, including `DATABASE_URL` and the S3 keys | `api`, `migrate` |
| `/etc/ekklesia/postgres.env` | `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB` | `postgres` |
| `/etc/ekklesia/object-storage.env` | The S3 key pair | `object-storage` |
| `/etc/ekklesia/tunnel.env` | `TUNNEL_TOKEN` | `tunnel` |
| `/etc/ekklesia/tls/` | The CA and server certificates and keys | mounted file by file |

The secret files are owned by root with mode 600, one per service, so no container receives another
service's credentials. None is committed, and `deploy/.env` is ignored by Git. `EKKLESIA_ENV_DIR`
and `EKKLESIA_TLS_DIR` in `deploy/.env` move the directories.

### First boot

1. **Get the image.** Build it ([Building](#building)) or pull it, and set `EKKLESIA_IMAGE` in
   `deploy/.env`.
2. **Create the TLS material**, as root:

   ```sh
   sudo sh deploy/scripts/init-tls.sh /etc/ekklesia/tls
   ```

   It creates the CA once and a certificate for each of `postgres` and `object-storage`, gives each
   key to its server's uid (PostgreSQL 70, SeaweedFS 1000) with mode 600, and prints the exact
   `EKKLESIA_DATABASE_CA="…"` line. It never overwrites an existing key or certificate and never
   prints a private key. `ca.key` is needed only to issue certificates: keep a copy offline.
3. **Generate secrets** with `sh deploy/scripts/gen-secrets.sh`, and paste each line into the file
   named above it.
4. **Write the env files** from [`env/`](env/), then install each one owned by root with mode 600:

   ```sh
   sudo install -d -m 755 /etc/ekklesia
   sudo install -o root -g root -m 600 ekklesia.env /etc/ekklesia/ekklesia.env
   ```

   `EKKLESIA_IDENTITY_BASE_URL` and `EKKLESIA_IDENTITY_TRUSTED_ORIGINS` are `https://` and your
   Ekklesia hostname. `S3_DELIVERY_ENDPOINT` is `https://` and your storage hostname.
5. **Configure the proxy:** copy `Caddyfile.example` to `Caddyfile`, and set `EKKLESIA_HOSTNAME`,
   `EKKLESIA_STORAGE_HOSTNAME`, and `EKKLESIA_ACME_EMAIL` in `deploy/.env`. Both hostnames must
   resolve to this host with ports 80 and 443 reachable for HTTP-01, or see [DNS-01](#dns-01).
6. **Start the data services, then migrate explicitly:**

   ```sh
   docker compose -f compose.yaml -f compose.proxy.yaml up -d --wait postgres object-storage
   docker compose run --rm migrate                                   # plan: read-only
   docker compose run --rm migrate node core/dist/cli/apply-migrations.js
   ```

   Nothing ever applies migrations automatically: the API never does, and `migrate` runs only when
   you run it.
7. **Start everything:** `docker compose -f compose.yaml -f compose.proxy.yaml up -d --wait`.
8. **Check it:** `curl https://<hostname>/api/v1/health/ready` answers `{"status":"ready"}`, and
   `docker compose run --rm migrate node core/dist/cli/doctor.js` reports `READY` (with warnings
   for anything optional you left unconfigured, such as email).

### The initial owner

```sh
docker compose run --rm -it migrate node core/dist/cli/create-initial-owner.js --email owner@example.org
```

It prompts for the password with hidden input and never accepts it on the command line. The owner
is created already verified, so they can sign in before any email relay exists
([ADR 0035](../docs/architecture/decisions/0035-initial-platform-owner-bootstrap.md)). To supply the
password from a secret manager instead, pipe it on standard input and use `-T` instead of `-it`.

### Upgrading

1. **Pull or build the new image**, and change `EKKLESIA_IMAGE` in `deploy/.env`. Keep the old tag.
2. **Back up** ([Backups](#backups)).
3. **Plan, review, then apply:** `docker compose run --rm migrate` with the new image, then
   `docker compose run --rm migrate node core/dist/cli/apply-migrations.js`.
4. **Restart on the new image:** `docker compose -f compose.yaml -f compose.proxy.yaml up -d --wait`.

### Rolling back

Set `EKKLESIA_IMAGE` back to the previous tag and run `up -d --wait` again. That is enough only when
the upgrade applied no migrations (the plan in step 3 showed none pending). Migrations are
forward-only. If any were applied, restore the backup taken in step 2 before starting the previous
image, because it was not built for the newer schema.

### Backups

Slice 6 adds the scheduled backup job and the restore procedure (ADR 0044, Backups). Until then,
take a `pg_dump` and a copy of every bucket before each upgrade by hand.

### Behind an existing reverse proxy

Run the base alone and attach `api` and `object-storage` to your proxy's network with an overlay of
your own, for example `compose.myproxy.yaml`:

```yaml
services:
  api:
    networks:
      proxy: {}
    environment:
      # Exactly one trusted peer: your proxy, at an address that cannot change.
      EKKLESIA_CLIENT_ADDRESS_MODE: trusted-proxy
      EKKLESIA_CLIENT_ADDRESS_HEADER: x-forwarded-for
      EKKLESIA_CLIENT_ADDRESS_TRUSTED_PROXIES: 10.0.100.254/32
  object-storage:
    networks:
      proxy: {}

networks:
  proxy:
    external: true
    name: your-proxy-network
```

Then make your proxy do what `Caddyfile.example` does:

- forward `/`, `/api/*`, `/display/*`, and `/assets/*` on the Ekklesia hostname to `api:3000`;
- **overwrite** `X-Forwarded-For` with the one client address it resolved, and strip
  `CF-Connecting-IP`, `X-Real-IP`, `Forwarded`, and `True-Client-IP`;
- stream responses on `/api/v1/module-realtime/*` without buffering (Server-Sent Events);
- set no request-body limit below 32 MiB; the API enforces its own ceilings;
- forward the storage hostname to `https://object-storage:8443`, verifying it against
  `/etc/ekklesia/tls/ca.crt` with server name `object-storage`, and **pass the original `Host`
  through unchanged**, or every signed URL fails (ADR 0031).

The proxy's address must be fixed and must never be handed to anything else. With Docker, give its
network an explicit subnet, limit `ip_range` to part of it, and fix the proxy's address outside that
range, as `compose.proxy.yaml` does.

### Fixed addresses on `edge`

`compose.proxy.yaml` gives `edge` the subnet `10.249.250.0/24` and lets Docker assign dynamic
addresses only from its lower half (`10.249.250.0/25`). The proxy (`.254`) and the tunnel connector
(`.253`) take fixed addresses from the top of the upper half, which Docker never assigns to anything
else. That is what makes "only the proxy" and "only the connector" expressible as `/32` trust
entries. If the subnet collides with one on your host, change `EKKLESIA_EDGE_SUBNET`,
`EKKLESIA_EDGE_DYNAMIC_RANGE`, `EKKLESIA_PROXY_ADDRESS`, and `EKKLESIA_TUNNEL_ADDRESS` together,
keeping the fixed addresses at the top and outside the dynamic range.

### Cloudflare Tunnel

`compose.tunnel.yaml` runs `cloudflared` with a token, so the tunnel is managed in the Cloudflare
dashboard. No Cloudflare API is called from here.

1. In Cloudflare Zero Trust, under **Networks → Tunnels**, create a Cloudflared tunnel. Copy its
   token into `/etc/ekklesia/tunnel.env` as `TUNNEL_TOKEN`.
2. Add a **public hostname** (shown as a *published application route* in newer dashboards) for
   the Ekklesia hostname. Service type **HTTPS**, URL `proxy:443`.
3. Under that hostname's **TLS** settings, set **Origin Server Name** to the same Ekklesia hostname,
   and leave **No TLS Verify** off. The connector then verifies the proxy's certificate for that name.
4. Add a second public hostname for the storage hostname, the same way, with its own origin server
   name.
5. Start with all three files. The proxy honors `CF-Connecting-IP` only from the connector's fixed
   address. From any other peer it is stripped.

The proxy's certificate must still be publicly trusted, and an HTTP-01 challenge cannot reach the
proxy through a tunnel, because Cloudflare terminates TLS at its edge. A tunnel-only installation
therefore uses [DNS-01](#dns-01). An installation that also serves local displays should give the
same hostnames a LAN path to the proxy (split-horizon DNS), so displays do not depend on the
Internet link.

### DNS-01

For an installation off the public Internet, LAN-only, or tunnel-only. DNS-01 needs a DNS provider
module, which the stock Caddy image does not contain:

1. Build a Caddy image with the module (for example
   `xcaddy build --with github.com/caddy-dns/cloudflare`) and pin it by digest.
2. Override the proxy's `image:` in an overlay of your own. Give it `CLOUDFLARE_API_TOKEN` (a token
   scoped to DNS edits for the zone) through a root-owned env file.
3. Enable the commented `acme_dns` line in your `Caddyfile`.

### Launch checklist

The ten items in [Security § launch checklist](../docs/development/security.md) mapped to this
deployment:

| # | Item | Where it is met |
| --- | --- | --- |
| 1 | The proxy forwards the Dashboard and `/api` on the intended origin | `Caddyfile.example` routes `/`, `/api/*`, `/display/*`, `/assets/*` on one hostname to `api`; the API serves the Dashboard itself. |
| 2 | The origin port is unreachable from the Internet | The base publishes no port; only `proxy` publishes 80 and 443. |
| 3 | The origin port is unreachable from untrusted networks | `api` is reachable only on Docker networks; `EKKLESIA_PUBLISH_ADDRESS` limits the proxy to one interface. Firewall the host itself. |
| 4 | The proxy overwrites the trusted client-IP header | `header_up X-Forwarded-For {client_ip}` in `Caddyfile.example`. |
| 5 | Rotating forwarding headers cannot rotate the rate-limit bucket | The same, plus stripping `CF-Connecting-IP`, `X-Real-IP`, `Forwarded`, and `True-Client-IP`; the API overwrites its internal header. Proven by the smoke test. |
| 6 | The proxy peer is the only trusted proxy | `compose.proxy.yaml` sets `EKKLESIA_CLIENT_ADDRESS_TRUSTED_PROXIES` to the proxy's fixed `/32`. |
| 7 | Proxy logs record no cookies, credentials, bodies, or forwarding chains | Caddy access logs are off in `Caddyfile.example`. If you enable them, exclude request headers. |
| 8 | PostgreSQL only from the API, over verified TLS | `postgres` is only on internal `data`; `pg_hba.conf` rejects non-TLS; `verify-full` with `EKKLESIA_DATABASE_CA`. Proven by the smoke test. |
| 9 | The deployed login and logout flow keeps its contract | Unchanged by the proxy; the smoke test signs in and out through it. |
| 10 | A supported way to create the first production identity | `create-initial-owner.js` through `migrate` ([The initial owner](#the-initial-owner)). Exercised by the smoke test. |

### Testing this deployment

- `pnpm deploy:config` validates the base, base + proxy, and base + proxy + tunnel with the example
  env files. CI runs it in `pnpm ci:verify`.
- `pnpm deploy:smoke` (`scripts/smoke-deploy.sh`) runs the whole deployment end to end, against a
  platform-only image and a disposable PostgreSQL and SeaweedFS that exist only for that run. It
  follows the first boot above and checks the following through the proxy:
  - readiness and the Dashboard;
  - a signed object URL fetched through the storage hostname;
  - the client-address chain, both from an ordinary peer and from the tunnel connector's address;
  - initial-owner sign-in and sign-out;
  - that the API's database connection is TLS 1.3, and that it is refused by a foreign CA and by a
    wrong host name;
  - Doctor.

  CI runs it on every change.
- **Test-only:** the smoke test runs Caddy with its internal CA (`local_certs`), because a test run
  has no public hostname. Never use it in an installation; `Caddyfile.example` does not contain it.

The smoke test uses no module, so it does not exercise realtime streams or large uploads. Through
this same proxy configuration, a scratch module confirmed both:

- a Server-Sent Event crossed the proxy in milliseconds;
- a 32 MiB streamed upload passed with and without `Content-Length`;
- one byte more was refused by the API's own `413 REQUEST_BODY_TOO_LARGE`.

## The production image

[`Dockerfile`](Dockerfile) builds **one image** for two jobs: the long-running `api` service, and
every operator-run command (migrations, Doctor, initial-owner creation, module enablement). There
is no second image and no second build.

### What it builds

The build context is an **already-assembled workspace**: this repository, plus zero or more module
packages already present in `modules/<id>`, plus a `pnpm-lock.yaml` describing exactly that set.
The Dockerfile clones nothing and chooses no modules. It builds what it is given. With no
`modules/` directory, it builds the platform alone; CI builds that case on every change.

Assembling the context (checking out the platform at a release tag, each module at a pinned tag or
commit, and the instance's own lockfile) belongs to the installation's instance composition, not to
this file.

| Stage | Does |
| --- | --- |
| `base` | `node:24.18.0-trixie-slim`, pinned by its multi-architecture index digest. Debian, so glibc, because module dependencies such as sharp ship glibc prebuilt binaries. Every stage starts here. |
| `toolchain` | Enables pnpm through Corepack, at exactly the root `packageManager` version. |
| `manifests` | Collects the lockfile, workspace configuration, `.npmrc`, and every workspace `package.json` at its repository path. |
| `build` | `pnpm install --frozen-lockfile`, then `pnpm boundaries`, `pnpm modules:validate`, `pnpm migrations:validate`, and `pnpm build`, against exactly the modules in the context. Never uses host `node_modules`: `.dockerignore` excludes them. |
| `prod-deps` | The same lockfile with `--prod`: production dependencies only. |
| `runtime` | Production dependencies plus built outputs, owned by root and run as the unprivileged `node` user (uid 1000). No pnpm, no compiler, no curl. |

The runtime keeps the repository layout at `/opt/ekklesia`, because that layout is how the platform
finds its files:

- the API takes the repository root as three directories above `apps/api/dist/`, and reads
  `package.json`, `packages/module-sdk/package.json`, and `modules/` beneath it;
- every CLI takes its **working directory** as the root and also reads Core's migrations in
  `core/drizzle/`, so commands must run from `/opt/ekklesia`, the image's `WORKDIR`;
- the Dashboard and display host files are read from `surfaces/shell/dist/browser/`;
- each module directory is copied whole, minus `node_modules`, because each manifest names its
  own entrypoints, UI artifacts, and migrations directory.

### Building

```sh
docker build \
  --file deploy/Dockerfile \
  --build-arg VERSION=0.1.0 \
  --build-arg REVISION="$(git rev-parse HEAD)" \
  --build-arg SOURCE=https://git.example.org/your-org/ekklesia \
  --tag ekklesia:0.1.0 \
  .
```

The three build arguments become the OCI labels `org.opencontainers.image.version`, `.revision`,
and `.source`. They have no defaults: the repository cannot know an installation's Git host. The
image also records its base image name and digest. BuildKit is required (Docker's default builder);
set `DOCKER_BUILDKIT=1` if your Docker CLI has no buildx plugin.

No secret is a build argument, and none enters any layer. `.env` files are excluded from the build
context entirely, and no registry credentials are needed.

**A module missing a production dependency fails the build, not the API.** The runtime receives
production dependencies only, so a module that imports a package at run time but declares it only
as a development or peer dependency would build and then fail at startup. `pnpm boundaries` runs
in the build stage and rejects that module first
(`runtime-imports-are-production-dependencies`, see
[Repository boundaries](../docs/architecture/repository-boundaries.md)). The same applies to every
other boundary rule: a module the platform's own CI would reject cannot be built into an image.

**A platform-only image does not exercise native module dependencies.** With no modules in the
context, nothing like sharp is installed, so the CI image build proves nothing about them. An
instance build that includes such a module must smoke-test the built image itself: start it,
confirm `/api/v1/health/ready` answers `200`, and exercise the module's native code path (for
example, process one image).

### Configuration

Configuration comes **only from the container environment**, never from a file inside the image.
The image sets exactly three variables itself:

| Variable | Image value | Why |
| --- | --- | --- |
| `NODE_ENV` | `production` | Stated rather than relied on. Anything other than `development` already validates as production. |
| `EKKLESIA_API_HOST` | `0.0.0.0` | Reachable from other containers. Outside a container the platform default stays `127.0.0.1`. |
| `EKKLESIA_API_PORT` | `3000` | The platform default, made explicit; `EXPOSE 3000`. |

Everything else is the ordinary host configuration in
[API § Required configuration](../docs/development/api.md#required-configuration), supplied by the
reference deployment's `ekklesia.env`. `EKKLESIA_DATABASE_CA` is the CA's PEM text itself, on one
double-quoted line with `\n` escapes. Compose expands those escapes; plain `docker run --env-file`
does not, which is why every command here goes through Compose.

Never publish the API's port to the host or the Internet. Only the proxy publishes ports.

### Process, signals, and the read-only posture

`compose.yaml` runs `api` and `migrate` with `init: true`, `read_only: true`, `tmpfs: /tmp`,
`cap_drop: [ALL]`, and `no-new-privileges`. That is the supported production posture:

- **`init: true`** runs a minimal init as PID 1 that forwards signals and reaps children, so
  `docker stop` reaches the API's own SIGTERM handler promptly. The API then drains in-flight
  requests, stops modules in reverse order, and closes its pool.
- **Read-only:** the image was verified under exactly these flags. The API reached `healthy`, and
  the migration and Doctor commands ran. Nothing wrote to `/tmp`, which stays mounted as headroom
  for module code that needs scratch space. The installation, `/home/node` included, is not
  writable, so anything that needs a writable path other than `/tmp` is a defect to report, not to
  work around.

### Health

The image's `HEALTHCHECK` calls `GET /api/v1/health/ready` on `127.0.0.1` with Node's built-in
`fetch`, every 30 seconds after a 30-second start period, with three retries. Readiness means the
database answered within 2 seconds and the instance is not shutting down; see
[API § Health](../docs/development/api.md#health). Because it probes `127.0.0.1`, keep
`EKKLESIA_API_HOST` at `0.0.0.0` (or any address that includes loopback) inside the container. The
API trusts only the proxy's address for client-address headers, so the loopback probe is an
ordinary direct peer.

An external monitor can use the same endpoint through the proxy. Use `/api/v1/health/live` to ask
only whether the process is serving.

### Operator commands

The same image runs every CLI, directly with `node core/dist/cli/<command>.js`, through the
`migrate` service. The `pnpm` scripts of the same names rebuild first and read `.env`, so they are
for a development checkout, not for the image.

`migrate` has `profiles: ["ops"]`, so `up` never starts it. It never restarts, has no health check,
and its default command is the read-only plan. **Nothing applies migrations automatically.**

| Task | Command (from `deploy/`) |
| --- | --- |
| Plan migrations (read-only) | `docker compose run --rm migrate` |
| Apply migrations | `docker compose run --rm migrate node core/dist/cli/apply-migrations.js` |
| Doctor (read-only report) | `docker compose run --rm migrate node core/dist/cli/doctor.js` |
| Create the initial owner (interactive) | `docker compose run --rm -it migrate node core/dist/cli/create-initial-owner.js --email owner@example.org` |
| Enable or disable a module | `docker compose run --rm migrate node core/dist/cli/set-module-enabled.js --enable --module <id>` (or `--disable`) |

The plan, apply, Doctor, and initial-owner commands are exercised by the smoke test; module
enablement is not.

### Size

The platform-only image is about 378 MB: the Debian slim Node base (about 230 MB), production
dependencies (about 141 MB), and built outputs (about 6 MB). A rebuild that reuses a warm pnpm store
cache can come out about 21 MB larger. When esbuild's install script runs, it hard-links its
`bin/esbuild` to the native binary it ships beside it; when pnpm restores esbuild from the store's
cached post-install state instead, that file is a second full copy. Both are esbuild, one of the
unused tools below.

About 57 MB of the production dependencies are development tools that no runtime code imports:
`vitest`, `vite`, `drizzle-kit`, `tsx`, `esbuild`, and their native bindings. They arrive as
optional peer dependencies of Better Auth (and `esbuild` through the module UI builder), which pnpm
resolves because the workspace already has them. Excluding them means changing dependency
resolution and the lockfile, which is separate work.