feat: one generic image, with installed modules in installation-owned storage (ADR 0048) #64

Merged

Zayin_Admin merged 2 commits from feat/installed-module-storage into main 2026-10-07 14:15:32 +00:00

Conversation 1 Commits 2 Files Changed 52

+2456 -383

ekklesia-bot commented 2026-10-07 13:38:49 +00:00

Member

Copy Link

Copy Source

Implements ADR 0048: one generic platform image with zero feature modules, and installed modules in installation-owned storage.

Out of scope, as the ADR defers them: marketplace, package archive or signing format, installer UI, purge, hot loading, and release publishing. rog-ekklesia-instance and Ekklesia_Signage are untouched.

Behaviour

Modules directory. Host configuration reads EKKLESIA_MODULES_DIR, and one Core function, resolveModulesDirectory, interprets it:

• unset: <repositoryRoot>/modules, so development is unchanged;
• set: an absolute path, normalized, and required to exist.

A relative, blank, padded or NUL-containing value fails with a sanitized MODULES_DIRECTORY_INVALID. A configured directory that is missing or unreadable fails with MODULES_DIRECTORY_UNREADABLE. Neither error ever contains the value. The API's test-only modulesDirectoryOverride still wins, and the configured value is then never resolved.

Every call site uses the resolver: API discovery, migration plan, apply and validate, validate-modules, set-module-enabled, and Doctor. A source-scan test fails if Core or the API joins "modules" onto a root anywhere else.

Doctor. It now receives Core's migrations directory explicitly; it used to derive it from the modules directory's parent. An unusable configured directory appears in the report as a sanitized module-discovery failure.

Retained migration history. Applied history for a module that is not installed is kept and reported ([retained] <id>: N applied migrations kept) in plan, apply and Doctor. It is not fatal and blocks neither Core's migrations nor any present module's.

• A module that is present, including one that has just returned, stays strict: a missing applied migration, a checksum mismatch, or a downgrade or incomplete history still fails.
• "Present" means discovered, not "has migrations".
• Core's history is never treated as an absent module's.
• Retained rows are still checked for shape and global consistency.
• No down migrations, no data deletion.

Image.

• deploy/Dockerfile copies no module, and installs neither the module template's manifest nor its dependencies.
• .dockerignore excludes /modules, and the build refuses a context that still contains it.
• The runtime has an empty, root-owned /var/lib/ekklesia/modules, and sets EKKLESIA_MODULES_DIR to it.

Compose. EKKLESIA_MODULES_HOST_DIR (default /var/lib/ekklesia/modules) is bind-mounted read-only into api and migrate, with create_host_path: false. EKKLESIA_MODULES_DIR is stated beside the mount. First boot gains a documented install -d step.

Test module packaging (test-only). scripts/lib/test-module-package.Dockerfile turns the template into a self-contained directory: pnpm deploy --prod --legacy, with pnpm's workspace self-link removed and any link that escapes the package refused. It produces a directory, never an image, and it is explicitly not a package format. The package carries the platform lockfile's Drizzle version, which the ADR records as a temporary constraint.

Conflicts between ADR 0048 and code before this PR, all resolved here

1. The Dockerfile copied modules/ into the image, and its prod-deps stage installed the template's dependencies.
2. The modules directory was hard-coded at six sites, and Doctor derived Core's migrations from its parent.
3. Discovery treated a missing directory as zero modules, where the ADR requires an explicitly configured one to fail closed.
4. The runner rejected any applied history missing from the inventory (MIGRATION_UNKNOWN_APPLIED).
5. The split smoke test baked the template into a second image.

One existing integration test asserted rule 4 for an absent ghost module. It now asserts retention instead, and a new test keeps the strict rejection for an unknown identity belonging to an installed module.

Tests, written first

• Resolver: unset, absolute (normalized), relative, dot-relative, parent-relative, home-relative, empty, whitespace, padded, NUL, never echoing the value, and discovery failing closed for a missing directory or a file.
• Host configuration: carries the value verbatim. Boundary test: the variable is interpreted in exactly one file.
• CLI helper: default, configured and refused values; migrations discovered from an external directory with Core's from the platform.
• API: reads a configured directory; fails closed on relative or missing values before any pool exists, without echoing them; the override takes precedence; environment parity.
• Doctor: an external directory with Core's migrations still found; unusable and missing directories reported without paths; retained history passes; Core's directory is passed to migration discovery.
• Runner: absent history retained, frozen and sorted; a later Core migration and present modules' migrations applied while a module is absent; a returning module's missing migration, checksum mismatch, or non-prefix history fails; a present module without migrations stays strict; Core is never absent; corrupt retained rows are rejected.
• Reports: plan and apply output include retained lines.
• Live PostgreSQL: real discovery, with Core's newest migration withheld at first:
  1. apply Core and the template from an external directory;
  2. insert a row;
  3. remove the module;
  4. plan (retained) and apply the later Core migration;
  5. the row survives;
  6. restore the module;
  7. the plan is strict and valid, the row is intact, and an edited migration is refused.
• Compose and Dockerfile (static): read-only mount in api and migrate, create_host_path: false, the mount only there, a host variable distinct from EKKLESIA_MODULES_DIR, .dockerignore, and the image environment and mount point.
• CI scripts: the image check is labelled and tagged per run and attempt; the smoke builds exactly one platform image and no module-in-image context.

Verification

All run locally on the final tree; Docker gates are against real Docker.

Gate	Result
pnpm verify	passed: core 1015, api 510, shell 219, root 91, other packages all green; boundaries, module and migration validation, identity schema, build
pnpm test:migrations:integration	passed: 14 files, 308 tests
pnpm deploy:config	all three combinations valid
bash scripts/check-image.sh	passed: both builds have no modules, no template, no manifest outside dependencies, an empty root-owned /var/lib/ekklesia/modules, and EKKLESIA_MODULES_DIR set; a populated modules/ gives an identical runtime file system (17,447 paths) and identical layers
pnpm deploy:smoke (same-origin)	passed on the generic image with empty module storage
EKKLESIA_SMOKE_TOPOLOGY=split pnpm deploy:smoke	passed: boots module-free, installs the external test module, migrates explicitly, restarts, runs the browser checks (UI, display, credentialed requests), removes it (Core runs, plan reports [retained], data survives), restores it with its data; Doctor READY

The split run uses one built generic image for its no-module boot and its external-module scenarios. It builds no second platform image. The same-origin CI leg builds the same Dockerfile in its own job, and check-image.sh proves that build is independent of modules/.

Notes for review

• CI job names are unchanged, so no branch-protection change is needed. image-build now builds twice (generic and populated context); the second build is a cache hit, and the job's 20-minute timeout is unchanged.
• EKKLESIA_SMOKE_IMAGE must now be a generic image; an older image with baked-in modules will not have them discovered. The opt-in display-device scenarios that expect Signage need Signage installed in module storage; that is unchanged opt-in behaviour, not exercised in CI.
• The image is about 400 MB with a warm pnpm cache, the same as before: the template's dependencies were already needed by the Dashboard shell.

🤖 Generated with Claude Code

Implements [ADR 0048](docs/architecture/decisions/0048-generic-image-and-installed-module-storage.md): one generic platform image with zero feature modules, and installed modules in installation-owned storage. Out of scope, as the ADR defers them: marketplace, package archive or signing format, installer UI, purge, hot loading, and release publishing. `rog-ekklesia-instance` and `Ekklesia_Signage` are untouched. ## Behaviour **Modules directory.** Host configuration reads `EKKLESIA_MODULES_DIR`, and one Core function, `resolveModulesDirectory`, interprets it: - unset: `<repositoryRoot>/modules`, so development is unchanged; - set: an absolute path, normalized, and required to exist. A relative, blank, padded or NUL-containing value fails with a sanitized `MODULES_DIRECTORY_INVALID`. A configured directory that is missing or unreadable fails with `MODULES_DIRECTORY_UNREADABLE`. Neither error ever contains the value. The API's test-only `modulesDirectoryOverride` still wins, and the configured value is then never resolved. **Every call site uses the resolver:** API discovery, migration plan, apply and validate, `validate-modules`, `set-module-enabled`, and Doctor. A source-scan test fails if Core or the API joins `"modules"` onto a root anywhere else. **Doctor.** It now receives Core's migrations directory explicitly; it used to derive it from the modules directory's parent. An unusable configured directory appears in the report as a sanitized module-discovery failure. **Retained migration history.** Applied history for a module that is not installed is kept and reported (`[retained] <id>: N applied migrations kept`) in plan, apply and Doctor. It is not fatal and blocks neither Core's migrations nor any present module's. - A module that is present, including one that has just returned, stays strict: a missing applied migration, a checksum mismatch, or a downgrade or incomplete history still fails. - "Present" means discovered, not "has migrations". - Core's history is never treated as an absent module's. - Retained rows are still checked for shape and global consistency. - No down migrations, no data deletion. **Image.** - `deploy/Dockerfile` copies no module, and installs neither the module template's manifest nor its dependencies. - `.dockerignore` excludes `/modules`, and the build refuses a context that still contains it. - The runtime has an empty, root-owned `/var/lib/ekklesia/modules`, and sets `EKKLESIA_MODULES_DIR` to it. **Compose.** `EKKLESIA_MODULES_HOST_DIR` (default `/var/lib/ekklesia/modules`) is bind-mounted read-only into `api` and `migrate`, with `create_host_path: false`. `EKKLESIA_MODULES_DIR` is stated beside the mount. First boot gains a documented `install -d` step. **Test module packaging (test-only).** `scripts/lib/test-module-package.Dockerfile` turns the template into a self-contained directory: `pnpm deploy --prod --legacy`, with pnpm's workspace self-link removed and any link that escapes the package refused. It produces a directory, never an image, and it is explicitly not a package format. The package carries the platform lockfile's Drizzle version, which the ADR records as a temporary constraint. ## Conflicts between ADR 0048 and code before this PR, all resolved here 1. The Dockerfile copied `modules/` into the image, and its `prod-deps` stage installed the template's dependencies. 2. The modules directory was hard-coded at six sites, and Doctor derived Core's migrations from its parent. 3. Discovery treated a missing directory as zero modules, where the ADR requires an explicitly configured one to fail closed. 4. The runner rejected any applied history missing from the inventory (`MIGRATION_UNKNOWN_APPLIED`). 5. The split smoke test baked the template into a second image. One existing integration test asserted rule 4 for an absent `ghost` module. It now asserts retention instead, and a new test keeps the strict rejection for an unknown identity belonging to an **installed** module. ## Tests, written first - **Resolver:** unset, absolute (normalized), relative, dot-relative, parent-relative, home-relative, empty, whitespace, padded, NUL, never echoing the value, and discovery failing closed for a missing directory or a file. - **Host configuration:** carries the value verbatim. Boundary test: the variable is interpreted in exactly one file. - **CLI helper:** default, configured and refused values; migrations discovered from an external directory with Core's from the platform. - **API:** reads a configured directory; fails closed on relative or missing values before any pool exists, without echoing them; the override takes precedence; environment parity. - **Doctor:** an external directory with Core's migrations still found; unusable and missing directories reported without paths; retained history passes; Core's directory is passed to migration discovery. - **Runner:** absent history retained, frozen and sorted; a later Core migration and present modules' migrations applied while a module is absent; a returning module's missing migration, checksum mismatch, or non-prefix history fails; a present module without migrations stays strict; Core is never absent; corrupt retained rows are rejected. - **Reports:** plan and apply output include retained lines. - **Live PostgreSQL:** real discovery, with Core's newest migration withheld at first: 1. apply Core and the template from an external directory; 2. insert a row; 3. remove the module; 4. plan (retained) and apply the later Core migration; 5. the row survives; 6. restore the module; 7. the plan is strict and valid, the row is intact, and an edited migration is refused. - **Compose and Dockerfile (static):** read-only mount in `api` and `migrate`, `create_host_path: false`, the mount only there, a host variable distinct from `EKKLESIA_MODULES_DIR`, `.dockerignore`, and the image environment and mount point. - **CI scripts:** the image check is labelled and tagged per run and attempt; the smoke builds exactly one platform image and no module-in-image context. ## Verification All run locally on the final tree; Docker gates are against real Docker. | Gate | Result | | --- | --- | | `pnpm verify` | passed: core 1015, api 510, shell 219, root 91, other packages all green; boundaries, module and migration validation, identity schema, build | | `pnpm test:migrations:integration` | passed: 14 files, 308 tests | | `pnpm deploy:config` | all three combinations valid | | `bash scripts/check-image.sh` | passed: both builds have no modules, no template, no manifest outside dependencies, an empty root-owned `/var/lib/ekklesia/modules`, and `EKKLESIA_MODULES_DIR` set; a populated `modules/` gives an **identical runtime file system (17,447 paths)** and identical layers | | `pnpm deploy:smoke` (same-origin) | passed on the generic image with empty module storage | | `EKKLESIA_SMOKE_TOPOLOGY=split pnpm deploy:smoke` | passed: boots module-free, installs the external test module, migrates explicitly, restarts, runs the browser checks (UI, display, credentialed requests), removes it (Core runs, plan reports `[retained]`, data survives), restores it with its data; Doctor READY | The split run uses **one built generic image** for its no-module boot and its external-module scenarios. It builds no second platform image. The same-origin CI leg builds the same Dockerfile in its own job, and `check-image.sh` proves that build is independent of `modules/`. ## Notes for review - CI job names are unchanged, so no branch-protection change is needed. `image-build` now builds twice (generic and populated context); the second build is a cache hit, and the job's 20-minute timeout is unchanged. - `EKKLESIA_SMOKE_IMAGE` must now be a generic image; an older image with baked-in modules will not have them discovered. The opt-in display-device scenarios that expect Signage need Signage installed in module storage; that is unchanged opt-in behaviour, not exercised in CI. - The image is about 400 MB with a warm pnpm cache, the same as before: the template's dependencies were already needed by the Dashboard shell. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

ekklesia-bot added 1 commit 2026-10-07 13:38:49 +00:00

feat: one generic image, with installed modules in installation-owned storage (ADR 0048) ... e36b70d8d4

The production image now contains zero feature modules and is the same for every
installation. Installed modules live in module storage that one Core resolver
selects from EKKLESIA_MODULES_DIR: the checkout's modules/ when unset, an
absolute path that must exist when set (/var/lib/ekklesia/modules in the image).
A relative, blank, missing, or unreadable configured directory fails closed with
a sanitized error that never contains the value.

- Host configuration reads EKKLESIA_MODULES_DIR. The API, migration plan, apply
  and validate, module validation, module enablement, and Doctor all resolve it
  the same way; the test-only API override still takes precedence.
- Doctor receives Core's migrations directory explicitly instead of deriving it
  from the modules directory's parent.
- The migration runner retains and reports the applied history of a module that
  is not installed, and plans and applies Core's and present modules' migrations
  around it. Core's history and every installed module's history stay strict: a
  missing or changed applied migration, or a downgrade, still fails. No down
  migrations, no data deletion.
- deploy/Dockerfile copies no module and installs nothing for the module
  template; .dockerignore excludes modules/ and the build refuses a context that
  contains it; the runtime has an empty root-owned /var/lib/ekklesia/modules and
  sets EKKLESIA_MODULES_DIR.
- deploy/compose.yaml bind-mounts EKKLESIA_MODULES_HOST_DIR read-only into api
  and migrate, with create_host_path disabled.
- scripts/check-image.sh proves the image holds no modules and that a populated
  modules/ directory leaves its runtime file system identical; CI runs it.
- The split deploy smoke now uses the same generic image: it boots with no
  modules, installs a self-contained test module packaged from the template
  (test-only), migrates and restarts, removes it (Core runs, planning reports the
  retained history, data survives), and restores it with its data.

Docs: deploy README (module storage, install/update/remove/restore, the runtime
modules target), .env.example, API configuration, module development guide,
testing, workflow, security, and the architecture overview.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Zayin_Admin added 1 commit 2026-10-07 14:08:01 +00:00

test: generate the external module fixture instead of copying ignored build output ... f122e910fc

The test proving that migrations of a module installed outside the repository
are discovered, while Core's come from the platform, copied templates/module
including dist/. dist/ is gitignored and never committed. CI runs `pnpm test`
before any build of the template, where only the template's own UI test may
have written dist/ui, dist/display, and dist/contributions, so discovery
failed with MODULE_ENTRYPOINT_MISSING for ./dist/server/index.js and
./dist/schema/index.js (and, had the UI test not run yet, the copy itself
would have failed). Locally a previous `pnpm build` had left dist/ complete.

The test now generates a minimal valid installed module in a temporary
directory with createMigrationModule, moved unchanged from the migration
discovery test into core/tests/support so both share it. It depends on no
build output, ignored file, or node_modules. It also asserts more precisely:
the module directory is outside the repository, Core's migrations match the
platform's own core/drizzle inventory by identity and checksum, and the
module's migration matches the SQL that was installed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Zayin_Admin approved these changes 2026-10-07 14:15:19 +00:00

Zayin_Admin merged commit fbf8f6edac into main 2026-10-07 14:15:32 +00:00

Zayin_Admin deleted branch feat/installed-module-storage 2026-10-07 14:15:32 +00:00

Zayin_Admin referenced this issue from a commit 2026-10-08 22:27:59 +00:00

chore(release): prepare 0.2.0

ekklesia-bot referenced this pull request 2026-10-08 22:28:17 +00:00

chore(release): prepare 0.2.0 #82

Sign in to join this conversation.

Reviewers

No Reviewers

Zayin_Admin

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

Joey Zayin_Admin ekklesia-bot

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Zayin/Ekklesia#64