feat: one generic image, with installed modules in installation-owned storage (ADR 0048)
CI / verify (pull_request) Failing after 47s
CI / infrastructure-smoke (pull_request) Skipped
CI / migration-smoke (pull_request) Skipped
CI / image-build (pull_request) Skipped
CI / deploy-smoke (same-origin) (pull_request) Skipped
CI / deploy-smoke (split) (pull_request) Skipped
CI / verify (pull_request) Failing after 47s
CI / infrastructure-smoke (pull_request) Skipped
CI / migration-smoke (pull_request) Skipped
CI / image-build (pull_request) Skipped
CI / deploy-smoke (same-origin) (pull_request) Skipped
CI / deploy-smoke (split) (pull_request) Skipped
The production image now contains zero feature modules and is the same for every installation. Installed modules live in module storage that one Core resolver selects from EKKLESIA_MODULES_DIR: the checkout's modules/ when unset, an absolute path that must exist when set (/var/lib/ekklesia/modules in the image). A relative, blank, missing, or unreadable configured directory fails closed with a sanitized error that never contains the value. - Host configuration reads EKKLESIA_MODULES_DIR. The API, migration plan, apply and validate, module validation, module enablement, and Doctor all resolve it the same way; the test-only API override still takes precedence. - Doctor receives Core's migrations directory explicitly instead of deriving it from the modules directory's parent. - The migration runner retains and reports the applied history of a module that is not installed, and plans and applies Core's and present modules' migrations around it. Core's history and every installed module's history stay strict: a missing or changed applied migration, or a downgrade, still fails. No down migrations, no data deletion. - deploy/Dockerfile copies no module and installs nothing for the module template; .dockerignore excludes modules/ and the build refuses a context that contains it; the runtime has an empty root-owned /var/lib/ekklesia/modules and sets EKKLESIA_MODULES_DIR. - deploy/compose.yaml bind-mounts EKKLESIA_MODULES_HOST_DIR read-only into api and migrate, with create_host_path disabled. - scripts/check-image.sh proves the image holds no modules and that a populated modules/ directory leaves its runtime file system identical; CI runs it. - The split deploy smoke now uses the same generic image: it boots with no modules, installs a self-contained test module packaged from the template (test-only), migrates and restarts, removes it (Core runs, planning reports the retained history, data survives), and restores it with its data. Docs: deploy README (module storage, install/update/remove/restore, the runtime modules target), .env.example, API configuration, module development guide, testing, workflow, security, and the architecture overview. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
4d28a82937
commit
e36b70d8d4
@@ -2,6 +2,11 @@
|
||||
# `pnpm install --frozen-lockfile` and builds its own outputs, so nothing installed or built on the
|
||||
# host may enter the context, and no secret or environment file may enter any layer.
|
||||
|
||||
# Installed modules never enter the platform image (ADR 0048): an installation keeps them in its own
|
||||
# module storage, mounted at run time. Excluding the checkout's `modules/` is what makes a build
|
||||
# identical whether or not modules happen to be present in the source tree.
|
||||
/modules
|
||||
|
||||
# Host-installed dependencies and build outputs: the image produces its own.
|
||||
**/node_modules
|
||||
**/dist
|
||||
|
||||
@@ -10,6 +10,12 @@
|
||||
# The plaintext loopback DATABASE_URL below is therefore only usable with this set.
|
||||
NODE_ENV=development
|
||||
|
||||
# Installed modules (ADR 0048). Unset, the checkout's own modules/ is used, as module development
|
||||
# and the platform's tests expect. Set, it must be an absolute path to an existing directory;
|
||||
# anything else stops the API and every module command with a sanitized error. The production
|
||||
# image sets /var/lib/ekklesia/modules.
|
||||
# EKKLESIA_MODULES_DIR=/absolute/path/to/modules
|
||||
|
||||
POSTGRES_DB=ekklesia
|
||||
POSTGRES_USER=ekklesia
|
||||
POSTGRES_PASSWORD=ekklesia-development-only
|
||||
|
||||
+12
-20
@@ -152,23 +152,14 @@ jobs:
|
||||
- name: Sweep stale CI stacks from the runner
|
||||
run: bash scripts/ci-sweep.sh --stale
|
||||
|
||||
# The platform-only image (ADR 0044): a checkout has no `modules/`, so this proves the image
|
||||
# builds, validates, and assembles with zero modules. Build only — no registry login, no
|
||||
# credentials, no push, no tag anywhere but this runner's local image store.
|
||||
# Tagged per run and attempt, not only per commit: a pull request and a push of the same commit
|
||||
# can build at once on one runner's Docker daemon, and neither may remove the other's image.
|
||||
# Labeled like every CI stack, so this job's final step removes it even if the build is killed.
|
||||
- name: Build the production image without modules
|
||||
shell: bash
|
||||
run: |
|
||||
source scripts/lib/ci-project.sh
|
||||
docker build \
|
||||
--file deploy/Dockerfile \
|
||||
--build-arg REVISION="$GITHUB_SHA" \
|
||||
--label "${EKKLESIA_CI_LABEL}=$(ci_project_name image)" \
|
||||
--label "${EKKLESIA_CI_CREATED_LABEL}=$(date +%s)" \
|
||||
--tag ekklesia-ci:"$GITHUB_SHA-${GITHUB_RUN_ID:-0}-${GITHUB_RUN_ATTEMPT:-1}" \
|
||||
.
|
||||
# The one generic platform image (ADR 0044, ADR 0048), proved rather than assumed: it contains no
|
||||
# module, no module template, and no module manifest; its module storage mount point is empty;
|
||||
# EKKLESIA_MODULES_DIR names it; and a checkout with a populated modules/ directory builds an
|
||||
# identical runtime file system. Build only — no registry login, no credentials, no push, no tag
|
||||
# anywhere but this runner's local image store. Tagged per run and attempt, and labeled like
|
||||
# every CI stack, so this job's final step removes the images even if a build is killed.
|
||||
- name: Build the generic platform image and prove it holds no modules
|
||||
run: bash scripts/check-image.sh
|
||||
|
||||
# Whatever happened above, this job's own stacks do not outlive it.
|
||||
- name: Remove this job's CI stacks
|
||||
@@ -179,8 +170,9 @@ jobs:
|
||||
needs: verify
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
# Both topologies (ADR 0045): same-origin, the default every installation gets, and split, which
|
||||
# adds a test-only module and drives a real headless browser across the two origins.
|
||||
# Both topologies (ADR 0045), each on the one generic image: same-origin, the default every
|
||||
# installation gets, and split, which installs a test-only module into module storage, removes
|
||||
# it, and restores it (ADR 0048), and drives a real headless browser across the two origins.
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
@@ -206,7 +198,7 @@ jobs:
|
||||
- name: Sweep stale CI stacks from the runner
|
||||
run: bash scripts/ci-sweep.sh --stale
|
||||
|
||||
# The reference deployment end to end (ADR 0044): a platform-only image, base + proxy with
|
||||
# The reference deployment end to end (ADR 0044): the generic platform image, base + proxy with
|
||||
# Caddy's internal CA (test-only), an explicit migration apply against this job's own fresh
|
||||
# PostgreSQL, then the API, Dashboard, storage delivery, client-address chain, and verify-full
|
||||
# database TLS through the proxy. Every secret is generated for this run and discarded with
|
||||
|
||||
@@ -35,6 +35,18 @@ concurrency and release, exact destructive approvals, and sanitization against t
|
||||
adapter. A Core-owned `pnpm doctor` dashboard combines the existing static discovery and read-only
|
||||
migration-planning services into one concise readiness report.
|
||||
|
||||
**Ekklesia ships one generic image, and installed modules live outside it
|
||||
([ADR 0048](docs/architecture/decisions/0048-generic-image-and-installed-module-storage.md)).** The
|
||||
production image built from [`deploy/Dockerfile`](deploy/Dockerfile) contains zero feature modules
|
||||
and is the same for every installation. Installed modules live in installation-owned module storage,
|
||||
selected by `EKKLESIA_MODULES_DIR` (`/var/lib/ekklesia/modules` in the image, the checkout's
|
||||
`modules/` when unset) and mounted read-only by the reference deployment. Each installed module is a
|
||||
self-contained directory carrying its own production dependencies. Installing, updating, or removing
|
||||
a module needs only a controlled restart and, when it has migrations, an explicit plan and apply.
|
||||
Nothing is hot loaded. Removing a module's code keeps its data and its stored state; its applied
|
||||
migration history is retained and reported, and blocks no other migration. Purge, a package format,
|
||||
signing, and any installer or marketplace do not exist yet.
|
||||
|
||||
Phase 4A added Core-owned identity foundations: a Core-owned Better Auth database schema
|
||||
(`ekklesia_core_user`, `ekklesia_core_session`, `ekklesia_core_account`,
|
||||
`ekklesia_core_verification`), a first-class Core-owned migration source that coexists with
|
||||
|
||||
@@ -6,6 +6,7 @@ import {
|
||||
composePlatformPermissionRegistry,
|
||||
discoverModules,
|
||||
type DiscoveredModule,
|
||||
resolveModulesDirectory,
|
||||
} from "@ekklesia/core/modules";
|
||||
|
||||
export type PlatformPermissionConfigurationErrorCode = "PLATFORM_PERMISSION_CONFIGURATION_INVALID";
|
||||
@@ -55,10 +56,18 @@ async function readPackageVersion(packagePath: string): Promise<string> {
|
||||
}
|
||||
|
||||
export interface PrepareStartupModuleInventoryOptions {
|
||||
/**
|
||||
* The deployment's `EKKLESIA_MODULES_DIR`, raw, exactly as host configuration read it (ADR 0048).
|
||||
* A production input, not a test seam: the executable entry point always passes it. Unset means
|
||||
* the repository's own `modules/`; set means that absolute directory, which must exist. It is
|
||||
* resolved by Core's one resolver, so an unusable value fails startup with a sanitized
|
||||
* `ModuleDiscoveryError` before any pool exists.
|
||||
*/
|
||||
readonly configuredModulesDirectory?: string | undefined;
|
||||
/** Test-only escape hatch: overrides the directory module discovery scans for installed
|
||||
* modules. Never set by the executable entry point — production always discovers the real
|
||||
* repository's `modules/` directory. Platform and Module SDK versions are always read from the
|
||||
* real repository's own `package.json` files regardless of this override. */
|
||||
* modules, taking precedence over `configuredModulesDirectory`, which is then never resolved.
|
||||
* Never set by the executable entry point. Platform and Module SDK versions are always read from
|
||||
* the real repository's own `package.json` files regardless of this override. */
|
||||
readonly modulesDirectoryOverride?: string;
|
||||
/** Test-only escape hatch: supplies an already-discovered module list directly, skipping the
|
||||
* filesystem discovery step entirely, while still running the real `composePlatformPermissionRegistry`
|
||||
@@ -109,6 +118,11 @@ export async function prepareStartupModuleInventory(
|
||||
}
|
||||
|
||||
const repositoryRoot = resolveRepositoryRoot();
|
||||
// Chosen before anything is read, so an unusable configured directory fails first and alone.
|
||||
const modulesDirectory =
|
||||
options.modulesDirectoryOverride === undefined
|
||||
? resolveModulesDirectory({ configured: options.configuredModulesDirectory, repositoryRoot })
|
||||
: { modulesDirectory: options.modulesDirectoryOverride, modulesDirectoryRequired: false };
|
||||
const [platformVersion, moduleSdkVersion] = await Promise.all([
|
||||
readPackageVersion(path.join(repositoryRoot, "package.json")),
|
||||
readPackageVersion(path.join(repositoryRoot, "packages/module-sdk/package.json")),
|
||||
@@ -116,7 +130,7 @@ export async function prepareStartupModuleInventory(
|
||||
|
||||
const modules = await discoverModules({
|
||||
moduleSdkVersion,
|
||||
modulesDirectory: options.modulesDirectoryOverride ?? path.join(repositoryRoot, "modules"),
|
||||
...modulesDirectory,
|
||||
platformVersion,
|
||||
});
|
||||
|
||||
|
||||
+16
-4
@@ -154,8 +154,12 @@ export interface StartApiOptions {
|
||||
* test needs somewhere to observe the operational failure the user is never shown.
|
||||
*/
|
||||
readonly reportNotificationFailure?: IdentityNotificationReporter;
|
||||
/** Test-only escape hatch: overrides the directory module discovery scans. Never set by the
|
||||
* executable entry point. See `prepareStartupModuleInventory`. */
|
||||
/** The deployment's raw `EKKLESIA_MODULES_DIR` (ADR 0048), a production input the executable
|
||||
* entry point always passes from host configuration. See `prepareStartupModuleInventory`. */
|
||||
readonly configuredModulesDirectory?: string | undefined;
|
||||
/** Test-only escape hatch: overrides the directory module discovery scans, taking precedence
|
||||
* over `configuredModulesDirectory`. Never set by the executable entry point. See
|
||||
* `prepareStartupModuleInventory`. */
|
||||
readonly modulesDirectoryOverride?: string;
|
||||
/** Test-only escape hatch: supplies an already-discovered module list directly, skipping
|
||||
* filesystem discovery entirely — this exact list drives both permission composition and module
|
||||
@@ -429,6 +433,7 @@ export async function startApi(
|
||||
options: StartApiOptions = {},
|
||||
): Promise<RunningApi> {
|
||||
const inventory = await prepareStartupModuleInventory({
|
||||
configuredModulesDirectory: options.configuredModulesDirectory,
|
||||
...(options.modulesDirectoryOverride === undefined
|
||||
? {}
|
||||
: { modulesDirectoryOverride: options.modulesDirectoryOverride }),
|
||||
@@ -769,6 +774,7 @@ export function readConfigFromEnvironment(processEnvironment: HostEnvironment):
|
||||
readonly input: Parameters<typeof createApiRuntimeConfig>[0];
|
||||
readonly environment: "development" | "production";
|
||||
readonly moduleSecrets: ModuleSecretSource;
|
||||
readonly modulesDirectory: string | undefined;
|
||||
} {
|
||||
const host = loadHostConfiguration(processEnvironment);
|
||||
return {
|
||||
@@ -784,13 +790,19 @@ export function readConfigFromEnvironment(processEnvironment: HostEnvironment):
|
||||
storage: host.storage,
|
||||
},
|
||||
moduleSecrets: host.moduleSecrets,
|
||||
modulesDirectory: host.modulesDirectory,
|
||||
};
|
||||
}
|
||||
|
||||
export async function main(): Promise<void> {
|
||||
const { environment, input, moduleSecrets } = readConfigFromEnvironment(process.env);
|
||||
const { environment, input, moduleSecrets, modulesDirectory } = readConfigFromEnvironment(
|
||||
process.env,
|
||||
);
|
||||
const config = createApiRuntimeConfig(input, { environment });
|
||||
const running = await startApi(config, { moduleSecrets });
|
||||
const running = await startApi(config, {
|
||||
configuredModulesDirectory: modulesDirectory,
|
||||
moduleSecrets,
|
||||
});
|
||||
|
||||
const address = running.server.address();
|
||||
const boundPort = address && typeof address === "object" ? address.port : config.port;
|
||||
|
||||
@@ -2590,6 +2590,7 @@ describe.sequential("live API — installed module permission end-to-end", () =>
|
||||
});
|
||||
|
||||
const report = await diagnoseEkklesia({
|
||||
coreMigrationsDirectory,
|
||||
moduleDiscovery: {
|
||||
moduleSdkVersion: "0.1.0",
|
||||
modulesDirectory,
|
||||
@@ -2599,10 +2600,8 @@ describe.sequential("live API — installed module permission end-to-end", () =>
|
||||
runtimeVersion: process.versions.node,
|
||||
});
|
||||
|
||||
// Only the module-discovery and permission-composition checks are asserted here — this
|
||||
// fixture's temporary directory has no `core/drizzle` sibling for migrations discovery to
|
||||
// find, which is unrelated to what this scenario proves and would otherwise make the check
|
||||
// fail for a reason that has nothing to do with permission composition.
|
||||
// Only the module-discovery and permission-composition checks are asserted here; nothing else
|
||||
// in the report bears on what this scenario proves.
|
||||
const moduleCheck = report.checks.find((entry) => entry.id === "modules.discovery");
|
||||
const permissionsCheck = report.checks.find((entry) => entry.id === "modules.permissions");
|
||||
expect(moduleCheck).toMatchObject({ status: "pass" });
|
||||
|
||||
@@ -151,6 +151,13 @@ describe("host configuration parity between startApi and Doctor", () => {
|
||||
);
|
||||
});
|
||||
|
||||
it("hands the API the same modules directory Doctor and the operator commands read", () => {
|
||||
expect(readConfigFromEnvironment(environment()).modulesDirectory).toBeUndefined();
|
||||
const env = environment({ EKKLESIA_MODULES_DIR: "/var/lib/ekklesia/modules" });
|
||||
expect(readConfigFromEnvironment(env).modulesDirectory).toBe("/var/lib/ekklesia/modules");
|
||||
expect(loadHostConfiguration(env).modulesDirectory).toBe("/var/lib/ekklesia/modules");
|
||||
});
|
||||
|
||||
it("reads only this listener's own address and port outside the shared loader", () => {
|
||||
const config = apiView(
|
||||
environment({ EKKLESIA_API_HOST: "0.0.0.0", EKKLESIA_API_PORT: "8080" }),
|
||||
|
||||
@@ -1004,6 +1004,73 @@ describe("Ekklesia API module permission composition at startup", () => {
|
||||
expect(poolEnd).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("discovers installed modules from the configured modules directory (ADR 0048)", async () => {
|
||||
const modulesDirectory = await createTemporaryModulesDirectory();
|
||||
await createFixtureModule(modulesDirectory, "incompatible", {
|
||||
compatibility: { moduleSdk: "^0.1.0", platform: ">=99.0.0" },
|
||||
});
|
||||
|
||||
// The module that fails discovery lives only in the configured directory, so this rejection
|
||||
// proves discovery read it rather than the repository's own `modules/`.
|
||||
await expect(
|
||||
startApi(validConfig(), { configuredModulesDirectory: modulesDirectory }),
|
||||
).rejects.toBeInstanceOf(ModuleDiscoveryError);
|
||||
expect(poolConstructor).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("starts with zero modules from an empty configured modules directory", async () => {
|
||||
const modulesDirectory = await createTemporaryModulesDirectory();
|
||||
poolConstructor.mockImplementation(function fakePool() {
|
||||
return { end: vi.fn().mockResolvedValue(undefined), on: vi.fn(), query: emptyQuery() };
|
||||
});
|
||||
betterAuthMock.mockReturnValue({ api: { getSession: vi.fn() }, handler: vi.fn() });
|
||||
mockSuccessfulServe();
|
||||
|
||||
const running = await startApi(validConfig(), { configuredModulesDirectory: modulesDirectory });
|
||||
|
||||
expect(poolConstructor).toHaveBeenCalledTimes(1);
|
||||
await running.shutdown();
|
||||
});
|
||||
|
||||
it.each([
|
||||
["a relative path", "relative/private-mount-5b2a", "MODULES_DIRECTORY_INVALID"],
|
||||
["a missing directory", "/nonexistent/private-mount-5b2a", "MODULES_DIRECTORY_UNREADABLE"],
|
||||
])("fails closed on %s, before any pool exists, naming no path", async (_label, configured, code) => {
|
||||
let captured: unknown;
|
||||
try {
|
||||
await startApi(validConfig(), { configuredModulesDirectory: configured });
|
||||
} catch (error) {
|
||||
captured = error;
|
||||
}
|
||||
|
||||
expect(captured).toBeInstanceOf(ModuleDiscoveryError);
|
||||
const issues = (captured as ModuleDiscoveryError).issues;
|
||||
expect(issues.map((issue) => issue.code)).toEqual([code]);
|
||||
expect(JSON.stringify(issues)).not.toContain("private-mount-5b2a");
|
||||
expect((captured as Error).message).not.toContain("private-mount-5b2a");
|
||||
expect(poolConstructor).not.toHaveBeenCalled();
|
||||
expect(betterAuthMock).not.toHaveBeenCalled();
|
||||
expect(serveMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("lets the test-only override take precedence over a configured directory", async () => {
|
||||
const modulesDirectory = await createTemporaryModulesDirectory();
|
||||
poolConstructor.mockImplementation(function fakePool() {
|
||||
return { end: vi.fn().mockResolvedValue(undefined), on: vi.fn(), query: emptyQuery() };
|
||||
});
|
||||
betterAuthMock.mockReturnValue({ api: { getSession: vi.fn() }, handler: vi.fn() });
|
||||
mockSuccessfulServe();
|
||||
|
||||
// The configured value is unusable; the override wins, so it is never even resolved.
|
||||
const running = await startApi(validConfig(), {
|
||||
configuredModulesDirectory: "relative/never-resolved",
|
||||
modulesDirectoryOverride: modulesDirectory,
|
||||
});
|
||||
|
||||
expect(poolConstructor).toHaveBeenCalledTimes(1);
|
||||
await running.shutdown();
|
||||
});
|
||||
|
||||
it("does not perform module discovery merely by importing the platform-permissions module", async () => {
|
||||
const module = await import("../src/platform-permissions.js");
|
||||
|
||||
|
||||
+23
-3
@@ -8,6 +8,7 @@ import {
|
||||
formatDoctorReport,
|
||||
} from "../diagnostics/index.js";
|
||||
import { type HostEnvironment, loadHostConfiguration } from "../host/index.js";
|
||||
import { type ModulesDirectorySelection, resolveModulesDirectory } from "../modules/index.js";
|
||||
|
||||
interface ModuleSdkPackageDocument {
|
||||
readonly version?: unknown;
|
||||
@@ -25,8 +26,10 @@ interface RootPackageDocument extends ModuleSdkPackageDocument {
|
||||
*/
|
||||
export interface DoctorCommandMetadata {
|
||||
readonly moduleSdkVersion: string;
|
||||
readonly modulesDirectory: string;
|
||||
readonly platformVersion: string;
|
||||
/** Where the platform is installed: Core's migrations live beneath it, and so does the
|
||||
* development default modules directory. Installed modules may live anywhere else. */
|
||||
readonly repositoryRoot: string;
|
||||
readonly runtimeRange: string;
|
||||
readonly runtimeVersion: string;
|
||||
}
|
||||
@@ -49,6 +52,22 @@ function requiredString(value: unknown): string {
|
||||
return value;
|
||||
}
|
||||
|
||||
/**
|
||||
* The modules directory exactly as `startApi` would choose it. An unusable configured value is not
|
||||
* thrown here, because Doctor must still produce a report: it is handed on as a required directory
|
||||
* for discovery to refuse, which reports it as a sanitized module-discovery failure.
|
||||
*/
|
||||
function doctorModulesDirectory(
|
||||
configured: string | undefined,
|
||||
repositoryRoot: string,
|
||||
): ModulesDirectorySelection {
|
||||
try {
|
||||
return resolveModulesDirectory({ configured, repositoryRoot });
|
||||
} catch {
|
||||
return { modulesDirectory: configured ?? "", modulesDirectoryRequired: true };
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Turns one deployment environment into Doctor's inputs.
|
||||
*
|
||||
@@ -69,6 +88,7 @@ export function createDoctorOptions(
|
||||
const host = loadHostConfiguration(environment);
|
||||
return {
|
||||
clientAddress: host.clientAddress,
|
||||
coreMigrationsDirectory: path.join(metadata.repositoryRoot, "core", "drizzle"),
|
||||
databaseEnvironment: host.environment,
|
||||
databaseTransport: host.databaseTransport,
|
||||
databaseUrl: host.databaseUrl,
|
||||
@@ -76,7 +96,7 @@ export function createDoctorOptions(
|
||||
mailConfiguration: host.mail,
|
||||
moduleDiscovery: {
|
||||
moduleSdkVersion: metadata.moduleSdkVersion,
|
||||
modulesDirectory: metadata.modulesDirectory,
|
||||
...doctorModulesDirectory(host.modulesDirectory, metadata.repositoryRoot),
|
||||
platformVersion: metadata.platformVersion,
|
||||
},
|
||||
moduleSecrets: host.moduleSecrets,
|
||||
@@ -97,8 +117,8 @@ export async function runDoctorCommand(repositoryRoot = process.cwd()): Promise<
|
||||
const report = await diagnoseEkklesia(
|
||||
createDoctorOptions(process.env, {
|
||||
moduleSdkVersion: requiredString(moduleSdkPackage.version),
|
||||
modulesDirectory: path.join(repositoryRoot, "modules"),
|
||||
platformVersion: requiredString(rootPackage.version),
|
||||
repositoryRoot,
|
||||
runtimeRange: requiredString(rootPackage.engines?.node),
|
||||
runtimeVersion: process.versions.node,
|
||||
}),
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
import { readFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import type { HostEnvironment } from "../host/index.js";
|
||||
import { discoverAllMigrations } from "../migrations/index.js";
|
||||
import type { ModuleMigrationAggregate } from "../modules/index.js";
|
||||
import { resolveCommandModulesDirectory } from "./modules-directory-environment.js";
|
||||
|
||||
interface PackageVersion {
|
||||
readonly version?: unknown;
|
||||
@@ -15,9 +17,16 @@ async function readPackageVersion(packagePath: string): Promise<string> {
|
||||
return packageData.version;
|
||||
}
|
||||
|
||||
/**
|
||||
* Core's migrations come from the platform (`<repositoryRoot>/core/drizzle`); installed modules'
|
||||
* come from the configured modules directory, wherever it is (ADR 0048). Neither is ever derived
|
||||
* from the other.
|
||||
*/
|
||||
export async function discoverRepositoryMigrations(
|
||||
repositoryRoot = process.cwd(),
|
||||
environment: HostEnvironment = process.env,
|
||||
): Promise<ModuleMigrationAggregate> {
|
||||
const modulesDirectory = resolveCommandModulesDirectory(environment, repositoryRoot);
|
||||
const [platformVersion, moduleSdkVersion] = await Promise.all([
|
||||
readPackageVersion(path.join(repositoryRoot, "package.json")),
|
||||
readPackageVersion(path.join(repositoryRoot, "packages/module-sdk/package.json")),
|
||||
@@ -27,7 +36,7 @@ export async function discoverRepositoryMigrations(
|
||||
coreMigrationsDirectory: path.join(repositoryRoot, "core", "drizzle"),
|
||||
moduleDiscovery: {
|
||||
moduleSdkVersion,
|
||||
modulesDirectory: path.join(repositoryRoot, "modules"),
|
||||
...modulesDirectory,
|
||||
platformVersion,
|
||||
},
|
||||
});
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
import type { ModuleMigrationAggregate } from "../modules/index.js";
|
||||
import type { MigrationApplyResult, MigrationPlanResult } from "../migrations/index.js";
|
||||
import type {
|
||||
MigrationApplyResult,
|
||||
MigrationPlanResult,
|
||||
RetainedMigrationHistory,
|
||||
} from "../migrations/index.js";
|
||||
|
||||
function plural(count: number, singular: string): string {
|
||||
return count === 1 ? singular : `${singular}s`;
|
||||
@@ -28,10 +32,19 @@ export function formatMigrationValidationReport(
|
||||
return Object.freeze(lines);
|
||||
}
|
||||
|
||||
/** An absent module's applied history (ADR 0048): kept, reported, and never a reason to stop. */
|
||||
function retainedLines(retained: readonly RetainedMigrationHistory[]): readonly string[] {
|
||||
return retained.map(
|
||||
(module) =>
|
||||
`- [retained] ${module.moduleId}: ${module.appliedCount} applied ${plural(module.appliedCount, "migration")} kept; the module is not installed`,
|
||||
);
|
||||
}
|
||||
|
||||
export function formatMigrationPlanReport(plan: MigrationPlanResult): readonly string[] {
|
||||
const readyCount = plan.pendingCount - plan.approvalRequiredCount;
|
||||
const retainedSummary = plan.retained.length === 0 ? "" : `, ${plan.retained.length} retained`;
|
||||
const lines = [
|
||||
`Migration plan (${plan.appliedCount} applied, ${readyCount} pending, ${plan.approvalRequiredCount} approval-blocked):`,
|
||||
`Migration plan (${plan.appliedCount} applied, ${readyCount} pending, ${plan.approvalRequiredCount} approval-blocked${retainedSummary}):`,
|
||||
];
|
||||
|
||||
for (const entry of plan.entries) {
|
||||
@@ -40,6 +53,7 @@ export function formatMigrationPlanReport(plan: MigrationPlanResult): readonly s
|
||||
if (plan.entries.length === 0) {
|
||||
lines.push("- No installed module migrations.");
|
||||
}
|
||||
lines.push(...retainedLines(plan.retained));
|
||||
lines.push(
|
||||
"Informational only: this plan is read-only and may become stale; apply revalidates history under the migration lock.",
|
||||
);
|
||||
@@ -49,7 +63,10 @@ export function formatMigrationPlanReport(plan: MigrationPlanResult): readonly s
|
||||
|
||||
export function formatMigrationApplyReport(result: MigrationApplyResult): readonly string[] {
|
||||
if (result.noOp) {
|
||||
return Object.freeze(["Migration apply completed: database is already current (no changes)."]);
|
||||
return Object.freeze([
|
||||
"Migration apply completed: database is already current (no changes).",
|
||||
...retainedLines(result.retained),
|
||||
]);
|
||||
}
|
||||
|
||||
const lines = [
|
||||
@@ -58,5 +75,6 @@ export function formatMigrationApplyReport(result: MigrationApplyResult): readon
|
||||
for (const migration of result.applied) {
|
||||
lines.push(`- ${migration.identity} sha256=${migration.checksum}`);
|
||||
}
|
||||
lines.push(...retainedLines(result.retained));
|
||||
return Object.freeze(lines);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
import { type HostEnvironment, loadHostConfiguration } from "../host/index.js";
|
||||
import { type ModulesDirectorySelection, resolveModulesDirectory } from "../modules/index.js";
|
||||
|
||||
/**
|
||||
* The modules directory an operator command discovers, read through the same host configuration
|
||||
* `startApi` and Doctor use and resolved by the one Core resolver (ADR 0048). Throws the resolver's
|
||||
* sanitized `ModuleDiscoveryError` for an unusable value, which every command already reports.
|
||||
*/
|
||||
export function resolveCommandModulesDirectory(
|
||||
environment: HostEnvironment,
|
||||
repositoryRoot: string,
|
||||
): ModulesDirectorySelection {
|
||||
return resolveModulesDirectory({
|
||||
configured: loadHostConfiguration(environment).modulesDirectory,
|
||||
repositoryRoot,
|
||||
});
|
||||
}
|
||||
@@ -12,6 +12,7 @@ import {
|
||||
resolveModuleEligibility,
|
||||
} from "../modules/index.js";
|
||||
import { resolveExecutableDatabaseTransport } from "./database-transport-environment.js";
|
||||
import { resolveCommandModulesDirectory } from "./modules-directory-environment.js";
|
||||
|
||||
/**
|
||||
* Enables or disables one installed module, out of band from the running API.
|
||||
@@ -99,7 +100,7 @@ async function main(): Promise<void> {
|
||||
// configurable here, which is the case this command exists for.
|
||||
const discovered = await discoverModules({
|
||||
moduleSdkVersion,
|
||||
modulesDirectory: path.join(repositoryRoot, "modules"),
|
||||
...resolveCommandModulesDirectory(process.env, repositoryRoot),
|
||||
platformVersion,
|
||||
});
|
||||
if (!discovered.some((module) => module.manifest.id === moduleId)) {
|
||||
|
||||
@@ -9,6 +9,7 @@ import {
|
||||
ModulePermissionCompositionError,
|
||||
ModuleSchemaDiscoveryError,
|
||||
} from "../modules/index.js";
|
||||
import { resolveCommandModulesDirectory } from "./modules-directory-environment.js";
|
||||
|
||||
interface PackageVersion {
|
||||
version?: unknown;
|
||||
@@ -30,7 +31,7 @@ async function main(): Promise<void> {
|
||||
]);
|
||||
const schemaAggregate = await discoverModuleSchemas({
|
||||
moduleSdkVersion,
|
||||
modulesDirectory: path.join(repositoryRoot, "modules"),
|
||||
...resolveCommandModulesDirectory(process.env, repositoryRoot),
|
||||
platformVersion,
|
||||
});
|
||||
const modules = schemaAggregate.discoveredModules;
|
||||
|
||||
@@ -1,4 +1,3 @@
|
||||
import path from "node:path";
|
||||
import {
|
||||
capabilityIdSchema,
|
||||
EMPTY_MODULE_SECRET_SOURCE,
|
||||
@@ -43,6 +42,7 @@ import {
|
||||
} from "../mail/index.js";
|
||||
import {
|
||||
createPostgresMigrationDatabase,
|
||||
type DiscoverAllMigrationsOptions,
|
||||
discoverAllMigrations,
|
||||
type MigrationDatabase,
|
||||
type MigrationPlanResult,
|
||||
@@ -64,6 +64,7 @@ import {
|
||||
isConfigurationBlockedDetail,
|
||||
isStorageBlockedDetail,
|
||||
type ModuleConfigurationSettings,
|
||||
ModuleDiscoveryError,
|
||||
type ModuleDiscoveryOptions,
|
||||
type ModuleEligibility,
|
||||
type ModuleLifecycleConfiguration,
|
||||
@@ -126,6 +127,9 @@ export interface DoctorIdentityConfigurationOptions {
|
||||
|
||||
export interface DoctorOptions {
|
||||
readonly clientAddress?: ClientAddressConfigInput | undefined;
|
||||
/** Core's own migrations, where the platform is installed. Never derived from the modules
|
||||
* directory: installed modules live in installation-owned storage, anywhere at all (ADR 0048). */
|
||||
readonly coreMigrationsDirectory: string;
|
||||
/** The environment the executable resolved, used only for transport policy evaluation; identity
|
||||
* configuration keeps its own `identityConfiguration.environment` for the same purpose. */
|
||||
readonly databaseEnvironment?: DatabaseTransportEnvironment | undefined;
|
||||
@@ -170,7 +174,7 @@ export interface DoctorDependencies {
|
||||
options: PostgresMigrationDatabaseOptions,
|
||||
) => MigrationDatabase;
|
||||
readonly discoverMigrations: (
|
||||
options: ModuleDiscoveryOptions,
|
||||
options: DiscoverAllMigrationsOptions,
|
||||
) => Promise<ModuleMigrationAggregate>;
|
||||
readonly discoverModules: (
|
||||
options: ModuleDiscoveryOptions,
|
||||
@@ -207,15 +211,6 @@ const checksumPattern = /^[a-f0-9]{64}$/u;
|
||||
const migrationIdentityPattern =
|
||||
/^[a-z][a-z0-9]*(?:-[a-z0-9]+)*:\d{4,}_[a-z][a-z0-9]*(?:_[a-z0-9]+)*$/u;
|
||||
|
||||
async function discoverAllMigrationsForRuntime(
|
||||
options: ModuleDiscoveryOptions,
|
||||
): Promise<ModuleMigrationAggregate> {
|
||||
return discoverAllMigrations({
|
||||
coreMigrationsDirectory: path.join(path.dirname(options.modulesDirectory), "core", "drizzle"),
|
||||
moduleDiscovery: options,
|
||||
});
|
||||
}
|
||||
|
||||
/** Opens its own short-lived connection, reads, and always closes it — Doctor never leaves a pool
|
||||
* behind, and never shares one with the migration planner's, whose lifetime it does not control. */
|
||||
async function readModuleLifecycleForRuntime(
|
||||
@@ -273,7 +268,7 @@ async function probeMailForRuntime(config: MailProviderConfig): Promise<void> {
|
||||
const productionDependencies: DoctorDependencies = Object.freeze({
|
||||
composePermissions: composePlatformPermissionRegistry,
|
||||
createMigrationDatabase: createPostgresMigrationDatabase,
|
||||
discoverMigrations: discoverAllMigrationsForRuntime,
|
||||
discoverMigrations: discoverAllMigrations,
|
||||
discoverModules,
|
||||
discoverSchemas: discoverModuleSchemas,
|
||||
planMigrations: planModuleMigrations,
|
||||
@@ -633,11 +628,23 @@ function storageBindingsCheck(
|
||||
details,
|
||||
);
|
||||
}
|
||||
function moduleDiscoveryFailureDetail(error: unknown): string {
|
||||
if (!(error instanceof ModuleDiscoveryError)) {
|
||||
return "Static module discovery could not be completed.";
|
||||
}
|
||||
// The modules directory itself, rather than anything in it (ADR 0048). Fixed wording, never the
|
||||
// configured value: a path is machine detail.
|
||||
if (error.issues.some((issue) => issue.code === "MODULES_DIRECTORY_INVALID")) {
|
||||
return "The configured modules directory is not usable.";
|
||||
}
|
||||
if (error.issues.some((issue) => issue.code === "MODULES_DIRECTORY_UNREADABLE")) {
|
||||
return "The configured modules directory could not be read.";
|
||||
}
|
||||
return "The installed module inventory is invalid.";
|
||||
}
|
||||
|
||||
function moduleDiscoveryFailure(error: unknown): DoctorCheck {
|
||||
const details =
|
||||
error instanceof Error && error.name === "ModuleDiscoveryError"
|
||||
? ["The installed module inventory is invalid."]
|
||||
: ["Static module discovery could not be completed."];
|
||||
const details = [moduleDiscoveryFailureDetail(error)];
|
||||
return freezeCheck("modules.discovery", "failure", "Module discovery failed", details);
|
||||
}
|
||||
|
||||
@@ -733,6 +740,14 @@ function migrationPlanCheck(plan: MigrationPlanResult): DoctorCheck {
|
||||
}
|
||||
return `${entry.status} · ${safeMigrationReference(entry.identity, entry.checksum)}`;
|
||||
}),
|
||||
// An absent module's applied history (ADR 0048) is information, never attention: removing a
|
||||
// module's code keeps its data, and nothing here is pending.
|
||||
...plan.retained.map((module) => {
|
||||
if (!/^[a-z][a-z0-9]*(?:-[a-z0-9]+)*$/u.test(module.moduleId)) {
|
||||
throw new TypeError("Unsafe migration plan diagnostic value.");
|
||||
}
|
||||
return `retained · ${module.moduleId} · ${module.appliedCount} applied ${plural(module.appliedCount, "migration")} kept; the module is not installed`;
|
||||
}),
|
||||
];
|
||||
|
||||
if (pendingCount + blockedCount === 0) {
|
||||
@@ -1132,7 +1147,10 @@ export async function diagnoseEkklesia(
|
||||
}
|
||||
|
||||
try {
|
||||
migrations = await dependencies.discoverMigrations(options.moduleDiscovery);
|
||||
migrations = await dependencies.discoverMigrations({
|
||||
coreMigrationsDirectory: options.coreMigrationsDirectory,
|
||||
moduleDiscovery: options.moduleDiscovery,
|
||||
});
|
||||
const inventory = migrations.migrations.map((migration) =>
|
||||
safeMigrationReference(migration.identity, migration.checksum),
|
||||
);
|
||||
|
||||
@@ -59,6 +59,10 @@ export interface HostConfiguration {
|
||||
/** The deployment's module secrets, resolvable only under names Core derives from a module's own
|
||||
* declared keys — there is no arbitrary variable lookup. */
|
||||
readonly moduleSecrets: ModuleSecretSource;
|
||||
/** `EKKLESIA_MODULES_DIR`, raw and unvalidated: `undefined` only when the variable is unset. Every
|
||||
* executable passes it to `resolveModulesDirectory`, the one place it is interpreted (ADR 0048),
|
||||
* so a relative or blank value is refused there rather than repaired here. */
|
||||
readonly modulesDirectory: string | undefined;
|
||||
readonly storage: StorageProviderConfigInput;
|
||||
}
|
||||
|
||||
@@ -144,6 +148,7 @@ export function loadHostConfiguration(environment: HostEnvironment): HostConfigu
|
||||
username: environment.EKKLESIA_SMTP_USERNAME,
|
||||
},
|
||||
moduleSecrets: createEnvironmentModuleSecretSource(environment),
|
||||
modulesDirectory: environment.EKKLESIA_MODULES_DIR,
|
||||
// The `S3_*` names ADR 0004 already established for this repository's Compose stack. Absent
|
||||
// endpoint means this installation has no object storage, which is a complete deployment
|
||||
// (ADR 0030).
|
||||
|
||||
@@ -2,6 +2,7 @@ import type {
|
||||
DiscoveredModuleMigration,
|
||||
ModuleMigrationAggregate,
|
||||
} from "../modules/migration-discovery.js";
|
||||
import { CORE_MIGRATION_SOURCE_ID } from "./core-source.js";
|
||||
|
||||
const DEFAULT_MIGRATION_LOCK_TIMEOUT_MS = 10_000;
|
||||
|
||||
@@ -86,11 +87,24 @@ export interface MigrationPlanEntry {
|
||||
readonly tag: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Applied history kept for a module that is not installed (ADR 0048): the module id and how many of
|
||||
* its migrations were applied. It is reported, never matched against an inventory, and never a
|
||||
* reason to refuse planning or applying anything else.
|
||||
*/
|
||||
export interface RetainedMigrationHistory {
|
||||
readonly appliedCount: number;
|
||||
readonly moduleId: string;
|
||||
}
|
||||
|
||||
export interface MigrationPlanResult {
|
||||
readonly appliedCount: number;
|
||||
readonly approvalRequiredCount: number;
|
||||
readonly entries: readonly MigrationPlanEntry[];
|
||||
readonly pendingCount: number;
|
||||
/** Absent modules' applied history, ordered by module id; empty when every applied migration
|
||||
* belongs to Core or an installed module. */
|
||||
readonly retained: readonly RetainedMigrationHistory[];
|
||||
}
|
||||
|
||||
export interface AppliedMigrationResult {
|
||||
@@ -105,6 +119,8 @@ export interface MigrationApplyResult {
|
||||
readonly applied: readonly AppliedMigrationResult[];
|
||||
readonly appliedCount: number;
|
||||
readonly noOp: boolean;
|
||||
/** The same report as `MigrationPlanResult.retained`, read under the migration lock. */
|
||||
readonly retained: readonly RetainedMigrationHistory[];
|
||||
}
|
||||
|
||||
export interface PlanModuleMigrationsOptions {
|
||||
@@ -204,10 +220,38 @@ function validateHistoryRecordShape(record: AppliedMigrationRecord): void {
|
||||
}
|
||||
}
|
||||
|
||||
interface ValidatedAppliedHistory {
|
||||
/** Identities of applied migrations that belong to Core or an installed module. */
|
||||
readonly applied: ReadonlySet<string>;
|
||||
readonly retained: readonly RetainedMigrationHistory[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Every module id this inventory treats as installed: each discovered module, whether or not it
|
||||
* owns migrations, every module that contributed migrations, and always Core. A module counts as
|
||||
* absent only when it is none of these, so a module that is installed but no longer ships its old
|
||||
* migrations stays strict, and Core's history can never be classified as an absent module's.
|
||||
*/
|
||||
function installedMigrationSources(aggregate: ModuleMigrationAggregate): ReadonlySet<string> {
|
||||
return new Set<string>([
|
||||
CORE_MIGRATION_SOURCE_ID,
|
||||
...aggregate.discoveredModules.map((module) => module.manifest.id),
|
||||
...aggregate.modules.map((module) => module.moduleId),
|
||||
]);
|
||||
}
|
||||
|
||||
function freezeRetained(counts: ReadonlyMap<string, number>): readonly RetainedMigrationHistory[] {
|
||||
return Object.freeze(
|
||||
[...counts.entries()]
|
||||
.sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0))
|
||||
.map(([moduleId, appliedCount]) => Object.freeze({ appliedCount, moduleId })),
|
||||
);
|
||||
}
|
||||
|
||||
function validateAppliedHistoryUnchecked(
|
||||
aggregate: ModuleMigrationAggregate,
|
||||
history: readonly AppliedMigrationRecord[],
|
||||
): ReadonlySet<string> {
|
||||
): ValidatedAppliedHistory {
|
||||
if (!Array.isArray(history)) {
|
||||
throw runnerError("MIGRATION_HISTORY_INVALID");
|
||||
}
|
||||
@@ -215,6 +259,9 @@ function validateAppliedHistoryUnchecked(
|
||||
const inventory = new Map(
|
||||
aggregate.migrations.map((migration) => [migration.identity, migration]),
|
||||
);
|
||||
const installed = installedMigrationSources(aggregate);
|
||||
const applied = new Set<string>();
|
||||
const retained = new Map<string, number>();
|
||||
const seenOrders = new Set<number>();
|
||||
const seenIdentities = new Set<string>();
|
||||
const seenModuleIndexes = new Set<string>();
|
||||
@@ -240,7 +287,19 @@ function validateAppliedHistoryUnchecked(
|
||||
|
||||
const migration = knownMigration(inventory, record.identity);
|
||||
if (!migration) {
|
||||
throw runnerError("MIGRATION_UNKNOWN_APPLIED");
|
||||
if (installed.has(record.moduleId)) {
|
||||
// Core, or a module that is installed now: its history is strict, exactly as ADR 0009
|
||||
// established. A migration it no longer ships is a downgrade or a rewritten package.
|
||||
throw runnerError("MIGRATION_UNKNOWN_APPLIED");
|
||||
}
|
||||
// An absent module's history (ADR 0048): retained and reported, never repaired, deleted, or
|
||||
// matched against anything. It must still name its own module and tag, so a corrupt row is
|
||||
// never mistaken for one.
|
||||
if (record.identity !== `${record.moduleId}:${record.tag}`) {
|
||||
throw runnerError("MIGRATION_HISTORY_INVALID");
|
||||
}
|
||||
retained.set(record.moduleId, (retained.get(record.moduleId) ?? 0) + 1);
|
||||
continue;
|
||||
}
|
||||
if (
|
||||
record.moduleId !== migration.moduleId ||
|
||||
@@ -252,11 +311,12 @@ function validateAppliedHistoryUnchecked(
|
||||
if (record.checksum !== migration.checksum) {
|
||||
throw runnerError("MIGRATION_CHECKSUM_MISMATCH", migration.identity);
|
||||
}
|
||||
applied.add(record.identity);
|
||||
}
|
||||
|
||||
for (const module of aggregate.modules) {
|
||||
const applied = history.filter((record) => record.moduleId === module.moduleId);
|
||||
for (const [position, record] of applied.entries()) {
|
||||
const moduleHistory = history.filter((record) => record.moduleId === module.moduleId);
|
||||
for (const [position, record] of moduleHistory.entries()) {
|
||||
if (record.identity !== module.migrations[position]?.identity) {
|
||||
const migration = knownMigration(inventory, record.identity);
|
||||
throw runnerError("MIGRATION_HISTORY_INVALID", migration?.identity);
|
||||
@@ -264,13 +324,13 @@ function validateAppliedHistoryUnchecked(
|
||||
}
|
||||
}
|
||||
|
||||
return seenIdentities;
|
||||
return Object.freeze({ applied, retained: freezeRetained(retained) });
|
||||
}
|
||||
|
||||
function validateAppliedHistory(
|
||||
aggregate: ModuleMigrationAggregate,
|
||||
history: readonly AppliedMigrationRecord[],
|
||||
): ReadonlySet<string> {
|
||||
): ValidatedAppliedHistory {
|
||||
try {
|
||||
return validateAppliedHistoryUnchecked(aggregate, history);
|
||||
} catch (error) {
|
||||
@@ -285,7 +345,7 @@ function planFromHistory(
|
||||
aggregate: ModuleMigrationAggregate,
|
||||
history: readonly AppliedMigrationRecord[],
|
||||
): MigrationPlanResult {
|
||||
const applied = validateAppliedHistory(aggregate, history);
|
||||
const { applied, retained } = validateAppliedHistory(aggregate, history);
|
||||
const entries = Object.freeze(
|
||||
aggregate.migrations.map((migration): MigrationPlanEntry => {
|
||||
const isApplied = applied.has(migration.identity);
|
||||
@@ -314,6 +374,7 @@ function planFromHistory(
|
||||
approvalRequiredCount,
|
||||
entries,
|
||||
pendingCount: entries.length - appliedCount,
|
||||
retained,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -472,14 +533,14 @@ export async function applyModuleMigrations(
|
||||
} catch {
|
||||
throw runnerError("MIGRATION_HISTORY_READ_FAILED");
|
||||
}
|
||||
const applied = validateAppliedHistory(options.aggregate, history);
|
||||
const { applied, retained } = validateAppliedHistory(options.aggregate, history);
|
||||
const pending = Object.freeze(
|
||||
options.aggregate.migrations.filter((migration) => !applied.has(migration.identity)),
|
||||
);
|
||||
await preflightApprovals(pending, options.approvals ?? denyAllMigrationApprovals);
|
||||
|
||||
if (pending.length === 0) {
|
||||
return Object.freeze({ applied: Object.freeze([]), appliedCount: 0, noOp: true });
|
||||
return Object.freeze({ applied: Object.freeze([]), appliedCount: 0, noOp: true, retained });
|
||||
}
|
||||
|
||||
let executingIdentity: string | undefined;
|
||||
@@ -508,6 +569,7 @@ export async function applyModuleMigrations(
|
||||
applied: resultMigrations,
|
||||
appliedCount: resultMigrations.length,
|
||||
noOp: false,
|
||||
retained,
|
||||
});
|
||||
} catch (error) {
|
||||
primaryError = error;
|
||||
|
||||
@@ -47,6 +47,7 @@ export type ModuleDiscoveryIssueCode =
|
||||
| "MODULE_PLATFORM_INCOMPATIBLE"
|
||||
| "MODULE_SDK_INCOMPATIBLE"
|
||||
| "MODULE_VERSION_MISMATCH"
|
||||
| "MODULES_DIRECTORY_INVALID"
|
||||
| "MODULES_DIRECTORY_UNREADABLE";
|
||||
|
||||
export interface ModuleDiscoveryIssue {
|
||||
@@ -117,9 +118,35 @@ export interface DiscoveredModule {
|
||||
export interface ModuleDiscoveryOptions {
|
||||
readonly moduleSdkVersion: string;
|
||||
readonly modulesDirectory: string;
|
||||
/**
|
||||
* True when an operator configured the directory explicitly (`EKKLESIA_MODULES_DIR`, ADR 0048).
|
||||
* A configured directory that is missing, not a directory, or not an absolute path then fails
|
||||
* discovery closed rather than reading as zero modules: a mistyped mount must never quietly start
|
||||
* the platform with nothing installed. False or omitted keeps the development default lenient,
|
||||
* where a checkout with no `modules/` simply has no modules.
|
||||
*/
|
||||
readonly modulesDirectoryRequired?: boolean;
|
||||
readonly platformVersion: string;
|
||||
}
|
||||
|
||||
/** The one sanitized message for an unusable configured modules directory. It never carries the
|
||||
* value: a path is machine detail, and a misplaced value could be anything at all. */
|
||||
export const MODULES_DIRECTORY_INVALID_MESSAGE = "EKKLESIA_MODULES_DIR must be an absolute path.";
|
||||
|
||||
/**
|
||||
* Whether a configured modules directory is usable as given: a non-empty absolute path with no
|
||||
* surrounding whitespace and no NUL byte. Deliberately strict rather than forgiving, because
|
||||
* trimming or resolving a near-miss would silently pick a directory the operator did not name.
|
||||
*/
|
||||
export function isUsableModulesDirectoryPath(value: string): boolean {
|
||||
return (
|
||||
value.length > 0 &&
|
||||
value.trim() === value &&
|
||||
!value.includes("\u0000") &&
|
||||
path.isAbsolute(value)
|
||||
);
|
||||
}
|
||||
|
||||
interface ModulePackageData {
|
||||
name: string;
|
||||
version: string;
|
||||
@@ -695,6 +722,13 @@ export async function discoverModules(
|
||||
});
|
||||
}
|
||||
}
|
||||
if (options.modulesDirectoryRequired && !isUsableModulesDirectoryPath(options.modulesDirectory)) {
|
||||
issues.push({
|
||||
code: "MODULES_DIRECTORY_INVALID",
|
||||
message: MODULES_DIRECTORY_INVALID_MESSAGE,
|
||||
moduleDirectory: ".",
|
||||
});
|
||||
}
|
||||
if (issues.length > 0) {
|
||||
throw new ModuleDiscoveryError(issues);
|
||||
}
|
||||
@@ -703,7 +737,7 @@ export async function discoverModules(
|
||||
try {
|
||||
directoryEntries = await readdir(options.modulesDirectory, { withFileTypes: true });
|
||||
} catch (error) {
|
||||
if (errorCode(error) === "ENOENT") {
|
||||
if (errorCode(error) === "ENOENT" && !options.modulesDirectoryRequired) {
|
||||
return Object.freeze([]);
|
||||
}
|
||||
throw new ModuleDiscoveryError([
|
||||
|
||||
@@ -7,6 +7,7 @@ export * from "./lifecycle.js";
|
||||
export * from "./lifecycle-schema.js";
|
||||
export * from "./lifecycle-store.js";
|
||||
export * from "./migration-discovery.js";
|
||||
export * from "./modules-directory.js";
|
||||
export * from "./permissions.js";
|
||||
export * from "./realtime.js";
|
||||
export * from "./runtime.js";
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
import path from "node:path";
|
||||
import {
|
||||
isUsableModulesDirectoryPath,
|
||||
MODULES_DIRECTORY_INVALID_MESSAGE,
|
||||
ModuleDiscoveryError,
|
||||
} from "./discovery.js";
|
||||
|
||||
/** The host environment variable that selects the installed-modules directory (ADR 0048). */
|
||||
export const MODULES_DIRECTORY_ENVIRONMENT_VARIABLE = "EKKLESIA_MODULES_DIR";
|
||||
|
||||
export interface ResolveModulesDirectoryOptions {
|
||||
/** The raw configured value, exactly as the host environment supplied it; `undefined` when the
|
||||
* variable is unset. */
|
||||
readonly configured: string | undefined;
|
||||
/** Where the platform itself is installed. Only consulted for the development default. */
|
||||
readonly repositoryRoot: string;
|
||||
}
|
||||
|
||||
/** Spreads directly into `ModuleDiscoveryOptions`, so every consumer discovers through the same
|
||||
* selection, including whether a missing directory is an error. */
|
||||
export interface ModulesDirectorySelection {
|
||||
readonly modulesDirectory: string;
|
||||
readonly modulesDirectoryRequired: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* The one place an installation's modules directory is chosen (ADR 0048).
|
||||
*
|
||||
* - Unset: `<repositoryRoot>/modules`, the development checkout's own directory. A checkout without
|
||||
* one has no modules, exactly as before.
|
||||
* - Set: an absolute path, normalized and used as given — never resolved against the repository or
|
||||
* the working directory — and required to exist when discovery reads it. The shipping image sets
|
||||
* it to `/var/lib/ekklesia/modules`.
|
||||
*
|
||||
* Anything else throws a `ModuleDiscoveryError` carrying `MODULES_DIRECTORY_INVALID` and a fixed
|
||||
* message, so every executable already reports it through the discovery error path it has, and the
|
||||
* supplied value never reaches an error, a log, or a report.
|
||||
*
|
||||
* Nothing here — and nothing that calls it — locates the platform's own files relative to the
|
||||
* result: Core's migrations and package metadata live under `repositoryRoot`, wherever modules are.
|
||||
*/
|
||||
export function resolveModulesDirectory(
|
||||
options: ResolveModulesDirectoryOptions,
|
||||
): ModulesDirectorySelection {
|
||||
if (options.configured === undefined) {
|
||||
return Object.freeze({
|
||||
modulesDirectory: path.join(options.repositoryRoot, "modules"),
|
||||
modulesDirectoryRequired: false,
|
||||
});
|
||||
}
|
||||
if (!isUsableModulesDirectoryPath(options.configured)) {
|
||||
throw new ModuleDiscoveryError([
|
||||
{
|
||||
code: "MODULES_DIRECTORY_INVALID",
|
||||
message: MODULES_DIRECTORY_INVALID_MESSAGE,
|
||||
moduleDirectory: ".",
|
||||
},
|
||||
]);
|
||||
}
|
||||
const normalized = path.normalize(options.configured);
|
||||
return Object.freeze({
|
||||
modulesDirectory:
|
||||
normalized.length > 1 && normalized.endsWith(path.sep) ? normalized.slice(0, -1) : normalized,
|
||||
modulesDirectoryRequired: true,
|
||||
});
|
||||
}
|
||||
@@ -8,10 +8,11 @@ import {
|
||||
diagnoseEkklesia,
|
||||
formatDoctorReport,
|
||||
} from "../src/diagnostics/index.js";
|
||||
import type {
|
||||
DiscoveredModule,
|
||||
ModuleMigrationAggregate,
|
||||
ModuleSchemaAggregate,
|
||||
import {
|
||||
type DiscoveredModule,
|
||||
discoverModules,
|
||||
type ModuleMigrationAggregate,
|
||||
type ModuleSchemaAggregate,
|
||||
} from "../src/modules/index.js";
|
||||
|
||||
/**
|
||||
@@ -34,8 +35,8 @@ const moduleSecret = "module-secret-material";
|
||||
|
||||
const metadata = Object.freeze({
|
||||
moduleSdkVersion: "0.1.0",
|
||||
modulesDirectory: "/not-rendered/modules",
|
||||
platformVersion: "0.1.0",
|
||||
repositoryRoot: "/not-rendered",
|
||||
runtimeRange: ">=24.11.0 <25",
|
||||
runtimeVersion: "24.15.0",
|
||||
});
|
||||
@@ -454,3 +455,33 @@ describe("Doctor host-configuration parity — sanitization", () => {
|
||||
expect(JSON.stringify(report)).not.toContain("S3_ACCESS_KEY");
|
||||
});
|
||||
});
|
||||
|
||||
describe("Doctor reads the modules directory from the same host configuration (ADR 0048)", () => {
|
||||
it("uses the repository's own modules directory when EKKLESIA_MODULES_DIR is unset", () => {
|
||||
const doctorOptions = createDoctorOptions(environment(), metadata);
|
||||
expect(doctorOptions.moduleDiscovery.modulesDirectory).toBe("/not-rendered/modules");
|
||||
expect(doctorOptions.moduleDiscovery.modulesDirectoryRequired).toBe(false);
|
||||
expect(doctorOptions.coreMigrationsDirectory).toBe("/not-rendered/core/drizzle");
|
||||
});
|
||||
|
||||
it("uses a configured directory, and still finds Core's migrations under the platform", () => {
|
||||
const doctorOptions = createDoctorOptions(
|
||||
environment({ EKKLESIA_MODULES_DIR: "/var/lib/ekklesia/modules" }),
|
||||
metadata,
|
||||
);
|
||||
expect(doctorOptions.moduleDiscovery.modulesDirectory).toBe("/var/lib/ekklesia/modules");
|
||||
expect(doctorOptions.moduleDiscovery.modulesDirectoryRequired).toBe(true);
|
||||
expect(doctorOptions.coreMigrationsDirectory).toBe("/not-rendered/core/drizzle");
|
||||
});
|
||||
|
||||
it("hands an unusable value to discovery to report, rather than throwing before a report exists", async () => {
|
||||
const doctorOptions = createDoctorOptions(
|
||||
environment({ EKKLESIA_MODULES_DIR: "relative/private-mount-91aa" }),
|
||||
metadata,
|
||||
);
|
||||
expect(doctorOptions.moduleDiscovery.modulesDirectoryRequired).toBe(true);
|
||||
const report = await diagnoseEkklesia(doctorOptions, dependencies({ discoverModules }));
|
||||
expect(report.checks.find((entry) => entry.id === "modules.discovery")?.status).toBe("failure");
|
||||
expect(JSON.stringify(report)).not.toContain("private-mount-91aa");
|
||||
});
|
||||
});
|
||||
|
||||
@@ -68,6 +68,7 @@ describe("Ekklesia Doctor terminal report", () => {
|
||||
it("renders the real zero-module and no-database shape exactly", async () => {
|
||||
const report = await diagnoseEkklesia(
|
||||
{
|
||||
coreMigrationsDirectory: "/not-rendered/core/drizzle",
|
||||
moduleDiscovery: {
|
||||
moduleSdkVersion: "0.1.0",
|
||||
modulesDirectory: "/not-rendered/modules",
|
||||
@@ -135,6 +136,7 @@ Result: READY WITH WARNINGS`);
|
||||
it("renders every check the report contains, so none can silently affect the result", async () => {
|
||||
const report = await diagnoseEkklesia(
|
||||
{
|
||||
coreMigrationsDirectory: "/not-rendered/core/drizzle",
|
||||
moduleDiscovery: {
|
||||
moduleSdkVersion: "0.1.0",
|
||||
modulesDirectory: "/not-rendered/modules",
|
||||
@@ -159,6 +161,7 @@ Result: READY WITH WARNINGS`);
|
||||
it("does not depend on NO_COLOR or terminal interactivity", async () => {
|
||||
const report = await diagnoseEkklesia(
|
||||
{
|
||||
coreMigrationsDirectory: "/not-rendered/core/drizzle",
|
||||
moduleDiscovery: {
|
||||
moduleSdkVersion: "0.1.0",
|
||||
modulesDirectory: "/not-rendered/modules",
|
||||
@@ -207,6 +210,7 @@ Result: READY WITH WARNINGS`);
|
||||
const report = await diagnoseEkklesia(
|
||||
{
|
||||
databaseUrl: "postgresql://another:[email protected]/db",
|
||||
coreMigrationsDirectory: "/not-rendered/core/drizzle",
|
||||
moduleDiscovery: {
|
||||
moduleSdkVersion: "0.1.0",
|
||||
modulesDirectory: "/not-rendered/modules",
|
||||
|
||||
+121
-2
@@ -1,5 +1,9 @@
|
||||
import { type ModuleManifest, parsePermissionId } from "@ekklesia/contracts";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { mkdtemp, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { createPermissionRegistry, PermissionRegistryError } from "../src/authorization/index.js";
|
||||
import {
|
||||
type DoctorCheck,
|
||||
@@ -114,7 +118,10 @@ function emptyDatabase(): MigrationDatabase {
|
||||
});
|
||||
}
|
||||
|
||||
function plan(entries: MigrationPlanResult["entries"] = []): MigrationPlanResult {
|
||||
function plan(
|
||||
entries: MigrationPlanResult["entries"] = [],
|
||||
retained: MigrationPlanResult["retained"] = [],
|
||||
): MigrationPlanResult {
|
||||
const appliedCount = entries.filter((entry) => entry.status === "applied").length;
|
||||
const approvalRequiredCount = entries.filter(
|
||||
(entry) => entry.status === "approval-blocked",
|
||||
@@ -124,6 +131,7 @@ function plan(entries: MigrationPlanResult["entries"] = []): MigrationPlanResult
|
||||
approvalRequiredCount,
|
||||
entries: Object.freeze([...entries]),
|
||||
pendingCount: entries.length - appliedCount,
|
||||
retained: Object.freeze([...retained]),
|
||||
});
|
||||
}
|
||||
|
||||
@@ -151,6 +159,7 @@ function dependencies(overrides: Partial<DoctorDependencies> = {}): DoctorDepend
|
||||
function options(databaseUrl?: string): DoctorOptions {
|
||||
return {
|
||||
...(databaseUrl === undefined ? {} : { databaseUrl }),
|
||||
coreMigrationsDirectory: "/not-rendered/core/drizzle",
|
||||
moduleDiscovery: {
|
||||
moduleSdkVersion: "0.1.0",
|
||||
modulesDirectory: "/not-rendered/modules",
|
||||
@@ -1657,3 +1666,113 @@ describe("Doctor reports declared browser contributions", () => {
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("Doctor with installation-owned module storage (ADR 0048)", () => {
|
||||
const platformCoreMigrations = path.join(
|
||||
path.dirname(fileURLToPath(import.meta.url)),
|
||||
"..",
|
||||
"drizzle",
|
||||
);
|
||||
const temporaryDirectories: string[] = [];
|
||||
|
||||
afterEach(async () => {
|
||||
await Promise.all(
|
||||
temporaryDirectories.splice(0).map((directory) => rm(directory, { recursive: true })),
|
||||
);
|
||||
});
|
||||
|
||||
it("finds Core's migrations where the platform is installed, not beside the modules", async () => {
|
||||
// A modules directory far from the platform, as the image mounts /var/lib/ekklesia/modules. Its
|
||||
// parent has no `core/drizzle`, so a Doctor that still derived Core's migrations from the
|
||||
// modules directory would fail migration discovery here.
|
||||
const external = await mkdtemp(path.join(tmpdir(), "ekklesia-external-modules-"));
|
||||
temporaryDirectories.push(external);
|
||||
const report = await diagnoseEkklesia({
|
||||
coreMigrationsDirectory: platformCoreMigrations,
|
||||
moduleDiscovery: {
|
||||
moduleSdkVersion: "0.1.0",
|
||||
modulesDirectory: external,
|
||||
modulesDirectoryRequired: true,
|
||||
platformVersion: "0.1.0",
|
||||
},
|
||||
runtimeRange: ">=24.11.0 <25",
|
||||
runtimeVersion: "24.15.0",
|
||||
});
|
||||
|
||||
expect(check(report, "modules.discovery").status).toBe("pass");
|
||||
const migrations = check(report, "migrations.discovery");
|
||||
expect(migrations.status).toBe("pass");
|
||||
expect(migrations.details.some((detail) => detail.startsWith("core:0000_"))).toBe(true);
|
||||
expect(JSON.stringify(report)).not.toContain(external);
|
||||
});
|
||||
|
||||
it("reports an unusable configured modules directory as a discovery failure, naming no path", async () => {
|
||||
const report = await diagnoseEkklesia({
|
||||
coreMigrationsDirectory: platformCoreMigrations,
|
||||
moduleDiscovery: {
|
||||
moduleSdkVersion: "0.1.0",
|
||||
modulesDirectory: "relative/private-mount-3c9d",
|
||||
modulesDirectoryRequired: true,
|
||||
platformVersion: "0.1.0",
|
||||
},
|
||||
runtimeRange: ">=24.11.0 <25",
|
||||
runtimeVersion: "24.15.0",
|
||||
});
|
||||
|
||||
const discovery = check(report, "modules.discovery");
|
||||
expect(discovery.status).toBe("failure");
|
||||
expect(discovery.details).toEqual(["The configured modules directory is not usable."]);
|
||||
expect(check(report, "migrations.discovery").status).toBe("skipped");
|
||||
expect(JSON.stringify(report)).not.toContain("private-mount-3c9d");
|
||||
});
|
||||
|
||||
it("reports a configured modules directory that does not exist, naming no path", async () => {
|
||||
const parent = await mkdtemp(path.join(tmpdir(), "ekklesia-missing-modules-"));
|
||||
temporaryDirectories.push(parent);
|
||||
const report = await diagnoseEkklesia({
|
||||
coreMigrationsDirectory: platformCoreMigrations,
|
||||
moduleDiscovery: {
|
||||
moduleSdkVersion: "0.1.0",
|
||||
modulesDirectory: path.join(parent, "absent"),
|
||||
modulesDirectoryRequired: true,
|
||||
platformVersion: "0.1.0",
|
||||
},
|
||||
runtimeRange: ">=24.11.0 <25",
|
||||
runtimeVersion: "24.15.0",
|
||||
});
|
||||
|
||||
const discovery = check(report, "modules.discovery");
|
||||
expect(discovery.status).toBe("failure");
|
||||
expect(discovery.details).toEqual(["The configured modules directory could not be read."]);
|
||||
expect(JSON.stringify(report)).not.toContain(parent);
|
||||
});
|
||||
|
||||
it("reports an absent module's retained history without blocking readiness", async () => {
|
||||
const report = await diagnoseEkklesia(
|
||||
options("postgres://[email protected]/ekklesia"),
|
||||
dependencies({
|
||||
planMigrations: vi.fn(async () =>
|
||||
plan(
|
||||
[planEntry("core:0000_identity_core_tables", "a".repeat(64), "applied")],
|
||||
[Object.freeze({ appliedCount: 2, moduleId: "calendar" })],
|
||||
),
|
||||
),
|
||||
}),
|
||||
);
|
||||
|
||||
const planCheck = check(report, "postgresql.plan");
|
||||
expect(planCheck.status).toBe("pass");
|
||||
expect(planCheck.details).toContain(
|
||||
"retained · calendar · 2 applied migrations kept; the module is not installed",
|
||||
);
|
||||
});
|
||||
|
||||
it("passes Core's own migration directory to migration discovery", async () => {
|
||||
const discoverMigrations = vi.fn(async () => migrationAggregate());
|
||||
await diagnoseEkklesia(options(), dependencies({ discoverMigrations }));
|
||||
expect(discoverMigrations).toHaveBeenCalledWith({
|
||||
coreMigrationsDirectory: "/not-rendered/core/drizzle",
|
||||
moduleDiscovery: options().moduleDiscovery,
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -68,6 +68,7 @@ const INTERPRETED_VARIABLES = Object.freeze([
|
||||
"EKKLESIA_IDENTITY_BASE_URL",
|
||||
"EKKLESIA_IDENTITY_SECRET",
|
||||
"EKKLESIA_IDENTITY_TRUSTED_ORIGINS",
|
||||
"EKKLESIA_MODULES_DIR",
|
||||
"EKKLESIA_PUBLIC_SIGNUP_ENABLED",
|
||||
"EKKLESIA_SMTP_FROM_ADDRESS",
|
||||
"EKKLESIA_SMTP_FROM_NAME",
|
||||
|
||||
@@ -48,6 +48,19 @@ describe("host configuration loading", () => {
|
||||
expect(host.identity.secret).toBeUndefined();
|
||||
});
|
||||
|
||||
it("carries the configured modules directory verbatim, and nothing when it is unset", () => {
|
||||
expect(loadHostConfiguration({}).modulesDirectory).toBeUndefined();
|
||||
expect(
|
||||
loadHostConfiguration({ EKKLESIA_MODULES_DIR: "/var/lib/ekklesia/modules" }).modulesDirectory,
|
||||
).toBe("/var/lib/ekklesia/modules");
|
||||
// Validation is the resolver's job (ADR 0048). The loader neither repairs a relative or blank
|
||||
// value nor treats a set-but-empty variable as unset, so the resolver can refuse it.
|
||||
expect(
|
||||
loadHostConfiguration({ EKKLESIA_MODULES_DIR: "relative/modules" }).modulesDirectory,
|
||||
).toBe("relative/modules");
|
||||
expect(loadHostConfiguration({ EKKLESIA_MODULES_DIR: "" }).modulesDirectory).toBe("");
|
||||
});
|
||||
|
||||
it("carries every object-storage field the storage passes established", () => {
|
||||
const host = loadHostConfiguration(productionEnvironment);
|
||||
|
||||
|
||||
@@ -40,6 +40,7 @@ describe("migration command reports", () => {
|
||||
}),
|
||||
]),
|
||||
pendingCount: 2,
|
||||
retained: Object.freeze([]),
|
||||
});
|
||||
|
||||
expect(formatMigrationPlanReport(plan)).toEqual([
|
||||
@@ -64,6 +65,7 @@ describe("migration command reports", () => {
|
||||
]),
|
||||
appliedCount: 1,
|
||||
noOp: false,
|
||||
retained: Object.freeze([]),
|
||||
});
|
||||
|
||||
expect(formatMigrationApplyReport(applied)).toEqual([
|
||||
@@ -72,7 +74,12 @@ describe("migration command reports", () => {
|
||||
]);
|
||||
expect(
|
||||
formatMigrationApplyReport(
|
||||
Object.freeze({ applied: Object.freeze([]), appliedCount: 0, noOp: true }),
|
||||
Object.freeze({
|
||||
applied: Object.freeze([]),
|
||||
appliedCount: 0,
|
||||
noOp: true,
|
||||
retained: Object.freeze([]),
|
||||
}),
|
||||
),
|
||||
).toEqual(["Migration apply completed: database is already current (no changes)."]);
|
||||
});
|
||||
@@ -85,6 +92,7 @@ describe("migration command reports", () => {
|
||||
approvalRequiredCount: 0,
|
||||
entries: Object.freeze([]),
|
||||
pendingCount: 0,
|
||||
retained: Object.freeze([]),
|
||||
}),
|
||||
),
|
||||
).toEqual([
|
||||
@@ -93,4 +101,62 @@ describe("migration command reports", () => {
|
||||
"Informational only: this plan is read-only and may become stale; apply revalidates history under the migration lock.",
|
||||
]);
|
||||
});
|
||||
|
||||
it("reports an absent module's retained history without treating it as work (ADR 0048)", () => {
|
||||
const retained = Object.freeze([
|
||||
Object.freeze({ appliedCount: 1, moduleId: "calendar" }),
|
||||
Object.freeze({ appliedCount: 3, moduleId: "signage" }),
|
||||
]);
|
||||
const retainedLines = [
|
||||
"- [retained] calendar: 1 applied migration kept; the module is not installed",
|
||||
"- [retained] signage: 3 applied migrations kept; the module is not installed",
|
||||
];
|
||||
|
||||
expect(
|
||||
formatMigrationPlanReport(
|
||||
Object.freeze({
|
||||
appliedCount: 0,
|
||||
approvalRequiredCount: 0,
|
||||
entries: Object.freeze([]),
|
||||
pendingCount: 0,
|
||||
retained,
|
||||
}),
|
||||
),
|
||||
).toEqual([
|
||||
"Migration plan (0 applied, 0 pending, 0 approval-blocked, 2 retained):",
|
||||
"- No installed module migrations.",
|
||||
...retainedLines,
|
||||
"Informational only: this plan is read-only and may become stale; apply revalidates history under the migration lock.",
|
||||
]);
|
||||
expect(
|
||||
formatMigrationApplyReport(
|
||||
Object.freeze({ applied: Object.freeze([]), appliedCount: 0, noOp: true, retained }),
|
||||
),
|
||||
).toEqual([
|
||||
"Migration apply completed: database is already current (no changes).",
|
||||
...retainedLines,
|
||||
]);
|
||||
expect(
|
||||
formatMigrationApplyReport(
|
||||
Object.freeze({
|
||||
applied: Object.freeze([
|
||||
Object.freeze({
|
||||
checksum: "e".repeat(64),
|
||||
identity: "core:0007_later",
|
||||
index: 7,
|
||||
moduleId: "core",
|
||||
tag: "0007_later",
|
||||
}),
|
||||
]),
|
||||
appliedCount: 1,
|
||||
noOp: false,
|
||||
retained,
|
||||
}),
|
||||
),
|
||||
).toEqual([
|
||||
"Migration apply completed (1 migration applied):",
|
||||
`- core:0007_later sha256=${"e".repeat(64)}`,
|
||||
...retainedLines,
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -13,6 +13,7 @@ import {
|
||||
type MigrationTransaction,
|
||||
planModuleMigrations,
|
||||
} from "../src/migrations/runner.js";
|
||||
import type { DiscoveredModule } from "../src/modules/discovery.js";
|
||||
import type {
|
||||
DiscoveredModuleMigration,
|
||||
ModuleMigrationAggregate,
|
||||
@@ -323,6 +324,7 @@ describe("module migration planning", () => {
|
||||
},
|
||||
],
|
||||
pendingCount: 1,
|
||||
retained: [],
|
||||
});
|
||||
expect(database.events).toEqual(["session:open", "history:read", "session:close"]);
|
||||
expect(database.historyTableExists).toBe(false);
|
||||
@@ -344,10 +346,11 @@ describe("module migration planning", () => {
|
||||
approvalRequiredCount: 0,
|
||||
entries: [],
|
||||
pendingCount: 0,
|
||||
retained: [],
|
||||
});
|
||||
expect(Object.isFrozen(result)).toBe(true);
|
||||
expect(Object.isFrozen(result.entries)).toBe(true);
|
||||
expect(applyResult).toEqual({ applied: [], appliedCount: 0, noOp: true });
|
||||
expect(applyResult).toEqual({ applied: [], appliedCount: 0, noOp: true, retained: [] });
|
||||
expect(database.historyTableExists).toBe(false);
|
||||
expect(database.events).not.toContain("transaction:begin");
|
||||
});
|
||||
@@ -548,6 +551,7 @@ describe("module migration apply", () => {
|
||||
],
|
||||
appliedCount: 1,
|
||||
noOp: false,
|
||||
retained: [],
|
||||
});
|
||||
});
|
||||
|
||||
@@ -580,7 +584,7 @@ describe("module migration apply", () => {
|
||||
database,
|
||||
});
|
||||
|
||||
expect(result).toEqual({ applied: [], appliedCount: 0, noOp: true });
|
||||
expect(result).toEqual({ applied: [], appliedCount: 0, noOp: true, retained: [] });
|
||||
expect(Object.isFrozen(result)).toBe(true);
|
||||
expect(Object.isFrozen(result.applied)).toBe(true);
|
||||
expectNoPersistentMutation(database);
|
||||
@@ -601,7 +605,7 @@ describe("module migration apply", () => {
|
||||
database,
|
||||
});
|
||||
|
||||
expect(result).toEqual({ applied: [], appliedCount: 0, noOp: true });
|
||||
expect(result).toEqual({ applied: [], appliedCount: 0, noOp: true, retained: [] });
|
||||
expect(database.historyTableExists).toBe(false);
|
||||
expect(database.events).toEqual([
|
||||
"session:open",
|
||||
@@ -980,3 +984,250 @@ describe("module migration failures and side effects", () => {
|
||||
expect(database.committedHistory).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* Marks modules as present in the discovered inventory without giving them migrations, the way a
|
||||
* real aggregate lists every discovered module whether or not it owns migration history. Only the
|
||||
* manifest id is read by the runner.
|
||||
*/
|
||||
function withDiscovered(
|
||||
inventory: ModuleMigrationAggregate,
|
||||
...moduleIds: readonly string[]
|
||||
): ModuleMigrationAggregate {
|
||||
return Object.freeze({
|
||||
...inventory,
|
||||
discoveredModules: Object.freeze(
|
||||
moduleIds.map((id) => ({ manifest: { id } }) as unknown as DiscoveredModule),
|
||||
),
|
||||
});
|
||||
}
|
||||
|
||||
describe("retained history for absent modules (ADR 0048)", () => {
|
||||
it("retains and reports an absent module's applied history without failing the plan", async () => {
|
||||
const coreInitial = migration("core", 0);
|
||||
const calendarInitial = migration("calendar", 0);
|
||||
const calendarLater = migration("calendar", 1);
|
||||
const history = [
|
||||
historyRecord(coreInitial, 1),
|
||||
historyRecord(calendarInitial, 2),
|
||||
historyRecord(calendarLater, 3),
|
||||
];
|
||||
const database = new FakeMigrationDatabase({ history });
|
||||
|
||||
const result = await planModuleMigrations({
|
||||
aggregate: aggregate({ id: "core", migrations: [coreInitial] }),
|
||||
database,
|
||||
});
|
||||
|
||||
expect(result.appliedCount).toBe(1);
|
||||
expect(result.pendingCount).toBe(0);
|
||||
expect(result.entries.map((entry) => entry.identity)).toEqual([coreInitial.identity]);
|
||||
expect(result.retained).toEqual([{ appliedCount: 2, moduleId: "calendar" }]);
|
||||
expect(Object.isFrozen(result.retained)).toBe(true);
|
||||
expect(Object.isFrozen(result.retained[0])).toBe(true);
|
||||
expect(database.committedHistory).toEqual(history);
|
||||
expectNoPersistentMutation(database);
|
||||
});
|
||||
|
||||
it("applies a later Core migration while a module is absent, leaving its history intact", async () => {
|
||||
const coreInitial = migration("core", 0);
|
||||
const coreLater = migration("core", 1);
|
||||
const calendarInitial = migration("calendar", 0);
|
||||
const history = [historyRecord(coreInitial, 1), historyRecord(calendarInitial, 2)];
|
||||
const database = new FakeMigrationDatabase({ history });
|
||||
|
||||
const result = await applyModuleMigrations({
|
||||
aggregate: aggregate({ id: "core", migrations: [coreInitial, coreLater] }),
|
||||
database,
|
||||
});
|
||||
|
||||
expect(result.applied.map((applied) => applied.identity)).toEqual([coreLater.identity]);
|
||||
expect(result.retained).toEqual([{ appliedCount: 1, moduleId: "calendar" }]);
|
||||
expect(database.executedStatements).toEqual([coreLater.sql]);
|
||||
expect(database.committedHistory.map((record) => record.identity)).toEqual([
|
||||
coreInitial.identity,
|
||||
calendarInitial.identity,
|
||||
coreLater.identity,
|
||||
]);
|
||||
});
|
||||
|
||||
it("applies a present module's migrations while another module is absent", async () => {
|
||||
const peopleInitial = migration("people", 0);
|
||||
const calendarInitial = migration("calendar", 0);
|
||||
const database = new FakeMigrationDatabase({ history: [historyRecord(calendarInitial, 1)] });
|
||||
|
||||
const result = await applyModuleMigrations({
|
||||
aggregate: aggregate({ id: "people", migrations: [peopleInitial] }),
|
||||
database,
|
||||
});
|
||||
|
||||
expect(result.applied.map((applied) => applied.identity)).toEqual([peopleInitial.identity]);
|
||||
expect(result.retained).toEqual([{ appliedCount: 1, moduleId: "calendar" }]);
|
||||
});
|
||||
|
||||
it("reports retained modules in a stable order with exact counts", async () => {
|
||||
const zeta = [migration("zeta", 0), migration("zeta", 1), migration("zeta", 2)];
|
||||
const alpha = migration("alpha", 0);
|
||||
const database = new FakeMigrationDatabase({
|
||||
history: [
|
||||
historyRecord(zeta[0] as DiscoveredModuleMigration, 1),
|
||||
historyRecord(alpha, 2),
|
||||
historyRecord(zeta[1] as DiscoveredModuleMigration, 3),
|
||||
historyRecord(zeta[2] as DiscoveredModuleMigration, 4),
|
||||
],
|
||||
});
|
||||
|
||||
const result = await planModuleMigrations({ aggregate: aggregate(), database });
|
||||
|
||||
expect(result.retained).toEqual([
|
||||
{ appliedCount: 1, moduleId: "alpha" },
|
||||
{ appliedCount: 3, moduleId: "zeta" },
|
||||
]);
|
||||
});
|
||||
|
||||
it("returns a module to strict validation: valid history plans cleanly and nothing is retained", async () => {
|
||||
const initial = migration("people", 0);
|
||||
const later = migration("people", 1);
|
||||
const database = new FakeMigrationDatabase({ history: [historyRecord(initial, 1)] });
|
||||
|
||||
const result = await planModuleMigrations({
|
||||
aggregate: aggregate({ id: "people", migrations: [initial, later] }),
|
||||
database,
|
||||
});
|
||||
|
||||
expect(result.retained).toEqual([]);
|
||||
expect(result.entries.map((entry) => [entry.identity, entry.status])).toEqual([
|
||||
[initial.identity, "applied"],
|
||||
[later.identity, "pending"],
|
||||
]);
|
||||
});
|
||||
|
||||
it("fails when a returning module no longer ships a previously applied migration", async () => {
|
||||
const initial = migration("people", 0);
|
||||
const later = migration("people", 1);
|
||||
const database = new FakeMigrationDatabase({
|
||||
history: [historyRecord(initial, 1), historyRecord(later, 2)],
|
||||
});
|
||||
const downgraded = aggregate({ id: "people", migrations: [initial] });
|
||||
|
||||
await captureRunnerError(
|
||||
planModuleMigrations({ aggregate: downgraded, database }),
|
||||
"MIGRATION_UNKNOWN_APPLIED",
|
||||
);
|
||||
await captureRunnerError(
|
||||
applyModuleMigrations({ aggregate: downgraded, database }),
|
||||
"MIGRATION_UNKNOWN_APPLIED",
|
||||
);
|
||||
expect(database.executedStatements).toEqual([]);
|
||||
});
|
||||
|
||||
it("fails when a returning module's applied migration has different contents", async () => {
|
||||
const initial = migration("people", 0);
|
||||
const database = new FakeMigrationDatabase({ history: [historyRecord(initial, 1)] });
|
||||
const edited = aggregate({
|
||||
id: "people",
|
||||
migrations: [migration("people", 0, { sql: "SELECT 'edited';\n" })],
|
||||
});
|
||||
|
||||
const error = await captureRunnerError(
|
||||
planModuleMigrations({ aggregate: edited, database }),
|
||||
"MIGRATION_CHECKSUM_MISMATCH",
|
||||
);
|
||||
expect(error.migrationIdentity).toBe(initial.identity);
|
||||
});
|
||||
|
||||
it("fails when a returning module's history is no longer a prefix of its journal", async () => {
|
||||
const initial = migration("people", 0);
|
||||
const later = migration("people", 1);
|
||||
const database = new FakeMigrationDatabase({ history: [historyRecord(later, 1)] });
|
||||
|
||||
await captureRunnerError(
|
||||
planModuleMigrations({
|
||||
aggregate: aggregate({ id: "people", migrations: [initial, later] }),
|
||||
database,
|
||||
}),
|
||||
"MIGRATION_HISTORY_INVALID",
|
||||
);
|
||||
});
|
||||
|
||||
it("keeps a present module that no longer ships any migrations strict", async () => {
|
||||
const initial = migration("people", 0);
|
||||
const database = new FakeMigrationDatabase({ history: [historyRecord(initial, 1)] });
|
||||
|
||||
await captureRunnerError(
|
||||
planModuleMigrations({ aggregate: withDiscovered(aggregate(), "people"), database }),
|
||||
"MIGRATION_UNKNOWN_APPLIED",
|
||||
);
|
||||
});
|
||||
|
||||
it("never treats Core's history as an absent module's", async () => {
|
||||
const coreInitial = migration("core", 0);
|
||||
const coreGhost = migration("core", 1);
|
||||
|
||||
// An inventory with no Core source at all: Core's rows are still strict, never retained.
|
||||
await captureRunnerError(
|
||||
planModuleMigrations({
|
||||
aggregate: aggregate({ id: "people", migrations: [] }),
|
||||
database: new FakeMigrationDatabase({ history: [historyRecord(coreInitial, 1)] }),
|
||||
}),
|
||||
"MIGRATION_UNKNOWN_APPLIED",
|
||||
);
|
||||
// A Core row missing from Core's own inventory.
|
||||
await captureRunnerError(
|
||||
planModuleMigrations({
|
||||
aggregate: aggregate({ id: "core", migrations: [coreInitial] }),
|
||||
database: new FakeMigrationDatabase({
|
||||
history: [historyRecord(coreInitial, 1), historyRecord(coreGhost, 2)],
|
||||
}),
|
||||
}),
|
||||
"MIGRATION_UNKNOWN_APPLIED",
|
||||
);
|
||||
});
|
||||
|
||||
it.each([
|
||||
[
|
||||
"an identity that does not name its own module and tag",
|
||||
(source: DiscoveredModuleMigration) => [
|
||||
historyRecord(source, 1, { identity: "calendar:0099_elsewhere" }),
|
||||
],
|
||||
],
|
||||
[
|
||||
"a duplicate applied order",
|
||||
(source: DiscoveredModuleMigration) => [
|
||||
historyRecord(source, 1),
|
||||
historyRecord(migration("calendar", 1), 1),
|
||||
],
|
||||
],
|
||||
[
|
||||
"a duplicate index",
|
||||
(source: DiscoveredModuleMigration) => [
|
||||
historyRecord(source, 1),
|
||||
historyRecord(migration("calendar", 1), 2, { index: 0 }),
|
||||
],
|
||||
],
|
||||
])("still rejects retained rows with %s", async (_label, makeHistory) => {
|
||||
const database = new FakeMigrationDatabase({
|
||||
history: makeHistory(migration("calendar", 0)),
|
||||
});
|
||||
|
||||
await captureRunnerError(
|
||||
planModuleMigrations({ aggregate: aggregate(), database }),
|
||||
"MIGRATION_HISTORY_INVALID",
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects a retained row that claims a present module's migration identity", async () => {
|
||||
const peopleInitial = migration("people", 0);
|
||||
const database = new FakeMigrationDatabase({
|
||||
history: [historyRecord(peopleInitial, 1, { moduleId: "calendar" })],
|
||||
});
|
||||
|
||||
await captureRunnerError(
|
||||
planModuleMigrations({
|
||||
aggregate: aggregate({ id: "people", migrations: [peopleInitial] }),
|
||||
database,
|
||||
}),
|
||||
"MIGRATION_HISTORY_INVALID",
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
import { cp, mkdtemp, readdir, readFile, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { discoverRepositoryMigrations } from "../src/cli/migration-command-support.js";
|
||||
import { resolveCommandModulesDirectory } from "../src/cli/modules-directory-environment.js";
|
||||
import { ModuleDiscoveryError } from "../src/modules/index.js";
|
||||
|
||||
const repositoryRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..");
|
||||
const templateModule = path.join(repositoryRoot, "templates", "module");
|
||||
const temporaryDirectories: string[] = [];
|
||||
|
||||
afterEach(async () => {
|
||||
await Promise.all(
|
||||
temporaryDirectories.splice(0).map((directory) => rm(directory, { recursive: true })),
|
||||
);
|
||||
});
|
||||
|
||||
async function externalModulesDirectory(): Promise<string> {
|
||||
const directory = await mkdtemp(path.join(tmpdir(), "ekklesia-installed-modules-"));
|
||||
temporaryDirectories.push(directory);
|
||||
return directory;
|
||||
}
|
||||
|
||||
/** Installs the module template's migration-bearing package files as module `example`. Migration
|
||||
* discovery imports nothing, so the template's manifest, package metadata, and migrations are all
|
||||
* it reads; the compiled artifacts discovery checks for are copied too when they exist. */
|
||||
async function installTemplate(modulesDirectory: string): Promise<void> {
|
||||
const target = path.join(modulesDirectory, "example");
|
||||
for (const entry of ["package.json", "ekklesia.module.json", "drizzle", "dist"]) {
|
||||
await cp(path.join(templateModule, entry), path.join(target, entry), { recursive: true });
|
||||
}
|
||||
}
|
||||
|
||||
describe("operator commands select the modules directory through host configuration", () => {
|
||||
it("defaults to the repository's own modules directory", () => {
|
||||
expect(resolveCommandModulesDirectory({}, "/opt/ekklesia")).toEqual({
|
||||
modulesDirectory: "/opt/ekklesia/modules",
|
||||
modulesDirectoryRequired: false,
|
||||
});
|
||||
});
|
||||
|
||||
it("honours EKKLESIA_MODULES_DIR", () => {
|
||||
expect(
|
||||
resolveCommandModulesDirectory(
|
||||
{ EKKLESIA_MODULES_DIR: "/var/lib/ekklesia/modules" },
|
||||
"/opt/ekklesia",
|
||||
),
|
||||
).toEqual({ modulesDirectory: "/var/lib/ekklesia/modules", modulesDirectoryRequired: true });
|
||||
});
|
||||
|
||||
it("refuses a relative EKKLESIA_MODULES_DIR without echoing it", () => {
|
||||
expect(() =>
|
||||
resolveCommandModulesDirectory({ EKKLESIA_MODULES_DIR: "modules-9e1f" }, "/opt/ekklesia"),
|
||||
).toThrow(ModuleDiscoveryError);
|
||||
try {
|
||||
resolveCommandModulesDirectory({ EKKLESIA_MODULES_DIR: "modules-9e1f" }, "/opt/ekklesia");
|
||||
} catch (error) {
|
||||
expect(JSON.stringify((error as ModuleDiscoveryError).issues)).not.toContain("9e1f");
|
||||
}
|
||||
});
|
||||
|
||||
it("plans migrations for modules installed outside the repository, with Core's from the platform", async () => {
|
||||
const modulesDirectory = await externalModulesDirectory();
|
||||
await installTemplate(modulesDirectory);
|
||||
|
||||
const aggregate = await discoverRepositoryMigrations(repositoryRoot, {
|
||||
EKKLESIA_MODULES_DIR: modulesDirectory,
|
||||
});
|
||||
|
||||
const sources = aggregate.modules.map((module) => module.moduleId);
|
||||
expect(sources[0]).toBe("core");
|
||||
expect(sources).toContain("example");
|
||||
expect(
|
||||
aggregate.migrations.some((migration) => migration.identity === "example:0000_initial"),
|
||||
).toBe(true);
|
||||
const coreTags = (await readdir(path.join(repositoryRoot, "core", "drizzle"))).filter((name) =>
|
||||
name.endsWith(".sql"),
|
||||
);
|
||||
expect(aggregate.modules[0]?.migrations).toHaveLength(coreTags.length);
|
||||
});
|
||||
|
||||
it("fails closed when the configured directory does not exist", async () => {
|
||||
const parent = await externalModulesDirectory();
|
||||
await expect(
|
||||
discoverRepositoryMigrations(repositoryRoot, {
|
||||
EKKLESIA_MODULES_DIR: path.join(parent, "absent"),
|
||||
}),
|
||||
).rejects.toBeInstanceOf(ModuleDiscoveryError);
|
||||
});
|
||||
});
|
||||
|
||||
describe("no source chooses a modules directory except the resolver", () => {
|
||||
async function sourceFiles(directory: string): Promise<readonly string[]> {
|
||||
const entries = await readdir(directory, { recursive: true, withFileTypes: true });
|
||||
return entries
|
||||
.filter((entry) => entry.isFile() && entry.name.endsWith(".ts"))
|
||||
.map((entry) => path.join(entry.parentPath, entry.name));
|
||||
}
|
||||
|
||||
it("never joins `modules` onto a root anywhere else in Core or the API", async () => {
|
||||
const files = [
|
||||
...(await sourceFiles(path.join(repositoryRoot, "core", "src"))),
|
||||
...(await sourceFiles(path.join(repositoryRoot, "apps", "api", "src"))),
|
||||
];
|
||||
const resolver = path.join(repositoryRoot, "core", "src", "modules", "modules-directory.ts");
|
||||
for (const file of files) {
|
||||
if (file === resolver) {
|
||||
continue;
|
||||
}
|
||||
const text = await readFile(file, "utf8");
|
||||
expect(text, path.relative(repositoryRoot, file)).not.toMatch(/,\s*"modules"\s*\)/u);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,159 @@
|
||||
import { mkdtemp, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import {
|
||||
discoverModules,
|
||||
ModuleDiscoveryError,
|
||||
type ModuleDiscoveryIssueCode,
|
||||
resolveModulesDirectory,
|
||||
} from "../src/modules/index.js";
|
||||
|
||||
const versions = { moduleSdkVersion: "0.1.0", platformVersion: "0.1.0" } as const;
|
||||
const repositoryRoot = path.join(path.sep, "opt", "ekklesia");
|
||||
const temporaryDirectories: string[] = [];
|
||||
|
||||
afterEach(async () => {
|
||||
await Promise.all(
|
||||
temporaryDirectories.splice(0).map((directory) => rm(directory, { recursive: true })),
|
||||
);
|
||||
});
|
||||
|
||||
async function temporaryDirectory(): Promise<string> {
|
||||
const directory = await mkdtemp(path.join(tmpdir(), "ekklesia-modules-directory-"));
|
||||
temporaryDirectories.push(directory);
|
||||
return directory;
|
||||
}
|
||||
|
||||
function captureDiscoveryError(operation: () => unknown): ModuleDiscoveryError {
|
||||
try {
|
||||
operation();
|
||||
} catch (error) {
|
||||
expect(error).toBeInstanceOf(ModuleDiscoveryError);
|
||||
return error as ModuleDiscoveryError;
|
||||
}
|
||||
throw new Error("Expected a ModuleDiscoveryError.");
|
||||
}
|
||||
|
||||
async function captureAsyncDiscoveryError(
|
||||
operation: Promise<unknown>,
|
||||
): Promise<ModuleDiscoveryError> {
|
||||
try {
|
||||
await operation;
|
||||
} catch (error) {
|
||||
expect(error).toBeInstanceOf(ModuleDiscoveryError);
|
||||
return error as ModuleDiscoveryError;
|
||||
}
|
||||
throw new Error("Expected a ModuleDiscoveryError.");
|
||||
}
|
||||
|
||||
function codes(error: ModuleDiscoveryError): readonly ModuleDiscoveryIssueCode[] {
|
||||
return error.issues.map((issue) => issue.code);
|
||||
}
|
||||
|
||||
/** Everything an operator, a log, or a test report could ever see from this error. */
|
||||
function rendered(error: ModuleDiscoveryError): string {
|
||||
return JSON.stringify({ issues: error.issues, message: error.message, stack: error.stack });
|
||||
}
|
||||
|
||||
describe("resolveModulesDirectory", () => {
|
||||
it("defaults to the repository's own modules directory when nothing is configured", () => {
|
||||
expect(resolveModulesDirectory({ configured: undefined, repositoryRoot })).toEqual({
|
||||
modulesDirectory: path.join(repositoryRoot, "modules"),
|
||||
modulesDirectoryRequired: false,
|
||||
});
|
||||
});
|
||||
|
||||
it("accepts an explicit absolute path, normalized, and requires it to exist", () => {
|
||||
expect(
|
||||
resolveModulesDirectory({ configured: "/var/lib/ekklesia/modules/", repositoryRoot }),
|
||||
).toEqual({ modulesDirectory: "/var/lib/ekklesia/modules", modulesDirectoryRequired: true });
|
||||
expect(
|
||||
resolveModulesDirectory({ configured: "/var/lib/./ekklesia//modules", repositoryRoot }),
|
||||
).toEqual({ modulesDirectory: "/var/lib/ekklesia/modules", modulesDirectoryRequired: true });
|
||||
});
|
||||
|
||||
it("never resolves a configured path against the repository or the working directory", () => {
|
||||
const resolved = resolveModulesDirectory({ configured: "/srv/modules", repositoryRoot });
|
||||
expect(resolved.modulesDirectory).toBe("/srv/modules");
|
||||
});
|
||||
|
||||
it.each([
|
||||
["a relative name", "modules"],
|
||||
["a dot-relative path", "./modules"],
|
||||
["a parent-relative path", "../private/modules"],
|
||||
["a home-relative path", "~/modules"],
|
||||
["the empty string", ""],
|
||||
["whitespace only", " "],
|
||||
["an absolute path with surrounding whitespace", " /var/lib/ekklesia/modules"],
|
||||
["a NUL byte", "/var/lib/ekklesia/\u0000modules"],
|
||||
])("rejects %s with a sanitized MODULES_DIRECTORY_INVALID", (_label, configured) => {
|
||||
const error = captureDiscoveryError(() =>
|
||||
resolveModulesDirectory({ configured, repositoryRoot }),
|
||||
);
|
||||
expect(codes(error)).toEqual(["MODULES_DIRECTORY_INVALID"]);
|
||||
expect(error.issues[0]?.message).toBe("EKKLESIA_MODULES_DIR must be an absolute path.");
|
||||
});
|
||||
|
||||
it("never echoes the supplied value in the error", () => {
|
||||
const configured = "relative/secret-token-a1b2c3/postgresql://admin:hunter2@db/modules";
|
||||
const error = captureDiscoveryError(() =>
|
||||
resolveModulesDirectory({ configured, repositoryRoot }),
|
||||
);
|
||||
const text = rendered(error);
|
||||
expect(text).not.toContain("secret-token-a1b2c3");
|
||||
expect(text).not.toContain("hunter2");
|
||||
expect(text).not.toContain("relative/");
|
||||
});
|
||||
});
|
||||
|
||||
describe("module discovery against a configured modules directory", () => {
|
||||
it("fails closed, without the path, when a configured directory does not exist", async () => {
|
||||
const parent = await temporaryDirectory();
|
||||
const missing = path.join(parent, "private-mount-name-7f3e");
|
||||
const error = await captureAsyncDiscoveryError(
|
||||
discoverModules({ ...versions, modulesDirectory: missing, modulesDirectoryRequired: true }),
|
||||
);
|
||||
expect(codes(error)).toEqual(["MODULES_DIRECTORY_UNREADABLE"]);
|
||||
expect(rendered(error)).not.toContain("private-mount-name-7f3e");
|
||||
expect(rendered(error)).not.toContain(parent);
|
||||
});
|
||||
|
||||
it("fails closed when a configured path is a file rather than a directory", async () => {
|
||||
const parent = await temporaryDirectory();
|
||||
const file = path.join(parent, "modules");
|
||||
await writeFile(file, "not a directory");
|
||||
const error = await captureAsyncDiscoveryError(
|
||||
discoverModules({ ...versions, modulesDirectory: file, modulesDirectoryRequired: true }),
|
||||
);
|
||||
expect(codes(error)).toEqual(["MODULES_DIRECTORY_UNREADABLE"]);
|
||||
expect(rendered(error)).not.toContain(parent);
|
||||
});
|
||||
|
||||
it("rejects a relative configured path even when it bypasses the resolver", async () => {
|
||||
const error = await captureAsyncDiscoveryError(
|
||||
discoverModules({ ...versions, modulesDirectory: "modules", modulesDirectoryRequired: true }),
|
||||
);
|
||||
expect(codes(error)).toEqual(["MODULES_DIRECTORY_INVALID"]);
|
||||
});
|
||||
|
||||
it("treats an existing, empty configured directory as zero installed modules", async () => {
|
||||
const directory = await temporaryDirectory();
|
||||
await expect(
|
||||
discoverModules({ ...versions, modulesDirectory: directory, modulesDirectoryRequired: true }),
|
||||
).resolves.toEqual([]);
|
||||
});
|
||||
|
||||
it("keeps the development default lenient: a missing default directory is zero modules", async () => {
|
||||
const parent = await temporaryDirectory();
|
||||
await expect(
|
||||
discoverModules({ ...versions, modulesDirectory: path.join(parent, "modules") }),
|
||||
).resolves.toEqual([]);
|
||||
});
|
||||
|
||||
it("discovers through the resolver's own selection unchanged", async () => {
|
||||
const directory = await temporaryDirectory();
|
||||
const selection = resolveModulesDirectory({ configured: directory, repositoryRoot });
|
||||
await expect(discoverModules({ ...versions, ...selection })).resolves.toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -1,5 +1,7 @@
|
||||
import { createHash, randomBytes } from "node:crypto";
|
||||
import { cp, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import net from "node:net";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { Client, type QueryResult, type QueryResultRow } from "pg";
|
||||
@@ -8,6 +10,7 @@ import { reportMigrationCommandError } from "../src/cli/migration-command-errors
|
||||
import {
|
||||
applyModuleMigrations,
|
||||
createPostgresMigrationDatabase,
|
||||
discoverAllMigrations,
|
||||
discoverCoreMigrations,
|
||||
type MigrationApproval,
|
||||
type MigrationApprovalProvider,
|
||||
@@ -764,7 +767,7 @@ describe.sequential("live PostgreSQL module migration execution", () => {
|
||||
database: migrationDatabase,
|
||||
});
|
||||
|
||||
expect(result).toEqual({ applied: [], appliedCount: 0, noOp: true });
|
||||
expect(result).toEqual({ applied: [], appliedCount: 0, noOp: true, retained: [] });
|
||||
expect(await databaseState(database.client)).toEqual(before);
|
||||
const afterData = await safeQuery<QueryResultRow>(
|
||||
database.client,
|
||||
@@ -834,7 +837,48 @@ describe.sequential("live PostgreSQL module migration execution", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("rejects an unknown applied identity without repair or further execution", async () => {
|
||||
it("retains an absent module's history and still applies later work (ADR 0048)", async () => {
|
||||
await withIsolatedDatabase("absent_history", async (database) => {
|
||||
const initial = peopleInitial();
|
||||
const later = peopleLater();
|
||||
const initialInventory = aggregate(moduleFixture("people", [initial]));
|
||||
const completeInventory = aggregate(moduleFixture("people", [initial, later]));
|
||||
const migrationDatabase = createPostgresMigrationDatabase(database.url);
|
||||
await applyModuleMigrations({ aggregate: initialInventory, database: migrationDatabase });
|
||||
// History left by a module that is no longer installed: kept, reported, never fatal.
|
||||
await safeQuery(
|
||||
database.client,
|
||||
`
|
||||
INSERT INTO public.ekklesia_core_migrations
|
||||
(identity, module_id, migration_tag, migration_index, checksum)
|
||||
VALUES ($1, $2, $3, $4, $5)
|
||||
`,
|
||||
["ghost:0000_unknown", "ghost", "0000_unknown", 0, "f".repeat(64)],
|
||||
);
|
||||
|
||||
const plan = await planModuleMigrations({
|
||||
aggregate: completeInventory,
|
||||
database: migrationDatabase,
|
||||
});
|
||||
expect(plan.retained).toEqual([{ appliedCount: 1, moduleId: "ghost" }]);
|
||||
expect(plan.pendingCount).toBe(1);
|
||||
|
||||
const result = await applyModuleMigrations({
|
||||
aggregate: completeInventory,
|
||||
database: migrationDatabase,
|
||||
});
|
||||
expect(result.applied.map((migration) => migration.identity)).toEqual([later.identity]);
|
||||
expect(result.retained).toEqual([{ appliedCount: 1, moduleId: "ghost" }]);
|
||||
expect(await columnExists(database.client, "people_records", "email")).toBe(true);
|
||||
expect((await readHistory(database.client)).map((row) => row.identity)).toEqual([
|
||||
initial.identity,
|
||||
"ghost:0000_unknown",
|
||||
later.identity,
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
it("rejects an unknown applied identity of an installed module without repair or further execution", async () => {
|
||||
await withIsolatedDatabase("unknown_history", async (database) => {
|
||||
const initial = peopleInitial();
|
||||
const later = peopleLater();
|
||||
@@ -849,7 +893,7 @@ describe.sequential("live PostgreSQL module migration execution", () => {
|
||||
(identity, module_id, migration_tag, migration_index, checksum)
|
||||
VALUES ($1, $2, $3, $4, $5)
|
||||
`,
|
||||
["ghost:0000_unknown", "ghost", "0000_unknown", 0, "f".repeat(64)],
|
||||
["people:0099_unknown", "people", "0099_unknown", 99, "f".repeat(64)],
|
||||
);
|
||||
const corrupted = await databaseState(database.client);
|
||||
|
||||
@@ -866,7 +910,7 @@ describe.sequential("live PostgreSQL module migration execution", () => {
|
||||
expect(await columnExists(database.client, "people_records", "email")).toBe(false);
|
||||
expect((await readHistory(database.client)).map((row) => row.identity)).toEqual([
|
||||
initial.identity,
|
||||
"ghost:0000_unknown",
|
||||
"people:0099_unknown",
|
||||
]);
|
||||
});
|
||||
});
|
||||
@@ -1358,7 +1402,7 @@ describe.sequential("live PostgreSQL Core identity migration execution", () => {
|
||||
database: migrationDatabase,
|
||||
});
|
||||
|
||||
expect(result).toEqual({ applied: [], appliedCount: 0, noOp: true });
|
||||
expect(result).toEqual({ applied: [], appliedCount: 0, noOp: true, retained: [] });
|
||||
expect(await databaseState(database.client)).toEqual(before);
|
||||
});
|
||||
});
|
||||
@@ -1851,3 +1895,151 @@ describe.sequential("live PostgreSQL migration timeout precedence", () => {
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* ADR 0048 end to end, with real discovery: Core's committed migrations, the module template
|
||||
* installed in a modules directory outside the repository, and a real PostgreSQL. Core's newest
|
||||
* migration is withheld from a copy of `core/drizzle` at first, so a genuinely later Core migration
|
||||
* can be applied while the module is absent.
|
||||
*/
|
||||
describe.sequential("live PostgreSQL retained history for an uninstalled module", () => {
|
||||
const repositoryRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..");
|
||||
const templateModule = path.join(repositoryRoot, "templates", "module");
|
||||
const versions = { moduleSdkVersion: "0.1.0", platformVersion: "0.1.0" } as const;
|
||||
|
||||
async function coreWithoutNewestMigration(directory: string): Promise<string> {
|
||||
const target = path.join(directory, "core-drizzle");
|
||||
await cp(coreMigrationsDirectory, target, { recursive: true });
|
||||
const journalPath = path.join(target, "meta", "_journal.json");
|
||||
const policyPath = path.join(target, "meta", "_ekklesia.json");
|
||||
const journal = JSON.parse(await readFile(journalPath, "utf8")) as {
|
||||
entries: { idx: number; tag: string }[];
|
||||
};
|
||||
const policy = JSON.parse(await readFile(policyPath, "utf8")) as { entries: unknown[] };
|
||||
const newest = journal.entries.pop();
|
||||
policy.entries.pop();
|
||||
if (!newest) {
|
||||
throw harnessError("Core has no migration to withhold.");
|
||||
}
|
||||
await writeFile(journalPath, `${JSON.stringify(journal, null, 2)}\n`);
|
||||
await writeFile(policyPath, `${JSON.stringify(policy, null, 2)}\n`);
|
||||
await rm(path.join(target, `${newest.tag}.sql`));
|
||||
await rm(path.join(target, "meta", `${String(newest.idx).padStart(4, "0")}_snapshot.json`));
|
||||
return target;
|
||||
}
|
||||
|
||||
async function installExample(modulesDirectory: string): Promise<void> {
|
||||
for (const entry of ["package.json", "ekklesia.module.json", "drizzle", "dist"]) {
|
||||
await cp(path.join(templateModule, entry), path.join(modulesDirectory, "example", entry), {
|
||||
recursive: true,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
async function inventory(coreDirectory: string, modulesDirectory: string) {
|
||||
return discoverAllMigrations({
|
||||
coreMigrationsDirectory: coreDirectory,
|
||||
moduleDiscovery: { ...versions, modulesDirectory, modulesDirectoryRequired: true },
|
||||
});
|
||||
}
|
||||
|
||||
async function exampleValues(client: Client): Promise<readonly string[]> {
|
||||
const result = await safeQuery<{ readonly value: string } & QueryResultRow>(
|
||||
client,
|
||||
"SELECT value FROM public.example_records ORDER BY value",
|
||||
);
|
||||
return result.rows.map((row) => row.value);
|
||||
}
|
||||
|
||||
it("keeps an uninstalled module's data and history, applies later Core work, and restores it", async () => {
|
||||
const scratch = await mkdtemp(path.join(tmpdir(), "ekklesia-retained-history-"));
|
||||
try {
|
||||
const earlyCore = await coreWithoutNewestMigration(scratch);
|
||||
const modulesDirectory = path.join(scratch, "modules");
|
||||
await mkdir(modulesDirectory);
|
||||
await installExample(modulesDirectory);
|
||||
|
||||
await withIsolatedDatabase("retained_module", async (database) => {
|
||||
const migrationDatabase = createPostgresMigrationDatabase(database.url);
|
||||
|
||||
// 1. Core (minus its newest migration) and the installed module, applied together.
|
||||
const first = await applyModuleMigrations({
|
||||
aggregate: await inventory(earlyCore, modulesDirectory),
|
||||
database: migrationDatabase,
|
||||
});
|
||||
expect(first.applied.map((migration) => migration.identity)).toContain(
|
||||
"example:0000_initial",
|
||||
);
|
||||
expect(first.retained).toEqual([]);
|
||||
|
||||
// 2. Module-owned data.
|
||||
await safeQuery(
|
||||
database.client,
|
||||
"INSERT INTO public.example_records (id, value) VALUES (gen_random_uuid(), 'kept')",
|
||||
);
|
||||
|
||||
// 3. The module is uninstalled: its directory is gone from module storage.
|
||||
await rm(path.join(modulesDirectory, "example"), { recursive: true });
|
||||
|
||||
// 4. A later Core migration plans and applies while the module is absent.
|
||||
const absentInventory = await inventory(coreMigrationsDirectory, modulesDirectory);
|
||||
const absentPlan = await planModuleMigrations({
|
||||
aggregate: absentInventory,
|
||||
database: migrationDatabase,
|
||||
});
|
||||
expect(absentPlan.retained).toEqual([{ appliedCount: 1, moduleId: "example" }]);
|
||||
expect(
|
||||
absentPlan.entries.filter((entry) => entry.status === "pending").map((e) => e.moduleId),
|
||||
).toEqual(["core"]);
|
||||
const later = await applyModuleMigrations({
|
||||
aggregate: absentInventory,
|
||||
database: migrationDatabase,
|
||||
});
|
||||
expect(later.applied).toHaveLength(1);
|
||||
expect(later.applied[0]?.moduleId).toBe("core");
|
||||
expect(later.retained).toEqual([{ appliedCount: 1, moduleId: "example" }]);
|
||||
|
||||
// 5. The module's table and rows are untouched, and so is its history.
|
||||
expect(await exampleValues(database.client)).toEqual(["kept"]);
|
||||
expect((await readHistory(database.client)).map((row) => row.identity)).toContain(
|
||||
"example:0000_initial",
|
||||
);
|
||||
|
||||
// 6. The module is reinstalled.
|
||||
await installExample(modulesDirectory);
|
||||
const restored = await inventory(coreMigrationsDirectory, modulesDirectory);
|
||||
|
||||
// 7. Its history is strict again, and valid; its data is where it left it.
|
||||
const restoredPlan = await planModuleMigrations({
|
||||
aggregate: restored,
|
||||
database: migrationDatabase,
|
||||
});
|
||||
expect(restoredPlan.retained).toEqual([]);
|
||||
expect(restoredPlan.pendingCount).toBe(0);
|
||||
expect(
|
||||
restoredPlan.entries.find((entry) => entry.identity === "example:0000_initial")?.status,
|
||||
).toBe("applied");
|
||||
const noOp = await applyModuleMigrations({
|
||||
aggregate: restored,
|
||||
database: migrationDatabase,
|
||||
});
|
||||
expect(noOp.noOp).toBe(true);
|
||||
expect(await exampleValues(database.client)).toEqual(["kept"]);
|
||||
|
||||
// A returning package whose applied migration changed is refused, exactly as before.
|
||||
const sqlPath = path.join(modulesDirectory, "example", "drizzle", "0000_initial.sql");
|
||||
await writeFile(sqlPath, `${await readFile(sqlPath, "utf8")}-- edited\n`);
|
||||
await captureRunnerError(
|
||||
planModuleMigrations({
|
||||
aggregate: await inventory(coreMigrationsDirectory, modulesDirectory),
|
||||
database: migrationDatabase,
|
||||
}),
|
||||
"MIGRATION_CHECKSUM_MISMATCH",
|
||||
);
|
||||
expect(await exampleValues(database.client)).toEqual(["kept"]);
|
||||
});
|
||||
} finally {
|
||||
await rm(scratch, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
+37
-29
@@ -1,9 +1,10 @@
|
||||
# Ekklesia's one production image (ADR 0044): the `api` service and every operator-run CLI, from one
|
||||
# build. See deploy/README.md for how to build and run it.
|
||||
# Ekklesia's one generic platform image (ADR 0044, ADR 0048): the `api` service and every operator-run
|
||||
# CLI, from one build, for every installation. See deploy/README.md for how to build and run it.
|
||||
#
|
||||
# The build context is an already-assembled workspace: the platform, zero or more module packages in
|
||||
# `modules/<id>`, and the `pnpm-lock.yaml` that describes exactly that set. This file clones nothing
|
||||
# and chooses no modules. With an empty or absent `modules/`, it builds the platform alone.
|
||||
# The build context is this repository. The image contains **zero feature modules**: installed
|
||||
# modules live in installation-owned storage, mounted read-only at /var/lib/ekklesia/modules
|
||||
# (EKKLESIA_MODULES_DIR) and discovered when a process starts. `.dockerignore` keeps a checkout's
|
||||
# `modules/` out of the context, so a build is the same whether or not one is present.
|
||||
#
|
||||
# No `# syntax=` directive, deliberately: it would make BuildKit pull an unpinned frontend image.
|
||||
# The built-in frontend supports everything below, including cache mounts.
|
||||
@@ -28,9 +29,10 @@ COPY package.json ./
|
||||
RUN corepack enable && corepack install
|
||||
|
||||
# ---------------------------------------------------------------------------------------------------
|
||||
# manifests: the lockfile, workspace configuration, and every workspace package.json, at their
|
||||
# repository paths. Dependency installation keys on these alone, so a source-only change reuses the
|
||||
# installed layers.
|
||||
# manifests: the lockfile, workspace configuration, and the package.json of every platform package
|
||||
# the runtime needs, at their repository paths. Dependency installation keys on these alone, so a
|
||||
# source-only change reuses the installed layers. The module template is a workspace package but not
|
||||
# part of the platform, so neither its manifest nor its dependencies reach the runtime.
|
||||
# ---------------------------------------------------------------------------------------------------
|
||||
FROM base AS manifests
|
||||
WORKDIR /context
|
||||
@@ -38,9 +40,8 @@ COPY . .
|
||||
RUN set -eu; \
|
||||
mkdir /manifests; \
|
||||
cp package.json pnpm-lock.yaml pnpm-workspace.yaml .npmrc /manifests/; \
|
||||
for manifest in apps/*/package.json builders/*/package.json core/package.json \
|
||||
modules/*/package.json packages/*/package.json surfaces/*/package.json \
|
||||
templates/*/package.json; do \
|
||||
for manifest in apps/*/package.json core/package.json packages/*/package.json \
|
||||
surfaces/*/package.json; do \
|
||||
if [ -f "$manifest" ]; then install -D -m 0644 "$manifest" "/manifests/$manifest"; fi; \
|
||||
done
|
||||
|
||||
@@ -52,37 +53,38 @@ FROM toolchain AS build
|
||||
# module UI builder's, which every module build invokes), and pnpm links a bin only when its target
|
||||
# already exists. The store cache mount keeps a rebuild off the network all the same.
|
||||
COPY . .
|
||||
# A second line of defence behind `.dockerignore`: a context that still carries installed modules is
|
||||
# refused rather than built into a platform image.
|
||||
RUN if [ -e modules ]; then \
|
||||
echo "The build context contains modules/. Installed modules never enter the platform image (ADR 0048)." >&2; \
|
||||
exit 1; \
|
||||
fi
|
||||
RUN --mount=type=cache,id=ekklesia-pnpm-store,target=/pnpm/store \
|
||||
pnpm install --frozen-lockfile --store-dir /pnpm/store
|
||||
# The same gates `pnpm verify` runs for boundaries, modules, and migrations, against exactly the
|
||||
# module set in this context. `pnpm boundaries` comes first and reads only sources: among its rules,
|
||||
# a module that imports something at run time without declaring it a production dependency fails
|
||||
# here, because the production-only install below would leave it unresolvable at API startup. Each
|
||||
# The same gates `pnpm verify` runs for boundaries, modules, and migrations. With no modules in the
|
||||
# context, module validation covers the empty set and migration validation covers Core's own. Each
|
||||
# validate script builds first; the final build is a Turbo cache hit that keeps the intent explicit.
|
||||
RUN pnpm boundaries && pnpm modules:validate && pnpm migrations:validate && pnpm build
|
||||
# What the runtime resolves at startup, at the paths it resolves them from:
|
||||
# - every workspace package's `dist/` (every package exports only from `dist/`), which includes
|
||||
# - every platform package's `dist/` (every package exports only from `dist/`), which includes
|
||||
# the Dashboard and display host assets under `surfaces/shell/dist/browser/`;
|
||||
# - Core's migrations, `core/drizzle/` with its `meta/` journal;
|
||||
# - each module directory whole, minus `node_modules`, because each manifest names its own
|
||||
# entrypoints, UI artifacts, and migrations directory.
|
||||
# `node_modules` comes from the production-only install below, never from this stage.
|
||||
# - Core's migrations, `core/drizzle/` with its `meta/` journal.
|
||||
# Never a module and never the module template. `node_modules` comes from the production-only
|
||||
# install below, never from this stage.
|
||||
RUN set -eu; \
|
||||
mkdir -p /runtime/core /runtime/modules; \
|
||||
mkdir -p /runtime/core; \
|
||||
for package in apps/* core packages/* surfaces/*; do \
|
||||
if [ -d "$package/dist" ]; then \
|
||||
mkdir -p "/runtime/$package"; \
|
||||
cp -a "$package/dist" "/runtime/$package/dist"; \
|
||||
fi; \
|
||||
done; \
|
||||
cp -a core/drizzle /runtime/core/drizzle; \
|
||||
if [ -d modules ]; then \
|
||||
tar -C modules --exclude='node_modules' -cf - . | tar -C /runtime/modules -xf -; \
|
||||
fi
|
||||
cp -a core/drizzle /runtime/core/drizzle
|
||||
|
||||
# ---------------------------------------------------------------------------------------------------
|
||||
# prod-deps: the same lockfile, production dependencies only. No compiler, bundler, or test runner
|
||||
# reaches the runtime image.
|
||||
# prod-deps: the same lockfile, production dependencies only, for the platform packages alone. No
|
||||
# compiler, bundler, or test runner reaches the runtime image, and neither does anything only the
|
||||
# module template depends on.
|
||||
# ---------------------------------------------------------------------------------------------------
|
||||
FROM toolchain AS prod-deps
|
||||
COPY --from=manifests /manifests/ ./
|
||||
@@ -109,13 +111,19 @@ LABEL org.opencontainers.image.title="Ekklesia" \
|
||||
# the API still binds 127.0.0.1. `production` is stated rather than relied on as the default.
|
||||
ENV NODE_ENV=production \
|
||||
EKKLESIA_API_HOST=0.0.0.0 \
|
||||
EKKLESIA_API_PORT=3000
|
||||
EKKLESIA_API_PORT=3000 \
|
||||
EKKLESIA_MODULES_DIR=/var/lib/ekklesia/modules
|
||||
|
||||
# The repository layout, preserved: the API finds the repository root three directories above
|
||||
# `apps/api/dist/`, and every CLI uses its working directory as the root.
|
||||
# `apps/api/dist/`, and every CLI uses its working directory as the root. Installed modules are not
|
||||
# under it: they are read from EKKLESIA_MODULES_DIR.
|
||||
WORKDIR /opt/ekklesia
|
||||
COPY --from=prod-deps /opt/ekklesia/ ./
|
||||
COPY --from=build /runtime/ ./
|
||||
# The mount point for installation-owned module storage (ADR 0048), empty in the image. An image run
|
||||
# with nothing mounted there has zero modules; the reference deployment mounts host storage
|
||||
# read-only over it.
|
||||
RUN install -d -o root -g root -m 0755 /var/lib/ekklesia/modules
|
||||
|
||||
# Files stay root-owned: the unprivileged runtime user can read the installation, never modify it.
|
||||
USER node
|
||||
|
||||
+155
-56
@@ -64,6 +64,7 @@ TLS is everywhere, without any exception in Core:
|
||||
| `/etc/ekklesia/object-storage.env` | The S3 key pair | `object-storage` |
|
||||
| `/etc/ekklesia/tunnel.env` | `TUNNEL_TOKEN` | `tunnel` |
|
||||
| `/etc/ekklesia/tls/` | The CA and server certificates and keys | mounted file by file |
|
||||
| `/var/lib/ekklesia/modules/` | Installed modules ([Installed modules](#installed-modules)); not a secret | `api`, `migrate`, read-only |
|
||||
|
||||
The secret files are owned by root with mode 600, one per service, so no container receives another
|
||||
service's credentials. None is committed, and `deploy/.env` is ignored by Git. `EKKLESIA_ENV_DIR`
|
||||
@@ -94,10 +95,19 @@ and `EKKLESIA_TLS_DIR` in `deploy/.env` move the directories.
|
||||
|
||||
`EKKLESIA_IDENTITY_BASE_URL` and `EKKLESIA_IDENTITY_TRUSTED_ORIGINS` are `https://` and your
|
||||
Ekklesia hostname. `S3_DELIVERY_ENDPOINT` is `https://` and your storage hostname.
|
||||
5. **Configure the proxy:** copy `Caddyfile.example` to `Caddyfile`, and set `EKKLESIA_HOSTNAME`,
|
||||
5. **Create module storage**, empty, even if you install no module yet:
|
||||
|
||||
```sh
|
||||
sudo install -d -o root -g root -m 755 /var/lib/ekklesia/modules
|
||||
```
|
||||
|
||||
`api` and `migrate` refuse to start without it: Compose never creates it for you, so a mistyped
|
||||
path cannot quietly become an empty directory. `EKKLESIA_MODULES_HOST_DIR` in `deploy/.env`
|
||||
moves it.
|
||||
6. **Configure the proxy:** copy `Caddyfile.example` to `Caddyfile`, and set `EKKLESIA_HOSTNAME`,
|
||||
`EKKLESIA_STORAGE_HOSTNAME`, and `EKKLESIA_ACME_EMAIL` in `deploy/.env`. Both hostnames must
|
||||
resolve to this host with ports 80 and 443 reachable for HTTP-01, or see [DNS-01](#dns-01).
|
||||
6. **Start the data services, then migrate explicitly:**
|
||||
7. **Start the data services, then migrate explicitly:**
|
||||
|
||||
```sh
|
||||
docker compose -f compose.yaml -f compose.proxy.yaml up -d --wait postgres object-storage
|
||||
@@ -107,8 +117,8 @@ and `EKKLESIA_TLS_DIR` in `deploy/.env` move the directories.
|
||||
|
||||
Nothing ever applies migrations automatically: the API never does, and `migrate` runs only when
|
||||
you run it.
|
||||
7. **Start everything:** `docker compose -f compose.yaml -f compose.proxy.yaml up -d --wait`.
|
||||
8. **Check it:** `curl https://<hostname>/api/v1/health/ready` answers `{"status":"ready"}`, and
|
||||
8. **Start everything:** `docker compose -f compose.yaml -f compose.proxy.yaml up -d --wait`.
|
||||
9. **Check it:** `curl https://<hostname>/api/v1/health/ready` answers `{"status":"ready"}`, and
|
||||
`docker compose run --rm migrate node core/dist/cli/doctor.js` reports `READY` (with warnings
|
||||
for anything optional you left unconfigured, such as email).
|
||||
|
||||
@@ -123,25 +133,93 @@ is created already verified, so they can sign in before any email relay exists
|
||||
([ADR 0035](../docs/architecture/decisions/0035-initial-platform-owner-bootstrap.md)). To supply the
|
||||
password from a secret manager instead, pipe it on standard input and use `-T` instead of `-it`.
|
||||
|
||||
### Installed modules
|
||||
|
||||
The image contains no modules
|
||||
([ADR 0048](../docs/architecture/decisions/0048-generic-image-and-installed-module-storage.md)).
|
||||
Installed modules live on the host, in module storage: `/var/lib/ekklesia/modules`, or
|
||||
`EKKLESIA_MODULES_HOST_DIR`. Compose mounts it **read-only** at `/var/lib/ekklesia/modules` in `api`
|
||||
and `migrate`, which is where `EKKLESIA_MODULES_DIR` points. Ekklesia never writes there.
|
||||
Installing, updating, or removing a module never needs a new image, only a restart.
|
||||
|
||||
**What belongs there.** One directory per module, each a self-contained installed module: its
|
||||
`package.json`, `ekklesia.module.json`, built artifacts, migrations, and the production
|
||||
dependencies its server code imports, in its own `node_modules`
|
||||
([Module development](../docs/module-development/README.md#an-installed-module-is-self-contained)).
|
||||
Copying a module's `dist/` alone is not enough. Files are owned by root and readable by everyone
|
||||
(directories 755, files 644), so the API's unprivileged user can read them and never change them.
|
||||
|
||||
**Nothing else belongs there.** Every non-hidden directory must be a valid module, or the API does
|
||||
not start. That includes `lost+found`, so if module storage is a file system of its own, mount it
|
||||
one level up and point `EKKLESIA_MODULES_HOST_DIR` at a directory inside it. Hidden entries are
|
||||
ignored, which is what makes staging possible.
|
||||
|
||||
**Changes take effect only at a restart.** A running API never notices module storage changing, and
|
||||
never loads or unloads a module while it runs. Change module storage only while the API is stopped:
|
||||
it reads browser artifacts from disk as it serves them, so files changed under a running API would
|
||||
serve one version's browser code beside another's server code.
|
||||
|
||||
To **install or update** a module, with `<id>` its module id:
|
||||
|
||||
```sh
|
||||
docker compose stop api
|
||||
# Stage under a hidden name, which discovery ignores, then rename it into place in one step.
|
||||
sudo cp -R ./<id> /var/lib/ekklesia/modules/.incoming-<id>
|
||||
sudo chown -R root:root /var/lib/ekklesia/modules/.incoming-<id>
|
||||
sudo chmod -R a+rX,go-w /var/lib/ekklesia/modules/.incoming-<id>
|
||||
sudo mv /var/lib/ekklesia/modules/<id> /var/lib/ekklesia/modules/.previous-<id> # update only
|
||||
sudo mv /var/lib/ekklesia/modules/.incoming-<id> /var/lib/ekklesia/modules/<id>
|
||||
docker compose run --rm migrate node core/dist/cli/validate-modules.js
|
||||
docker compose run --rm migrate # plan: review it
|
||||
docker compose run --rm migrate node core/dist/cli/apply-migrations.js # only when the plan is right
|
||||
docker compose up -d --wait api
|
||||
```
|
||||
|
||||
Back up before an update that brings migrations: migrations are forward-only. A newly installed
|
||||
module is enabled unless an administrator disabled it before; set its configuration and grants as
|
||||
its documentation says. Keep `.previous-<id>` until the update has proven itself, then delete it.
|
||||
|
||||
To **remove** a module, stop the API, delete (or move away) its directory, and start the API again.
|
||||
Remove first any module that *requires* it, or the API refuses to start. Removing a module's code
|
||||
deletes nothing else. Its tables and rows, its buckets and objects, its enablement, configuration,
|
||||
Dashboard placement, and grants all stay. Its applied migration history stays too, and the
|
||||
migration plan and Doctor report it as `retained` history for a module that is not installed. That
|
||||
is information, not a problem, and it blocks no other migration. To stop a module without removing
|
||||
it, disable it instead (`set-module-enabled.js --disable`, [Operator commands](#operator-commands)).
|
||||
There is no purge: deleting a module's data is a separate operation that does not exist yet.
|
||||
|
||||
To **restore** a removed module, install it again as above, at the same version or a newer one. It
|
||||
finds its data, its settings, and its history where it left them. Its history is strict again: a
|
||||
package that no longer contains a migration that was applied, or contains one with different
|
||||
contents, is refused by the migration plan, so an older version cannot be installed over newer data.
|
||||
|
||||
Module storage is installation state: back it up with the database and the buckets.
|
||||
|
||||
### Upgrading
|
||||
|
||||
1. **Pull or build the new image**, and change `EKKLESIA_IMAGE` in `deploy/.env`. Keep the old tag.
|
||||
Installed modules stay where they are, in module storage.
|
||||
2. **Back up** ([Backups](#backups)).
|
||||
3. **Plan, review, then apply:** `docker compose run --rm migrate` with the new image, then
|
||||
3. **Check the installed modules against the new platform:**
|
||||
`docker compose run --rm migrate node core/dist/cli/validate-modules.js`. A module whose
|
||||
`compatibility.platform` range excludes the new version fails here, and would stop the API from
|
||||
starting: update it first, or stay on the old image.
|
||||
4. **Plan, review, then apply:** `docker compose run --rm migrate` with the new image, then
|
||||
`docker compose run --rm migrate node core/dist/cli/apply-migrations.js`.
|
||||
4. **Restart on the new image:** `docker compose -f compose.yaml -f compose.proxy.yaml up -d --wait`.
|
||||
5. **Restart on the new image:** `docker compose -f compose.yaml -f compose.proxy.yaml up -d --wait`.
|
||||
|
||||
### Rolling back
|
||||
|
||||
Set `EKKLESIA_IMAGE` back to the previous tag and run `up -d --wait` again. That is enough only when
|
||||
the upgrade applied no migrations (the plan in step 3 showed none pending). Migrations are
|
||||
the upgrade applied no migrations (the plan in step 4 showed none pending). Migrations are
|
||||
forward-only. If any were applied, restore the backup taken in step 2 before starting the previous
|
||||
image, because it was not built for the newer schema.
|
||||
image, because it was not built for the newer schema. Module storage is not part of the image, so a
|
||||
rollback does not change installed modules.
|
||||
|
||||
### Backups
|
||||
|
||||
Slice 6 adds the scheduled backup job and the restore procedure (ADR 0044, Backups). Until then,
|
||||
take a `pg_dump` and a copy of every bucket before each upgrade by hand.
|
||||
take a `pg_dump`, a copy of every bucket, and a copy of module storage before each upgrade by hand.
|
||||
|
||||
### Split Dashboard and API origins
|
||||
|
||||
@@ -309,9 +387,13 @@ deployment:
|
||||
|
||||
- `pnpm deploy:config` validates the base, base + proxy, and base + proxy + tunnel with the example
|
||||
env files. CI runs it in `pnpm ci:verify`.
|
||||
- `pnpm deploy:smoke` (`scripts/smoke-deploy.sh`) runs the whole deployment end to end, against a
|
||||
platform-only image and a disposable PostgreSQL and SeaweedFS that exist only for that run. It
|
||||
follows the first boot above and checks the following through the proxy:
|
||||
- `bash scripts/check-image.sh` builds the image and proves it is generic: no `/opt/ekklesia/modules`,
|
||||
no module template, no `ekklesia.module.json` outside installed dependencies, an empty root-owned
|
||||
`/var/lib/ekklesia/modules`, and `EKKLESIA_MODULES_DIR` set. It then builds again from a checkout
|
||||
with a populated `modules/` directory and requires an identical runtime file system. CI runs it.
|
||||
- `pnpm deploy:smoke` (`scripts/smoke-deploy.sh`) runs the whole deployment end to end, against the
|
||||
generic image, empty module storage, and a disposable PostgreSQL and SeaweedFS that exist only for
|
||||
that run. It follows the first boot above and checks the following through the proxy:
|
||||
- readiness and the Dashboard;
|
||||
- a signed object URL fetched through the storage hostname, and the same image loaded twice over
|
||||
HTTP/2, the second time as a conditional request that must be a clean `304`;
|
||||
@@ -324,12 +406,23 @@ deployment:
|
||||
- Doctor.
|
||||
|
||||
`EKKLESIA_SMOKE_TOPOLOGY=split` runs the [split topology](#split-dashboard-and-api-origins)
|
||||
instead: an image with the repository's module template installed as a test-only module, every
|
||||
hostname's served and refused paths, exact CORS for the Dashboard's origin and none for others,
|
||||
and then a real headless Chromium (pinned by digest, driven over the DevTools protocol) that signs
|
||||
in on the Dashboard's hostname, checks the session cookie lives only on the API's, mounts the
|
||||
module's UI from the API's origin and sees its own credentialed requests succeed, mounts its display
|
||||
on the API's hostname with its service worker, and signs out.
|
||||
instead, on the **same generic image**, and also proves [Installed modules](#installed-modules)
|
||||
end to end:
|
||||
- the image boots with no module;
|
||||
- the repository's module template, packaged as a self-contained test module
|
||||
(`scripts/lib/test-module-package.Dockerfile`, test-only and not a package format), is installed
|
||||
into module storage, its migration is planned and explicitly applied, the API restarts, and the
|
||||
module reads data it owns;
|
||||
- every hostname's served and refused paths, exact CORS for the Dashboard's origin and none for
|
||||
others;
|
||||
- a real headless Chromium (pinned by digest, driven over the DevTools protocol) signs in on the
|
||||
Dashboard's hostname, checks the session cookie lives only on the API's, mounts the module's UI
|
||||
from the API's origin and sees its own credentialed requests succeed, mounts its display on the
|
||||
API's hostname with its service worker, and signs out;
|
||||
- the module is removed and the API restarted: Core still runs, the module's routes are gone,
|
||||
migration planning and applying succeed and report its retained history, and its data is intact;
|
||||
- the module is restored and the API restarted: its history validates strictly and it returns its
|
||||
previous data.
|
||||
|
||||
CI runs both topologies on every change.
|
||||
- **Test-only:** the smoke test runs Caddy with its internal CA (`local_certs`), because a test run
|
||||
@@ -350,40 +443,37 @@ is no second image and no second build.
|
||||
|
||||
### What it builds
|
||||
|
||||
The build context is an **already-assembled workspace**: this repository, plus zero or more module
|
||||
packages already present in `modules/<id>`, plus a `pnpm-lock.yaml` describing exactly that set.
|
||||
The Dockerfile clones nothing and chooses no modules. It builds what it is given. With no
|
||||
`modules/` directory, it builds the platform alone; CI builds that case on every change.
|
||||
The build context is **this repository**, and the image is the same for every installation
|
||||
([ADR 0048](../docs/architecture/decisions/0048-generic-image-and-installed-module-storage.md)). It
|
||||
contains **zero feature modules**: installed modules live in module storage outside it
|
||||
([Installed modules](#installed-modules)). No input is specific to an installation: no module list,
|
||||
no lockfile of its own, no branding, no configuration.
|
||||
|
||||
Assembling the context (checking out the platform at a release tag, each module at a pinned tag or
|
||||
commit, and the instance's own lockfile) belongs to the installation's instance composition, not to
|
||||
this file.
|
||||
|
||||
> **Being replaced.** [ADR 0048](../docs/architecture/decisions/0048-generic-image-and-installed-module-storage.md)
|
||||
> supersedes this composition model. The image will contain no modules and ignore `modules/` in its
|
||||
> build context. Installed modules will live in installation-owned storage
|
||||
> (`EKKLESIA_MODULES_DIR`, `/var/lib/ekklesia/modules` in the image), mounted read-only. Until that
|
||||
> lands, the image behaves exactly as described here. Do not start a new per-installation assembly.
|
||||
A `modules/` directory in the checkout never reaches the image. `.dockerignore` keeps it out of the
|
||||
build context, and the build refuses a context that still contains one. The module template is a
|
||||
workspace package, so it is built and validated with everything else, but neither its files nor its
|
||||
dependencies reach the runtime image. `scripts/check-image.sh` proves all of this on every change
|
||||
([Testing this deployment](#testing-this-deployment)).
|
||||
|
||||
| Stage | Does |
|
||||
| --- | --- |
|
||||
| `base` | `node:24.18.0-trixie-slim`, pinned by its multi-architecture index digest. Debian, so glibc, because module dependencies such as sharp ship glibc prebuilt binaries. Every stage starts here. |
|
||||
| `toolchain` | Enables pnpm through Corepack, at exactly the root `packageManager` version. |
|
||||
| `manifests` | Collects the lockfile, workspace configuration, `.npmrc`, and every workspace `package.json` at its repository path. |
|
||||
| `build` | `pnpm install --frozen-lockfile`, then `pnpm boundaries`, `pnpm modules:validate`, `pnpm migrations:validate`, and `pnpm build`, against exactly the modules in the context. Never uses host `node_modules`: `.dockerignore` excludes them. |
|
||||
| `prod-deps` | The same lockfile with `--prod`: production dependencies only. |
|
||||
| `runtime` | Production dependencies plus built outputs, owned by root and run as the unprivileged `node` user (uid 1000). No pnpm, no compiler, no curl. |
|
||||
| `manifests` | Collects the lockfile, workspace configuration, `.npmrc`, and the `package.json` of every platform package (not the module template's) at its repository path. |
|
||||
| `build` | Refuses a context containing `modules/`, then `pnpm install --frozen-lockfile`, `pnpm boundaries`, `pnpm modules:validate`, `pnpm migrations:validate`, and `pnpm build`. Never uses host `node_modules`: `.dockerignore` excludes them. |
|
||||
| `prod-deps` | The same lockfile with `--prod`, for the platform packages only: production dependencies only. |
|
||||
| `runtime` | Production dependencies plus built outputs, owned by root and run as the unprivileged `node` user (uid 1000), and an empty, root-owned `/var/lib/ekklesia/modules`. No pnpm, no compiler, no curl. |
|
||||
|
||||
The runtime keeps the repository layout at `/opt/ekklesia`, because that layout is how the platform
|
||||
finds its files:
|
||||
|
||||
- the API takes the repository root as three directories above `apps/api/dist/`, and reads
|
||||
`package.json`, `packages/module-sdk/package.json`, and `modules/` beneath it;
|
||||
`package.json` and `packages/module-sdk/package.json` beneath it;
|
||||
- every CLI takes its **working directory** as the root and also reads Core's migrations in
|
||||
`core/drizzle/`, so commands must run from `/opt/ekklesia`, the image's `WORKDIR`;
|
||||
- the Dashboard and display host files are read from `surfaces/shell/dist/browser/`;
|
||||
- each module directory is copied whole, minus `node_modules`, because each manifest names its
|
||||
own entrypoints, UI artifacts, and migrations directory.
|
||||
- installed modules are read from `EKKLESIA_MODULES_DIR`, `/var/lib/ekklesia/modules`, never from
|
||||
under `/opt/ekklesia`. With nothing mounted there, the image runs with no modules.
|
||||
|
||||
### Building
|
||||
|
||||
@@ -405,30 +495,37 @@ set `DOCKER_BUILDKIT=1` if your Docker CLI has no buildx plugin.
|
||||
No secret is a build argument, and none enters any layer. `.env` files are excluded from the build
|
||||
context entirely, and no registry credentials are needed.
|
||||
|
||||
**A module missing a production dependency fails the build, not the API.** The runtime receives
|
||||
production dependencies only, so a module that imports a package at run time but declares it only
|
||||
as a development or peer dependency would build and then fail at startup. `pnpm boundaries` runs
|
||||
in the build stage and rejects that module first
|
||||
(`runtime-imports-are-production-dependencies`, see
|
||||
[Repository boundaries](../docs/architecture/repository-boundaries.md)). The same applies to every
|
||||
other boundary rule: a module the platform's own CI would reject cannot be built into an image.
|
||||
### The runtime an installed module targets
|
||||
|
||||
**A platform-only image does not exercise native module dependencies.** With no modules in the
|
||||
context, nothing like sharp is installed, so the CI image build proves nothing about them. An
|
||||
instance build that includes such a module must smoke-test the built image itself: start it,
|
||||
confirm `/api/v1/health/ready` answers `200`, and exercise the module's native code path (for
|
||||
example, process one image).
|
||||
An installed module runs on the image's Node.js, inside the API process, from its own directory
|
||||
([ADR 0048](../docs/architecture/decisions/0048-generic-image-and-installed-module-storage.md)):
|
||||
|
||||
- **Node.js 24.18.0 on Debian 13, so glibc**, for the architecture the image was built for. A native
|
||||
dependency such as sharp must be built or downloaded for exactly that.
|
||||
- **Its own dependencies only.** Node resolves a module's imports from the module's directory, and
|
||||
the platform's `node_modules` is never on that path. A module whose server code imports something
|
||||
its directory does not contain stops the API from starting, with a sanitized
|
||||
`MODULE_RUNTIME_IMPORT_FAILED`.
|
||||
- **The platform's exact Drizzle version, for now.** Core runs a module's queries through its own
|
||||
Drizzle instance over tables the module defined with its own copy, so the two copies must be
|
||||
interchangeable. That is a constraint of the current implementation, not a permanent contract.
|
||||
|
||||
The repository's boundary checks (`pnpm boundaries`) prove only the source modules in a workspace.
|
||||
An installed module's conformance is its own CI's job, so check a module before installing it:
|
||||
`validate-modules.js` ([Operator commands](#operator-commands)) validates what can be checked on an
|
||||
installed directory, and the API's startup refuses anything it cannot load.
|
||||
|
||||
### Configuration
|
||||
|
||||
Configuration comes **only from the container environment**, never from a file inside the image.
|
||||
The image sets exactly three variables itself:
|
||||
The image sets exactly four variables itself:
|
||||
|
||||
| Variable | Image value | Why |
|
||||
| --- | --- | --- |
|
||||
| `NODE_ENV` | `production` | Stated rather than relied on. Anything other than `development` already validates as production. |
|
||||
| `EKKLESIA_API_HOST` | `0.0.0.0` | Reachable from other containers. Outside a container the platform default stays `127.0.0.1`. |
|
||||
| `EKKLESIA_API_PORT` | `3000` | The platform default, made explicit; `EXPOSE 3000`. |
|
||||
| `EKKLESIA_MODULES_DIR` | `/var/lib/ekklesia/modules` | Where installed modules are read from (ADR 0048). Outside a container, unset means the checkout's own `modules/`. Compose states the same value beside the mount. |
|
||||
|
||||
Everything else is the ordinary host configuration in
|
||||
[API § Required configuration](../docs/development/api.md#required-configuration), supplied by the
|
||||
@@ -449,8 +546,8 @@ Never publish the API's port to the host or the Internet. Only the proxy publish
|
||||
- **Read-only:** the image was verified under exactly these flags. The API reached `healthy`, and
|
||||
the migration and Doctor commands ran. Nothing wrote to `/tmp`, which stays mounted as headroom
|
||||
for module code that needs scratch space. The installation, `/home/node` included, is not
|
||||
writable, so anything that needs a writable path other than `/tmp` is a defect to report, not to
|
||||
work around.
|
||||
writable, and neither is module storage, which is mounted read-only. Anything that needs a
|
||||
writable path other than `/tmp` is a defect to report, not to work around.
|
||||
|
||||
### Health
|
||||
|
||||
@@ -488,14 +585,16 @@ reports direct mode for an API that trusts its proxy. The running API's own valu
|
||||
| Apply migrations | `docker compose run --rm migrate node core/dist/cli/apply-migrations.js` |
|
||||
| Doctor (read-only report) | `docker compose run --rm migrate node core/dist/cli/doctor.js` |
|
||||
| Create the initial owner (interactive) | `docker compose run --rm -it migrate node core/dist/cli/create-initial-owner.js --email [email protected]` |
|
||||
| Validate the installed modules (read-only) | `docker compose run --rm migrate node core/dist/cli/validate-modules.js` |
|
||||
| Enable or disable a module | `docker compose run --rm migrate node core/dist/cli/set-module-enabled.js --enable --module <id>` (or `--disable`) |
|
||||
|
||||
The plan, apply, Doctor, and initial-owner commands are exercised by the smoke test; module
|
||||
enablement is not.
|
||||
Every command reads installed modules from the same `EKKLESIA_MODULES_DIR` the API does. The plan,
|
||||
apply, Doctor, and initial-owner commands are exercised by the smoke test, and so are installing,
|
||||
removing, and restoring a module; module validation and enablement are not.
|
||||
|
||||
### Size
|
||||
|
||||
The platform-only image is about 378 MB: the Debian slim Node base (about 230 MB), production
|
||||
The image is about 378 MB, and it contains no module: the Debian slim Node base (about 230 MB), production
|
||||
dependencies (about 141 MB), and built outputs (about 6 MB). A rebuild that reuses a warm pnpm store
|
||||
cache can come out about 21 MB larger. When esbuild's install script runs, it hard-links its
|
||||
`bin/esbuild` to the native binary it ships beside it; when pnpm restores esbuild from the store's
|
||||
|
||||
@@ -32,8 +32,20 @@ x-ekklesia-app: &ekklesia-app
|
||||
# without any storage TLS exception. The database CA travels separately, as PEM text in
|
||||
# EKKLESIA_DATABASE_CA, and is the only CA the database connection trusts.
|
||||
NODE_EXTRA_CA_CERTS: /run/ekklesia/tls/ca.crt
|
||||
# Installed modules (ADR 0048). The image sets the same value; stating it beside the mount below
|
||||
# keeps the variable and the mount target from ever drifting apart.
|
||||
EKKLESIA_MODULES_DIR: /var/lib/ekklesia/modules
|
||||
volumes:
|
||||
- ${EKKLESIA_TLS_DIR:-/etc/ekklesia/tls}/ca.crt:/run/ekklesia/tls/ca.crt:ro
|
||||
# Installation-owned module storage, outside the image (ADR 0048). Read-only: Ekklesia never
|
||||
# writes installed code. The host directory must already exist; a missing one stops the
|
||||
# container from starting instead of Compose quietly creating an empty, root-owned directory.
|
||||
- type: bind
|
||||
source: ${EKKLESIA_MODULES_HOST_DIR:-/var/lib/ekklesia/modules}
|
||||
target: /var/lib/ekklesia/modules
|
||||
read_only: true
|
||||
bind:
|
||||
create_host_path: false
|
||||
networks:
|
||||
data: {}
|
||||
storage: {}
|
||||
|
||||
Vendored
+6
@@ -15,6 +15,12 @@ EKKLESIA_IMAGE=
|
||||
EKKLESIA_ENV_DIR=/etc/ekklesia
|
||||
EKKLESIA_TLS_DIR=/etc/ekklesia/tls
|
||||
|
||||
# Installed modules (ADR 0048): the host directory mounted read-only into `api` and `migrate` at
|
||||
# /var/lib/ekklesia/modules. It must already exist; Compose never creates it. Create it once, owned
|
||||
# by root, before the first `up`:
|
||||
# sudo install -d -o root -g root -m 755 /var/lib/ekklesia/modules
|
||||
# EKKLESIA_MODULES_HOST_DIR=/var/lib/ekklesia/modules
|
||||
|
||||
# --- compose.proxy.yaml ---------------------------------------------------------------------------
|
||||
# The Ekklesia hostname and the storage hostname (signed object URLs). In the default same-origin
|
||||
# topology (Caddyfile.example) EKKLESIA_HOSTNAME serves the Dashboard, the API, and displays.
|
||||
|
||||
Vendored
+8
-3
@@ -6,9 +6,14 @@
|
||||
# Never commit the filled-in file. Generate secrets with scripts/gen-secrets.sh, and the database CA
|
||||
# line with scripts/init-tls.sh.
|
||||
#
|
||||
# The image already sets NODE_ENV=production, EKKLESIA_API_HOST=0.0.0.0, and EKKLESIA_API_PORT=3000.
|
||||
# Compose sets NODE_EXTRA_CA_CERTS. The proxy overlay sets the three EKKLESIA_CLIENT_ADDRESS_*
|
||||
# variables. None of those belong here.
|
||||
# The image already sets NODE_ENV=production, EKKLESIA_API_HOST=0.0.0.0, EKKLESIA_API_PORT=3000, and
|
||||
# EKKLESIA_MODULES_DIR=/var/lib/ekklesia/modules. Compose sets NODE_EXTRA_CA_CERTS and mounts module
|
||||
# storage at that path. The proxy overlay sets the three EKKLESIA_CLIENT_ADDRESS_* variables. None of
|
||||
# those belong here.
|
||||
#
|
||||
# Installed modules (ADR 0048) are read from EKKLESIA_MODULES_DIR, which must name the mounted
|
||||
# directory. Move the host directory with EKKLESIA_MODULES_HOST_DIR in deploy/.env instead.
|
||||
# EKKLESIA_MODULES_DIR=/var/lib/ekklesia/modules
|
||||
#
|
||||
# Values that are part of the reference topology are filled in; everything installation-specific is
|
||||
# blank. Full reference: docs/development/api.md#required-configuration.
|
||||
|
||||
@@ -652,3 +652,23 @@ Still deliberately absent after Phase 8: a theme editor, a brand builder, or any
|
||||
module store or installer; a standalone host; signed write URLs, resumable uploads, or any body above
|
||||
the absolute ceiling; module-owned offline *content*; and every real feature module, Signage
|
||||
included. Nothing in this pass claims any of them exists.
|
||||
|
||||
[ADR 0048](decisions/0048-generic-image-and-installed-module-storage.md) separates the platform
|
||||
artifact from what an installation installs:
|
||||
|
||||
- **One generic image.** `deploy/Dockerfile` builds one image with zero feature modules, the same
|
||||
for every installation; a checkout's `modules/` never reaches it.
|
||||
- **Module storage.** Installed modules live in installation-owned storage that one Core resolver
|
||||
selects from `EKKLESIA_MODULES_DIR`. The API, every operator command, and Doctor use the same
|
||||
resolver, and Core's own migrations are always found under the platform, never beside the
|
||||
modules. A configured directory that is relative, blank, missing, or unreadable fails closed
|
||||
with a sanitized error.
|
||||
- **Self-contained modules.** An installed module is a self-contained directory with its own
|
||||
production dependencies; the platform lends it none. Discovery still happens only when a process
|
||||
starts, so module changes take effect at a controlled restart and nothing is hot loaded.
|
||||
- **Removing keeps data.** Removing a module's code keeps its data and stored state. The migration
|
||||
runner retains and reports the applied history of a module that is absent, instead of refusing
|
||||
to plan, while Core's history and every installed module's history stay strict, so a returning
|
||||
module that lost or changed an applied migration is still refused.
|
||||
- **Trust is unchanged.** Installed modules are still trusted code; module storage is mounted
|
||||
read-only, which keeps Ekklesia from modifying installed code but does not sandbox it.
|
||||
|
||||
@@ -60,6 +60,7 @@ No real values are shown here — configure per environment and never commit the
|
||||
| `EKKLESIA_DATABASE_CA` | Optional. PEM text of a private certificate authority to trust in addition to the system store, for a database using an internal CA. |
|
||||
| `EKKLESIA_API_HOST` | Optional. Listen address. Defaults to loopback (`127.0.0.1`). An explicit non-loopback address is permitted for future deployment. |
|
||||
| `EKKLESIA_API_PORT` | Optional. Listen port, `0`–`65535` (`0` requests an OS-assigned port). Defaults to `3000`. |
|
||||
| `EKKLESIA_MODULES_DIR` | Optional. The installed-modules directory, as an absolute path ([ADR 0048](../architecture/decisions/0048-generic-image-and-installed-module-storage.md)). Unset means the checkout's own `modules/`, where a missing directory is simply zero modules. Set, the directory must exist: a relative, blank, or unreadable value, or a missing directory, stops startup before any database connection with a sanitized `MODULES_DIRECTORY_INVALID` or `MODULES_DIRECTORY_UNREADABLE` that never contains the value. The production image sets `/var/lib/ekklesia/modules`. Every operator command and Doctor read the same value. |
|
||||
| `EKKLESIA_IDENTITY_SECRET` | At least 32 characters, high entropy (for example `openssl rand -base64 32`). Same requirement as Phase 4A. |
|
||||
| `EKKLESIA_IDENTITY_BASE_URL` | Exactly one canonical `http`/`https` origin — scheme, host, and non-default port only, with no path, trailing slash, query, fragment, credentials, or explicit default port. `https` is required in production; loopback and unspecified hosts are rejected in production. |
|
||||
| `EKKLESIA_IDENTITY_TRUSTED_ORIGINS` | Comma-separated list of canonical origins, validated identically to the base URL (no wildcards). Used for both Better Auth's own CSRF/origin checks and the API's CORS policy — there is only one trusted-origin list. The base URL's own origin is always trusted as well and need not be listed ([ADR 0045](../architecture/decisions/0045-split-dashboard-and-api-origins.md)). |
|
||||
|
||||
@@ -173,8 +173,9 @@ integration layer depends on authorization).
|
||||
exact same installed-module inventory every time it starts, in a fixed order, before anything
|
||||
else:
|
||||
|
||||
1. Resolve module-discovery configuration (the repository's `modules/` directory, and the
|
||||
platform/Module SDK versions from their own `package.json` files).
|
||||
1. Resolve module-discovery configuration: the installed-modules directory, chosen by Core's one
|
||||
resolver from `EKKLESIA_MODULES_DIR` (the repository's own `modules/` when it is unset, ADR
|
||||
0048), and the platform/Module SDK versions from the platform's own `package.json` files.
|
||||
2. Discover installed modules exactly once (`discoverModules`).
|
||||
3. Compose the platform permission registry from that exact discovered list
|
||||
(`composePlatformPermissionRegistry`).
|
||||
@@ -199,7 +200,8 @@ never falls back to `CORE_PERMISSION_REGISTRY` or any Core-only registry. The re
|
||||
resolved from this compiled/source file's own location (two directories up from
|
||||
`apps/api/dist/*.js` or `apps/api/src/*.ts`, which sit at identical depth), never `process.cwd()`
|
||||
— a package-scoped test run has `apps/api` as its working directory, which would otherwise point
|
||||
discovery at the wrong `modules/` directory.
|
||||
discovery at the wrong `modules/` directory. A configured `EKKLESIA_MODULES_DIR` that is not an
|
||||
absolute path to an existing directory fails at step 1, before anything else exists.
|
||||
|
||||
The resulting registry is a **static, immutable snapshot** taken once at startup. There is no hot
|
||||
installation, removal, or dynamic mutation of installed modules or their permissions while the
|
||||
|
||||
@@ -35,8 +35,10 @@ The lint, typecheck, and test commands run both repository-root checks and every
|
||||
package-level Turborepo task. New packages add their own `lint`, `typecheck`, and `test` scripts;
|
||||
the root verification command discovers them automatically.
|
||||
|
||||
`pnpm modules:validate` builds the workspace, then scans `modules/*` without a central registry. A
|
||||
missing or empty modules directory is valid and reports zero installed modules. Installed modules
|
||||
`pnpm modules:validate` builds the workspace, then scans the installed-modules directory without a
|
||||
central registry: the checkout's own `modules/`, or `EKKLESIA_MODULES_DIR` when it is set (ADR
|
||||
0048). A missing or empty `modules/` is valid and reports zero installed modules; a configured
|
||||
directory must exist. Installed modules
|
||||
must produce the compiled JavaScript entrypoints declared by their static manifests. Each module's
|
||||
declared browser contributions are listed with their target capability and range, and marked
|
||||
according to whether a compatible provider is installed — a contribution nobody provides is otherwise
|
||||
@@ -113,9 +115,12 @@ identity secret, not a module secret value.
|
||||
|
||||
Every check has a stable ID and one of four states: pass, warning, failure, or skipped. Independent
|
||||
checks continue after a failure; a failed prerequisite causes only its dependent checks to be
|
||||
skipped. Pending or approval-blocked migrations are warnings and are never applied. Invalid
|
||||
history, checksum disagreement, unknown applied migrations, or a failed configured connection are
|
||||
failures. If `DATABASE_URL` is missing, PostgreSQL is skipped and all static checks still run.
|
||||
skipped. Pending or approval-blocked migrations are warnings and are never applied. Applied
|
||||
history that belongs to a module which is not installed is reported as `retained` and is neither a
|
||||
warning nor a failure (ADR 0048). Invalid history, checksum disagreement, an applied migration that
|
||||
an installed module or Core no longer contains, or a failed configured connection are failures. An
|
||||
unusable configured `EKKLESIA_MODULES_DIR` fails module discovery. If `DATABASE_URL` is missing,
|
||||
PostgreSQL is skipped and all static checks still run.
|
||||
|
||||
`READY` means every check passed. `READY WITH WARNINGS` means no check failed but at least one check
|
||||
warned or was skipped. Both results exit zero. `NOT READY` means at least one check failed and exits
|
||||
|
||||
@@ -472,6 +472,15 @@ CI builds and smoke-tests end to end on every change against disposable services
|
||||
publishes, or runs it against a real installation. No `.env` has ever been committed; only
|
||||
`.env.example`, which the image build context excludes.
|
||||
|
||||
The image contains no modules
|
||||
([ADR 0048](../architecture/decisions/0048-generic-image-and-installed-module-storage.md)).
|
||||
Installed modules come from installation-owned module storage, which the reference deployment mounts
|
||||
**read-only** into `api` and `migrate`, so the Ekklesia process cannot modify installed code. That
|
||||
is an integrity property, not a sandbox: an installed module is still trusted code with the API's
|
||||
database role, exactly as above. The repository's boundary checks prove only source modules in a
|
||||
workspace, and an installed package usually ships none, so installing one means trusting that
|
||||
module's own CI and whoever placed it in module storage.
|
||||
|
||||
Two production defaults are relevant but are **not** by themselves a deployment guarantee:
|
||||
|
||||
- `apps/api` defaults its listen address to loopback (`127.0.0.1`), so it is not reachable off-host
|
||||
|
||||
@@ -171,6 +171,16 @@ whole-batch DDL rollback, Drizzle breakpoint execution, real advisory-lock concu
|
||||
timeouts, lock release after failure, exact destructive approvals, and public-error sanitization.
|
||||
It does not make migrations automatic during Core startup.
|
||||
|
||||
It also proves ADR 0048's retained history for an uninstalled module end to end, with real
|
||||
discovery: Core's committed migrations (the newest one withheld at first) and the module template
|
||||
installed in a modules directory outside the repository are applied, the module writes a row, and
|
||||
its directory is removed. Planning then reports the module's history as retained, and the withheld
|
||||
Core migration applies with the module absent. The module's table and row survive. Reinstalled, the
|
||||
module's history validates strictly again with nothing pending, its row is intact, and a reinstalled
|
||||
package whose applied migration changed is refused. A history row for an absent module is retained
|
||||
and reported, never fatal, while an unknown row belonging to an installed module is still rejected
|
||||
without repair.
|
||||
|
||||
A dedicated set of scenarios exercises the committed Core identity migration against real
|
||||
PostgreSQL: read-only planning on a fresh database, applying it and confirming exactly the four
|
||||
expected `ekklesia_core_*` tables and their indexes exist, a genuine no-op on a second apply, Core
|
||||
@@ -554,6 +564,33 @@ the following pull-request status context to `main` branch protection:
|
||||
CI / migration-smoke (pull_request)
|
||||
```
|
||||
|
||||
## The generic platform image and the reference deployment
|
||||
|
||||
These need Docker and build images locally; nothing is pushed or published.
|
||||
|
||||
- `bash scripts/check-image.sh` builds the production image and proves it is the one generic
|
||||
platform artifact ADR 0048 requires. It checks that the image has:
|
||||
- no `/opt/ekklesia/modules`;
|
||||
- no module template;
|
||||
- no `ekklesia.module.json` outside installed dependencies;
|
||||
- an empty, root-owned `/var/lib/ekklesia/modules`;
|
||||
- `EKKLESIA_MODULES_DIR` set, in the image configuration and in a running process.
|
||||
|
||||
It then builds again from the same checkout with a populated `modules/` directory, and requires
|
||||
the same checks and an identical runtime file system. CI's `image-build` job runs it.
|
||||
- `pnpm deploy:smoke` (`scripts/smoke-deploy.sh`) runs the reference deployment end to end on the
|
||||
generic image with empty module storage. With `EKKLESIA_SMOKE_TOPOLOGY=split`, the same image also
|
||||
runs the installed-module lifecycle:
|
||||
- the module template, packaged as a self-contained test module
|
||||
(`scripts/lib/test-module-package.Dockerfile`, test-only), is installed into module storage;
|
||||
- its migration is planned and explicitly applied, and the API restarted;
|
||||
- a real browser mounts it;
|
||||
- it is removed and the API restarted, and Core still runs, migrations still plan and apply, and
|
||||
its data survives;
|
||||
- it is restored, with its data.
|
||||
|
||||
See [deploy/README.md](../../deploy/README.md#testing-this-deployment).
|
||||
|
||||
## Safe troubleshooting
|
||||
|
||||
Confirm the prerequisites without printing configuration:
|
||||
|
||||
@@ -26,7 +26,10 @@ application's. A module consumes those; it never redefines or bypasses them.
|
||||
|
||||
## Required package shape
|
||||
|
||||
Each immediate child of `modules/` must be an ESM workspace package with a unique package name, a
|
||||
A module is developed as an ESM workspace package in a checkout's `modules/` directory, and it is
|
||||
installed, self-contained, into an installation's module storage
|
||||
([An installed module is self-contained](#an-installed-module-is-self-contained)). In a checkout,
|
||||
each immediate child of `modules/` must be an ESM workspace package with a unique package name, a
|
||||
strict semantic version, and the module layer marker:
|
||||
|
||||
```text
|
||||
@@ -66,6 +69,72 @@ module with no migration history omits `entrypoints.migrations` and needs no pla
|
||||
The package version must match the manifest and runtime-definition versions. Internal package
|
||||
dependencies use `workspace:` ranges and supported public package entrypoints.
|
||||
|
||||
### An installed module is self-contained
|
||||
|
||||
Ekklesia's image contains no modules
|
||||
([ADR 0048](../architecture/decisions/0048-generic-image-and-installed-module-storage.md)). An
|
||||
installation keeps installed modules in its own module storage, the directory `EKKLESIA_MODULES_DIR`
|
||||
names (`/var/lib/ekklesia/modules` in the production image, the checkout's `modules/` when it is
|
||||
unset). Discovery reads it only when a process starts. Nothing is hot loaded, and an installed module
|
||||
is read-only to the platform.
|
||||
|
||||
An installed module directory is **self-contained for runtime use**. Everything the platform reads
|
||||
or loads from it, and everything those files import at run time, resolves inside it:
|
||||
|
||||
- `package.json`, with no `workspace:` range needed at run time, and `ekklesia.module.json`;
|
||||
- every built artifact the manifest declares: `dist/server`, `dist/schema`, jobs, and the UI,
|
||||
display, and contribution bundles;
|
||||
- the `drizzle/` migrations directory, with its journal and `_ekklesia.json`;
|
||||
- a `node_modules` holding the production dependencies of your server, schema, and jobs code,
|
||||
including your own copies of `@ekklesia/module-sdk`, `@ekklesia/contracts`, and `drizzle-orm`.
|
||||
|
||||
**Copying `dist/` is not an installed module.** Your compiled server code imports
|
||||
`@ekklesia/module-sdk` and `drizzle-orm` by name, and Node resolves those only from your module's
|
||||
own directory upwards. The platform's `node_modules` is never on that path, and Ekklesia adds it
|
||||
through no link, `NODE_PATH`, loader hook, or alias. A module whose imports do not resolve stops the
|
||||
API from starting with a sanitized `MODULE_RUNTIME_IMPORT_FAILED`, so test the directory you ship,
|
||||
not your workspace.
|
||||
|
||||
**Native dependencies** must match the platform image's runtime: Node.js 24.18.0 on Debian 13
|
||||
(glibc), on the image's CPU architecture.
|
||||
|
||||
**Drizzle, for now.** You define tables with your own copy of Drizzle, and Core runs your queries
|
||||
through the platform's Drizzle instance (see [Database access](#database-access)). Objects from two
|
||||
copies interoperate only when the copies are interchangeable, so the current implementation requires
|
||||
**exactly the Drizzle version the platform uses**: the one `@ekklesia/module-sdk` depends on. This is
|
||||
a constraint of today's implementation, not a permanent contract.
|
||||
|
||||
Browser artifacts are already self-contained bundles, so nothing changes for them.
|
||||
|
||||
From a workspace, `pnpm --filter <your package> deploy --prod --legacy <directory>` produces such a
|
||||
directory; remove the self-reference pnpm leaves at
|
||||
`node_modules/.pnpm/node_modules/<your package>`, which points back into the workspace. The
|
||||
repository's own smoke test packages the module template this way
|
||||
(`scripts/lib/test-module-package.Dockerfile`). That is a test fixture, not a package format: the
|
||||
format modules are delivered in, how they are signed, and how they reach an installation are not
|
||||
decided yet.
|
||||
|
||||
The repository's boundary checks (`pnpm boundaries`) prove only source modules in a workspace. An
|
||||
installed package normally has no source, so the platform cannot check it the same way. Your
|
||||
module's own CI is where its boundary rules, its production dependencies, and its behaviour inside
|
||||
Ekklesia are proven ([ADR 0047](../architecture/decisions/0047-module-ci-harness.md)).
|
||||
|
||||
### Removing a module keeps its data
|
||||
|
||||
Removing your module's directory from module storage is not deleting its data:
|
||||
|
||||
- your tables and rows, buckets and objects stay;
|
||||
- the platform keeps its enablement decision, your configuration, its Dashboard placement, and
|
||||
permission grants;
|
||||
- your applied migration history stays, and the migration plan reports it as retained history for
|
||||
a module that is not installed.
|
||||
|
||||
Reinstalled with the same module id, your module finds all of it again. Its history is validated
|
||||
strictly again, so a reinstalled package must still contain every migration that was ever applied,
|
||||
unchanged. Never delete or edit a released migration, and never ship an older version over newer
|
||||
data. Disabling is different from removing: a disabled module stays installed and is simply not
|
||||
loaded. Deleting a module's data (purge) does not exist yet.
|
||||
|
||||
## Canonical manifest
|
||||
|
||||
`ekklesia.module.json` is inert and authoritative, so Core can validate every installed module
|
||||
@@ -1447,7 +1516,8 @@ pnpm migrations:plan
|
||||
pnpm verify
|
||||
```
|
||||
|
||||
`pnpm modules:validate` builds the workspace and then atomically validates every installed module.
|
||||
`pnpm modules:validate` builds the workspace and then atomically validates every installed module,
|
||||
from the checkout's `modules/` or from `EKKLESIA_MODULES_DIR` when it is set.
|
||||
It rejects malformed manifests, duplicate IDs or package names, version mismatches, incompatible
|
||||
platform or SDK ranges, missing required modules or capabilities, incompatible installed
|
||||
dependencies, cycles, and missing, escaping, or non-JavaScript server or schema entrypoints.
|
||||
|
||||
Executable
+146
@@ -0,0 +1,146 @@
|
||||
#!/usr/bin/env bash
|
||||
# Proves that Ekklesia's image is the one generic platform artifact ADR 0048 requires:
|
||||
#
|
||||
# - it contains zero feature modules: no /opt/ekklesia/modules, no module template, and no
|
||||
# ekklesia.module.json anywhere outside installed dependencies;
|
||||
# - its module storage mount point, /var/lib/ekklesia/modules, exists, is root-owned, and is empty;
|
||||
# - EKKLESIA_MODULES_DIR names that directory, in the image configuration and in a running process;
|
||||
# - a checkout with a populated modules/ directory builds the same image: the runtime file system
|
||||
# of both builds is identical, and no module from that directory enters it.
|
||||
#
|
||||
# bash scripts/check-image.sh builds both images from this checkout
|
||||
# EKKLESIA_CHECK_IMAGE=<image> bash scripts/check-image.sh checks one existing image's contents
|
||||
#
|
||||
# Both builds use a copy of the files Git would track (committed, staged, or untracked but not
|
||||
# ignored), so they differ in exactly one thing: the second context also has modules/example. Images
|
||||
# are built and inspected locally only; nothing is pushed, published, or logged in to.
|
||||
|
||||
set -Eeuo pipefail
|
||||
|
||||
repository_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$repository_root"
|
||||
|
||||
# shellcheck source=scripts/lib/ci-project.sh
|
||||
source "$repository_root/scripts/lib/ci-project.sh"
|
||||
|
||||
command -v docker >/dev/null || {
|
||||
echo "check-image: docker is required." >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
project="$(ci_project_name image)"
|
||||
modules_directory="/var/lib/ekklesia/modules"
|
||||
work="$(mktemp -d)"
|
||||
built_images=()
|
||||
|
||||
cleanup() {
|
||||
local status=$?
|
||||
trap - EXIT INT TERM
|
||||
for built in "${built_images[@]}"; do
|
||||
docker image rm --force "$built" >/dev/null 2>&1 || true
|
||||
done
|
||||
rm -rf "$work" || true
|
||||
exit "$status"
|
||||
}
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
step() {
|
||||
echo "==> $*"
|
||||
}
|
||||
|
||||
fail() {
|
||||
echo "check-image: $*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
# check_contents <image>: every content guarantee, inside the image and in its configuration.
|
||||
check_contents() {
|
||||
local image="$1" configured
|
||||
configured="$(docker image inspect --format '{{range .Config.Env}}{{println .}}{{end}}' "$image")" ||
|
||||
fail "could not inspect ${image}."
|
||||
grep -qx "EKKLESIA_MODULES_DIR=${modules_directory}" <<<"$configured" ||
|
||||
fail "the image does not set EKKLESIA_MODULES_DIR=${modules_directory}."
|
||||
|
||||
# As root, so a directory the runtime user cannot read cannot hide a manifest. No network.
|
||||
docker run --rm --network none --user 0:0 --entrypoint sh "$image" -ec '
|
||||
fail() { echo "check-image: $*" >&2; exit 1; }
|
||||
[ ! -e /opt/ekklesia/modules ] || fail "/opt/ekklesia/modules exists in the image."
|
||||
[ ! -e /opt/ekklesia/templates ] || fail "the module template is in the image."
|
||||
[ -d /var/lib/ekklesia/modules ] || fail "/var/lib/ekklesia/modules is missing."
|
||||
[ "$(stat -c %U:%G:%a /var/lib/ekklesia/modules)" = "root:root:755" ] ||
|
||||
fail "/var/lib/ekklesia/modules is not root-owned with mode 755."
|
||||
[ -z "$(ls -A /var/lib/ekklesia/modules)" ] || fail "/var/lib/ekklesia/modules is not empty."
|
||||
manifests="$(find / -xdev -name ekklesia.module.json -not -path "*/node_modules/*" 2>/dev/null)"
|
||||
[ -z "$manifests" ] || fail "a module manifest is in the image: ${manifests}"
|
||||
[ "$EKKLESIA_MODULES_DIR" = /var/lib/ekklesia/modules ] ||
|
||||
fail "a process in the image does not see EKKLESIA_MODULES_DIR."
|
||||
' || fail "${image} is not the generic platform image."
|
||||
echo "ok ${image}: no modules, no template, empty root-owned module storage, EKKLESIA_MODULES_DIR set"
|
||||
}
|
||||
|
||||
# runtime_listing <image>: every path the platform and its module storage consist of, with its
|
||||
# type, mode, and size, in a stable order. Two builds that differ here differ in content.
|
||||
runtime_listing() {
|
||||
docker run --rm --network none --user 0:0 --entrypoint sh "$1" -ec \
|
||||
'find /opt/ekklesia /var/lib/ekklesia -xdev -printf "%y %m %s %p\n" | LC_ALL=C sort'
|
||||
}
|
||||
|
||||
if [[ -n "${EKKLESIA_CHECK_IMAGE:-}" ]]; then
|
||||
check_contents "$EKKLESIA_CHECK_IMAGE"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
command -v git >/dev/null || fail "git is required to copy the build context."
|
||||
|
||||
# context_copy <directory>: the files Git would track, and nothing it ignores.
|
||||
context_copy() {
|
||||
mkdir -p "$1"
|
||||
git ls-files -z --cached --others --exclude-standard |
|
||||
tar --null --files-from=- --create --file=- |
|
||||
tar -C "$1" -xf -
|
||||
}
|
||||
|
||||
tag_base="ekklesia-ci:${GITHUB_SHA:-local}-${GITHUB_RUN_ID:-0}-${GITHUB_RUN_ATTEMPT:-1}-"
|
||||
generic_image="${tag_base}generic-$$"
|
||||
populated_image="${tag_base}populated-$$"
|
||||
|
||||
# build_image <context> <tag>
|
||||
build_image() {
|
||||
built_images+=("$2")
|
||||
DOCKER_BUILDKIT=1 docker build --quiet --file "$1/deploy/Dockerfile" \
|
||||
--build-arg REVISION="${GITHUB_SHA:-}" \
|
||||
--label "${EKKLESIA_CI_LABEL}=${project}" --label "${EKKLESIA_CI_CREATED_LABEL}=$(date +%s)" \
|
||||
--tag "$2" "$1" >/dev/null || fail "the image did not build from ${1##*/}."
|
||||
}
|
||||
|
||||
step "Building the platform image from this checkout"
|
||||
context_copy "$work/generic"
|
||||
[[ ! -e "$work/generic/modules" ]] || fail "this checkout itself has a modules/ directory."
|
||||
build_image "$work/generic" "$generic_image"
|
||||
check_contents "$generic_image"
|
||||
|
||||
step "Building again from the same checkout with a populated modules/ directory"
|
||||
context_copy "$work/populated"
|
||||
mkdir -p "$work/populated/modules"
|
||||
cp -R "$work/populated/templates/module" "$work/populated/modules/example"
|
||||
printf 'must never reach an image\n' >"$work/populated/modules/example/canary.txt"
|
||||
build_image "$work/populated" "$populated_image"
|
||||
check_contents "$populated_image"
|
||||
|
||||
step "Comparing the two images' runtime file systems"
|
||||
runtime_listing "$generic_image" >"$work/generic.listing"
|
||||
runtime_listing "$populated_image" >"$work/populated.listing"
|
||||
[[ -s "$work/generic.listing" ]] || fail "could not list the image's file system."
|
||||
if ! diff -u "$work/generic.listing" "$work/populated.listing" >"$work/listing.diff"; then
|
||||
head -40 "$work/listing.diff" >&2
|
||||
fail "a populated modules/ directory changed the image."
|
||||
fi
|
||||
echo "ok both builds have the identical runtime file system ($(wc -l <"$work/generic.listing") paths)"
|
||||
if [[ "$(docker image inspect --format '{{json .RootFS.Layers}}' "$generic_image")" == \
|
||||
"$(docker image inspect --format '{{json .RootFS.Layers}}' "$populated_image")" ]]; then
|
||||
echo "ok both builds share every layer"
|
||||
else
|
||||
echo "note the two builds' layers differ (a cold build cache); their file systems do not"
|
||||
fi
|
||||
|
||||
echo "Generic platform image check passed."
|
||||
@@ -6,6 +6,9 @@
|
||||
// node deploy-smoke-client.mjs edge an ordinary, untrusted peer of the proxy
|
||||
// node deploy-smoke-client.mjs tunnel the Cloudflare Tunnel connector's fixed address
|
||||
// node deploy-smoke-client.mjs session the initial owner signing in and out through the proxy
|
||||
// node deploy-smoke-client.mjs module the test module `example` as the initial owner sees it:
|
||||
// SMOKE_MODULE_EXPECT is "absent", or a JSON array of the
|
||||
// exact values its records route must return (ADR 0048)
|
||||
//
|
||||
// Either topology (ADR 0045): SMOKE_HOSTNAME is the API's hostname, and SMOKE_DASHBOARD_HOSTNAME,
|
||||
// when set, the Dashboard's. Unset means the same-origin topology, where both are one hostname. Every
|
||||
@@ -443,14 +446,72 @@ async function sessionChecks() {
|
||||
);
|
||||
}
|
||||
|
||||
/** Signs the initial owner in, through the proxy, and returns the session cookie. */
|
||||
async function ownerCookie() {
|
||||
const signIn = await fetch(`${apiOrigin}/api/auth/sign-in/email`, {
|
||||
body: JSON.stringify({
|
||||
email: required("SMOKE_OWNER_EMAIL"),
|
||||
password: required("SMOKE_OWNER_PASSWORD"),
|
||||
}),
|
||||
headers: { "content-type": "application/json", origin: dashboardOrigin },
|
||||
method: "POST",
|
||||
});
|
||||
await signIn.arrayBuffer();
|
||||
check(signIn.status === 200, "the initial owner signs in");
|
||||
return (signIn.headers.get("set-cookie") ?? "").split(";")[0] ?? "";
|
||||
}
|
||||
|
||||
/**
|
||||
* The installed-module checks (ADR 0048): whether `example` is in the module inventory, and what
|
||||
* its own route returns. Absent, it is in no inventory and its routes are not mounted at all;
|
||||
* present, its records route returns exactly the expected values, which proves its tables, its
|
||||
* dependencies, and the platform's database facade all work from module storage.
|
||||
*/
|
||||
async function moduleChecks() {
|
||||
const expectation = required("SMOKE_MODULE_EXPECT");
|
||||
const cookie = await ownerCookie();
|
||||
const headers = { cookie, origin: dashboardOrigin };
|
||||
|
||||
const inventory = await fetch(`${apiOrigin}/api/v1/module-ui`, { headers });
|
||||
const inventoryBody = await inventory.json();
|
||||
const installed =
|
||||
inventory.status === 200 && inventoryBody.modules.some((module) => module.id === "example");
|
||||
const records = await fetch(`${apiOrigin}/api/v1/modules/example/records`, { headers });
|
||||
|
||||
if (expectation === "absent") {
|
||||
check(inventory.status === 200 && !installed, "the module inventory has no `example` module");
|
||||
check(
|
||||
records.status === 404 && (await records.text()) === '{"code":"NOT_FOUND"}',
|
||||
"the absent module's routes are not mounted",
|
||||
);
|
||||
} else {
|
||||
const expected = JSON.parse(expectation);
|
||||
check(installed, "the module inventory lists the `example` module");
|
||||
const values = records.status === 200 ? (await records.json()).map((row) => row.value) : null;
|
||||
check(
|
||||
JSON.stringify(values) === JSON.stringify(expected),
|
||||
`the module's records route returns ${JSON.stringify(expected)}`,
|
||||
);
|
||||
}
|
||||
|
||||
const signOut = await fetch(`${apiOrigin}/api/auth/sign-out`, {
|
||||
body: "{}",
|
||||
headers: { ...headers, "content-type": "application/json" },
|
||||
method: "POST",
|
||||
});
|
||||
await signOut.arrayBuffer();
|
||||
}
|
||||
|
||||
if (mode === "edge") {
|
||||
await edgeChecks();
|
||||
} else if (mode === "tunnel") {
|
||||
await tunnelChecks();
|
||||
} else if (mode === "session") {
|
||||
await sessionChecks();
|
||||
} else if (mode === "module") {
|
||||
await moduleChecks();
|
||||
} else {
|
||||
throw new Error('Mode must be "edge", "tunnel", or "session".');
|
||||
throw new Error('Mode must be "edge", "tunnel", "session", or "module".');
|
||||
}
|
||||
|
||||
if (failures > 0) {
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
# TEST-ONLY. Builds the repository's module template as one self-contained installed-module directory
|
||||
# for scripts/smoke-deploy.sh, which installs it into a deployment's module storage (ADR 0048). It is
|
||||
# not a module package format and defines none: the archive format, signing, and transport of real
|
||||
# module packages are deferred. No image is produced; the build's only output is the directory:
|
||||
#
|
||||
# DOCKER_BUILDKIT=1 docker build --file scripts/lib/test-module-package.Dockerfile \
|
||||
# --output type=local,dest=<directory> .
|
||||
#
|
||||
# writes <directory>/example: the module's package.json, manifest, built artifacts, migrations, and a
|
||||
# node_modules holding exactly its production dependencies, including its own copies of the Module
|
||||
# SDK, the contracts, and Drizzle. Nothing in it resolves to anything outside it, because the
|
||||
# platform lends an installed module nothing.
|
||||
#
|
||||
# The same pinned Node.js base as deploy/Dockerfile, so the package targets the runtime it is
|
||||
# installed into. Drizzle comes from the platform's own lockfile, which is what keeps it the exact
|
||||
# version the platform uses: a constraint of the current implementation (ADR 0048), not a contract.
|
||||
FROM node:24.18.0-trixie-slim@sha256:ae91dcc111a68c9d2d81ff2a17bda61be126426176fde6fe7d08ab13b7f50573 AS build
|
||||
ENV COREPACK_ENABLE_DOWNLOAD_PROMPT=0 \
|
||||
PNPM_HOME=/pnpm \
|
||||
TURBO_TELEMETRY_DISABLED=1 \
|
||||
DO_NOT_TRACK=1
|
||||
WORKDIR /opt/source
|
||||
COPY package.json ./
|
||||
RUN corepack enable && corepack install
|
||||
COPY . .
|
||||
RUN --mount=type=cache,id=ekklesia-pnpm-store,target=/pnpm/store \
|
||||
pnpm install --frozen-lockfile --store-dir /pnpm/store
|
||||
RUN pnpm exec turbo run build --filter=@ekklesia/module-template
|
||||
# `--legacy` deploys from this workspace without switching it to injected workspace packages; the
|
||||
# Module SDK and every other workspace dependency are still copied into the package, built.
|
||||
RUN --mount=type=cache,id=ekklesia-pnpm-store,target=/pnpm/store \
|
||||
set -eu; \
|
||||
pnpm --filter @ekklesia/module-template deploy --prod --legacy --store-dir /pnpm/store \
|
||||
/package/example; \
|
||||
rm -f /package/example/README.md; \
|
||||
# pnpm's hoisted self-reference points back into this workspace; an installed module has none.
|
||||
rm -f /package/example/node_modules/.pnpm/node_modules/@ekklesia/module-template; \
|
||||
escaping=""; \
|
||||
for link in $(find /package/example -type l); do \
|
||||
case "$(realpath -m "$link")" in \
|
||||
/package/example/*) ;; \
|
||||
*) escaping="$escaping $link" ;; \
|
||||
esac; \
|
||||
done; \
|
||||
if [ -n "$escaping" ]; then echo "Links escape the package:$escaping" >&2; exit 1; fi; \
|
||||
chmod -R a+rX,go-w /package/example; \
|
||||
cd /package/example && node -e 'import("./dist/server/index.js").then((m) => { if (m.default.id !== "example") process.exit(1); })'
|
||||
|
||||
FROM scratch AS package
|
||||
COPY --from=build /package/ /
|
||||
+134
-38
@@ -1,19 +1,30 @@
|
||||
#!/usr/bin/env bash
|
||||
# End-to-end smoke test of the reference deployment (ADR 0044): deploy/compose.yaml with
|
||||
# deploy/compose.proxy.yaml, a platform-only image, and a disposable PostgreSQL and SeaweedFS that
|
||||
# exist only for this run. CI runs it; it is safe to run on a workstation.
|
||||
# End-to-end smoke test of the reference deployment (ADR 0044, ADR 0048): deploy/compose.yaml with
|
||||
# deploy/compose.proxy.yaml, the one generic platform image, installation-owned module storage, and
|
||||
# a disposable PostgreSQL and SeaweedFS that exist only for this run. CI runs it; it is safe to run
|
||||
# on a workstation.
|
||||
#
|
||||
# bash scripts/smoke-deploy.sh builds the image first
|
||||
# EKKLESIA_SMOKE_IMAGE=<image> bash scripts/smoke-deploy.sh
|
||||
# EKKLESIA_SMOKE_TOPOLOGY=split bash scripts/smoke-deploy.sh
|
||||
#
|
||||
# Either topology (ADR 0045). `same-origin` (the default) serves the Dashboard, the API, and displays
|
||||
# from one hostname with Caddyfile.example and a platform-only image. `split` serves the Dashboard on
|
||||
# its own hostname with Caddyfile.split.example, builds an image with the repository's module
|
||||
# template installed as a test-only module (so a module UI, its API routes, and a display exist), and
|
||||
# then drives a real headless Chromium through the Dashboard, the module, the display, and sign-out.
|
||||
# With EKKLESIA_SMOKE_IMAGE, the given image is used as is in either topology; a split run then
|
||||
# exercises whichever of its modules publishes UI and a display.
|
||||
# Either topology (ADR 0045), and in both the same generic image, built once from deploy/Dockerfile.
|
||||
# It contains no module; module storage starts empty, as on a fresh installation. `same-origin` (the
|
||||
# default) serves the Dashboard, the API, and displays from one hostname with Caddyfile.example.
|
||||
# `split` serves the Dashboard on its own hostname with Caddyfile.split.example and then exercises
|
||||
# ADR 0048 end to end with that same image:
|
||||
#
|
||||
# - it boots with no modules;
|
||||
# - the repository's module template, packaged as one self-contained directory
|
||||
# (scripts/lib/test-module-package.Dockerfile, TEST-ONLY), is installed into module storage, its
|
||||
# migrations are planned and explicitly applied, and the API is restarted;
|
||||
# - a real headless Chromium drives the Dashboard, the module, its display, and sign-out;
|
||||
# - the module is removed and the API restarted: Core still runs, migration planning reports the
|
||||
# module's retained history, and its data survives;
|
||||
# - the module is restored and the API restarted: it returns with its previous data.
|
||||
#
|
||||
# EKKLESIA_SMOKE_IMAGE uses the given image instead of building one. It must be a generic image
|
||||
# (ADR 0048): modules come from module storage, never from the image.
|
||||
#
|
||||
# Opt-in, split topology only: EKKLESIA_SMOKE_DISPLAY_DEVICE=1 adds the display-device scenarios in
|
||||
# scripts/lib/deploy-smoke-display.mjs (pairing a kiosk browser with a persistent profile, a clean
|
||||
@@ -113,6 +124,8 @@ choose_edge_network() {
|
||||
session_client_address="10.249.${octet}.252"
|
||||
# The headless browser's fixed address, likewise (split topology only).
|
||||
browser_address="10.249.${octet}.251"
|
||||
# The installed-module checks' own client address, so their sign-ins have their own bucket.
|
||||
module_client_address="10.249.${octet}.249"
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
@@ -166,7 +179,7 @@ cleanup() {
|
||||
# job itself runs as root), so only the staged copy on the Docker host needs this.
|
||||
if [[ "$daemon_work" == "$work" ]]; then
|
||||
docker run --rm --entrypoint rm -v "$work:/work" "$postgres_image" -rf /work/tls /work/env \
|
||||
>/dev/null 2>&1 || true
|
||||
/work/modules >/dev/null 2>&1 || true
|
||||
else
|
||||
docker run --rm --entrypoint rm -v /tmp:/host-tmp "$postgres_image" -rf "/host-tmp/${project}" \
|
||||
>/dev/null 2>&1 || true
|
||||
@@ -188,41 +201,38 @@ fail() {
|
||||
exit 1
|
||||
}
|
||||
|
||||
if [[ -z "$image" && "$topology" == same-origin ]]; then
|
||||
step "Building the platform-only image"
|
||||
if [[ -z "$image" ]]; then
|
||||
# The one generic platform image, for both topologies. Nothing about either topology, and no
|
||||
# module, is built into it.
|
||||
step "Building the generic platform image"
|
||||
built_image="ekklesia-deploy-smoke:${run_id}"
|
||||
DOCKER_BUILDKIT=1 docker build --quiet --file deploy/Dockerfile --tag "$built_image" \
|
||||
--label "${EKKLESIA_CI_LABEL}=${project}" --label "${EKKLESIA_CI_CREATED_LABEL}=$(date +%s)" \
|
||||
. >/dev/null
|
||||
image="$built_image"
|
||||
elif [[ -z "$image" ]]; then
|
||||
# TEST-ONLY: the repository's module template, installed as the module `example`, in a copy of the
|
||||
# working tree. Its lockfile importer moves with it; the depth is the same, so every relative link
|
||||
# in that importer stays correct and the install stays frozen.
|
||||
step "Building an image with the module template installed as a test-only module"
|
||||
command -v git >/dev/null || fail "git is required to build the split topology's test image."
|
||||
context="$work/build-context"
|
||||
mkdir -p "$context"
|
||||
git ls-files -z --cached --others --exclude-standard |
|
||||
tar --null --files-from=- --create --file=- |
|
||||
tar -C "$context" -xf -
|
||||
mkdir -p "$context/modules"
|
||||
mv "$context/templates/module" "$context/modules/example"
|
||||
[[ "$(grep -c '^ templates/module:$' "$context/pnpm-lock.yaml")" == 1 ]] ||
|
||||
fail "the lockfile has no single templates/module importer to move."
|
||||
sed -i 's|^ templates/module:$| modules/example:|' "$context/pnpm-lock.yaml"
|
||||
built_image="ekklesia-deploy-smoke:${run_id}"
|
||||
DOCKER_BUILDKIT=1 docker build --quiet --file "$context/deploy/Dockerfile" --tag "$built_image" \
|
||||
--label "${EKKLESIA_CI_LABEL}=${project}" --label "${EKKLESIA_CI_CREATED_LABEL}=$(date +%s)" \
|
||||
"$context" >/dev/null || fail "the test-module image did not build."
|
||||
rm -rf "$context"
|
||||
image="$built_image"
|
||||
fi
|
||||
|
||||
if [[ "$topology" == split ]]; then
|
||||
# TEST-ONLY: the module template as one self-contained installed-module directory, built outside
|
||||
# the platform image and installed into module storage below. Never a second platform image.
|
||||
step "Packaging the module template as a self-contained test module"
|
||||
DOCKER_BUILDKIT=1 docker build --quiet --file scripts/lib/test-module-package.Dockerfile \
|
||||
--label "${EKKLESIA_CI_LABEL}=${project}" \
|
||||
--output "type=local,dest=${work}/module-package" . >/dev/null ||
|
||||
fail "the test module package did not build."
|
||||
[[ -f "$work/module-package/example/ekklesia.module.json" ]] ||
|
||||
fail "the test module package has no manifest."
|
||||
fi
|
||||
|
||||
# The compose project directory: its relative bind mount (postgres/pg_hba.conf) resolves here.
|
||||
cp -R deploy "$work/deploy"
|
||||
stage deploy
|
||||
|
||||
# Installation-owned module storage (ADR 0048), empty, as an operator creates it before the first
|
||||
# `up`: Compose refuses to start `api` or `migrate` without it, and never creates it.
|
||||
mkdir -m 755 "$work/modules"
|
||||
stage modules
|
||||
|
||||
step "Creating the installation-private CA and server certificates"
|
||||
init_status=0
|
||||
sh deploy/scripts/init-tls.sh "$work/tls" >/dev/null 2>&1 || init_status=$?
|
||||
@@ -309,6 +319,7 @@ EKKLESIA_EDGE_SUBNET=${edge_subnet}
|
||||
EKKLESIA_EDGE_DYNAMIC_RANGE=${edge_range}
|
||||
EKKLESIA_PROXY_ADDRESS=${proxy_address}
|
||||
EKKLESIA_TUNNEL_ADDRESS=${tunnel_address}
|
||||
EKKLESIA_MODULES_HOST_DIR=${daemon_work}/modules
|
||||
EOF
|
||||
|
||||
step "Starting PostgreSQL and object storage"
|
||||
@@ -398,7 +409,70 @@ client session --ip "$session_client_address" \
|
||||
-e SMOKE_OWNER_EMAIL="$owner_email" -e SMOKE_OWNER_PASSWORD="$owner_password" ||
|
||||
fail "session checks failed."
|
||||
|
||||
psql_as_superuser() {
|
||||
"${compose[@]}" exec -T postgres psql -U ekklesia -d ekklesia -Atc "$1"
|
||||
}
|
||||
|
||||
# module_storage <script>: runs a root shell over this run's module storage, as an operator changing
|
||||
# installed modules on the host would. The API never writes it; it is mounted read-only.
|
||||
module_storage() {
|
||||
docker run --rm -i --network none --entrypoint sh -v "$daemon_work/modules:/modules" \
|
||||
"$postgres_image" -ec "$1"
|
||||
}
|
||||
|
||||
# install_test_module: stages the package under a hidden name, which discovery ignores, then renames
|
||||
# it into place in one step, owned by root and readable by the API's unprivileged user.
|
||||
install_test_module() {
|
||||
tar -C "$work/module-package" -cf - example |
|
||||
module_storage 'rm -rf /modules/.incoming && mkdir /modules/.incoming &&
|
||||
tar -xf - -C /modules/.incoming && chown -R 0:0 /modules/.incoming/example &&
|
||||
chmod -R a+rX,go-w /modules/.incoming/example &&
|
||||
mv /modules/.incoming/example /modules/example && rmdir /modules/.incoming' ||
|
||||
fail "the test module could not be installed into module storage."
|
||||
}
|
||||
|
||||
# restart_api <command...>: stops the API, runs the command, and starts the API again. A controlled
|
||||
# restart is the only way a module change is picked up; nothing is ever hot loaded.
|
||||
restart_api() {
|
||||
"${compose[@]}" stop api >/dev/null 2>&1 || fail "the API did not stop."
|
||||
"$@"
|
||||
if ! up_output="$("${compose[@]}" up --detach --wait --wait-timeout 180 api 2>&1)"; then
|
||||
compose_report_not_ready "$up_output" "${compose[@]}"
|
||||
fail "the API did not become healthy after a restart."
|
||||
fi
|
||||
}
|
||||
|
||||
# module_check <expectation>: "absent", or the exact JSON array of values `example` must return.
|
||||
module_check() {
|
||||
client module --ip "$module_client_address" -e SMOKE_MODULE_EXPECT="$1" \
|
||||
-e SMOKE_OWNER_EMAIL="$owner_email" -e SMOKE_OWNER_PASSWORD="$owner_password"
|
||||
}
|
||||
|
||||
# plan_output: the read-only migration plan, as `docker compose run --rm migrate` prints it.
|
||||
plan_output() {
|
||||
"${compose[@]}" run --rm --no-deps migrate 2>&1 || fail "migration planning failed."
|
||||
}
|
||||
|
||||
if [[ "$topology" == split ]]; then
|
||||
step "Booting the generic image with empty module storage"
|
||||
module_check absent || fail "the image without modules did not run module-free."
|
||||
|
||||
step "Installing the test module into module storage, migrating explicitly, and restarting the API"
|
||||
install_module_and_migrate() {
|
||||
install_test_module
|
||||
local plan
|
||||
plan="$(plan_output)"
|
||||
grep -q '^- \[pending\] example:0000_initial ' <<<"$plan" ||
|
||||
fail "the plan does not list the installed module's migration as pending."
|
||||
"${compose[@]}" run --rm --no-deps migrate node core/dist/cli/apply-migrations.js >/dev/null ||
|
||||
fail "applying the installed module's migration failed."
|
||||
}
|
||||
restart_api install_module_and_migrate
|
||||
module_check '[]' || fail "the installed module did not run from module storage."
|
||||
psql_as_superuser "INSERT INTO example_records (id, value) VALUES (gen_random_uuid(), 'kept')" \
|
||||
>/dev/null || fail "could not write the module's own data."
|
||||
module_check '["Hello, kept"]' || fail "the installed module did not read its own data."
|
||||
|
||||
step "Driving a real browser through the split Dashboard, a module, its display, and sign-out"
|
||||
# Chromium trusts this run's proxy through the public-key hash of Caddy's internal intermediate CA,
|
||||
# which signs every certificate the test proxy serves: no CA is installed anywhere, and nothing but
|
||||
@@ -501,6 +575,31 @@ if [[ "$topology" == split ]]; then
|
||||
while read -r ts uri; do echo " $(date -u -d "@${ts}" +%T) ${uri}"; done || true
|
||||
((display_status == 0)) || fail "display-device checks failed."
|
||||
fi
|
||||
|
||||
step "Removing the module from module storage and restarting: Core runs, its data stays"
|
||||
restart_api module_storage 'rm -rf /modules/example'
|
||||
module_check absent || fail "the removed module is still running."
|
||||
plan="$(plan_output)"
|
||||
grep -q '^- \[retained\] example: 1 applied migration kept; the module is not installed$' \
|
||||
<<<"$plan" || fail "migration planning did not report the removed module's retained history."
|
||||
grep -q '^Migration plan (.*0 pending, 0 approval-blocked, 1 retained):$' <<<"$plan" ||
|
||||
fail "migration planning did not succeed cleanly with the module absent."
|
||||
"${compose[@]}" run --rm --no-deps migrate node core/dist/cli/apply-migrations.js >/dev/null ||
|
||||
fail "applying migrations failed with the module absent."
|
||||
[[ "$(psql_as_superuser "SELECT value FROM example_records")" == "kept" ]] ||
|
||||
fail "the removed module's data did not survive."
|
||||
echo "ok the module is gone, Core runs, its history is retained, and its data survives"
|
||||
|
||||
step "Restoring the module and restarting: it returns with its previous data"
|
||||
restart_api install_test_module
|
||||
plan="$(plan_output)"
|
||||
grep -q '^- \[applied\] example:0000_initial ' <<<"$plan" ||
|
||||
fail "the restored module's migration history is not valid."
|
||||
if grep -q '\[retained\]\|\[pending\]' <<<"$plan"; then
|
||||
fail "the restored module left retained or pending migrations."
|
||||
fi
|
||||
module_check '["Hello, kept"]' || fail "the restored module did not return with its data."
|
||||
echo "ok the restored module runs again, with its history strict and its data intact"
|
||||
fi
|
||||
|
||||
step "Proving the API's database connection is certificate-verified TLS"
|
||||
@@ -510,9 +609,6 @@ api_data_address="$(
|
||||
"$api_container"
|
||||
)"
|
||||
[[ -n "$api_data_address" ]] || fail "could not find the API's database-network address."
|
||||
psql_as_superuser() {
|
||||
"${compose[@]}" exec -T postgres psql -U ekklesia -d ekklesia -Atc "$1"
|
||||
}
|
||||
tls_connections="$(psql_as_superuser "SELECT count(*) FROM pg_stat_ssl s JOIN pg_stat_activity a USING (pid) WHERE a.client_addr = '${api_data_address}' AND s.ssl AND s.version = 'TLSv1.3'")"
|
||||
plain_connections="$(psql_as_superuser "SELECT count(*) FROM pg_stat_ssl s JOIN pg_stat_activity a USING (pid) WHERE a.client_addr = '${api_data_address}' AND NOT s.ssl")"
|
||||
((tls_connections > 0)) || fail "the API holds no TLS 1.3 database connection."
|
||||
|
||||
@@ -53,6 +53,13 @@ To create a module inside this repository:
|
||||
root. The declared production entrypoint is the compiled `dist/server/index.js`, not source
|
||||
TypeScript; the schema entrypoint is likewise compiled to `dist/schema/index.js`.
|
||||
|
||||
That `modules/<module-id>` directory is where a module is **developed**. An installation never runs
|
||||
it from there: Ekklesia's image contains no modules, and an installation installs a self-contained
|
||||
copy of the module, with its own production dependencies, into its module storage
|
||||
([An installed module is self-contained](../../docs/module-development/README.md#an-installed-module-is-self-contained),
|
||||
ADR 0048). Copying `dist/` alone is not enough, because the compiled server code imports the Module
|
||||
SDK and Drizzle by name.
|
||||
|
||||
The module owns its feature screens and display renderer. Do not add platform navigation, the
|
||||
containing shell, global theme, authentication screens, or top-level routing; those belong to
|
||||
surfaces and Core. A module UI asks its host for relative API paths and realtime state, and never
|
||||
|
||||
@@ -97,7 +97,7 @@ describe("CI scripts on a shared Docker daemon", () => {
|
||||
>;
|
||||
};
|
||||
const startsStacks =
|
||||
/smoke-infrastructure\.sh|test:migrations:integration|smoke-deploy\.sh|docker build/u;
|
||||
/smoke-infrastructure\.sh|test:migrations:integration|smoke-deploy\.sh|check-image\.sh|docker build/u;
|
||||
const jobs = Object.entries(workflow.jobs).filter(([, job]) =>
|
||||
job.steps.some((step) => startsStacks.test(step.run ?? "")),
|
||||
);
|
||||
@@ -132,12 +132,37 @@ describe("CI scripts on a shared Docker daemon", () => {
|
||||
|
||||
it("tag the CI image per run and attempt, so concurrent runs of one commit cannot remove it", async () => {
|
||||
const workflow = await source(".github/workflows/ci.yml");
|
||||
expect(workflow).toContain(
|
||||
'ekklesia-ci:"$GITHUB_SHA-${GITHUB_RUN_ID:-0}-${GITHUB_RUN_ATTEMPT:-1}"',
|
||||
const imageCheck = await source("scripts/check-image.sh");
|
||||
expect(workflow).toContain("bash scripts/check-image.sh");
|
||||
expect(imageCheck).toContain(
|
||||
'tag_base="ekklesia-ci:${GITHUB_SHA:-local}-${GITHUB_RUN_ID:-0}-${GITHUB_RUN_ATTEMPT:-1}-"',
|
||||
);
|
||||
expect(workflow).not.toContain('ekklesia-ci:"$GITHUB_SHA"');
|
||||
});
|
||||
|
||||
it("label every image the image check builds, so a sweep can find it", async () => {
|
||||
const imageCheck = (await source("scripts/check-image.sh")).replace(/\\\n\s*/gu, " ");
|
||||
expect(imageCheck).toContain('source "$repository_root/scripts/lib/ci-project.sh"');
|
||||
expect(imageCheck).toMatch(/^project="\$\(ci_project_name image\)"$/mu);
|
||||
const builds = imageCheck.split("\n").filter((line) => /\bdocker build\b/u.test(line));
|
||||
expect(builds.length).toBeGreaterThanOrEqual(1);
|
||||
for (const line of builds) {
|
||||
expect(line).toContain('--label "${EKKLESIA_CI_LABEL}=${project}"');
|
||||
}
|
||||
});
|
||||
|
||||
it("run every deploy smoke against the one generic image, never a second image with a module in it", async () => {
|
||||
// ADR 0048: the split topology installs its test module into module storage at run time.
|
||||
const deploy = (await source("scripts/smoke-deploy.sh")).replace(/\\\n\s*/gu, " ");
|
||||
const imageBuilds = deploy
|
||||
.split("\n")
|
||||
.filter((line) => /\bdocker build\b/u.test(line) && line.includes("deploy/Dockerfile"));
|
||||
expect(imageBuilds).toHaveLength(1);
|
||||
expect(deploy).not.toContain('mv "$context/templates/module" "$context/modules/');
|
||||
expect(deploy).not.toMatch(/sed -i 's\|\^ {2}templates\/module:/u);
|
||||
expect(deploy).toContain("EKKLESIA_MODULES_HOST_DIR=");
|
||||
});
|
||||
|
||||
it("report every readiness failure with the project's containers and logs, through the redactor only", async () => {
|
||||
for (const relativePath of ciScripts) {
|
||||
const text = await source(relativePath);
|
||||
|
||||
@@ -26,6 +26,15 @@ interface ComposeService {
|
||||
read_only?: boolean;
|
||||
security_opt?: string[];
|
||||
tmpfs?: string[];
|
||||
volumes?: (string | ComposeVolume)[];
|
||||
}
|
||||
|
||||
interface ComposeVolume {
|
||||
bind?: { create_host_path?: boolean };
|
||||
read_only?: boolean;
|
||||
source?: string;
|
||||
target?: string;
|
||||
type?: string;
|
||||
}
|
||||
|
||||
interface ComposeDocument {
|
||||
@@ -234,3 +243,68 @@ describe("reference deployment env examples", () => {
|
||||
expect(example).toContain("EKKLESIA_MODULE_SECRET__<MODULE_ID>__<KEY>=");
|
||||
});
|
||||
});
|
||||
|
||||
describe("installation-owned module storage (ADR 0048)", () => {
|
||||
const modulesTarget = "/var/lib/ekklesia/modules";
|
||||
|
||||
function moduleMount(service: ComposeService | undefined): ComposeVolume | undefined {
|
||||
return (service?.volumes ?? []).find(
|
||||
(volume): volume is ComposeVolume =>
|
||||
typeof volume === "object" && volume.target === modulesTarget,
|
||||
);
|
||||
}
|
||||
|
||||
it.each([
|
||||
"api",
|
||||
"migrate",
|
||||
])("mounts host module storage read-only into %s, and never lets Compose create it", async (name) => {
|
||||
const service = (await compose("compose.yaml")).services?.[name];
|
||||
expect(moduleMount(service)).toEqual({
|
||||
bind: { create_host_path: false },
|
||||
read_only: true,
|
||||
source: "${EKKLESIA_MODULES_HOST_DIR:-/var/lib/ekklesia/modules}",
|
||||
target: modulesTarget,
|
||||
type: "bind",
|
||||
});
|
||||
// The process reads exactly the directory the mount provides.
|
||||
expect(service?.environment?.EKKLESIA_MODULES_DIR).toBe(modulesTarget);
|
||||
// Read-only everywhere else too: the mount is the only module storage there is.
|
||||
expect(service?.read_only).toBe(true);
|
||||
});
|
||||
|
||||
it("mounts module storage into nothing but the API and the operator commands", async () => {
|
||||
const services = (await compose("compose.yaml")).services ?? {};
|
||||
for (const [name, service] of Object.entries(services)) {
|
||||
if (name !== "api" && name !== "migrate") {
|
||||
expect(moduleMount(service), name).toBeUndefined();
|
||||
}
|
||||
}
|
||||
for (const overlay of ["compose.proxy.yaml", "compose.tunnel.yaml"]) {
|
||||
for (const [name, service] of Object.entries((await compose(overlay)).services ?? {})) {
|
||||
expect(moduleMount(service), `${overlay} ${name}`).toBeUndefined();
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
it("names the host-side directory with its own variable, never EKKLESIA_MODULES_DIR", async () => {
|
||||
const composeEnvironment = await text("env/compose.env.example");
|
||||
expect(composeEnvironment).toMatch(
|
||||
/^# ?EKKLESIA_MODULES_HOST_DIR=\/var\/lib\/ekklesia\/modules$/mu,
|
||||
);
|
||||
expect(composeEnvironment).not.toMatch(/^#? ?EKKLESIA_MODULES_DIR=/mu);
|
||||
// The API's own env file must not move the directory away from the mount.
|
||||
expect(envAssignments(await text("env/ekklesia.env.example")).has("EKKLESIA_MODULES_DIR")).toBe(
|
||||
false,
|
||||
);
|
||||
});
|
||||
|
||||
it("builds an image that holds no modules and points at empty module storage", async () => {
|
||||
const dockerfile = await text("Dockerfile");
|
||||
const ignore = await readFile(path.join(repositoryRoot, ".dockerignore"), "utf8");
|
||||
expect(ignore).toMatch(/^\/modules$/mu);
|
||||
expect(dockerfile).toMatch(/EKKLESIA_MODULES_DIR=\/var\/lib\/ekklesia\/modules/u);
|
||||
expect(dockerfile).toContain("install -d -o root -g root -m 0755 /var/lib/ekklesia/modules");
|
||||
// Nothing from a module or the module template is copied into, or installed for, the runtime.
|
||||
expect(dockerfile).not.toMatch(/\/runtime\/modules|modules\/\*\/package\.json|templates\/\*/u);
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user