feat: Ekklesia Module Package v1, the canonical packager and verifier (ADR 0049)
CI / verify (pull_request) Successful in 57s
CI / infrastructure-smoke (pull_request) Successful in 35s
CI / migration-smoke (pull_request) Successful in 1m42s
CI / image-build (pull_request) Successful in 29s
CI / deploy-smoke (same-origin) (pull_request) Successful in 51s
CI / deploy-smoke (split) (pull_request) Failing after 54s
CI / verify (pull_request) Successful in 57s
CI / infrastructure-smoke (pull_request) Successful in 35s
CI / migration-smoke (pull_request) Successful in 1m42s
CI / image-build (pull_request) Successful in 29s
CI / deploy-smoke (same-origin) (pull_request) Successful in 51s
CI / deploy-smoke (split) (pull_request) Failing after 54s
A separately versioned module is now distributed as one deterministic .tar.gz and its SHA-256, produced by an Ekklesia-owned packager and checked by Core before anything is installed. This slice implements the portable `any` target, with the module template as the end-to-end proof. ADR 0049 is accepted. @ekklesia/contracts: the Module Package v1 contract. The strict ekklesia.package.json schema (packageFormat 1, identity, target, runtime, provenance, incorporated Ekklesia packages, strict transitional facts, an exact byte-wise-sorted file list with size and SHA-256), target names, and the generated runtime package.json. @ekklesia/module-pack (new, tooling): `ekklesia-pack-module --module <dir> --target any --out <dir> --source-revision <commit>`. - Bundles server, schema, and jobs with esbuild, SDK, contracts, and Drizzle included; only Node built-ins remain imports. Any warning, unresolved import, external dependency, or native or platform-specific package fails. - Copies browser artifacts as built, after proving them current, and the migrations; generates package.json and ekklesia.package.json; never ships src, tests, declarations, source maps, or workspace ranges. - THIRD_PARTY_NOTICES: every third-party package traced from the bundles' own inputs, credited once with its declared licence and every licence, copying, copyright, and notice file it ships. A missing licence fails; a missing text fails unless its standard text is complete without copyright lines (Apache-2.0, vendored, needed by drizzle-orm). - Deterministic ustar and gzip: sorted paths, uid/gid 0, empty names, mtime 0, 0755/0644, no links. Byte-identical across runs and machines. - Writes only the archive and its .sha256; never deploys, links, or installs, so the workspace's install state is untouched. Core: verifyModulePackage and `verify-module-package.js <archive> [--sha256 | --sha256-file] [--extract-to <new dir>]`. Reads the archive in memory and refuses links, special entries, setuid/setgid/sticky modes, unsafe or escaping paths, and a second top-level directory; checks metadata, every file, identity, target, Node.js range, SDK, and the transitional Drizzle agreement; runs the platform's module and migration discovery; loads the backend once in a forked process with an empty environment; only then extracts, with normalized modes, into a new directory. Errors are sanitized. @ekklesia/module-ui-build: analyzeModuleUi reports each artifact's inputs and whether the built file is current. The nested stylesheet build now works from the module directory, so artifacts no longer depend on the caller's working directory. The split deploy smoke builds the package with the canonical packager (scripts/lib/build-module-package.Dockerfile replaces the test-only pnpm deploy packaging), verifies it with the same generic image, installs the verified tree, removes it, and reinstalls the exact same archive. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
45d6506c10
commit
35ef31df9f
@@ -0,0 +1,31 @@
|
||||
# Builds the repository's module template and packages it with the canonical Ekklesia packager
|
||||
# (`@ekklesia/module-pack`, ADR 0049), in a clean container from a clean copy of the checkout. The
|
||||
# deploy smoke test uses it because a CI job that drives Docker has no Node.js of its own. The
|
||||
# packaging itself is exactly what a module's CI runs; this file only provides the environment.
|
||||
#
|
||||
# DOCKER_BUILDKIT=1 docker build --file scripts/lib/build-module-package.Dockerfile \
|
||||
# --build-arg SOURCE_REVISION=<full commit id> --output type=local,dest=<directory> .
|
||||
#
|
||||
# writes <directory>/example-<version>-any.tar.gz and its .sha256 line. No image is produced.
|
||||
#
|
||||
# The same pinned Node.js base as deploy/Dockerfile, so the package is built by the runtime it is
|
||||
# installed into.
|
||||
FROM node:24.18.0-trixie-slim@sha256:ae91dcc111a68c9d2d81ff2a17bda61be126426176fde6fe7d08ab13b7f50573 AS build
|
||||
ENV COREPACK_ENABLE_DOWNLOAD_PROMPT=0 \
|
||||
PNPM_HOME=/pnpm \
|
||||
TURBO_TELEMETRY_DISABLED=1 \
|
||||
DO_NOT_TRACK=1
|
||||
WORKDIR /opt/source
|
||||
COPY package.json ./
|
||||
RUN corepack enable && corepack install
|
||||
COPY . .
|
||||
RUN --mount=type=cache,id=ekklesia-pnpm-store,target=/pnpm/store \
|
||||
pnpm install --frozen-lockfile --store-dir /pnpm/store
|
||||
RUN pnpm exec turbo run build --filter=@ekklesia/module-template --filter=@ekklesia/module-pack
|
||||
ARG SOURCE_REVISION
|
||||
RUN test -n "$SOURCE_REVISION" && \
|
||||
node packages/module-pack/bin/ekklesia-pack-module.mjs --module templates/module --target any \
|
||||
--out /package --source-revision "$SOURCE_REVISION"
|
||||
|
||||
FROM scratch AS package
|
||||
COPY --from=build /package/ /
|
||||
@@ -1,50 +0,0 @@
|
||||
# TEST-ONLY. Builds the repository's module template as one self-contained installed-module directory
|
||||
# for scripts/smoke-deploy.sh, which installs it into a deployment's module storage (ADR 0048). It is
|
||||
# not a module package format and defines none: the archive format, signing, and transport of real
|
||||
# module packages are deferred. No image is produced; the build's only output is the directory:
|
||||
#
|
||||
# DOCKER_BUILDKIT=1 docker build --file scripts/lib/test-module-package.Dockerfile \
|
||||
# --output type=local,dest=<directory> .
|
||||
#
|
||||
# writes <directory>/example: the module's package.json, manifest, built artifacts, migrations, and a
|
||||
# node_modules holding exactly its production dependencies, including its own copies of the Module
|
||||
# SDK, the contracts, and Drizzle. Nothing in it resolves to anything outside it, because the
|
||||
# platform lends an installed module nothing.
|
||||
#
|
||||
# The same pinned Node.js base as deploy/Dockerfile, so the package targets the runtime it is
|
||||
# installed into. Drizzle comes from the platform's own lockfile, which is what keeps it the exact
|
||||
# version the platform uses: a constraint of the current implementation (ADR 0048), not a contract.
|
||||
FROM node:24.18.0-trixie-slim@sha256:ae91dcc111a68c9d2d81ff2a17bda61be126426176fde6fe7d08ab13b7f50573 AS build
|
||||
ENV COREPACK_ENABLE_DOWNLOAD_PROMPT=0 \
|
||||
PNPM_HOME=/pnpm \
|
||||
TURBO_TELEMETRY_DISABLED=1 \
|
||||
DO_NOT_TRACK=1
|
||||
WORKDIR /opt/source
|
||||
COPY package.json ./
|
||||
RUN corepack enable && corepack install
|
||||
COPY . .
|
||||
RUN --mount=type=cache,id=ekklesia-pnpm-store,target=/pnpm/store \
|
||||
pnpm install --frozen-lockfile --store-dir /pnpm/store
|
||||
RUN pnpm exec turbo run build --filter=@ekklesia/module-template
|
||||
# `--legacy` deploys from this workspace without switching it to injected workspace packages; the
|
||||
# Module SDK and every other workspace dependency are still copied into the package, built.
|
||||
RUN --mount=type=cache,id=ekklesia-pnpm-store,target=/pnpm/store \
|
||||
set -eu; \
|
||||
pnpm --filter @ekklesia/module-template deploy --prod --legacy --store-dir /pnpm/store \
|
||||
/package/example; \
|
||||
rm -f /package/example/README.md; \
|
||||
# pnpm's hoisted self-reference points back into this workspace; an installed module has none.
|
||||
rm -f /package/example/node_modules/.pnpm/node_modules/@ekklesia/module-template; \
|
||||
escaping=""; \
|
||||
for link in $(find /package/example -type l); do \
|
||||
case "$(realpath -m "$link")" in \
|
||||
/package/example/*) ;; \
|
||||
*) escaping="$escaping $link" ;; \
|
||||
esac; \
|
||||
done; \
|
||||
if [ -n "$escaping" ]; then echo "Links escape the package:$escaping" >&2; exit 1; fi; \
|
||||
chmod -R a+rX,go-w /package/example; \
|
||||
cd /package/example && node -e 'import("./dist/server/index.js").then((m) => { if (m.default.id !== "example") process.exit(1); })'
|
||||
|
||||
FROM scratch AS package
|
||||
COPY --from=build /package/ /
|
||||
+48
-22
@@ -15,13 +15,15 @@
|
||||
# ADR 0048 end to end with that same image:
|
||||
#
|
||||
# - it boots with no modules;
|
||||
# - the repository's module template, packaged as one self-contained directory
|
||||
# (scripts/lib/test-module-package.Dockerfile, TEST-ONLY), is installed into module storage, its
|
||||
# migrations are planned and explicitly applied, and the API is restarted;
|
||||
# - the repository's module template is packaged by the canonical packager as an Ekklesia Module
|
||||
# Package v1 (ADR 0049; scripts/lib/build-module-package.Dockerfile only provides the build
|
||||
# environment), verified by the same image's own verifier, extracted to staging, installed into
|
||||
# module storage, its migrations planned and explicitly applied, and the API restarted;
|
||||
# - a real headless Chromium drives the Dashboard, the module, its display, and sign-out;
|
||||
# - the module is removed and the API restarted: Core still runs, migration planning reports the
|
||||
# module's retained history, and its data survives;
|
||||
# - the module is restored and the API restarted: it returns with its previous data.
|
||||
# - the exact same package is verified and installed again, and the API restarted: the module
|
||||
# returns with its previous data.
|
||||
#
|
||||
# EKKLESIA_SMOKE_IMAGE uses the given image instead of building one. It must be a generic image
|
||||
# (ADR 0048): modules come from module storage, never from the image.
|
||||
@@ -179,7 +181,7 @@ cleanup() {
|
||||
# job itself runs as root), so only the staged copy on the Docker host needs this.
|
||||
if [[ "$daemon_work" == "$work" ]]; then
|
||||
docker run --rm --entrypoint rm -v "$work:/work" "$postgres_image" -rf /work/tls /work/env \
|
||||
/work/modules >/dev/null 2>&1 || true
|
||||
/work/modules /work/staging >/dev/null 2>&1 || true
|
||||
else
|
||||
docker run --rm --entrypoint rm -v /tmp:/host-tmp "$postgres_image" -rf "/host-tmp/${project}" \
|
||||
>/dev/null 2>&1 || true
|
||||
@@ -213,15 +215,20 @@ if [[ -z "$image" ]]; then
|
||||
fi
|
||||
|
||||
if [[ "$topology" == split ]]; then
|
||||
# TEST-ONLY: the module template as one self-contained installed-module directory, built outside
|
||||
# the platform image and installed into module storage below. Never a second platform image.
|
||||
step "Packaging the module template as a self-contained test module"
|
||||
DOCKER_BUILDKIT=1 docker build --quiet --file scripts/lib/test-module-package.Dockerfile \
|
||||
--label "${EKKLESIA_CI_LABEL}=${project}" \
|
||||
# The module template as a Module Package v1, from the canonical packager. Never a second platform
|
||||
# image: the package is installed into module storage below, after the image verifies it.
|
||||
step "Packaging the module template with the canonical Ekklesia packager"
|
||||
source_revision="${GITHUB_SHA:-$(git rev-parse HEAD 2>/dev/null || true)}"
|
||||
[[ "$source_revision" =~ ^[0-9a-f]{40}$ ]] || fail "no source revision to record in the package."
|
||||
DOCKER_BUILDKIT=1 docker build --quiet --file scripts/lib/build-module-package.Dockerfile \
|
||||
--label "${EKKLESIA_CI_LABEL}=${project}" --build-arg SOURCE_REVISION="$source_revision" \
|
||||
--output "type=local,dest=${work}/module-package" . >/dev/null ||
|
||||
fail "the test module package did not build."
|
||||
[[ -f "$work/module-package/example/ekklesia.module.json" ]] ||
|
||||
fail "the test module package has no manifest."
|
||||
fail "the module package did not build."
|
||||
package_archive="$(cd "$work/module-package" && ls -- *-any.tar.gz)"
|
||||
[[ "$package_archive" =~ ^example-[0-9]+\.[0-9]+\.[0-9]+-any\.tar\.gz$ &&
|
||||
-f "$work/module-package/${package_archive}.sha256" ]] ||
|
||||
fail "the packager did not produce exactly one archive and its digest."
|
||||
echo "ok packaged ${package_archive} ($(wc -c <"$work/module-package/$package_archive") bytes, sha256 $(cut -d' ' -f1 "$work/module-package/${package_archive}.sha256"))"
|
||||
fi
|
||||
|
||||
# The compose project directory: its relative bind mount (postgres/pg_hba.conf) resolves here.
|
||||
@@ -232,6 +239,13 @@ stage deploy
|
||||
# `up`: Compose refuses to start `api` or `migrate` without it, and never creates it.
|
||||
mkdir -m 755 "$work/modules"
|
||||
stage modules
|
||||
if [[ "$topology" == split ]]; then
|
||||
stage module-package
|
||||
# TEST-ONLY staging the verifier extracts into, as the image's unprivileged user. Nothing here is
|
||||
# module storage: the install step below moves a verified tree into module storage as root.
|
||||
mkdir -m 777 "$work/staging"
|
||||
stage staging
|
||||
fi
|
||||
|
||||
step "Creating the installation-private CA and server certificates"
|
||||
init_status=0
|
||||
@@ -420,15 +434,27 @@ module_storage() {
|
||||
"$postgres_image" -ec "$1"
|
||||
}
|
||||
|
||||
# install_test_module: stages the package under a hidden name, which discovery ignores, then renames
|
||||
# it into place in one step, owned by root and readable by the API's unprivileged user.
|
||||
# install_test_module: verifies the package with the generic image's own verifier, which writes the
|
||||
# verified module directory to staging, then moves it into module storage under a hidden name, which
|
||||
# discovery ignores, and renames it into place in one step, owned by root and readable by the API's
|
||||
# unprivileged user. Nothing unverified ever reaches module storage.
|
||||
install_test_module() {
|
||||
tar -C "$work/module-package" -cf - example |
|
||||
module_storage 'rm -rf /modules/.incoming && mkdir /modules/.incoming &&
|
||||
tar -xf - -C /modules/.incoming && chown -R 0:0 /modules/.incoming/example &&
|
||||
chmod -R a+rX,go-w /modules/.incoming/example &&
|
||||
mv /modules/.incoming/example /modules/example && rmdir /modules/.incoming' ||
|
||||
fail "the test module could not be installed into module storage."
|
||||
local verified
|
||||
verified="$(
|
||||
docker run --rm --network none --read-only --tmpfs /tmp --cap-drop ALL \
|
||||
--security-opt no-new-privileges \
|
||||
-v "$daemon_work/module-package:/package:ro" -v "$daemon_work/staging:/staging" \
|
||||
--entrypoint node "$image" core/dist/cli/verify-module-package.js "/package/${package_archive}" \
|
||||
--sha256-file "/package/${package_archive}.sha256" --extract-to /staging/example 2>&1
|
||||
)" || fail "the module package did not verify: ${verified}"
|
||||
echo "ok ${verified%%$'\n'*}"
|
||||
docker run --rm --network none --entrypoint sh -v "$daemon_work/modules:/modules" \
|
||||
-v "$daemon_work/staging:/staging" "$postgres_image" -ec '
|
||||
rm -rf /modules/.incoming-example && cp -R /staging/example /modules/.incoming-example &&
|
||||
rm -rf /staging/example && chown -R 0:0 /modules/.incoming-example &&
|
||||
chmod -R a+rX,go-w /modules/.incoming-example &&
|
||||
mv /modules/.incoming-example /modules/example' ||
|
||||
fail "the verified module could not be installed into module storage."
|
||||
}
|
||||
|
||||
# restart_api <command...>: stops the API, runs the command, and starts the API again. A controlled
|
||||
@@ -590,7 +616,7 @@ if [[ "$topology" == split ]]; then
|
||||
fail "the removed module's data did not survive."
|
||||
echo "ok the module is gone, Core runs, its history is retained, and its data survives"
|
||||
|
||||
step "Restoring the module and restarting: it returns with its previous data"
|
||||
step "Reinstalling the exact same package and restarting: the module returns with its previous data"
|
||||
restart_api install_test_module
|
||||
plan="$(plan_output)"
|
||||
grep -q '^- \[applied\] example:0000_initial ' <<<"$plan" ||
|
||||
|
||||
Reference in New Issue
Block a user