feat: native linux-x64-glibc module packages and reviewed license supplements (ADR 0049, ADR 0052)
CI / verify (pull_request) Successful in 1m6s
CI / infrastructure-smoke (pull_request) Successful in 27s
CI / migration-smoke (pull_request) Successful in 1m54s
CI / image-build (pull_request) Successful in 29s
CI / deploy-smoke (same-origin) (pull_request) Successful in 43s
CI / deploy-smoke (split) (pull_request) Successful in 1m30s

Implements ADR 0049's deferred native target and ADR 0052's reviewed
license supplements. The proof case is a real module using sharp.
packageFormat stays 1, and `any` is unchanged.

Packager (@ekklesia/module-pack):
- --target linux-x64-glibc. Native or platform-specific packages, found by
  the same generic rule that refuses them in `any`, stay outside the bundle.
- Each native package's production closure is walked through its own
  dependencies, installed optionalDependencies that match the target, and
  required peers. It is copied as real files flat under node_modules/<name>,
  with no links, dev dependencies, declarations, source maps, or other
  platforms' packages. Missing or foreign-platform required dependencies,
  version conflicts, links, foreign machines, and Node.js range conflicts
  fail closed.
- runtime.nodeApi, nodeAbi, and libcVersion are read from the binaries by a
  small ELF reader: the addon's node_api_module_get_api_version_v1 and the
  newest GLIBC_ symbol version needed. No shelling out. A native target with
  no addon is refused.
- ADR 0052 supplements: package-provided material stays primary. An exact
  match on name, version, and ekklesia-content-v1 digest (sorted normalized
  paths with each file's size and SHA-256; no metadata) adds labelled
  material after the package's own. Any mismatch, a declared-licence
  conflict, or a malformed or unreviewed supplement file fails. Missing
  material with no supplement still fails.

Supplement for @img/[email protected]:
- 184 components and 143 unique texts, assembled from the exact upstream
  sources sharp-libvips v1.3.3 builds linux-x64 from: 27 C/C++ libraries,
  glib's gvdb, librsvg's Rust crates (reproduced with cargo tree), the Rust
  standard library at the rustc commit recorded in the binary, plus the
  LGPL-3.0, GPL-3.0, and MPL-2.0 texts. Each source is recorded with its
  SHA-256 in PROVENANCE.md.
- It pins digest ekklesia-content-v1:sha256:50647b0b…, which is identical
  from the installed copy and from the registry tarball.
- LGPL/MPL source-availability determinations stay open, as ADR 0052
  states. Nothing claims they are met.

Verifier (Core):
- linux-x64-glibc is accepted only on a host whose platform, arch, glibc
  family and version, and Node-API version (from Node itself) match.
- Every ELF file must be x86-64; every .node must be an ELF object.
- `any` packages now also refuse node_modules/ and ELF files.
- Contract: native metadata must state nodeApi or nodeAbi, and Linux targets
  libcVersion (ADR 0049).

Proof:
- A checked-in AGPL native fixture packs byte-identically from unrelated
  copies (8.86 MB) and verifies with Core's verifier.
- Extracted elsewhere, it runs a sharp resize and PNG round trip in an
  empty-environment process. Every resolution stays in the package, and
  only the package's addon and libvips plus host system libraries load.

Signage is untouched.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
ekklesia
2026-10-07 20:16:53 -05:00
co-authored by Claude Opus 5.5
parent f53506b3f0
commit 44d385cd7f
173 changed files with 20638 additions and 53 deletions
+3 -3
View File
@@ -8,17 +8,17 @@ import { ModulePackError, type PackModuleOptions, packModule } from "./pack-modu
/**
* `ekklesia-pack-module` — the canonical Ekklesia module packager (ADR 0049).
*
* ekklesia-pack-module --module <built module directory> --target any --out <directory>
* ekklesia-pack-module --module <built module directory> --target <any|linux-x64-glibc> --out <directory>
* --license-file <file> --source-revision <full commit id> [--repository <url>]
*
* Writes `<id>-<version>-any.tar.gz` and its `.sha256` line to the output directory, and nothing
* Writes `<id>-<version>-<target>.tar.gz` and its `.sha256` line to the output directory, and nothing
* else anywhere. Run it in the module's CI after building the module.
*/
class PackArgumentsError extends Error {}
const USAGE =
"Usage: ekklesia-pack-module --module <directory> --target any --out <directory> --license-file <file> --source-revision <commit> [--repository <url>]";
"Usage: ekklesia-pack-module --module <directory> --target <any|linux-x64-glibc> --out <directory> --license-file <file> --source-revision <commit> [--repository <url>]";
const OPTIONS: Readonly<Record<string, keyof PackModuleOptions>> = Object.freeze({
"--license-file": "licenseFile",
+188
View File
@@ -0,0 +1,188 @@
/**
* Just enough of the ELF format to describe a native file the packager is about to copy (ADR 0049):
* which machine it was built for, which shared libraries and C library symbol versions it needs, and
* which Node.js addon interface it exports. Read from the file's own bytes, never by running it or
* shelling out to a system tool.
*
* Only 64-bit little-endian ELF is understood, because that is every target this packager produces.
* Anything else that starts with the ELF magic is reported as unsupported rather than guessed at.
*/
export class ElfFormatError extends Error {
constructor() {
super("The file is not a 64-bit little-endian ELF object this packager can read.");
this.name = "ElfFormatError";
}
}
/** `EM_X86_64` in the ELF machine field. */
export const ELF_MACHINE_X86_64 = 62;
export interface ElfDescription {
readonly machine: number;
/** `DT_NEEDED` entries: the shared libraries the file links against. */
readonly needed: readonly string[];
/** Symbol version names the file requires, for example `GLIBC_2.28`. */
readonly requiredVersions: readonly string[];
/** Names of the dynamic symbols the file itself defines. */
readonly exported: ReadonlySet<string>;
/** The value a Node-API addon's `node_api_module_get_api_version_v1` returns, when it exports
* that function and its body is the plain constant return every compiler emits for it.
* `undefined` when the function is absent; `"unrecognized"` when its body is something else. */
readonly nodeApiModuleVersion: number | "unrecognized" | undefined;
}
const SHT_NOBITS = 8;
const SHT_DYNSYM = 11;
const SHT_DYNAMIC = 6;
const SHT_GNU_VERNEED = 0x6ffffffe;
const DT_NULL = 0;
const DT_NEEDED = 1;
export function isElf(data: Uint8Array): boolean {
return (
data.length >= 4 && data[0] === 0x7f && data[1] === 0x45 && data[2] === 0x4c && data[3] === 0x46
);
}
interface Section {
readonly type: number;
readonly address: number;
readonly offset: number;
readonly size: number;
readonly link: number;
readonly entrySize: number;
}
export function describeElf(input: Uint8Array): ElfDescription {
const data = Buffer.from(input.buffer, input.byteOffset, input.byteLength);
if (!isElf(data) || data.length < 64 || data[4] !== 2 || data[5] !== 1) {
throw new ElfFormatError();
}
const u16 = (offset: number): number => {
if (offset < 0 || offset + 2 > data.length) throw new ElfFormatError();
return data.readUInt16LE(offset);
};
const u32 = (offset: number): number => {
if (offset < 0 || offset + 4 > data.length) throw new ElfFormatError();
return data.readUInt32LE(offset);
};
const u64 = (offset: number): number => {
if (offset < 0 || offset + 8 > data.length) throw new ElfFormatError();
const value = data.readBigUInt64LE(offset);
if (value > BigInt(Number.MAX_SAFE_INTEGER)) throw new ElfFormatError();
return Number(value);
};
const string = (tableOffset: number, tableSize: number, index: number): string => {
if (index >= tableSize) throw new ElfFormatError();
const start = tableOffset + index;
const end = data.indexOf(0, start);
if (end === -1 || end > tableOffset + tableSize) throw new ElfFormatError();
return data.toString("latin1", start, end);
};
const machine = u16(18);
const sectionOffset = u64(40);
const sectionEntrySize = u16(58);
const sectionCount = u16(60);
if (sectionEntrySize < 64 || sectionCount === 0) {
throw new ElfFormatError();
}
const sections: Section[] = [];
for (let index = 0; index < sectionCount; index += 1) {
const base = sectionOffset + index * sectionEntrySize;
sections.push({
address: u64(base + 16),
entrySize: u64(base + 56),
link: u32(base + 40),
offset: u64(base + 24),
size: u64(base + 32),
type: u32(base + 4),
});
}
const linkedStrings = (section: Section): { offset: number; size: number } => {
const table = sections[section.link];
if (table === undefined || table.offset + table.size > data.length) throw new ElfFormatError();
return { offset: table.offset, size: table.size };
};
const needed: string[] = [];
const requiredVersions: string[] = [];
const exported = new Set<string>();
let apiVersionAddress: number | undefined;
for (const section of sections) {
if (section.type === SHT_DYNAMIC) {
const strings = linkedStrings(section);
for (let entry = section.offset; entry + 16 <= section.offset + section.size; entry += 16) {
const tag = u64(entry);
if (tag === DT_NULL) break;
if (tag === DT_NEEDED) needed.push(string(strings.offset, strings.size, u64(entry + 8)));
}
} else if (section.type === SHT_DYNSYM) {
const strings = linkedStrings(section);
const size = section.entrySize === 0 ? 24 : section.entrySize;
for (let entry = section.offset; entry + 24 <= section.offset + section.size; entry += size) {
const nameIndex = u32(entry);
const sectionIndex = u16(entry + 6);
if (nameIndex === 0 || sectionIndex === 0) continue;
const name = string(strings.offset, strings.size, nameIndex);
exported.add(name);
if (name === "node_api_module_get_api_version_v1") {
apiVersionAddress = u64(entry + 8);
}
}
} else if (section.type === SHT_GNU_VERNEED) {
const strings = linkedStrings(section);
let need = section.offset;
for (let guard = 0; guard < 4096; guard += 1) {
const count = u16(need + 2);
let aux = need + u32(need + 8);
for (let index = 0; index < count; index += 1) {
requiredVersions.push(string(strings.offset, strings.size, u32(aux + 8)));
const next = u32(aux + 12);
if (next === 0) break;
aux += next;
}
const next = u32(need + 12);
if (next === 0) break;
need += next;
}
}
}
let nodeApiModuleVersion: number | "unrecognized" | undefined;
if (apiVersionAddress !== undefined) {
const text = sections.find(
(section) =>
section.type !== SHT_NOBITS &&
section.address <= apiVersionAddress &&
apiVersionAddress < section.address + section.size,
);
nodeApiModuleVersion = "unrecognized";
if (text !== undefined) {
let offset = text.offset + (apiVersionAddress - text.address);
// An optional `endbr64` (Intel CET), then `mov eax, imm32; ret`: what every compiler emits for
// a function that returns the NAPI_VERSION constant. Anything else is not guessed at.
if (
data[offset] === 0xf3 &&
data[offset + 1] === 0x0f &&
data[offset + 2] === 0x1e &&
data[offset + 3] === 0xfa
) {
offset += 4;
}
if (data[offset] === 0xb8 && data[offset + 5] === 0xc3) {
nodeApiModuleVersion = u32(offset + 1);
}
}
}
return Object.freeze({
exported,
machine,
needed: Object.freeze(needed),
nodeApiModuleVersion,
requiredVersions: Object.freeze(requiredVersions),
});
}
+357
View File
@@ -0,0 +1,357 @@
import { lstat, readdir, readFile, realpath } from "node:fs/promises";
import path from "node:path";
import { describeElf, ELF_MACHINE_X86_64, ElfFormatError, isElf } from "./elf.js";
/**
* Native targets of Ekklesia Module Package v1 (ADR 0049): what a `<platform>-<arch>[-<libc>]`
* package may contain beyond bundled JavaScript, and how the packager establishes it.
*
* A native or platform-specific dependency cannot be bundled. The packager keeps it outside the
* bundle and copies it, with exactly its own production runtime dependencies, as real files flat
* under `node_modules/<name>`. Nothing is installed, linked, or downloaded. Which packages qualify is
* decided generically, from each package's own metadata and files, never from a list of names.
*/
export interface NativeTarget {
readonly id: string;
readonly os: string;
readonly cpu: string;
readonly libc: string;
readonly elfMachine: number;
/** Shared libraries whose presence means a binary was linked against another C library. */
readonly foreignLibc: RegExp;
readonly libcVersionPrefix: string;
}
/** The native targets this packager produces. Every other target is refused. */
export const NATIVE_TARGETS: ReadonlyMap<string, NativeTarget> = new Map([
[
"linux-x64-glibc",
Object.freeze({
cpu: "x64",
elfMachine: ELF_MACHINE_X86_64,
foreignLibc: /^(?:libc\.musl-|ld-musl-)/u,
id: "linux-x64-glibc",
libc: "glibc",
libcVersionPrefix: "GLIBC_",
os: "linux",
}),
],
]);
export type NativeErrorCode =
| "MODULE_PACK_NATIVE_DEPENDENCY_MISSING"
| "MODULE_PACK_NATIVE_LAYOUT_CONFLICT"
| "MODULE_PACK_NATIVE_PACKAGE_INVALID"
| "MODULE_PACK_NATIVE_RUNTIME_UNKNOWN"
| "MODULE_PACK_NATIVE_TARGET_MISMATCH";
export class NativeError extends Error {
readonly code: NativeErrorCode;
readonly detail: string;
constructor(code: NativeErrorCode, detail: string) {
super(code);
this.name = "NativeError";
this.code = code;
this.detail = detail;
}
}
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === "object" && value !== null && !Array.isArray(value);
}
function compare(left: string, right: string): number {
return left < right ? -1 : left > right ? 1 : 0;
}
/** npm's `os`, `cpu`, and `libc` fields: absent means any; entries are names or `!`-negations. */
function fieldAllows(field: unknown, value: string): boolean {
if (!Array.isArray(field) || field.length === 0) {
return true;
}
const entries = field.filter((entry): entry is string => typeof entry === "string");
if (entries.includes(`!${value}`)) {
return false;
}
const positive = entries.filter((entry) => !entry.startsWith("!"));
return positive.length === 0 || positive.includes(value);
}
export function packageSupportsTarget(
manifest: Readonly<Record<string, unknown>>,
target: NativeTarget,
): boolean {
return (
fieldAllows(manifest.os, target.os) &&
fieldAllows(manifest.cpu, target.cpu) &&
fieldAllows(manifest.libc, target.libc)
);
}
async function readManifest(directory: string): Promise<Record<string, unknown> | undefined> {
try {
const parsed = JSON.parse(
await readFile(path.join(directory, "package.json"), "utf8"),
) as unknown;
return isRecord(parsed) ? parsed : undefined;
} catch {
return undefined;
}
}
/** The installed package `name` resolves to from `fromDirectory`, by Node's own directory walk. */
async function resolveInstalled(fromDirectory: string, name: string): Promise<string | undefined> {
let directory = fromDirectory;
for (;;) {
if (path.basename(directory) !== "node_modules") {
const candidate = path.join(directory, "node_modules", ...name.split("/"));
if (await readManifest(candidate)) {
return realpath(candidate);
}
}
const parent = path.dirname(directory);
if (parent === directory) {
return undefined;
}
directory = parent;
}
}
/** Declarations and source maps never ship (ADR 0049); everything else a package publishes does,
* because the packager cannot know which of a third-party package's files it reads at run time. */
function excludedFromCopy(relative: string): boolean {
return /\.(?:d\.ts|d\.mts|d\.cts|map)$/u.test(relative);
}
/** Every regular file a package publishes, by path relative to its root, nested `node_modules`
* excluded (those are separate packages). A link or special file fails: none may enter a package. */
async function readPackageFiles(root: string, packageId: string): Promise<Map<string, Buffer>> {
const files = new Map<string, Buffer>();
const walk = async (relative: string): Promise<void> => {
const directory = relative === "" ? root : path.join(root, relative);
const entries = (await readdir(directory, { withFileTypes: true })).sort((left, right) =>
compare(left.name, right.name),
);
for (const entry of entries) {
const childRelative = relative === "" ? entry.name : `${relative}/${entry.name}`;
const stats = await lstat(path.join(directory, entry.name));
if (stats.isDirectory()) {
if (entry.name !== "node_modules") {
await walk(childRelative);
}
} else if (stats.isFile()) {
if (!excludedFromCopy(childRelative)) {
files.set(childRelative, await readFile(path.join(directory, entry.name)));
}
} else {
throw new NativeError("MODULE_PACK_NATIVE_PACKAGE_INVALID", packageId);
}
}
};
await walk("");
return files;
}
/** A package copied into the module package's `node_modules`. */
export interface NativePackage {
readonly name: string;
readonly version: string;
readonly root: string;
readonly manifest: Readonly<Record<string, unknown>>;
/** The files copied, by path relative to the package root: exactly what is redistributed. */
readonly files: ReadonlyMap<string, Buffer>;
}
export interface NativeRuntime {
readonly nodeApi?: number;
readonly nodeAbi?: string;
readonly libcVersion?: string;
}
function dependencyNames(manifest: Readonly<Record<string, unknown>>): {
readonly required: readonly string[];
readonly optional: readonly string[];
} {
const names = (field: unknown): string[] => (isRecord(field) ? Object.keys(field) : []);
const optional = new Set(names(manifest.optionalDependencies));
const peerMeta = isRecord(manifest.peerDependenciesMeta) ? manifest.peerDependenciesMeta : {};
for (const peer of names(manifest.peerDependencies)) {
const meta = peerMeta[peer];
if (isRecord(meta) && meta.optional === true) {
optional.add(peer);
}
}
const required = new Set(
[...names(manifest.dependencies), ...names(manifest.peerDependencies)].filter(
(name) => !optional.has(name),
),
);
return {
optional: [...optional].sort(compare),
required: [...required].sort(compare),
};
}
/**
* The runtime closure of the native roots for `target`: each root, its production `dependencies`,
* its installed `optionalDependencies` whose `os`/`cpu`/`libc` admit the target, and its required
* peers, recursively. Development dependencies are never followed. A required dependency that is
* missing or built for another platform fails; an optional one for another platform is left out.
* Two different versions of one name cannot share the flat layout and fail.
*/
export async function nativeClosure(
roots: readonly { readonly name: string; readonly root: string }[],
target: NativeTarget,
): Promise<readonly NativePackage[]> {
const byName = new Map<string, NativePackage>();
const queue = [...roots].sort((left, right) => compare(left.name, right.name));
for (let next = queue.shift(); next !== undefined; next = queue.shift()) {
const root = await realpath(next.root);
const manifest = await readManifest(root);
if (
manifest === undefined ||
typeof manifest.name !== "string" ||
typeof manifest.version !== "string" ||
manifest.name !== next.name
) {
throw new NativeError("MODULE_PACK_NATIVE_PACKAGE_INVALID", next.name);
}
const packageId = `${manifest.name}@${manifest.version}`;
const existing = byName.get(manifest.name);
if (existing !== undefined) {
if (existing.version !== manifest.version) {
throw new NativeError("MODULE_PACK_NATIVE_LAYOUT_CONFLICT", manifest.name);
}
continue;
}
if (!packageSupportsTarget(manifest, target)) {
throw new NativeError("MODULE_PACK_NATIVE_TARGET_MISMATCH", packageId);
}
byName.set(
manifest.name,
Object.freeze({
files: await readPackageFiles(root, packageId),
manifest: Object.freeze(manifest),
name: manifest.name,
root,
version: manifest.version,
}),
);
const { optional, required } = dependencyNames(manifest);
for (const name of required) {
const resolved = await resolveInstalled(root, name);
if (resolved === undefined) {
throw new NativeError("MODULE_PACK_NATIVE_DEPENDENCY_MISSING", `${packageId} -> ${name}`);
}
queue.push({ name, root: resolved });
}
for (const name of optional) {
const resolved = await resolveInstalled(root, name);
const dependency = resolved === undefined ? undefined : await readManifest(resolved);
if (resolved !== undefined && dependency && packageSupportsTarget(dependency, target)) {
queue.push({ name, root: resolved });
}
}
}
return [...byName.values()].sort((left, right) => compare(left.name, right.name));
}
/** Compares `GLIBC_2.28`-style versions numerically. */
function compareVersions(left: readonly number[], right: readonly number[]): number {
for (let index = 0; index < Math.max(left.length, right.length); index += 1) {
const difference = (left[index] ?? 0) - (right[index] ?? 0);
if (difference !== 0) {
return difference;
}
}
return 0;
}
/**
* The runtime requirements of the copied native files, read from the binaries themselves (ADR 0049):
*
* - every ELF file must be built for the target's machine and must not link another C library;
* - every `.node` file must be an ELF Node.js addon;
* - `nodeApi` is the highest Node-API version a Node-API addon declares through its exported
* `node_api_module_get_api_version_v1` (an addon without that export is version 8, as Node.js
* itself treats it); an ABI-specific addon records its `node_register_module_v<ABI>` as `nodeAbi`;
* - `libcVersion` is the highest C library symbol version any file requires.
*
* Anything that cannot be established exactly fails closed.
*/
export function nativeRuntime(
packages: readonly NativePackage[],
target: NativeTarget,
): NativeRuntime & { readonly addons: number } {
let nodeApi: number | undefined;
let nodeAbi: string | undefined;
let libc: number[] | undefined;
let addons = 0;
for (const nativePackage of packages) {
for (const [relative, data] of nativePackage.files) {
const fileId = `${nativePackage.name}@${nativePackage.version}/${relative}`;
const addon = relative.toLowerCase().endsWith(".node");
if (!isElf(data)) {
if (addon) {
throw new NativeError("MODULE_PACK_NATIVE_TARGET_MISMATCH", fileId);
}
continue;
}
let elf: ReturnType<typeof describeElf>;
try {
elf = describeElf(data);
} catch (error) {
if (error instanceof ElfFormatError) {
throw new NativeError("MODULE_PACK_NATIVE_TARGET_MISMATCH", fileId);
}
throw error;
}
if (
elf.machine !== target.elfMachine ||
elf.needed.some((name) => target.foreignLibc.test(name))
) {
throw new NativeError("MODULE_PACK_NATIVE_TARGET_MISMATCH", fileId);
}
for (const version of elf.requiredVersions) {
if (!version.startsWith(target.libcVersionPrefix)) {
continue;
}
const parts = version.slice(target.libcVersionPrefix.length).split(".").map(Number);
if (parts.some((part) => !Number.isInteger(part))) {
continue;
}
if (libc === undefined || compareVersions(parts, libc) > 0) {
libc = parts;
}
}
if (!addon) {
continue;
}
addons += 1;
if (elf.exported.has("napi_register_module_v1")) {
const declared = elf.nodeApiModuleVersion;
if (declared === "unrecognized" || (typeof declared === "number" && declared < 1)) {
throw new NativeError("MODULE_PACK_NATIVE_RUNTIME_UNKNOWN", fileId);
}
const version = declared ?? 8;
nodeApi = nodeApi === undefined ? version : Math.max(nodeApi, version);
} else {
const abi = [...elf.exported]
.map((name) => /^node_register_module_v(\d+)$/u.exec(name)?.[1])
.find((value) => value !== undefined);
if (abi === undefined || (nodeAbi !== undefined && nodeAbi !== abi)) {
throw new NativeError("MODULE_PACK_NATIVE_RUNTIME_UNKNOWN", fileId);
}
nodeAbi = abi;
}
}
}
return {
addons,
...(nodeApi === undefined ? {} : { nodeApi }),
...(nodeAbi === undefined ? {} : { nodeAbi }),
...(libc === undefined ? {} : { libcVersion: `>=${[...libc, 0, 0].slice(0, 3).join(".")}` }),
};
}
+72 -4
View File
@@ -2,6 +2,7 @@ import { readdir, readFile } from "node:fs/promises";
import path from "node:path";
import { fileURLToPath } from "node:url";
import type { BundledPackage } from "./packages.js";
import { contentDigest, type LicenseSupplement } from "./supplements.js";
/**
* `THIRD_PARTY_NOTICES` (ADR 0049): the licence and notice text that redistributing a module
@@ -24,11 +25,20 @@ import type { BundledPackage } from "./packages.js";
* (Apache-2.0). An MIT, ISC, or BSD package without its file is refused, because its copyright line
* exists only in that file.
*
* - **Reviewed supplements** (ADR 0052). When a reviewed license supplement exists for a package's
* name, it must match the package exactly (version and content digest over the files
* redistributed for it) and must name the licence the package declares, or packaging fails. A
* matching supplement's material follows the package's own, labelled as supplement material; it
* stands in for the licence text the package omits, and never changes the declared licence.
*
* Everything is sorted, and line endings are normalized, so the file is byte-identical for identical
* inputs.
*/
export type NoticeErrorCode = "MODULE_PACK_LICENCE_TEXT_MISSING" | "MODULE_PACK_LICENCE_UNKNOWN";
export type NoticeErrorCode =
| "MODULE_PACK_LICENCE_SUPPLEMENT_MISMATCH"
| "MODULE_PACK_LICENCE_TEXT_MISSING"
| "MODULE_PACK_LICENCE_UNKNOWN";
export class NoticeError extends Error {
readonly code: NoticeErrorCode;
@@ -42,11 +52,16 @@ export class NoticeError extends Error {
}
}
export type NoticeUse = "backend" | "browser";
/** Where in the package a component is used: bundled into the backend or a browser artifact, or
* copied as files under `node_modules` (a native target's runtime dependency). */
export type NoticeUse = "backend" | "browser" | "node_modules";
export interface NoticeComponent {
readonly bundled: BundledPackage;
readonly uses: ReadonlySet<NoticeUse>;
/** The files redistributed for this component, by path relative to its root: what a supplement's
* content digest is computed over. Read only when a supplement names the component. */
readonly redistributedFiles: () => Promise<ReadonlyMap<string, Uint8Array>>;
}
/** Licences whose standard text is complete without any project-specific line, so supplying it
@@ -114,8 +129,47 @@ async function standardText(licence: string): Promise<string | undefined> {
: normalized(await readFile(path.join(packageRoot, "licenses", file), "utf8"));
}
function supplementSection(supplement: LicenseSupplement): string[] {
const lines = [
"[Reviewed license supplement]",
"",
`This material did not come from ${supplement.name} ${supplement.version}. The package omits licence`,
"and notice material that redistribution requires, so Ekklesia's canonical packager adds it from a",
"reviewed license supplement matched to this exact package: its name, its version, and its content",
`digest ${supplement.contentDigest}. The supplement does not change the licence`,
"the package declares. Review of a supplement is project and code review of a build input. It is",
"not legal review, and it does not establish that every redistribution obligation is met.",
"",
normalized(supplement.statement).replace(/\n$/u, ""),
"",
"Components covered:",
"",
];
for (const component of supplement.components) {
lines.push(` ${component.name} ${component.version}: ${component.license}`);
lines.push(` Source: ${component.source}`);
}
lines.push("");
const printed = new Map<string, string>();
for (const component of supplement.components) {
for (const material of component.material) {
const label = `${component.name} ${component.version}: ${material.title}`;
const first = printed.get(material.sha256);
lines.push(`[${label}]`, "");
if (first === undefined) {
printed.set(material.sha256, label);
lines.push(normalized(material.text));
} else {
lines.push(`Identical to the text printed above under [${first}].`, "");
}
}
}
return lines;
}
export async function thirdPartyNotices(
components: ReadonlyMap<string, NoticeComponent>,
supplements: ReadonlyMap<string, readonly LicenseSupplement[]> = new Map(),
): Promise<string> {
const ordered = [...components.values()].sort(
(left, right) =>
@@ -123,15 +177,28 @@ export async function thirdPartyNotices(
compare(left.bundled.version, right.bundled.version),
);
const sections: string[] = [];
for (const { bundled, uses } of ordered) {
for (const { bundled, redistributedFiles, uses } of ordered) {
const component = `${bundled.name}@${bundled.version}`;
const licence = licenceOf(bundled);
if (licence === undefined || /^UNLICEN[CS]ED$/iu.test(licence)) {
throw new NoticeError("MODULE_PACK_LICENCE_UNKNOWN", component);
}
let supplement: LicenseSupplement | undefined;
const candidates = supplements.get(bundled.name);
if (candidates !== undefined && candidates.length > 0) {
// A supplement exists for this name: only an exact match may be used, and anything else
// fails rather than falling back to a nearby version or to no supplement at all.
const digest = contentDigest(await redistributedFiles());
supplement = candidates.find(
(candidate) => candidate.version === bundled.version && candidate.contentDigest === digest,
);
if (supplement === undefined || supplement.license !== licence) {
throw new NoticeError("MODULE_PACK_LICENCE_SUPPLEMENT_MISMATCH", component);
}
}
let texts = await licenceTexts(bundled, licence);
// A NOTICE file alone is not the licence: the licence text must still come from somewhere.
if (!texts.some((text) => !/^notice/iu.test(text.title))) {
if (supplement === undefined && !texts.some((text) => !/^notice/iu.test(text.title))) {
const standard = await standardText(licence);
if (standard === undefined) {
throw new NoticeError("MODULE_PACK_LICENCE_TEXT_MISSING", component);
@@ -153,6 +220,7 @@ export async function thirdPartyNotices(
RULE,
"",
...texts.flatMap((text) => [`[${text.title}]`, "", text.text]),
...(supplement === undefined ? [] : supplementSection(supplement)),
].join("\n"),
);
}
+239 -25
View File
@@ -17,9 +17,23 @@ import {
} from "@ekklesia/contracts/module-package";
import { analyzeModuleUi, ModuleUiBuildError } from "@ekklesia/module-ui-build";
import { build, type Message, type Plugin } from "esbuild";
import { minVersion, satisfies, validRange } from "semver";
import { minVersion, satisfies, subset, validRange } from "semver";
import {
NATIVE_TARGETS,
type NativePackage,
type NativeTarget,
NativeError,
type NativeErrorCode,
nativeClosure,
nativeRuntime,
} from "./native.js";
import { type NoticeComponent, NoticeError, type NoticeUse, thirdPartyNotices } from "./notices.js";
import { BundledPackageIndex, isNativeOrPlatformSpecific } from "./packages.js";
import {
type BundledPackage,
BundledPackageIndex,
isNativeOrPlatformSpecific,
} from "./packages.js";
import { loadLicenseSupplements, SupplementError } from "./supplements.js";
import { deterministicTarGz, TarPathError } from "./tar.js";
/**
@@ -28,10 +42,12 @@ import { deterministicTarGz, TarPathError } from "./tar.js";
*
* It reads the module directory and the dependencies its workspace installed, and writes only the
* two output files. It never installs, links, or deploys anything, so the workspace it runs in is
* left exactly as it was. In this release it produces the portable `any` target only.
* left exactly as it was. It produces the portable `any` target, and the native `linux-x64-glibc`
* target, whose native runtime dependencies are copied rather than bundled (see `./native.ts`).
*/
export type ModulePackErrorCode =
| NativeErrorCode
| "MODULE_PACK_BUNDLE_FAILED"
| "MODULE_PACK_DATABASE_RUNTIME_CONFLICT"
| "MODULE_PACK_ENTRYPOINT_MISSING"
@@ -43,12 +59,16 @@ export type ModulePackErrorCode =
| "MODULE_PACK_LAYOUT_CONFLICT"
| "MODULE_PACK_MODULE_LICENSE_FILE_INVALID"
| "MODULE_PACK_MODULE_LICENSE_INVALID"
| "MODULE_PACK_LICENCE_SUPPLEMENT_INVALID"
| "MODULE_PACK_LICENCE_SUPPLEMENT_MISMATCH"
| "MODULE_PACK_LICENCE_TEXT_MISSING"
| "MODULE_PACK_LICENCE_UNKNOWN"
| "MODULE_PACK_MANIFEST_INVALID"
| "MODULE_PACK_METADATA_INVALID"
| "MODULE_PACK_MIGRATIONS_INVALID"
| "MODULE_PACK_NATIVE_DEPENDENCY"
| "MODULE_PACK_NATIVE_NODE_INCOMPATIBLE"
| "MODULE_PACK_NATIVE_TARGET_UNNEEDED"
| "MODULE_PACK_OUTPUT_FAILED"
| "MODULE_PACK_PATH_UNSUPPORTED"
| "MODULE_PACK_REVISION_INVALID"
@@ -78,6 +98,10 @@ const messages: Readonly<Record<ModulePackErrorCode, string>> = Object.freeze({
"The module license text must be supplied as an explicit, non-empty, readable regular file.",
MODULE_PACK_MODULE_LICENSE_INVALID:
"The source package.json must declare a valid SPDX license expression as a string.",
MODULE_PACK_LICENCE_SUPPLEMENT_INVALID:
"A reviewed license supplement is malformed or does not match its reviewed files.",
MODULE_PACK_LICENCE_SUPPLEMENT_MISMATCH:
"A license supplement names this package, but not its exact version, content digest, and declared licence.",
MODULE_PACK_LICENCE_TEXT_MISSING:
"A bundled package ships no licence text, and its licence has no complete standard text to supply.",
MODULE_PACK_LICENCE_UNKNOWN: "A bundled package declares no licence that permits redistribution.",
@@ -87,6 +111,20 @@ const messages: Readonly<Record<ModulePackErrorCode, string>> = Object.freeze({
"The migrations directory must hold only regular files and directories.",
MODULE_PACK_NATIVE_DEPENDENCY:
"The backend bundles a native or platform-specific dependency, which an any-target package cannot contain.",
MODULE_PACK_NATIVE_DEPENDENCY_MISSING:
"A native runtime dependency requires a package that is not installed.",
MODULE_PACK_NATIVE_LAYOUT_CONFLICT:
"Two versions of one package would share the native node_modules layout.",
MODULE_PACK_NATIVE_NODE_INCOMPATIBLE:
"A native runtime dependency does not support every Node.js version the package targets.",
MODULE_PACK_NATIVE_PACKAGE_INVALID:
"A native runtime dependency could not be read, or contains a link or special file.",
MODULE_PACK_NATIVE_RUNTIME_UNKNOWN:
"A native addon's Node.js interface could not be established exactly.",
MODULE_PACK_NATIVE_TARGET_MISMATCH:
"A native runtime dependency, or a file in one, is not built for the package's target.",
MODULE_PACK_NATIVE_TARGET_UNNEEDED:
"A native target needs a native addon; a package without one uses the any target.",
MODULE_PACK_OUTPUT_FAILED: "The package could not be written to the output directory.",
MODULE_PACK_PATH_UNSUPPORTED:
"A package path is not portable printable ASCII that fits the archive format.",
@@ -95,7 +133,8 @@ const messages: Readonly<Record<ModulePackErrorCode, string>> = Object.freeze({
"The Node.js range could not be determined. Run inside an Ekklesia workspace, or pass one.",
MODULE_PACK_SDK_INCOMPATIBLE:
"The bundled Module SDK is outside the manifest's own compatibility.moduleSdk range.",
MODULE_PACK_TARGET_UNSUPPORTED: "This packager produces the any target only.",
MODULE_PACK_TARGET_UNSUPPORTED:
"This packager produces the any and linux-x64-glibc targets only.",
MODULE_PACK_UI_STALE:
"A declared browser artifact is missing or out of date. Build the module before packaging it.",
});
@@ -124,7 +163,7 @@ export interface PackModuleOptions {
readonly moduleDirectory: string;
/** Explicit license-text input. Its bytes are copied unchanged to the package root as LICENSE. */
readonly licenseFile: string;
/** `any` only, in this release. */
/** `any`, or a native target this packager supports (`linux-x64-glibc`). */
readonly target: string;
readonly outDirectory: string;
/** The full commit id the module was built from. */
@@ -134,6 +173,8 @@ export interface PackModuleOptions {
/** The Node.js range the package is built for. Defaults to the enclosing workspace's
* `engines.node`, the toolchain the module was built and tested with. */
readonly nodeRange?: string;
/** Reviewed license supplements (ADR 0052). Defaults to the ones this packager ships. */
readonly supplementsDirectory?: string;
}
export interface ModulePackResult {
@@ -254,15 +295,93 @@ async function readTree(root: string, relative: string, files: Map<string, Buffe
}
}
/** The files redistributed for a component, relative to its root: what it copies, or the source
* files of it that were bundled. The content digest a license supplement names is over these. */
async function redistributedFiles(component: {
readonly bundled: BundledPackage;
readonly files: ReadonlySet<string>;
readonly copied?: ReadonlyMap<string, Buffer>;
}): Promise<ReadonlyMap<string, Uint8Array>> {
if (component.copied !== undefined) {
return component.copied;
}
const files = new Map<string, Uint8Array>();
for (const file of component.files) {
const real = await realpath(file).catch(() => file);
files.set(
path.relative(component.bundled.root, real).split(path.sep).join("/"),
await readFile(real),
);
}
return files;
}
interface BackendBundle {
readonly outputs: ReadonlyMap<string, Buffer>;
readonly inputs: readonly string[];
}
const NATIVE_EXTERNALS = "ekklesia-native-externals";
/** The bare specifier's package name: `sharp/x` → `sharp`, `@scope/name/x` → `@scope/name`. */
function specifierPackage(specifier: string): string {
const segments = specifier.split("/");
return specifier.startsWith("@") ? segments.slice(0, 2).join("/") : (segments[0] ?? "");
}
/**
* For a native target: keeps every native or platform-specific third-party package out of the
* bundle, as an import of the package itself, and records it as a root of the copied runtime
* closure. Decided per resolved package, by the same rule that refuses one in an `any` package.
*/
function nativeExternals(
index: BundledPackageIndex,
externals: Map<string, BundledPackage>,
): Plugin {
return {
name: NATIVE_EXTERNALS,
setup(pluginBuild) {
pluginBuild.onResolve({ filter: /^[^./]/ }, async (args) => {
if (
(args.pluginData as { ekklesiaNative?: boolean } | undefined)?.ekklesiaNative ||
isBuiltin(args.path)
) {
return undefined;
}
const resolved = await pluginBuild.resolve(args.path, {
importer: args.importer,
kind: args.kind,
pluginData: { ekklesiaNative: true },
resolveDir: args.resolveDir,
});
if (resolved.errors.length > 0 || resolved.external || resolved.path === "") {
return undefined;
}
const owner = await index.packageOf(resolved.path);
if (
owner === undefined ||
!owner.thirdParty ||
!(await isNativeOrPlatformSpecific(owner))
) {
return undefined;
}
if (specifierPackage(args.path) !== owner.name) {
return {
errors: [{ text: "A native package must be imported by its own package name." }],
};
}
externals.set(owner.name, owner);
return { external: true, path: args.path };
});
},
};
}
async function bundleBackend(
moduleDirectory: string,
manifest: ModuleManifest,
nodeRange: string,
native?: { readonly index: BundledPackageIndex; readonly externals: Map<string, BundledPackage> },
): Promise<BackendBundle> {
const declared = [
manifest.entrypoints.server,
@@ -303,7 +422,10 @@ async function bundleBackend(
minify: false,
outdir: moduleDirectory,
platform: "node",
plugins: [nativeAddonGuard],
plugins:
native === undefined
? [nativeAddonGuard]
: [nativeExternals(native.index, native.externals), nativeAddonGuard],
sourcemap: false,
splitting: true,
target: `node${minimum?.version ?? "24"}`,
@@ -330,7 +452,11 @@ async function bundleBackend(
}
for (const output of Object.values(metafile.outputs)) {
for (const imported of output.imports) {
if (imported.external && !isBuiltin(imported.path)) {
if (
imported.external &&
!isBuiltin(imported.path) &&
!native?.externals.has(specifierPackage(imported.path))
) {
throw new ModulePackError("MODULE_PACK_EXTERNAL_DEPENDENCY", imported.path);
}
}
@@ -349,9 +475,11 @@ async function bundleBackend(
}
export async function packModule(options: PackModuleOptions): Promise<ModulePackResult> {
if (options.target !== MODULE_PACKAGE_ANY_TARGET) {
const nativeTarget: NativeTarget | undefined = NATIVE_TARGETS.get(options.target);
if (options.target !== MODULE_PACKAGE_ANY_TARGET && nativeTarget === undefined) {
throw new ModulePackError("MODULE_PACK_TARGET_UNSUPPORTED", options.target);
}
const target = options.target;
if (!/^(?:[0-9a-f]{40}|[0-9a-f]{64})$/u.test(options.sourceRevision)) {
throw new ModulePackError("MODULE_PACK_REVISION_INVALID");
}
@@ -414,8 +542,53 @@ export async function packModule(options: PackModuleOptions): Promise<ModulePack
throw new ModulePackError("MODULE_PACK_RUNTIME_UNKNOWN");
}
// 1. The backend, bundled with every pure-JavaScript dependency it uses.
const backend = await bundleBackend(moduleDirectory, manifest, nodeRange);
let supplements: Awaited<ReturnType<typeof loadLicenseSupplements>>;
try {
supplements = await loadLicenseSupplements(options.supplementsDirectory);
} catch (error) {
throw new ModulePackError(
"MODULE_PACK_LICENCE_SUPPLEMENT_INVALID",
error instanceof SupplementError ? error.supplement : undefined,
);
}
// 1. The backend, bundled with every pure-JavaScript dependency it uses. For a native target,
// native packages stay outside the bundle and are copied with their runtime closure instead.
const index = new BundledPackageIndex();
const externals = new Map<string, BundledPackage>();
const backend = await bundleBackend(
moduleDirectory,
manifest,
nodeRange,
nativeTarget === undefined ? undefined : { externals, index },
);
let nativePackages: readonly NativePackage[] = [];
let runtimeRequirements: Record<string, unknown> = {};
if (nativeTarget !== undefined) {
try {
nativePackages = await nativeClosure([...externals.values()], nativeTarget);
const { addons, ...requirements } = nativeRuntime(nativePackages, nativeTarget);
if (addons === 0) {
throw new ModulePackError("MODULE_PACK_NATIVE_TARGET_UNNEEDED", options.target);
}
runtimeRequirements = requirements;
} catch (error) {
if (error instanceof NativeError) {
throw new ModulePackError(error.code, error.detail);
}
throw error;
}
for (const nativePackage of nativePackages) {
const engines = nativePackage.manifest.engines;
const engine = isRecord(engines) ? engines.node : undefined;
if (typeof engine === "string" && !subset(nodeRange, engine)) {
throw new ModulePackError(
"MODULE_PACK_NATIVE_NODE_INCOMPATIBLE",
`${nativePackage.name}@${nativePackage.version}`,
);
}
}
}
// 2. The browser artifacts, exactly as built, with what went into each.
let ui: Awaited<ReturnType<typeof analyzeModuleUi>>;
@@ -428,11 +601,15 @@ export async function packModule(options: PackModuleOptions): Promise<ModulePack
);
}
// 3. Every bundled source file traced to its package.
const index = new BundledPackageIndex();
// 3. Every bundled source file traced to its package, and every copied package.
const components = new Map<
string,
{ bundled: Awaited<ReturnType<typeof index.packageOf>>; uses: Set<NoticeUse> }
{
bundled: BundledPackage;
uses: Set<NoticeUse>;
files: Set<string>;
copied?: ReadonlyMap<string, Buffer>;
}
>();
const incorporates = new Map<string, string>();
const drizzleVersions = new Set<string>();
@@ -464,13 +641,32 @@ export async function packModule(options: PackModuleOptions): Promise<ModulePack
const existing = components.get(key);
if (existing) {
existing.uses.add(use);
existing.files.add(file);
} else {
if (use === "backend" && (await isNativeOrPlatformSpecific(bundled))) {
throw new ModulePackError("MODULE_PACK_NATIVE_DEPENDENCY", key);
}
components.set(key, { bundled, uses: new Set([use]) });
components.set(key, { bundled, files: new Set([file]), uses: new Set([use]) });
}
}
for (const nativePackage of nativePackages) {
const key = `${nativePackage.name}@${nativePackage.version}`;
const existing = components.get(key);
const entry = existing ?? {
bundled: {
manifest: nativePackage.manifest,
name: nativePackage.name,
root: nativePackage.root,
thirdParty: true,
version: nativePackage.version,
},
files: new Set<string>(),
uses: new Set<NoticeUse>(),
};
entry.uses.add("node_modules");
entry.copied = nativePackage.files;
components.set(key, entry);
}
const usesDatabaseAbi = manifest.database?.abi === 1;
if (!usesDatabaseAbi && drizzleVersions.size > 1) {
throw new ModulePackError("MODULE_PACK_DATABASE_RUNTIME_CONFLICT");
@@ -482,11 +678,11 @@ export async function packModule(options: PackModuleOptions): Promise<ModulePack
// 4. The payload.
const files = new Map<string, Buffer>();
const add = (relative: string, data: Buffer): void => {
const add = (relative: string, data: Buffer, copied = false): void => {
if (!portable(relative)) {
throw new ModulePackError("MODULE_PACK_PATH_UNSUPPORTED", relative);
}
if (forbidden(relative)) {
if (!copied && forbidden(relative)) {
throw new ModulePackError("MODULE_PACK_FORBIDDEN_FILE", relative);
}
if (files.has(relative) || relative === MODULE_PACKAGE_METADATA_FILENAME) {
@@ -523,6 +719,14 @@ export async function packModule(options: PackModuleOptions): Promise<ModulePack
add(relative, data);
}
}
// A native target's runtime dependencies, as real files, flat under node_modules/<name>. They are
// third-party packages carried as published, so the module's own source and test path rules do
// not apply to their contents; declarations and source maps were already left out.
for (const nativePackage of nativePackages) {
for (const [relative, data] of nativePackage.files) {
add(`node_modules/${nativePackage.name}/${relative}`, data, true);
}
}
add("ekklesia.module.json", manifestBytes);
add(MODULE_PACKAGE_LICENSE_FILENAME, licenseBytes);
add(
@@ -538,7 +742,21 @@ export async function packModule(options: PackModuleOptions): Promise<ModulePack
try {
add(
MODULE_PACKAGE_NOTICES_FILENAME,
Buffer.from(await thirdPartyNotices(components as ReadonlyMap<string, NoticeComponent>)),
Buffer.from(
await thirdPartyNotices(
new Map(
[...components].map(([key, component]): [string, NoticeComponent] => [
key,
{
bundled: component.bundled,
redistributedFiles: () => redistributedFiles(component),
uses: component.uses,
},
]),
),
supplements,
),
),
);
} catch (error) {
if (error instanceof NoticeError) {
@@ -555,8 +773,8 @@ export async function packModule(options: PackModuleOptions): Promise<ModulePack
packageFormat: MODULE_PACKAGE_FORMAT,
module: { id: manifest.id, version: manifest.version },
license,
target: MODULE_PACKAGE_ANY_TARGET,
runtime: { node: nodeRange },
target,
runtime: { node: nodeRange, ...runtimeRequirements },
provenance: {
revision: options.sourceRevision,
...(options.repository === undefined ? {} : { repository: options.repository }),
@@ -590,11 +808,7 @@ export async function packModule(options: PackModuleOptions): Promise<ModulePack
throw error;
}
const digest = sha256(archive);
const fileName = modulePackageArchiveFileName(
manifest.id,
manifest.version,
MODULE_PACKAGE_ANY_TARGET,
);
const fileName = modulePackageArchiveFileName(manifest.id, manifest.version, target);
const outDirectory = path.resolve(options.outDirectory);
const archivePath = path.join(outDirectory, fileName);
const sha256Path = `${archivePath}.sha256`;
@@ -616,7 +830,7 @@ export async function packModule(options: PackModuleOptions): Promise<ModulePack
moduleId: manifest.id,
sha256: digest,
sha256Path,
target: MODULE_PACKAGE_ANY_TARGET,
target,
version: manifest.version,
});
}
+280
View File
@@ -0,0 +1,280 @@
import { createHash } from "node:crypto";
import { lstat, readdir, readFile } from "node:fs/promises";
import path from "node:path";
import { fileURLToPath } from "node:url";
/**
* Reviewed license supplements (ADR 0052).
*
* When a redistributed dependency's own package omits licence or notice material that
* redistribution requires, the canonical packager may add material from a checked-in, reviewed
* supplement. A supplement is data, not code: nothing here knows about any particular package.
*
* A supplement lives at `license-supplements/<package name>/<version>/` beside this package. It
* holds `supplement.json` and the text files that file lists:
*
* ```json
* {
* "supplementFormat": 1,
* "package": {
* "name": "…", "version": "…", "license": "<SPDX the package declares>",
* "contentDigest": "ekklesia-content-v1:sha256:<hex>"
* },
* "statement": "<what the supplement covers, printed with it>",
* "components": [
* {
* "name": "…", "version": "…", "license": "<as stated by its source>",
* "source": "<where the material came from>",
* "material": [{ "title": "LICENSE", "text": "texts/<sha256 of the text>.txt" }]
* }
* ]
* }
* ```
*
* Every text file is named by its own SHA-256. The loader checks every name against its bytes, and
* refuses a file the supplement does not list, so a supplement can only change through review.
*
* Repository review of a supplement is project and code review of a build input. It is not legal
* review, and carrying a supplement does not establish that any redistribution obligation is met.
*/
export const CONTENT_DIGEST_ALGORITHM = "ekklesia-content-v1";
export const SUPPLEMENT_FORMAT = 1;
/**
* The content digest a supplement is matched against (ADR 0052), over exactly the files the
* packager redistributes for a package, by path relative to the package root.
*
* 1. Each path is normalized: `/` separators, no leading `./`, no empty, `.`, or `..` segment.
* 2. Paths are sorted by their UTF-8 bytes.
* 3. For each file, in that order, this line is appended to a manifest:
* `<path> NUL <decimal byte length> NUL <lowercase hex SHA-256 of the bytes> LF`.
* 4. The digest is `ekklesia-content-v1:sha256:` followed by the lowercase hex SHA-256 of the
* manifest's UTF-8 bytes.
*
* Nothing else is an input: not timestamps, ownership, permissions, the directory the package was
* installed in, the package manager, or the order files were read or extracted in.
*/
export function contentDigest(files: ReadonlyMap<string, Uint8Array>): string {
const entries = [...files].map(([relative, data]) => {
const normalized = relative.split(path.sep).join("/").replace(/^\.\//u, "");
if (
normalized.split("/").some((segment) => segment === "" || segment === "." || segment === "..")
) {
throw new Error("A redistributed path must be relative and normalized.");
}
return [Buffer.from(normalized, "utf8"), data] as const;
});
entries.sort(([left], [right]) => Buffer.compare(left, right));
const manifest = createHash("sha256");
for (const [name, data] of entries) {
const digest = createHash("sha256").update(data).digest("hex");
manifest.update(name);
manifest.update(`\0${data.length}\0${digest}\n`);
}
return `${CONTENT_DIGEST_ALGORITHM}:sha256:${manifest.digest("hex")}`;
}
export interface SupplementMaterial {
readonly title: string;
/** The text, exactly as reviewed. */
readonly text: string;
/** SHA-256 of the text's bytes, so identical texts are recognized. */
readonly sha256: string;
}
export interface SupplementComponent {
readonly name: string;
readonly version: string;
readonly license: string;
readonly source: string;
readonly material: readonly SupplementMaterial[];
}
export interface LicenseSupplement {
readonly name: string;
readonly version: string;
readonly license: string;
readonly contentDigest: string;
readonly statement: string;
readonly components: readonly SupplementComponent[];
}
export class SupplementError extends Error {
readonly supplement: string;
constructor(supplement: string) {
super("A license supplement is malformed or does not match its reviewed files.");
this.name = "SupplementError";
this.supplement = supplement;
}
}
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === "object" && value !== null && !Array.isArray(value);
}
function nonEmptyString(value: unknown): value is string {
return typeof value === "string" && value.trim() !== "";
}
function strictKeys(value: Record<string, unknown>, keys: readonly string[]): boolean {
return Object.keys(value).every((key) => keys.includes(key)) && keys.every((key) => key in value);
}
export const DEFAULT_SUPPLEMENTS_DIRECTORY = path.resolve(
path.dirname(fileURLToPath(import.meta.url)),
"..",
"license-supplements",
);
async function listFiles(root: string, relative = ""): Promise<string[]> {
const directory = relative === "" ? root : path.join(root, relative);
const found: string[] = [];
for (const entry of await readdir(directory, { withFileTypes: true })) {
const childRelative = relative === "" ? entry.name : `${relative}/${entry.name}`;
const stats = await lstat(path.join(directory, entry.name));
if (stats.isDirectory()) {
found.push(...(await listFiles(root, childRelative)));
} else if (stats.isFile()) {
found.push(childRelative);
} else {
throw new SupplementError(childRelative);
}
}
return found;
}
async function loadOne(directory: string, label: string): Promise<LicenseSupplement> {
let raw: unknown;
try {
raw = JSON.parse(await readFile(path.join(directory, "supplement.json"), "utf8")) as unknown;
} catch {
throw new SupplementError(label);
}
if (
!isRecord(raw) ||
!strictKeys(raw, ["supplementFormat", "package", "statement", "components"]) ||
raw.supplementFormat !== SUPPLEMENT_FORMAT ||
!isRecord(raw.package) ||
!strictKeys(raw.package, ["name", "version", "license", "contentDigest"]) ||
!nonEmptyString(raw.statement) ||
!Array.isArray(raw.components) ||
raw.components.length === 0
) {
throw new SupplementError(label);
}
const { contentDigest: digest, license, name, version } = raw.package;
if (
!nonEmptyString(name) ||
!nonEmptyString(version) ||
!nonEmptyString(license) ||
typeof digest !== "string" ||
!new RegExp(`^${CONTENT_DIGEST_ALGORITHM}:sha256:[0-9a-f]{64}$`, "u").test(digest) ||
`${name}/${version}` !== label
) {
throw new SupplementError(label);
}
const listed = new Set<string>();
const components: SupplementComponent[] = [];
for (const component of raw.components) {
if (
!isRecord(component) ||
!strictKeys(component, ["name", "version", "license", "source", "material"]) ||
!nonEmptyString(component.name) ||
!nonEmptyString(component.version) ||
!nonEmptyString(component.license) ||
!nonEmptyString(component.source) ||
!Array.isArray(component.material)
) {
throw new SupplementError(label);
}
const material: SupplementMaterial[] = [];
for (const item of component.material) {
if (
!isRecord(item) ||
!strictKeys(item, ["title", "text"]) ||
!nonEmptyString(item.title) ||
typeof item.text !== "string" ||
!/^texts\/[0-9a-f]{64}\.txt$/u.test(item.text)
) {
throw new SupplementError(label);
}
const expected = item.text.slice("texts/".length, -".txt".length);
let bytes: Buffer;
try {
bytes = await readFile(path.join(directory, ...item.text.split("/")));
} catch {
throw new SupplementError(label);
}
if (bytes.length === 0 || createHash("sha256").update(bytes).digest("hex") !== expected) {
throw new SupplementError(label);
}
listed.add(item.text);
material.push(
Object.freeze({ sha256: expected, text: bytes.toString("utf8"), title: item.title }),
);
}
components.push(
Object.freeze({
license: component.license,
material: Object.freeze(material),
name: component.name,
source: component.source,
version: component.version,
}),
);
}
// Only reviewed files: supplement.json, the texts it lists, and a human-readable provenance note.
for (const file of await listFiles(directory)) {
if (file !== "supplement.json" && file !== "PROVENANCE.md" && !listed.has(file)) {
throw new SupplementError(label);
}
}
return Object.freeze({
components: Object.freeze(components),
contentDigest: digest,
license,
name,
statement: raw.statement,
version,
});
}
/**
* Loads every supplement under `directory`, indexed by package name. A directory that does not exist
* holds none. Any malformed supplement fails the whole load: a broken reviewed input is never
* skipped.
*/
export async function loadLicenseSupplements(
directory: string = DEFAULT_SUPPLEMENTS_DIRECTORY,
): Promise<ReadonlyMap<string, readonly LicenseSupplement[]>> {
const index = new Map<string, LicenseSupplement[]>();
let files: string[];
try {
files = await listFiles(directory);
} catch (error) {
if (isRecord(error) && error.code === "ENOENT") {
return index;
}
throw error;
}
const roots = files
.filter((file) => file.endsWith("/supplement.json"))
.map((file) => file.slice(0, -"/supplement.json".length))
.sort();
for (const label of roots) {
const supplement = await loadOne(path.join(directory, ...label.split("/")), label);
const list = index.get(supplement.name) ?? [];
list.push(supplement);
index.set(supplement.name, list);
}
// Every file must belong to some supplement directory.
for (const file of files) {
if (!roots.some((root) => file.startsWith(`${root}/`))) {
throw new SupplementError(file);
}
}
return index;
}