feat: native linux-x64-glibc module packages and reviewed license supplements (ADR 0049, ADR 0052)
CI / verify (pull_request) Successful in 1m6s
CI / infrastructure-smoke (pull_request) Successful in 27s
CI / migration-smoke (pull_request) Successful in 1m54s
CI / image-build (pull_request) Successful in 29s
CI / deploy-smoke (same-origin) (pull_request) Successful in 43s
CI / deploy-smoke (split) (pull_request) Successful in 1m30s
CI / verify (pull_request) Successful in 1m6s
CI / infrastructure-smoke (pull_request) Successful in 27s
CI / migration-smoke (pull_request) Successful in 1m54s
CI / image-build (pull_request) Successful in 29s
CI / deploy-smoke (same-origin) (pull_request) Successful in 43s
CI / deploy-smoke (split) (pull_request) Successful in 1m30s
Implements ADR 0049's deferred native target and ADR 0052's reviewed license supplements. The proof case is a real module using sharp. packageFormat stays 1, and `any` is unchanged. Packager (@ekklesia/module-pack): - --target linux-x64-glibc. Native or platform-specific packages, found by the same generic rule that refuses them in `any`, stay outside the bundle. - Each native package's production closure is walked through its own dependencies, installed optionalDependencies that match the target, and required peers. It is copied as real files flat under node_modules/<name>, with no links, dev dependencies, declarations, source maps, or other platforms' packages. Missing or foreign-platform required dependencies, version conflicts, links, foreign machines, and Node.js range conflicts fail closed. - runtime.nodeApi, nodeAbi, and libcVersion are read from the binaries by a small ELF reader: the addon's node_api_module_get_api_version_v1 and the newest GLIBC_ symbol version needed. No shelling out. A native target with no addon is refused. - ADR 0052 supplements: package-provided material stays primary. An exact match on name, version, and ekklesia-content-v1 digest (sorted normalized paths with each file's size and SHA-256; no metadata) adds labelled material after the package's own. Any mismatch, a declared-licence conflict, or a malformed or unreviewed supplement file fails. Missing material with no supplement still fails. Supplement for @img/[email protected]: - 184 components and 143 unique texts, assembled from the exact upstream sources sharp-libvips v1.3.3 builds linux-x64 from: 27 C/C++ libraries, glib's gvdb, librsvg's Rust crates (reproduced with cargo tree), the Rust standard library at the rustc commit recorded in the binary, plus the LGPL-3.0, GPL-3.0, and MPL-2.0 texts. Each source is recorded with its SHA-256 in PROVENANCE.md. - It pins digest ekklesia-content-v1:sha256:50647b0b…, which is identical from the installed copy and from the registry tarball. - LGPL/MPL source-availability determinations stay open, as ADR 0052 states. Nothing claims they are met. Verifier (Core): - linux-x64-glibc is accepted only on a host whose platform, arch, glibc family and version, and Node-API version (from Node itself) match. - Every ELF file must be x86-64; every .node must be an ELF object. - `any` packages now also refuse node_modules/ and ELF files. - Contract: native metadata must state nodeApi or nodeAbi, and Linux targets libcVersion (ADR 0049). Proof: - A checked-in AGPL native fixture packs byte-identically from unrelated copies (8.86 MB) and verifies with Core's verifier. - Extracted elsewhere, it runs a sharp resize and PNG round trip in an empty-environment process. Every resolution stays in the package, and only the package's addon and libvips plus host system libraries load. Signage is untouched. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
f53506b3f0
commit
44d385cd7f
@@ -8,17 +8,17 @@ import { ModulePackError, type PackModuleOptions, packModule } from "./pack-modu
|
||||
/**
|
||||
* `ekklesia-pack-module` — the canonical Ekklesia module packager (ADR 0049).
|
||||
*
|
||||
* ekklesia-pack-module --module <built module directory> --target any --out <directory>
|
||||
* ekklesia-pack-module --module <built module directory> --target <any|linux-x64-glibc> --out <directory>
|
||||
* --license-file <file> --source-revision <full commit id> [--repository <url>]
|
||||
*
|
||||
* Writes `<id>-<version>-any.tar.gz` and its `.sha256` line to the output directory, and nothing
|
||||
* Writes `<id>-<version>-<target>.tar.gz` and its `.sha256` line to the output directory, and nothing
|
||||
* else anywhere. Run it in the module's CI after building the module.
|
||||
*/
|
||||
|
||||
class PackArgumentsError extends Error {}
|
||||
|
||||
const USAGE =
|
||||
"Usage: ekklesia-pack-module --module <directory> --target any --out <directory> --license-file <file> --source-revision <commit> [--repository <url>]";
|
||||
"Usage: ekklesia-pack-module --module <directory> --target <any|linux-x64-glibc> --out <directory> --license-file <file> --source-revision <commit> [--repository <url>]";
|
||||
|
||||
const OPTIONS: Readonly<Record<string, keyof PackModuleOptions>> = Object.freeze({
|
||||
"--license-file": "licenseFile",
|
||||
|
||||
@@ -0,0 +1,188 @@
|
||||
/**
|
||||
* Just enough of the ELF format to describe a native file the packager is about to copy (ADR 0049):
|
||||
* which machine it was built for, which shared libraries and C library symbol versions it needs, and
|
||||
* which Node.js addon interface it exports. Read from the file's own bytes, never by running it or
|
||||
* shelling out to a system tool.
|
||||
*
|
||||
* Only 64-bit little-endian ELF is understood, because that is every target this packager produces.
|
||||
* Anything else that starts with the ELF magic is reported as unsupported rather than guessed at.
|
||||
*/
|
||||
|
||||
export class ElfFormatError extends Error {
|
||||
constructor() {
|
||||
super("The file is not a 64-bit little-endian ELF object this packager can read.");
|
||||
this.name = "ElfFormatError";
|
||||
}
|
||||
}
|
||||
|
||||
/** `EM_X86_64` in the ELF machine field. */
|
||||
export const ELF_MACHINE_X86_64 = 62;
|
||||
|
||||
export interface ElfDescription {
|
||||
readonly machine: number;
|
||||
/** `DT_NEEDED` entries: the shared libraries the file links against. */
|
||||
readonly needed: readonly string[];
|
||||
/** Symbol version names the file requires, for example `GLIBC_2.28`. */
|
||||
readonly requiredVersions: readonly string[];
|
||||
/** Names of the dynamic symbols the file itself defines. */
|
||||
readonly exported: ReadonlySet<string>;
|
||||
/** The value a Node-API addon's `node_api_module_get_api_version_v1` returns, when it exports
|
||||
* that function and its body is the plain constant return every compiler emits for it.
|
||||
* `undefined` when the function is absent; `"unrecognized"` when its body is something else. */
|
||||
readonly nodeApiModuleVersion: number | "unrecognized" | undefined;
|
||||
}
|
||||
|
||||
const SHT_NOBITS = 8;
|
||||
const SHT_DYNSYM = 11;
|
||||
const SHT_DYNAMIC = 6;
|
||||
const SHT_GNU_VERNEED = 0x6ffffffe;
|
||||
const DT_NULL = 0;
|
||||
const DT_NEEDED = 1;
|
||||
|
||||
export function isElf(data: Uint8Array): boolean {
|
||||
return (
|
||||
data.length >= 4 && data[0] === 0x7f && data[1] === 0x45 && data[2] === 0x4c && data[3] === 0x46
|
||||
);
|
||||
}
|
||||
|
||||
interface Section {
|
||||
readonly type: number;
|
||||
readonly address: number;
|
||||
readonly offset: number;
|
||||
readonly size: number;
|
||||
readonly link: number;
|
||||
readonly entrySize: number;
|
||||
}
|
||||
|
||||
export function describeElf(input: Uint8Array): ElfDescription {
|
||||
const data = Buffer.from(input.buffer, input.byteOffset, input.byteLength);
|
||||
if (!isElf(data) || data.length < 64 || data[4] !== 2 || data[5] !== 1) {
|
||||
throw new ElfFormatError();
|
||||
}
|
||||
const u16 = (offset: number): number => {
|
||||
if (offset < 0 || offset + 2 > data.length) throw new ElfFormatError();
|
||||
return data.readUInt16LE(offset);
|
||||
};
|
||||
const u32 = (offset: number): number => {
|
||||
if (offset < 0 || offset + 4 > data.length) throw new ElfFormatError();
|
||||
return data.readUInt32LE(offset);
|
||||
};
|
||||
const u64 = (offset: number): number => {
|
||||
if (offset < 0 || offset + 8 > data.length) throw new ElfFormatError();
|
||||
const value = data.readBigUInt64LE(offset);
|
||||
if (value > BigInt(Number.MAX_SAFE_INTEGER)) throw new ElfFormatError();
|
||||
return Number(value);
|
||||
};
|
||||
const string = (tableOffset: number, tableSize: number, index: number): string => {
|
||||
if (index >= tableSize) throw new ElfFormatError();
|
||||
const start = tableOffset + index;
|
||||
const end = data.indexOf(0, start);
|
||||
if (end === -1 || end > tableOffset + tableSize) throw new ElfFormatError();
|
||||
return data.toString("latin1", start, end);
|
||||
};
|
||||
|
||||
const machine = u16(18);
|
||||
const sectionOffset = u64(40);
|
||||
const sectionEntrySize = u16(58);
|
||||
const sectionCount = u16(60);
|
||||
if (sectionEntrySize < 64 || sectionCount === 0) {
|
||||
throw new ElfFormatError();
|
||||
}
|
||||
const sections: Section[] = [];
|
||||
for (let index = 0; index < sectionCount; index += 1) {
|
||||
const base = sectionOffset + index * sectionEntrySize;
|
||||
sections.push({
|
||||
address: u64(base + 16),
|
||||
entrySize: u64(base + 56),
|
||||
link: u32(base + 40),
|
||||
offset: u64(base + 24),
|
||||
size: u64(base + 32),
|
||||
type: u32(base + 4),
|
||||
});
|
||||
}
|
||||
const linkedStrings = (section: Section): { offset: number; size: number } => {
|
||||
const table = sections[section.link];
|
||||
if (table === undefined || table.offset + table.size > data.length) throw new ElfFormatError();
|
||||
return { offset: table.offset, size: table.size };
|
||||
};
|
||||
|
||||
const needed: string[] = [];
|
||||
const requiredVersions: string[] = [];
|
||||
const exported = new Set<string>();
|
||||
let apiVersionAddress: number | undefined;
|
||||
|
||||
for (const section of sections) {
|
||||
if (section.type === SHT_DYNAMIC) {
|
||||
const strings = linkedStrings(section);
|
||||
for (let entry = section.offset; entry + 16 <= section.offset + section.size; entry += 16) {
|
||||
const tag = u64(entry);
|
||||
if (tag === DT_NULL) break;
|
||||
if (tag === DT_NEEDED) needed.push(string(strings.offset, strings.size, u64(entry + 8)));
|
||||
}
|
||||
} else if (section.type === SHT_DYNSYM) {
|
||||
const strings = linkedStrings(section);
|
||||
const size = section.entrySize === 0 ? 24 : section.entrySize;
|
||||
for (let entry = section.offset; entry + 24 <= section.offset + section.size; entry += size) {
|
||||
const nameIndex = u32(entry);
|
||||
const sectionIndex = u16(entry + 6);
|
||||
if (nameIndex === 0 || sectionIndex === 0) continue;
|
||||
const name = string(strings.offset, strings.size, nameIndex);
|
||||
exported.add(name);
|
||||
if (name === "node_api_module_get_api_version_v1") {
|
||||
apiVersionAddress = u64(entry + 8);
|
||||
}
|
||||
}
|
||||
} else if (section.type === SHT_GNU_VERNEED) {
|
||||
const strings = linkedStrings(section);
|
||||
let need = section.offset;
|
||||
for (let guard = 0; guard < 4096; guard += 1) {
|
||||
const count = u16(need + 2);
|
||||
let aux = need + u32(need + 8);
|
||||
for (let index = 0; index < count; index += 1) {
|
||||
requiredVersions.push(string(strings.offset, strings.size, u32(aux + 8)));
|
||||
const next = u32(aux + 12);
|
||||
if (next === 0) break;
|
||||
aux += next;
|
||||
}
|
||||
const next = u32(need + 12);
|
||||
if (next === 0) break;
|
||||
need += next;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let nodeApiModuleVersion: number | "unrecognized" | undefined;
|
||||
if (apiVersionAddress !== undefined) {
|
||||
const text = sections.find(
|
||||
(section) =>
|
||||
section.type !== SHT_NOBITS &&
|
||||
section.address <= apiVersionAddress &&
|
||||
apiVersionAddress < section.address + section.size,
|
||||
);
|
||||
nodeApiModuleVersion = "unrecognized";
|
||||
if (text !== undefined) {
|
||||
let offset = text.offset + (apiVersionAddress - text.address);
|
||||
// An optional `endbr64` (Intel CET), then `mov eax, imm32; ret`: what every compiler emits for
|
||||
// a function that returns the NAPI_VERSION constant. Anything else is not guessed at.
|
||||
if (
|
||||
data[offset] === 0xf3 &&
|
||||
data[offset + 1] === 0x0f &&
|
||||
data[offset + 2] === 0x1e &&
|
||||
data[offset + 3] === 0xfa
|
||||
) {
|
||||
offset += 4;
|
||||
}
|
||||
if (data[offset] === 0xb8 && data[offset + 5] === 0xc3) {
|
||||
nodeApiModuleVersion = u32(offset + 1);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return Object.freeze({
|
||||
exported,
|
||||
machine,
|
||||
needed: Object.freeze(needed),
|
||||
nodeApiModuleVersion,
|
||||
requiredVersions: Object.freeze(requiredVersions),
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,357 @@
|
||||
import { lstat, readdir, readFile, realpath } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { describeElf, ELF_MACHINE_X86_64, ElfFormatError, isElf } from "./elf.js";
|
||||
|
||||
/**
|
||||
* Native targets of Ekklesia Module Package v1 (ADR 0049): what a `<platform>-<arch>[-<libc>]`
|
||||
* package may contain beyond bundled JavaScript, and how the packager establishes it.
|
||||
*
|
||||
* A native or platform-specific dependency cannot be bundled. The packager keeps it outside the
|
||||
* bundle and copies it, with exactly its own production runtime dependencies, as real files flat
|
||||
* under `node_modules/<name>`. Nothing is installed, linked, or downloaded. Which packages qualify is
|
||||
* decided generically, from each package's own metadata and files, never from a list of names.
|
||||
*/
|
||||
|
||||
export interface NativeTarget {
|
||||
readonly id: string;
|
||||
readonly os: string;
|
||||
readonly cpu: string;
|
||||
readonly libc: string;
|
||||
readonly elfMachine: number;
|
||||
/** Shared libraries whose presence means a binary was linked against another C library. */
|
||||
readonly foreignLibc: RegExp;
|
||||
readonly libcVersionPrefix: string;
|
||||
}
|
||||
|
||||
/** The native targets this packager produces. Every other target is refused. */
|
||||
export const NATIVE_TARGETS: ReadonlyMap<string, NativeTarget> = new Map([
|
||||
[
|
||||
"linux-x64-glibc",
|
||||
Object.freeze({
|
||||
cpu: "x64",
|
||||
elfMachine: ELF_MACHINE_X86_64,
|
||||
foreignLibc: /^(?:libc\.musl-|ld-musl-)/u,
|
||||
id: "linux-x64-glibc",
|
||||
libc: "glibc",
|
||||
libcVersionPrefix: "GLIBC_",
|
||||
os: "linux",
|
||||
}),
|
||||
],
|
||||
]);
|
||||
|
||||
export type NativeErrorCode =
|
||||
| "MODULE_PACK_NATIVE_DEPENDENCY_MISSING"
|
||||
| "MODULE_PACK_NATIVE_LAYOUT_CONFLICT"
|
||||
| "MODULE_PACK_NATIVE_PACKAGE_INVALID"
|
||||
| "MODULE_PACK_NATIVE_RUNTIME_UNKNOWN"
|
||||
| "MODULE_PACK_NATIVE_TARGET_MISMATCH";
|
||||
|
||||
export class NativeError extends Error {
|
||||
readonly code: NativeErrorCode;
|
||||
readonly detail: string;
|
||||
|
||||
constructor(code: NativeErrorCode, detail: string) {
|
||||
super(code);
|
||||
this.name = "NativeError";
|
||||
this.code = code;
|
||||
this.detail = detail;
|
||||
}
|
||||
}
|
||||
|
||||
function isRecord(value: unknown): value is Record<string, unknown> {
|
||||
return typeof value === "object" && value !== null && !Array.isArray(value);
|
||||
}
|
||||
|
||||
function compare(left: string, right: string): number {
|
||||
return left < right ? -1 : left > right ? 1 : 0;
|
||||
}
|
||||
|
||||
/** npm's `os`, `cpu`, and `libc` fields: absent means any; entries are names or `!`-negations. */
|
||||
function fieldAllows(field: unknown, value: string): boolean {
|
||||
if (!Array.isArray(field) || field.length === 0) {
|
||||
return true;
|
||||
}
|
||||
const entries = field.filter((entry): entry is string => typeof entry === "string");
|
||||
if (entries.includes(`!${value}`)) {
|
||||
return false;
|
||||
}
|
||||
const positive = entries.filter((entry) => !entry.startsWith("!"));
|
||||
return positive.length === 0 || positive.includes(value);
|
||||
}
|
||||
|
||||
export function packageSupportsTarget(
|
||||
manifest: Readonly<Record<string, unknown>>,
|
||||
target: NativeTarget,
|
||||
): boolean {
|
||||
return (
|
||||
fieldAllows(manifest.os, target.os) &&
|
||||
fieldAllows(manifest.cpu, target.cpu) &&
|
||||
fieldAllows(manifest.libc, target.libc)
|
||||
);
|
||||
}
|
||||
|
||||
async function readManifest(directory: string): Promise<Record<string, unknown> | undefined> {
|
||||
try {
|
||||
const parsed = JSON.parse(
|
||||
await readFile(path.join(directory, "package.json"), "utf8"),
|
||||
) as unknown;
|
||||
return isRecord(parsed) ? parsed : undefined;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
/** The installed package `name` resolves to from `fromDirectory`, by Node's own directory walk. */
|
||||
async function resolveInstalled(fromDirectory: string, name: string): Promise<string | undefined> {
|
||||
let directory = fromDirectory;
|
||||
for (;;) {
|
||||
if (path.basename(directory) !== "node_modules") {
|
||||
const candidate = path.join(directory, "node_modules", ...name.split("/"));
|
||||
if (await readManifest(candidate)) {
|
||||
return realpath(candidate);
|
||||
}
|
||||
}
|
||||
const parent = path.dirname(directory);
|
||||
if (parent === directory) {
|
||||
return undefined;
|
||||
}
|
||||
directory = parent;
|
||||
}
|
||||
}
|
||||
|
||||
/** Declarations and source maps never ship (ADR 0049); everything else a package publishes does,
|
||||
* because the packager cannot know which of a third-party package's files it reads at run time. */
|
||||
function excludedFromCopy(relative: string): boolean {
|
||||
return /\.(?:d\.ts|d\.mts|d\.cts|map)$/u.test(relative);
|
||||
}
|
||||
|
||||
/** Every regular file a package publishes, by path relative to its root, nested `node_modules`
|
||||
* excluded (those are separate packages). A link or special file fails: none may enter a package. */
|
||||
async function readPackageFiles(root: string, packageId: string): Promise<Map<string, Buffer>> {
|
||||
const files = new Map<string, Buffer>();
|
||||
const walk = async (relative: string): Promise<void> => {
|
||||
const directory = relative === "" ? root : path.join(root, relative);
|
||||
const entries = (await readdir(directory, { withFileTypes: true })).sort((left, right) =>
|
||||
compare(left.name, right.name),
|
||||
);
|
||||
for (const entry of entries) {
|
||||
const childRelative = relative === "" ? entry.name : `${relative}/${entry.name}`;
|
||||
const stats = await lstat(path.join(directory, entry.name));
|
||||
if (stats.isDirectory()) {
|
||||
if (entry.name !== "node_modules") {
|
||||
await walk(childRelative);
|
||||
}
|
||||
} else if (stats.isFile()) {
|
||||
if (!excludedFromCopy(childRelative)) {
|
||||
files.set(childRelative, await readFile(path.join(directory, entry.name)));
|
||||
}
|
||||
} else {
|
||||
throw new NativeError("MODULE_PACK_NATIVE_PACKAGE_INVALID", packageId);
|
||||
}
|
||||
}
|
||||
};
|
||||
await walk("");
|
||||
return files;
|
||||
}
|
||||
|
||||
/** A package copied into the module package's `node_modules`. */
|
||||
export interface NativePackage {
|
||||
readonly name: string;
|
||||
readonly version: string;
|
||||
readonly root: string;
|
||||
readonly manifest: Readonly<Record<string, unknown>>;
|
||||
/** The files copied, by path relative to the package root: exactly what is redistributed. */
|
||||
readonly files: ReadonlyMap<string, Buffer>;
|
||||
}
|
||||
|
||||
export interface NativeRuntime {
|
||||
readonly nodeApi?: number;
|
||||
readonly nodeAbi?: string;
|
||||
readonly libcVersion?: string;
|
||||
}
|
||||
|
||||
function dependencyNames(manifest: Readonly<Record<string, unknown>>): {
|
||||
readonly required: readonly string[];
|
||||
readonly optional: readonly string[];
|
||||
} {
|
||||
const names = (field: unknown): string[] => (isRecord(field) ? Object.keys(field) : []);
|
||||
const optional = new Set(names(manifest.optionalDependencies));
|
||||
const peerMeta = isRecord(manifest.peerDependenciesMeta) ? manifest.peerDependenciesMeta : {};
|
||||
for (const peer of names(manifest.peerDependencies)) {
|
||||
const meta = peerMeta[peer];
|
||||
if (isRecord(meta) && meta.optional === true) {
|
||||
optional.add(peer);
|
||||
}
|
||||
}
|
||||
const required = new Set(
|
||||
[...names(manifest.dependencies), ...names(manifest.peerDependencies)].filter(
|
||||
(name) => !optional.has(name),
|
||||
),
|
||||
);
|
||||
return {
|
||||
optional: [...optional].sort(compare),
|
||||
required: [...required].sort(compare),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The runtime closure of the native roots for `target`: each root, its production `dependencies`,
|
||||
* its installed `optionalDependencies` whose `os`/`cpu`/`libc` admit the target, and its required
|
||||
* peers, recursively. Development dependencies are never followed. A required dependency that is
|
||||
* missing or built for another platform fails; an optional one for another platform is left out.
|
||||
* Two different versions of one name cannot share the flat layout and fail.
|
||||
*/
|
||||
export async function nativeClosure(
|
||||
roots: readonly { readonly name: string; readonly root: string }[],
|
||||
target: NativeTarget,
|
||||
): Promise<readonly NativePackage[]> {
|
||||
const byName = new Map<string, NativePackage>();
|
||||
const queue = [...roots].sort((left, right) => compare(left.name, right.name));
|
||||
for (let next = queue.shift(); next !== undefined; next = queue.shift()) {
|
||||
const root = await realpath(next.root);
|
||||
const manifest = await readManifest(root);
|
||||
if (
|
||||
manifest === undefined ||
|
||||
typeof manifest.name !== "string" ||
|
||||
typeof manifest.version !== "string" ||
|
||||
manifest.name !== next.name
|
||||
) {
|
||||
throw new NativeError("MODULE_PACK_NATIVE_PACKAGE_INVALID", next.name);
|
||||
}
|
||||
const packageId = `${manifest.name}@${manifest.version}`;
|
||||
const existing = byName.get(manifest.name);
|
||||
if (existing !== undefined) {
|
||||
if (existing.version !== manifest.version) {
|
||||
throw new NativeError("MODULE_PACK_NATIVE_LAYOUT_CONFLICT", manifest.name);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (!packageSupportsTarget(manifest, target)) {
|
||||
throw new NativeError("MODULE_PACK_NATIVE_TARGET_MISMATCH", packageId);
|
||||
}
|
||||
byName.set(
|
||||
manifest.name,
|
||||
Object.freeze({
|
||||
files: await readPackageFiles(root, packageId),
|
||||
manifest: Object.freeze(manifest),
|
||||
name: manifest.name,
|
||||
root,
|
||||
version: manifest.version,
|
||||
}),
|
||||
);
|
||||
const { optional, required } = dependencyNames(manifest);
|
||||
for (const name of required) {
|
||||
const resolved = await resolveInstalled(root, name);
|
||||
if (resolved === undefined) {
|
||||
throw new NativeError("MODULE_PACK_NATIVE_DEPENDENCY_MISSING", `${packageId} -> ${name}`);
|
||||
}
|
||||
queue.push({ name, root: resolved });
|
||||
}
|
||||
for (const name of optional) {
|
||||
const resolved = await resolveInstalled(root, name);
|
||||
const dependency = resolved === undefined ? undefined : await readManifest(resolved);
|
||||
if (resolved !== undefined && dependency && packageSupportsTarget(dependency, target)) {
|
||||
queue.push({ name, root: resolved });
|
||||
}
|
||||
}
|
||||
}
|
||||
return [...byName.values()].sort((left, right) => compare(left.name, right.name));
|
||||
}
|
||||
|
||||
/** Compares `GLIBC_2.28`-style versions numerically. */
|
||||
function compareVersions(left: readonly number[], right: readonly number[]): number {
|
||||
for (let index = 0; index < Math.max(left.length, right.length); index += 1) {
|
||||
const difference = (left[index] ?? 0) - (right[index] ?? 0);
|
||||
if (difference !== 0) {
|
||||
return difference;
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* The runtime requirements of the copied native files, read from the binaries themselves (ADR 0049):
|
||||
*
|
||||
* - every ELF file must be built for the target's machine and must not link another C library;
|
||||
* - every `.node` file must be an ELF Node.js addon;
|
||||
* - `nodeApi` is the highest Node-API version a Node-API addon declares through its exported
|
||||
* `node_api_module_get_api_version_v1` (an addon without that export is version 8, as Node.js
|
||||
* itself treats it); an ABI-specific addon records its `node_register_module_v<ABI>` as `nodeAbi`;
|
||||
* - `libcVersion` is the highest C library symbol version any file requires.
|
||||
*
|
||||
* Anything that cannot be established exactly fails closed.
|
||||
*/
|
||||
export function nativeRuntime(
|
||||
packages: readonly NativePackage[],
|
||||
target: NativeTarget,
|
||||
): NativeRuntime & { readonly addons: number } {
|
||||
let nodeApi: number | undefined;
|
||||
let nodeAbi: string | undefined;
|
||||
let libc: number[] | undefined;
|
||||
let addons = 0;
|
||||
for (const nativePackage of packages) {
|
||||
for (const [relative, data] of nativePackage.files) {
|
||||
const fileId = `${nativePackage.name}@${nativePackage.version}/${relative}`;
|
||||
const addon = relative.toLowerCase().endsWith(".node");
|
||||
if (!isElf(data)) {
|
||||
if (addon) {
|
||||
throw new NativeError("MODULE_PACK_NATIVE_TARGET_MISMATCH", fileId);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
let elf: ReturnType<typeof describeElf>;
|
||||
try {
|
||||
elf = describeElf(data);
|
||||
} catch (error) {
|
||||
if (error instanceof ElfFormatError) {
|
||||
throw new NativeError("MODULE_PACK_NATIVE_TARGET_MISMATCH", fileId);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
if (
|
||||
elf.machine !== target.elfMachine ||
|
||||
elf.needed.some((name) => target.foreignLibc.test(name))
|
||||
) {
|
||||
throw new NativeError("MODULE_PACK_NATIVE_TARGET_MISMATCH", fileId);
|
||||
}
|
||||
for (const version of elf.requiredVersions) {
|
||||
if (!version.startsWith(target.libcVersionPrefix)) {
|
||||
continue;
|
||||
}
|
||||
const parts = version.slice(target.libcVersionPrefix.length).split(".").map(Number);
|
||||
if (parts.some((part) => !Number.isInteger(part))) {
|
||||
continue;
|
||||
}
|
||||
if (libc === undefined || compareVersions(parts, libc) > 0) {
|
||||
libc = parts;
|
||||
}
|
||||
}
|
||||
if (!addon) {
|
||||
continue;
|
||||
}
|
||||
addons += 1;
|
||||
if (elf.exported.has("napi_register_module_v1")) {
|
||||
const declared = elf.nodeApiModuleVersion;
|
||||
if (declared === "unrecognized" || (typeof declared === "number" && declared < 1)) {
|
||||
throw new NativeError("MODULE_PACK_NATIVE_RUNTIME_UNKNOWN", fileId);
|
||||
}
|
||||
const version = declared ?? 8;
|
||||
nodeApi = nodeApi === undefined ? version : Math.max(nodeApi, version);
|
||||
} else {
|
||||
const abi = [...elf.exported]
|
||||
.map((name) => /^node_register_module_v(\d+)$/u.exec(name)?.[1])
|
||||
.find((value) => value !== undefined);
|
||||
if (abi === undefined || (nodeAbi !== undefined && nodeAbi !== abi)) {
|
||||
throw new NativeError("MODULE_PACK_NATIVE_RUNTIME_UNKNOWN", fileId);
|
||||
}
|
||||
nodeAbi = abi;
|
||||
}
|
||||
}
|
||||
}
|
||||
return {
|
||||
addons,
|
||||
...(nodeApi === undefined ? {} : { nodeApi }),
|
||||
...(nodeAbi === undefined ? {} : { nodeAbi }),
|
||||
...(libc === undefined ? {} : { libcVersion: `>=${[...libc, 0, 0].slice(0, 3).join(".")}` }),
|
||||
};
|
||||
}
|
||||
@@ -2,6 +2,7 @@ import { readdir, readFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import type { BundledPackage } from "./packages.js";
|
||||
import { contentDigest, type LicenseSupplement } from "./supplements.js";
|
||||
|
||||
/**
|
||||
* `THIRD_PARTY_NOTICES` (ADR 0049): the licence and notice text that redistributing a module
|
||||
@@ -24,11 +25,20 @@ import type { BundledPackage } from "./packages.js";
|
||||
* (Apache-2.0). An MIT, ISC, or BSD package without its file is refused, because its copyright line
|
||||
* exists only in that file.
|
||||
*
|
||||
* - **Reviewed supplements** (ADR 0052). When a reviewed license supplement exists for a package's
|
||||
* name, it must match the package exactly (version and content digest over the files
|
||||
* redistributed for it) and must name the licence the package declares, or packaging fails. A
|
||||
* matching supplement's material follows the package's own, labelled as supplement material; it
|
||||
* stands in for the licence text the package omits, and never changes the declared licence.
|
||||
*
|
||||
* Everything is sorted, and line endings are normalized, so the file is byte-identical for identical
|
||||
* inputs.
|
||||
*/
|
||||
|
||||
export type NoticeErrorCode = "MODULE_PACK_LICENCE_TEXT_MISSING" | "MODULE_PACK_LICENCE_UNKNOWN";
|
||||
export type NoticeErrorCode =
|
||||
| "MODULE_PACK_LICENCE_SUPPLEMENT_MISMATCH"
|
||||
| "MODULE_PACK_LICENCE_TEXT_MISSING"
|
||||
| "MODULE_PACK_LICENCE_UNKNOWN";
|
||||
|
||||
export class NoticeError extends Error {
|
||||
readonly code: NoticeErrorCode;
|
||||
@@ -42,11 +52,16 @@ export class NoticeError extends Error {
|
||||
}
|
||||
}
|
||||
|
||||
export type NoticeUse = "backend" | "browser";
|
||||
/** Where in the package a component is used: bundled into the backend or a browser artifact, or
|
||||
* copied as files under `node_modules` (a native target's runtime dependency). */
|
||||
export type NoticeUse = "backend" | "browser" | "node_modules";
|
||||
|
||||
export interface NoticeComponent {
|
||||
readonly bundled: BundledPackage;
|
||||
readonly uses: ReadonlySet<NoticeUse>;
|
||||
/** The files redistributed for this component, by path relative to its root: what a supplement's
|
||||
* content digest is computed over. Read only when a supplement names the component. */
|
||||
readonly redistributedFiles: () => Promise<ReadonlyMap<string, Uint8Array>>;
|
||||
}
|
||||
|
||||
/** Licences whose standard text is complete without any project-specific line, so supplying it
|
||||
@@ -114,8 +129,47 @@ async function standardText(licence: string): Promise<string | undefined> {
|
||||
: normalized(await readFile(path.join(packageRoot, "licenses", file), "utf8"));
|
||||
}
|
||||
|
||||
function supplementSection(supplement: LicenseSupplement): string[] {
|
||||
const lines = [
|
||||
"[Reviewed license supplement]",
|
||||
"",
|
||||
`This material did not come from ${supplement.name} ${supplement.version}. The package omits licence`,
|
||||
"and notice material that redistribution requires, so Ekklesia's canonical packager adds it from a",
|
||||
"reviewed license supplement matched to this exact package: its name, its version, and its content",
|
||||
`digest ${supplement.contentDigest}. The supplement does not change the licence`,
|
||||
"the package declares. Review of a supplement is project and code review of a build input. It is",
|
||||
"not legal review, and it does not establish that every redistribution obligation is met.",
|
||||
"",
|
||||
normalized(supplement.statement).replace(/\n$/u, ""),
|
||||
"",
|
||||
"Components covered:",
|
||||
"",
|
||||
];
|
||||
for (const component of supplement.components) {
|
||||
lines.push(` ${component.name} ${component.version}: ${component.license}`);
|
||||
lines.push(` Source: ${component.source}`);
|
||||
}
|
||||
lines.push("");
|
||||
const printed = new Map<string, string>();
|
||||
for (const component of supplement.components) {
|
||||
for (const material of component.material) {
|
||||
const label = `${component.name} ${component.version}: ${material.title}`;
|
||||
const first = printed.get(material.sha256);
|
||||
lines.push(`[${label}]`, "");
|
||||
if (first === undefined) {
|
||||
printed.set(material.sha256, label);
|
||||
lines.push(normalized(material.text));
|
||||
} else {
|
||||
lines.push(`Identical to the text printed above under [${first}].`, "");
|
||||
}
|
||||
}
|
||||
}
|
||||
return lines;
|
||||
}
|
||||
|
||||
export async function thirdPartyNotices(
|
||||
components: ReadonlyMap<string, NoticeComponent>,
|
||||
supplements: ReadonlyMap<string, readonly LicenseSupplement[]> = new Map(),
|
||||
): Promise<string> {
|
||||
const ordered = [...components.values()].sort(
|
||||
(left, right) =>
|
||||
@@ -123,15 +177,28 @@ export async function thirdPartyNotices(
|
||||
compare(left.bundled.version, right.bundled.version),
|
||||
);
|
||||
const sections: string[] = [];
|
||||
for (const { bundled, uses } of ordered) {
|
||||
for (const { bundled, redistributedFiles, uses } of ordered) {
|
||||
const component = `${bundled.name}@${bundled.version}`;
|
||||
const licence = licenceOf(bundled);
|
||||
if (licence === undefined || /^UNLICEN[CS]ED$/iu.test(licence)) {
|
||||
throw new NoticeError("MODULE_PACK_LICENCE_UNKNOWN", component);
|
||||
}
|
||||
let supplement: LicenseSupplement | undefined;
|
||||
const candidates = supplements.get(bundled.name);
|
||||
if (candidates !== undefined && candidates.length > 0) {
|
||||
// A supplement exists for this name: only an exact match may be used, and anything else
|
||||
// fails rather than falling back to a nearby version or to no supplement at all.
|
||||
const digest = contentDigest(await redistributedFiles());
|
||||
supplement = candidates.find(
|
||||
(candidate) => candidate.version === bundled.version && candidate.contentDigest === digest,
|
||||
);
|
||||
if (supplement === undefined || supplement.license !== licence) {
|
||||
throw new NoticeError("MODULE_PACK_LICENCE_SUPPLEMENT_MISMATCH", component);
|
||||
}
|
||||
}
|
||||
let texts = await licenceTexts(bundled, licence);
|
||||
// A NOTICE file alone is not the licence: the licence text must still come from somewhere.
|
||||
if (!texts.some((text) => !/^notice/iu.test(text.title))) {
|
||||
if (supplement === undefined && !texts.some((text) => !/^notice/iu.test(text.title))) {
|
||||
const standard = await standardText(licence);
|
||||
if (standard === undefined) {
|
||||
throw new NoticeError("MODULE_PACK_LICENCE_TEXT_MISSING", component);
|
||||
@@ -153,6 +220,7 @@ export async function thirdPartyNotices(
|
||||
RULE,
|
||||
"",
|
||||
...texts.flatMap((text) => [`[${text.title}]`, "", text.text]),
|
||||
...(supplement === undefined ? [] : supplementSection(supplement)),
|
||||
].join("\n"),
|
||||
);
|
||||
}
|
||||
|
||||
@@ -17,9 +17,23 @@ import {
|
||||
} from "@ekklesia/contracts/module-package";
|
||||
import { analyzeModuleUi, ModuleUiBuildError } from "@ekklesia/module-ui-build";
|
||||
import { build, type Message, type Plugin } from "esbuild";
|
||||
import { minVersion, satisfies, validRange } from "semver";
|
||||
import { minVersion, satisfies, subset, validRange } from "semver";
|
||||
import {
|
||||
NATIVE_TARGETS,
|
||||
type NativePackage,
|
||||
type NativeTarget,
|
||||
NativeError,
|
||||
type NativeErrorCode,
|
||||
nativeClosure,
|
||||
nativeRuntime,
|
||||
} from "./native.js";
|
||||
import { type NoticeComponent, NoticeError, type NoticeUse, thirdPartyNotices } from "./notices.js";
|
||||
import { BundledPackageIndex, isNativeOrPlatformSpecific } from "./packages.js";
|
||||
import {
|
||||
type BundledPackage,
|
||||
BundledPackageIndex,
|
||||
isNativeOrPlatformSpecific,
|
||||
} from "./packages.js";
|
||||
import { loadLicenseSupplements, SupplementError } from "./supplements.js";
|
||||
import { deterministicTarGz, TarPathError } from "./tar.js";
|
||||
|
||||
/**
|
||||
@@ -28,10 +42,12 @@ import { deterministicTarGz, TarPathError } from "./tar.js";
|
||||
*
|
||||
* It reads the module directory and the dependencies its workspace installed, and writes only the
|
||||
* two output files. It never installs, links, or deploys anything, so the workspace it runs in is
|
||||
* left exactly as it was. In this release it produces the portable `any` target only.
|
||||
* left exactly as it was. It produces the portable `any` target, and the native `linux-x64-glibc`
|
||||
* target, whose native runtime dependencies are copied rather than bundled (see `./native.ts`).
|
||||
*/
|
||||
|
||||
export type ModulePackErrorCode =
|
||||
| NativeErrorCode
|
||||
| "MODULE_PACK_BUNDLE_FAILED"
|
||||
| "MODULE_PACK_DATABASE_RUNTIME_CONFLICT"
|
||||
| "MODULE_PACK_ENTRYPOINT_MISSING"
|
||||
@@ -43,12 +59,16 @@ export type ModulePackErrorCode =
|
||||
| "MODULE_PACK_LAYOUT_CONFLICT"
|
||||
| "MODULE_PACK_MODULE_LICENSE_FILE_INVALID"
|
||||
| "MODULE_PACK_MODULE_LICENSE_INVALID"
|
||||
| "MODULE_PACK_LICENCE_SUPPLEMENT_INVALID"
|
||||
| "MODULE_PACK_LICENCE_SUPPLEMENT_MISMATCH"
|
||||
| "MODULE_PACK_LICENCE_TEXT_MISSING"
|
||||
| "MODULE_PACK_LICENCE_UNKNOWN"
|
||||
| "MODULE_PACK_MANIFEST_INVALID"
|
||||
| "MODULE_PACK_METADATA_INVALID"
|
||||
| "MODULE_PACK_MIGRATIONS_INVALID"
|
||||
| "MODULE_PACK_NATIVE_DEPENDENCY"
|
||||
| "MODULE_PACK_NATIVE_NODE_INCOMPATIBLE"
|
||||
| "MODULE_PACK_NATIVE_TARGET_UNNEEDED"
|
||||
| "MODULE_PACK_OUTPUT_FAILED"
|
||||
| "MODULE_PACK_PATH_UNSUPPORTED"
|
||||
| "MODULE_PACK_REVISION_INVALID"
|
||||
@@ -78,6 +98,10 @@ const messages: Readonly<Record<ModulePackErrorCode, string>> = Object.freeze({
|
||||
"The module license text must be supplied as an explicit, non-empty, readable regular file.",
|
||||
MODULE_PACK_MODULE_LICENSE_INVALID:
|
||||
"The source package.json must declare a valid SPDX license expression as a string.",
|
||||
MODULE_PACK_LICENCE_SUPPLEMENT_INVALID:
|
||||
"A reviewed license supplement is malformed or does not match its reviewed files.",
|
||||
MODULE_PACK_LICENCE_SUPPLEMENT_MISMATCH:
|
||||
"A license supplement names this package, but not its exact version, content digest, and declared licence.",
|
||||
MODULE_PACK_LICENCE_TEXT_MISSING:
|
||||
"A bundled package ships no licence text, and its licence has no complete standard text to supply.",
|
||||
MODULE_PACK_LICENCE_UNKNOWN: "A bundled package declares no licence that permits redistribution.",
|
||||
@@ -87,6 +111,20 @@ const messages: Readonly<Record<ModulePackErrorCode, string>> = Object.freeze({
|
||||
"The migrations directory must hold only regular files and directories.",
|
||||
MODULE_PACK_NATIVE_DEPENDENCY:
|
||||
"The backend bundles a native or platform-specific dependency, which an any-target package cannot contain.",
|
||||
MODULE_PACK_NATIVE_DEPENDENCY_MISSING:
|
||||
"A native runtime dependency requires a package that is not installed.",
|
||||
MODULE_PACK_NATIVE_LAYOUT_CONFLICT:
|
||||
"Two versions of one package would share the native node_modules layout.",
|
||||
MODULE_PACK_NATIVE_NODE_INCOMPATIBLE:
|
||||
"A native runtime dependency does not support every Node.js version the package targets.",
|
||||
MODULE_PACK_NATIVE_PACKAGE_INVALID:
|
||||
"A native runtime dependency could not be read, or contains a link or special file.",
|
||||
MODULE_PACK_NATIVE_RUNTIME_UNKNOWN:
|
||||
"A native addon's Node.js interface could not be established exactly.",
|
||||
MODULE_PACK_NATIVE_TARGET_MISMATCH:
|
||||
"A native runtime dependency, or a file in one, is not built for the package's target.",
|
||||
MODULE_PACK_NATIVE_TARGET_UNNEEDED:
|
||||
"A native target needs a native addon; a package without one uses the any target.",
|
||||
MODULE_PACK_OUTPUT_FAILED: "The package could not be written to the output directory.",
|
||||
MODULE_PACK_PATH_UNSUPPORTED:
|
||||
"A package path is not portable printable ASCII that fits the archive format.",
|
||||
@@ -95,7 +133,8 @@ const messages: Readonly<Record<ModulePackErrorCode, string>> = Object.freeze({
|
||||
"The Node.js range could not be determined. Run inside an Ekklesia workspace, or pass one.",
|
||||
MODULE_PACK_SDK_INCOMPATIBLE:
|
||||
"The bundled Module SDK is outside the manifest's own compatibility.moduleSdk range.",
|
||||
MODULE_PACK_TARGET_UNSUPPORTED: "This packager produces the any target only.",
|
||||
MODULE_PACK_TARGET_UNSUPPORTED:
|
||||
"This packager produces the any and linux-x64-glibc targets only.",
|
||||
MODULE_PACK_UI_STALE:
|
||||
"A declared browser artifact is missing or out of date. Build the module before packaging it.",
|
||||
});
|
||||
@@ -124,7 +163,7 @@ export interface PackModuleOptions {
|
||||
readonly moduleDirectory: string;
|
||||
/** Explicit license-text input. Its bytes are copied unchanged to the package root as LICENSE. */
|
||||
readonly licenseFile: string;
|
||||
/** `any` only, in this release. */
|
||||
/** `any`, or a native target this packager supports (`linux-x64-glibc`). */
|
||||
readonly target: string;
|
||||
readonly outDirectory: string;
|
||||
/** The full commit id the module was built from. */
|
||||
@@ -134,6 +173,8 @@ export interface PackModuleOptions {
|
||||
/** The Node.js range the package is built for. Defaults to the enclosing workspace's
|
||||
* `engines.node`, the toolchain the module was built and tested with. */
|
||||
readonly nodeRange?: string;
|
||||
/** Reviewed license supplements (ADR 0052). Defaults to the ones this packager ships. */
|
||||
readonly supplementsDirectory?: string;
|
||||
}
|
||||
|
||||
export interface ModulePackResult {
|
||||
@@ -254,15 +295,93 @@ async function readTree(root: string, relative: string, files: Map<string, Buffe
|
||||
}
|
||||
}
|
||||
|
||||
/** The files redistributed for a component, relative to its root: what it copies, or the source
|
||||
* files of it that were bundled. The content digest a license supplement names is over these. */
|
||||
async function redistributedFiles(component: {
|
||||
readonly bundled: BundledPackage;
|
||||
readonly files: ReadonlySet<string>;
|
||||
readonly copied?: ReadonlyMap<string, Buffer>;
|
||||
}): Promise<ReadonlyMap<string, Uint8Array>> {
|
||||
if (component.copied !== undefined) {
|
||||
return component.copied;
|
||||
}
|
||||
const files = new Map<string, Uint8Array>();
|
||||
for (const file of component.files) {
|
||||
const real = await realpath(file).catch(() => file);
|
||||
files.set(
|
||||
path.relative(component.bundled.root, real).split(path.sep).join("/"),
|
||||
await readFile(real),
|
||||
);
|
||||
}
|
||||
return files;
|
||||
}
|
||||
|
||||
interface BackendBundle {
|
||||
readonly outputs: ReadonlyMap<string, Buffer>;
|
||||
readonly inputs: readonly string[];
|
||||
}
|
||||
|
||||
const NATIVE_EXTERNALS = "ekklesia-native-externals";
|
||||
|
||||
/** The bare specifier's package name: `sharp/x` → `sharp`, `@scope/name/x` → `@scope/name`. */
|
||||
function specifierPackage(specifier: string): string {
|
||||
const segments = specifier.split("/");
|
||||
return specifier.startsWith("@") ? segments.slice(0, 2).join("/") : (segments[0] ?? "");
|
||||
}
|
||||
|
||||
/**
|
||||
* For a native target: keeps every native or platform-specific third-party package out of the
|
||||
* bundle, as an import of the package itself, and records it as a root of the copied runtime
|
||||
* closure. Decided per resolved package, by the same rule that refuses one in an `any` package.
|
||||
*/
|
||||
function nativeExternals(
|
||||
index: BundledPackageIndex,
|
||||
externals: Map<string, BundledPackage>,
|
||||
): Plugin {
|
||||
return {
|
||||
name: NATIVE_EXTERNALS,
|
||||
setup(pluginBuild) {
|
||||
pluginBuild.onResolve({ filter: /^[^./]/ }, async (args) => {
|
||||
if (
|
||||
(args.pluginData as { ekklesiaNative?: boolean } | undefined)?.ekklesiaNative ||
|
||||
isBuiltin(args.path)
|
||||
) {
|
||||
return undefined;
|
||||
}
|
||||
const resolved = await pluginBuild.resolve(args.path, {
|
||||
importer: args.importer,
|
||||
kind: args.kind,
|
||||
pluginData: { ekklesiaNative: true },
|
||||
resolveDir: args.resolveDir,
|
||||
});
|
||||
if (resolved.errors.length > 0 || resolved.external || resolved.path === "") {
|
||||
return undefined;
|
||||
}
|
||||
const owner = await index.packageOf(resolved.path);
|
||||
if (
|
||||
owner === undefined ||
|
||||
!owner.thirdParty ||
|
||||
!(await isNativeOrPlatformSpecific(owner))
|
||||
) {
|
||||
return undefined;
|
||||
}
|
||||
if (specifierPackage(args.path) !== owner.name) {
|
||||
return {
|
||||
errors: [{ text: "A native package must be imported by its own package name." }],
|
||||
};
|
||||
}
|
||||
externals.set(owner.name, owner);
|
||||
return { external: true, path: args.path };
|
||||
});
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function bundleBackend(
|
||||
moduleDirectory: string,
|
||||
manifest: ModuleManifest,
|
||||
nodeRange: string,
|
||||
native?: { readonly index: BundledPackageIndex; readonly externals: Map<string, BundledPackage> },
|
||||
): Promise<BackendBundle> {
|
||||
const declared = [
|
||||
manifest.entrypoints.server,
|
||||
@@ -303,7 +422,10 @@ async function bundleBackend(
|
||||
minify: false,
|
||||
outdir: moduleDirectory,
|
||||
platform: "node",
|
||||
plugins: [nativeAddonGuard],
|
||||
plugins:
|
||||
native === undefined
|
||||
? [nativeAddonGuard]
|
||||
: [nativeExternals(native.index, native.externals), nativeAddonGuard],
|
||||
sourcemap: false,
|
||||
splitting: true,
|
||||
target: `node${minimum?.version ?? "24"}`,
|
||||
@@ -330,7 +452,11 @@ async function bundleBackend(
|
||||
}
|
||||
for (const output of Object.values(metafile.outputs)) {
|
||||
for (const imported of output.imports) {
|
||||
if (imported.external && !isBuiltin(imported.path)) {
|
||||
if (
|
||||
imported.external &&
|
||||
!isBuiltin(imported.path) &&
|
||||
!native?.externals.has(specifierPackage(imported.path))
|
||||
) {
|
||||
throw new ModulePackError("MODULE_PACK_EXTERNAL_DEPENDENCY", imported.path);
|
||||
}
|
||||
}
|
||||
@@ -349,9 +475,11 @@ async function bundleBackend(
|
||||
}
|
||||
|
||||
export async function packModule(options: PackModuleOptions): Promise<ModulePackResult> {
|
||||
if (options.target !== MODULE_PACKAGE_ANY_TARGET) {
|
||||
const nativeTarget: NativeTarget | undefined = NATIVE_TARGETS.get(options.target);
|
||||
if (options.target !== MODULE_PACKAGE_ANY_TARGET && nativeTarget === undefined) {
|
||||
throw new ModulePackError("MODULE_PACK_TARGET_UNSUPPORTED", options.target);
|
||||
}
|
||||
const target = options.target;
|
||||
if (!/^(?:[0-9a-f]{40}|[0-9a-f]{64})$/u.test(options.sourceRevision)) {
|
||||
throw new ModulePackError("MODULE_PACK_REVISION_INVALID");
|
||||
}
|
||||
@@ -414,8 +542,53 @@ export async function packModule(options: PackModuleOptions): Promise<ModulePack
|
||||
throw new ModulePackError("MODULE_PACK_RUNTIME_UNKNOWN");
|
||||
}
|
||||
|
||||
// 1. The backend, bundled with every pure-JavaScript dependency it uses.
|
||||
const backend = await bundleBackend(moduleDirectory, manifest, nodeRange);
|
||||
let supplements: Awaited<ReturnType<typeof loadLicenseSupplements>>;
|
||||
try {
|
||||
supplements = await loadLicenseSupplements(options.supplementsDirectory);
|
||||
} catch (error) {
|
||||
throw new ModulePackError(
|
||||
"MODULE_PACK_LICENCE_SUPPLEMENT_INVALID",
|
||||
error instanceof SupplementError ? error.supplement : undefined,
|
||||
);
|
||||
}
|
||||
|
||||
// 1. The backend, bundled with every pure-JavaScript dependency it uses. For a native target,
|
||||
// native packages stay outside the bundle and are copied with their runtime closure instead.
|
||||
const index = new BundledPackageIndex();
|
||||
const externals = new Map<string, BundledPackage>();
|
||||
const backend = await bundleBackend(
|
||||
moduleDirectory,
|
||||
manifest,
|
||||
nodeRange,
|
||||
nativeTarget === undefined ? undefined : { externals, index },
|
||||
);
|
||||
let nativePackages: readonly NativePackage[] = [];
|
||||
let runtimeRequirements: Record<string, unknown> = {};
|
||||
if (nativeTarget !== undefined) {
|
||||
try {
|
||||
nativePackages = await nativeClosure([...externals.values()], nativeTarget);
|
||||
const { addons, ...requirements } = nativeRuntime(nativePackages, nativeTarget);
|
||||
if (addons === 0) {
|
||||
throw new ModulePackError("MODULE_PACK_NATIVE_TARGET_UNNEEDED", options.target);
|
||||
}
|
||||
runtimeRequirements = requirements;
|
||||
} catch (error) {
|
||||
if (error instanceof NativeError) {
|
||||
throw new ModulePackError(error.code, error.detail);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
for (const nativePackage of nativePackages) {
|
||||
const engines = nativePackage.manifest.engines;
|
||||
const engine = isRecord(engines) ? engines.node : undefined;
|
||||
if (typeof engine === "string" && !subset(nodeRange, engine)) {
|
||||
throw new ModulePackError(
|
||||
"MODULE_PACK_NATIVE_NODE_INCOMPATIBLE",
|
||||
`${nativePackage.name}@${nativePackage.version}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 2. The browser artifacts, exactly as built, with what went into each.
|
||||
let ui: Awaited<ReturnType<typeof analyzeModuleUi>>;
|
||||
@@ -428,11 +601,15 @@ export async function packModule(options: PackModuleOptions): Promise<ModulePack
|
||||
);
|
||||
}
|
||||
|
||||
// 3. Every bundled source file traced to its package.
|
||||
const index = new BundledPackageIndex();
|
||||
// 3. Every bundled source file traced to its package, and every copied package.
|
||||
const components = new Map<
|
||||
string,
|
||||
{ bundled: Awaited<ReturnType<typeof index.packageOf>>; uses: Set<NoticeUse> }
|
||||
{
|
||||
bundled: BundledPackage;
|
||||
uses: Set<NoticeUse>;
|
||||
files: Set<string>;
|
||||
copied?: ReadonlyMap<string, Buffer>;
|
||||
}
|
||||
>();
|
||||
const incorporates = new Map<string, string>();
|
||||
const drizzleVersions = new Set<string>();
|
||||
@@ -464,13 +641,32 @@ export async function packModule(options: PackModuleOptions): Promise<ModulePack
|
||||
const existing = components.get(key);
|
||||
if (existing) {
|
||||
existing.uses.add(use);
|
||||
existing.files.add(file);
|
||||
} else {
|
||||
if (use === "backend" && (await isNativeOrPlatformSpecific(bundled))) {
|
||||
throw new ModulePackError("MODULE_PACK_NATIVE_DEPENDENCY", key);
|
||||
}
|
||||
components.set(key, { bundled, uses: new Set([use]) });
|
||||
components.set(key, { bundled, files: new Set([file]), uses: new Set([use]) });
|
||||
}
|
||||
}
|
||||
for (const nativePackage of nativePackages) {
|
||||
const key = `${nativePackage.name}@${nativePackage.version}`;
|
||||
const existing = components.get(key);
|
||||
const entry = existing ?? {
|
||||
bundled: {
|
||||
manifest: nativePackage.manifest,
|
||||
name: nativePackage.name,
|
||||
root: nativePackage.root,
|
||||
thirdParty: true,
|
||||
version: nativePackage.version,
|
||||
},
|
||||
files: new Set<string>(),
|
||||
uses: new Set<NoticeUse>(),
|
||||
};
|
||||
entry.uses.add("node_modules");
|
||||
entry.copied = nativePackage.files;
|
||||
components.set(key, entry);
|
||||
}
|
||||
const usesDatabaseAbi = manifest.database?.abi === 1;
|
||||
if (!usesDatabaseAbi && drizzleVersions.size > 1) {
|
||||
throw new ModulePackError("MODULE_PACK_DATABASE_RUNTIME_CONFLICT");
|
||||
@@ -482,11 +678,11 @@ export async function packModule(options: PackModuleOptions): Promise<ModulePack
|
||||
|
||||
// 4. The payload.
|
||||
const files = new Map<string, Buffer>();
|
||||
const add = (relative: string, data: Buffer): void => {
|
||||
const add = (relative: string, data: Buffer, copied = false): void => {
|
||||
if (!portable(relative)) {
|
||||
throw new ModulePackError("MODULE_PACK_PATH_UNSUPPORTED", relative);
|
||||
}
|
||||
if (forbidden(relative)) {
|
||||
if (!copied && forbidden(relative)) {
|
||||
throw new ModulePackError("MODULE_PACK_FORBIDDEN_FILE", relative);
|
||||
}
|
||||
if (files.has(relative) || relative === MODULE_PACKAGE_METADATA_FILENAME) {
|
||||
@@ -523,6 +719,14 @@ export async function packModule(options: PackModuleOptions): Promise<ModulePack
|
||||
add(relative, data);
|
||||
}
|
||||
}
|
||||
// A native target's runtime dependencies, as real files, flat under node_modules/<name>. They are
|
||||
// third-party packages carried as published, so the module's own source and test path rules do
|
||||
// not apply to their contents; declarations and source maps were already left out.
|
||||
for (const nativePackage of nativePackages) {
|
||||
for (const [relative, data] of nativePackage.files) {
|
||||
add(`node_modules/${nativePackage.name}/${relative}`, data, true);
|
||||
}
|
||||
}
|
||||
add("ekklesia.module.json", manifestBytes);
|
||||
add(MODULE_PACKAGE_LICENSE_FILENAME, licenseBytes);
|
||||
add(
|
||||
@@ -538,7 +742,21 @@ export async function packModule(options: PackModuleOptions): Promise<ModulePack
|
||||
try {
|
||||
add(
|
||||
MODULE_PACKAGE_NOTICES_FILENAME,
|
||||
Buffer.from(await thirdPartyNotices(components as ReadonlyMap<string, NoticeComponent>)),
|
||||
Buffer.from(
|
||||
await thirdPartyNotices(
|
||||
new Map(
|
||||
[...components].map(([key, component]): [string, NoticeComponent] => [
|
||||
key,
|
||||
{
|
||||
bundled: component.bundled,
|
||||
redistributedFiles: () => redistributedFiles(component),
|
||||
uses: component.uses,
|
||||
},
|
||||
]),
|
||||
),
|
||||
supplements,
|
||||
),
|
||||
),
|
||||
);
|
||||
} catch (error) {
|
||||
if (error instanceof NoticeError) {
|
||||
@@ -555,8 +773,8 @@ export async function packModule(options: PackModuleOptions): Promise<ModulePack
|
||||
packageFormat: MODULE_PACKAGE_FORMAT,
|
||||
module: { id: manifest.id, version: manifest.version },
|
||||
license,
|
||||
target: MODULE_PACKAGE_ANY_TARGET,
|
||||
runtime: { node: nodeRange },
|
||||
target,
|
||||
runtime: { node: nodeRange, ...runtimeRequirements },
|
||||
provenance: {
|
||||
revision: options.sourceRevision,
|
||||
...(options.repository === undefined ? {} : { repository: options.repository }),
|
||||
@@ -590,11 +808,7 @@ export async function packModule(options: PackModuleOptions): Promise<ModulePack
|
||||
throw error;
|
||||
}
|
||||
const digest = sha256(archive);
|
||||
const fileName = modulePackageArchiveFileName(
|
||||
manifest.id,
|
||||
manifest.version,
|
||||
MODULE_PACKAGE_ANY_TARGET,
|
||||
);
|
||||
const fileName = modulePackageArchiveFileName(manifest.id, manifest.version, target);
|
||||
const outDirectory = path.resolve(options.outDirectory);
|
||||
const archivePath = path.join(outDirectory, fileName);
|
||||
const sha256Path = `${archivePath}.sha256`;
|
||||
@@ -616,7 +830,7 @@ export async function packModule(options: PackModuleOptions): Promise<ModulePack
|
||||
moduleId: manifest.id,
|
||||
sha256: digest,
|
||||
sha256Path,
|
||||
target: MODULE_PACKAGE_ANY_TARGET,
|
||||
target,
|
||||
version: manifest.version,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -0,0 +1,280 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { lstat, readdir, readFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
/**
|
||||
* Reviewed license supplements (ADR 0052).
|
||||
*
|
||||
* When a redistributed dependency's own package omits licence or notice material that
|
||||
* redistribution requires, the canonical packager may add material from a checked-in, reviewed
|
||||
* supplement. A supplement is data, not code: nothing here knows about any particular package.
|
||||
*
|
||||
* A supplement lives at `license-supplements/<package name>/<version>/` beside this package. It
|
||||
* holds `supplement.json` and the text files that file lists:
|
||||
*
|
||||
* ```json
|
||||
* {
|
||||
* "supplementFormat": 1,
|
||||
* "package": {
|
||||
* "name": "…", "version": "…", "license": "<SPDX the package declares>",
|
||||
* "contentDigest": "ekklesia-content-v1:sha256:<hex>"
|
||||
* },
|
||||
* "statement": "<what the supplement covers, printed with it>",
|
||||
* "components": [
|
||||
* {
|
||||
* "name": "…", "version": "…", "license": "<as stated by its source>",
|
||||
* "source": "<where the material came from>",
|
||||
* "material": [{ "title": "LICENSE", "text": "texts/<sha256 of the text>.txt" }]
|
||||
* }
|
||||
* ]
|
||||
* }
|
||||
* ```
|
||||
*
|
||||
* Every text file is named by its own SHA-256. The loader checks every name against its bytes, and
|
||||
* refuses a file the supplement does not list, so a supplement can only change through review.
|
||||
*
|
||||
* Repository review of a supplement is project and code review of a build input. It is not legal
|
||||
* review, and carrying a supplement does not establish that any redistribution obligation is met.
|
||||
*/
|
||||
|
||||
export const CONTENT_DIGEST_ALGORITHM = "ekklesia-content-v1";
|
||||
export const SUPPLEMENT_FORMAT = 1;
|
||||
|
||||
/**
|
||||
* The content digest a supplement is matched against (ADR 0052), over exactly the files the
|
||||
* packager redistributes for a package, by path relative to the package root.
|
||||
*
|
||||
* 1. Each path is normalized: `/` separators, no leading `./`, no empty, `.`, or `..` segment.
|
||||
* 2. Paths are sorted by their UTF-8 bytes.
|
||||
* 3. For each file, in that order, this line is appended to a manifest:
|
||||
* `<path> NUL <decimal byte length> NUL <lowercase hex SHA-256 of the bytes> LF`.
|
||||
* 4. The digest is `ekklesia-content-v1:sha256:` followed by the lowercase hex SHA-256 of the
|
||||
* manifest's UTF-8 bytes.
|
||||
*
|
||||
* Nothing else is an input: not timestamps, ownership, permissions, the directory the package was
|
||||
* installed in, the package manager, or the order files were read or extracted in.
|
||||
*/
|
||||
export function contentDigest(files: ReadonlyMap<string, Uint8Array>): string {
|
||||
const entries = [...files].map(([relative, data]) => {
|
||||
const normalized = relative.split(path.sep).join("/").replace(/^\.\//u, "");
|
||||
if (
|
||||
normalized.split("/").some((segment) => segment === "" || segment === "." || segment === "..")
|
||||
) {
|
||||
throw new Error("A redistributed path must be relative and normalized.");
|
||||
}
|
||||
return [Buffer.from(normalized, "utf8"), data] as const;
|
||||
});
|
||||
entries.sort(([left], [right]) => Buffer.compare(left, right));
|
||||
const manifest = createHash("sha256");
|
||||
for (const [name, data] of entries) {
|
||||
const digest = createHash("sha256").update(data).digest("hex");
|
||||
manifest.update(name);
|
||||
manifest.update(`\0${data.length}\0${digest}\n`);
|
||||
}
|
||||
return `${CONTENT_DIGEST_ALGORITHM}:sha256:${manifest.digest("hex")}`;
|
||||
}
|
||||
|
||||
export interface SupplementMaterial {
|
||||
readonly title: string;
|
||||
/** The text, exactly as reviewed. */
|
||||
readonly text: string;
|
||||
/** SHA-256 of the text's bytes, so identical texts are recognized. */
|
||||
readonly sha256: string;
|
||||
}
|
||||
|
||||
export interface SupplementComponent {
|
||||
readonly name: string;
|
||||
readonly version: string;
|
||||
readonly license: string;
|
||||
readonly source: string;
|
||||
readonly material: readonly SupplementMaterial[];
|
||||
}
|
||||
|
||||
export interface LicenseSupplement {
|
||||
readonly name: string;
|
||||
readonly version: string;
|
||||
readonly license: string;
|
||||
readonly contentDigest: string;
|
||||
readonly statement: string;
|
||||
readonly components: readonly SupplementComponent[];
|
||||
}
|
||||
|
||||
export class SupplementError extends Error {
|
||||
readonly supplement: string;
|
||||
|
||||
constructor(supplement: string) {
|
||||
super("A license supplement is malformed or does not match its reviewed files.");
|
||||
this.name = "SupplementError";
|
||||
this.supplement = supplement;
|
||||
}
|
||||
}
|
||||
|
||||
function isRecord(value: unknown): value is Record<string, unknown> {
|
||||
return typeof value === "object" && value !== null && !Array.isArray(value);
|
||||
}
|
||||
|
||||
function nonEmptyString(value: unknown): value is string {
|
||||
return typeof value === "string" && value.trim() !== "";
|
||||
}
|
||||
|
||||
function strictKeys(value: Record<string, unknown>, keys: readonly string[]): boolean {
|
||||
return Object.keys(value).every((key) => keys.includes(key)) && keys.every((key) => key in value);
|
||||
}
|
||||
|
||||
export const DEFAULT_SUPPLEMENTS_DIRECTORY = path.resolve(
|
||||
path.dirname(fileURLToPath(import.meta.url)),
|
||||
"..",
|
||||
"license-supplements",
|
||||
);
|
||||
|
||||
async function listFiles(root: string, relative = ""): Promise<string[]> {
|
||||
const directory = relative === "" ? root : path.join(root, relative);
|
||||
const found: string[] = [];
|
||||
for (const entry of await readdir(directory, { withFileTypes: true })) {
|
||||
const childRelative = relative === "" ? entry.name : `${relative}/${entry.name}`;
|
||||
const stats = await lstat(path.join(directory, entry.name));
|
||||
if (stats.isDirectory()) {
|
||||
found.push(...(await listFiles(root, childRelative)));
|
||||
} else if (stats.isFile()) {
|
||||
found.push(childRelative);
|
||||
} else {
|
||||
throw new SupplementError(childRelative);
|
||||
}
|
||||
}
|
||||
return found;
|
||||
}
|
||||
|
||||
async function loadOne(directory: string, label: string): Promise<LicenseSupplement> {
|
||||
let raw: unknown;
|
||||
try {
|
||||
raw = JSON.parse(await readFile(path.join(directory, "supplement.json"), "utf8")) as unknown;
|
||||
} catch {
|
||||
throw new SupplementError(label);
|
||||
}
|
||||
if (
|
||||
!isRecord(raw) ||
|
||||
!strictKeys(raw, ["supplementFormat", "package", "statement", "components"]) ||
|
||||
raw.supplementFormat !== SUPPLEMENT_FORMAT ||
|
||||
!isRecord(raw.package) ||
|
||||
!strictKeys(raw.package, ["name", "version", "license", "contentDigest"]) ||
|
||||
!nonEmptyString(raw.statement) ||
|
||||
!Array.isArray(raw.components) ||
|
||||
raw.components.length === 0
|
||||
) {
|
||||
throw new SupplementError(label);
|
||||
}
|
||||
const { contentDigest: digest, license, name, version } = raw.package;
|
||||
if (
|
||||
!nonEmptyString(name) ||
|
||||
!nonEmptyString(version) ||
|
||||
!nonEmptyString(license) ||
|
||||
typeof digest !== "string" ||
|
||||
!new RegExp(`^${CONTENT_DIGEST_ALGORITHM}:sha256:[0-9a-f]{64}$`, "u").test(digest) ||
|
||||
`${name}/${version}` !== label
|
||||
) {
|
||||
throw new SupplementError(label);
|
||||
}
|
||||
|
||||
const listed = new Set<string>();
|
||||
const components: SupplementComponent[] = [];
|
||||
for (const component of raw.components) {
|
||||
if (
|
||||
!isRecord(component) ||
|
||||
!strictKeys(component, ["name", "version", "license", "source", "material"]) ||
|
||||
!nonEmptyString(component.name) ||
|
||||
!nonEmptyString(component.version) ||
|
||||
!nonEmptyString(component.license) ||
|
||||
!nonEmptyString(component.source) ||
|
||||
!Array.isArray(component.material)
|
||||
) {
|
||||
throw new SupplementError(label);
|
||||
}
|
||||
const material: SupplementMaterial[] = [];
|
||||
for (const item of component.material) {
|
||||
if (
|
||||
!isRecord(item) ||
|
||||
!strictKeys(item, ["title", "text"]) ||
|
||||
!nonEmptyString(item.title) ||
|
||||
typeof item.text !== "string" ||
|
||||
!/^texts\/[0-9a-f]{64}\.txt$/u.test(item.text)
|
||||
) {
|
||||
throw new SupplementError(label);
|
||||
}
|
||||
const expected = item.text.slice("texts/".length, -".txt".length);
|
||||
let bytes: Buffer;
|
||||
try {
|
||||
bytes = await readFile(path.join(directory, ...item.text.split("/")));
|
||||
} catch {
|
||||
throw new SupplementError(label);
|
||||
}
|
||||
if (bytes.length === 0 || createHash("sha256").update(bytes).digest("hex") !== expected) {
|
||||
throw new SupplementError(label);
|
||||
}
|
||||
listed.add(item.text);
|
||||
material.push(
|
||||
Object.freeze({ sha256: expected, text: bytes.toString("utf8"), title: item.title }),
|
||||
);
|
||||
}
|
||||
components.push(
|
||||
Object.freeze({
|
||||
license: component.license,
|
||||
material: Object.freeze(material),
|
||||
name: component.name,
|
||||
source: component.source,
|
||||
version: component.version,
|
||||
}),
|
||||
);
|
||||
}
|
||||
// Only reviewed files: supplement.json, the texts it lists, and a human-readable provenance note.
|
||||
for (const file of await listFiles(directory)) {
|
||||
if (file !== "supplement.json" && file !== "PROVENANCE.md" && !listed.has(file)) {
|
||||
throw new SupplementError(label);
|
||||
}
|
||||
}
|
||||
return Object.freeze({
|
||||
components: Object.freeze(components),
|
||||
contentDigest: digest,
|
||||
license,
|
||||
name,
|
||||
statement: raw.statement,
|
||||
version,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Loads every supplement under `directory`, indexed by package name. A directory that does not exist
|
||||
* holds none. Any malformed supplement fails the whole load: a broken reviewed input is never
|
||||
* skipped.
|
||||
*/
|
||||
export async function loadLicenseSupplements(
|
||||
directory: string = DEFAULT_SUPPLEMENTS_DIRECTORY,
|
||||
): Promise<ReadonlyMap<string, readonly LicenseSupplement[]>> {
|
||||
const index = new Map<string, LicenseSupplement[]>();
|
||||
let files: string[];
|
||||
try {
|
||||
files = await listFiles(directory);
|
||||
} catch (error) {
|
||||
if (isRecord(error) && error.code === "ENOENT") {
|
||||
return index;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
const roots = files
|
||||
.filter((file) => file.endsWith("/supplement.json"))
|
||||
.map((file) => file.slice(0, -"/supplement.json".length))
|
||||
.sort();
|
||||
for (const label of roots) {
|
||||
const supplement = await loadOne(path.join(directory, ...label.split("/")), label);
|
||||
const list = index.get(supplement.name) ?? [];
|
||||
list.push(supplement);
|
||||
index.set(supplement.name, list);
|
||||
}
|
||||
// Every file must belong to some supplement directory.
|
||||
for (const file of files) {
|
||||
if (!roots.some((root) => file.startsWith(`${root}/`))) {
|
||||
throw new SupplementError(file);
|
||||
}
|
||||
}
|
||||
return index;
|
||||
}
|
||||
Reference in New Issue
Block a user