feat: corresponding-source artifacts for native packages that redistribute copyleft code (ADR 0055)
CI / verify (pull_request) Successful in 1m3s
CI / infrastructure-smoke (pull_request) Successful in 36s
CI / migration-smoke (pull_request) Successful in 1m53s
CI / package-release (pull_request) Successful in 36s
CI / package-publication (pull_request) Canceled after 0s
CI / deploy-smoke (same-origin) (pull_request) Canceled after 0s
CI / deploy-smoke (split) (pull_request) Canceled after 0s
CI / image-build (pull_request) Canceled after 30s
CI / verify (pull_request) Successful in 1m3s
CI / infrastructure-smoke (pull_request) Successful in 36s
CI / migration-smoke (pull_request) Successful in 1m53s
CI / package-release (pull_request) Successful in 36s
CI / package-publication (pull_request) Canceled after 0s
CI / deploy-smoke (same-origin) (pull_request) Canceled after 0s
CI / deploy-smoke (split) (pull_request) Canceled after 0s
CI / image-build (pull_request) Canceled after 30s
Ekklesia may publicly distribute a native Module Package containing LGPL or MPL components only beside a version-specific corresponding-source artifact. The artifact carries the exact source, patches, build definitions, and rebuild and relink instructions for those components. ADR 0055 (Accepted) records the policy: project compliance policy, not legal certification. It decides the item ADR 0052 deferred, and extends ADR 0049's distributable-release gate. Module Package v1, packageFormat 1, and both libvips supplements are unchanged. The mechanism is generic and lives in @ekklesia/module-pack: - Source recipes: reviewed data under corresponding-sources/<package>/<version>/. recipe.json names the exact package (name, version, content digest) and the exact supplement it was prepared with, and lists every component with its role, licence, licence material, and relationship to the binary. Every source file and patch is recorded by origin, size, and SHA-256; documents are hashed too, and every exclusion has a reason. Loading is strict and refuses anything unlisted. - Acquisition (the only network step) fetches each recorded file once into a cache keyed by SHA-256. It uses node:https with exact headers and no decoding, retries transient failures, and stores only bytes that match. - Construction is offline. From a built package it finds the covered native dependencies (by declared licence, or by a supplement component's licence), verifies every byte, and writes four sibling artifacts: the deterministic <package>-corresponding-source.tar.gz and its .sha256; <package>.redistribution.json, linking package, dependencies, supplement, recipe, and source by digest; and <package>.source-notice.txt, the release notice. - The public-distribution gate (verifyRedistribution, `ekklesia-corresponding-source verify`) approves only when the recipe, supplement, canonical archive, manifest, and metadata identify exactly these files. It refuses bundled copyleft code, and approves packages with no covered native code without an artifact. packModule is untouched. First complete proof: @img/[email protected] (sharp 0.35.5), from the reviewed supplement and provenance. - 383 components and 387 source files, about 208 MB: the sharp-libvips v1.3.4 build definitions, the 27 library archives, the five build-time patches, the 353 crates librsvg's resolved Cargo.lock records, and rust-src for nightly-2026-09-27 (rustc 75a75c3e0), which vendors std's dependencies. - BUILDING.md and two helpers point sharp-libvips's unchanged build logic at those files. - Followed in a Rocky Linux 8 container with networking disabled, they rebuilt libvips-cpp.so.8.18.7, byte-identically across two builds. Swapped into a verified, extracted native package, the rebuilt library was the one loaded, and sharp ran. - The native proof's source artifact, native-proof-0.1.0-linux-x64-glibc-corresponding-source.tar.gz, is 207242785 bytes, sha256 cc405492032b04a5391ab84e64f0e5e50f78e571ef7818204782c25a03eb50b7. It is byte-identical from unrelated directories and from a cache acquired fresh from upstream. Tests: - corresponding-source.test.ts runs in CI: synthetic inputs, every negative case, acquisition with retries and refusal, and determinism. - corresponding-source-native.test.ts checks the real recipe everywhere. With a verified cache (`pnpm test:corresponding-source`, which acquires once) it builds the real artifact twice, gates it, fails it closed, and runs sharp from the package. Docs: the module guide's distribution section, the README and architecture notes, and an ADR 0052 cross-reference. No Signage change, version bump, publication, or tag. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
4919db6168
commit
e083c44e05
@@ -0,0 +1,20 @@
|
||||
#!/usr/bin/env node
|
||||
// Committed launcher, deliberately not generated: see ekklesia-pack-module.mjs for why. It defers to
|
||||
// the compiled CLI at run time.
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const compiledCli = new URL("../dist/corresponding-source-cli.js", import.meta.url);
|
||||
|
||||
try {
|
||||
await import(compiledCli.href);
|
||||
} catch (error) {
|
||||
if (error?.code === "ERR_MODULE_NOT_FOUND") {
|
||||
process.stderr.write(
|
||||
`@ekklesia/module-pack is not built yet (${fileURLToPath(compiledCli)}).\n` +
|
||||
"Run `pnpm build` from the repository root first.\n",
|
||||
);
|
||||
process.exitCode = 1;
|
||||
} else {
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
+144
@@ -0,0 +1,144 @@
|
||||
# Rebuilding `@img/sharp-libvips-linux-x64` 1.3.4
|
||||
|
||||
This directory is the corresponding source for one redistributed native binary. These instructions
|
||||
say how to rebuild it from the source here, and how to use a rebuilt or modified library in place of
|
||||
the redistributed one. They are technical instructions. They are not a legal opinion, and they have
|
||||
not been legally certified.
|
||||
|
||||
## What this corresponds to
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| Package | `@img/sharp-libvips-linux-x64` 1.3.4, from the npm registry |
|
||||
| Declared licence | `LGPL-3.0-or-later` |
|
||||
| Content digest | `ekklesia-content-v1:sha256:2927fe11493461fa34847191d0b9b1c6441007c3e99f306e5708cb07fb0a5e59` |
|
||||
| Binary | `lib/libvips-cpp.so.8.18.7` (SONAME `libvips-cpp.so.8.18.7`) |
|
||||
| Target | Linux, x86-64, glibc 2.28 or newer |
|
||||
| Used by | `sharp` 0.35.5, through `@img/sharp-linux-x64` 0.35.5 |
|
||||
|
||||
`lib/libvips-cpp.so.8.18.7` is one shared object. libvips and the libraries it uses are built as
|
||||
static libraries and linked into it: 27 C and C++ libraries, glib's bundled gvdb, librsvg with its
|
||||
Rust crates, and the Rust standard library. `recipe.json` lists every component, its licence, its
|
||||
source file and patches with their SHA-256, and its relationship to the binary. `PROVENANCE.md`
|
||||
records how all of this was established, and `licenses/` holds the reviewed licence and notice
|
||||
material.
|
||||
|
||||
## What is here
|
||||
|
||||
- `sources/sharp-libvips-1.3.4.tar.gz`: the build definitions that produced the binary. They are
|
||||
`build.sh`, `build/posix.sh`, `platforms/linux-x64/` (a Dockerfile and its toolchain files), and
|
||||
`versions.properties`.
|
||||
- `sources/*.tar.*`: the source archive of every compiled library, exactly as `build/posix.sh`
|
||||
downloads it.
|
||||
- `patches/`: the five patches `build/posix.sh` applies: glib, mozjpeg, librsvg, libultrahdr, and
|
||||
libvips.
|
||||
- `sources/crates/`: all 353 Rust crates librsvg's resolved `Cargo.lock` records.
|
||||
- 147 are compiled into the binary.
|
||||
- 17 are needed only to build it: build-script dependencies, and one procedural macro that adds no
|
||||
code.
|
||||
- 189 serve librsvg's other workspace members, its tests, or other platforms. Cargo resolves the
|
||||
whole workspace lockfile even when it builds one package, so it needs them present to build
|
||||
offline. None of them is compiled into the binary.
|
||||
|
||||
`recipe.json` records each crate's role.
|
||||
- `sources/rust/rust-src-nightly-2026-09-27.tar.xz`: the standard-library source of the Rust toolchain
|
||||
that built librsvg, `rustc 1.101.0-nightly (75a75c3e0 2026-09-26)`.
|
||||
- `build/`: the resolved librsvg `Cargo.lock`, and the two scripts below that point the unchanged build
|
||||
logic at these files.
|
||||
|
||||
Every file's SHA-256 is in `recipe.json` and in the archive's top-level `corresponding-source.json`.
|
||||
Check them before building.
|
||||
|
||||
## How the original was built
|
||||
|
||||
sharp-libvips builds the binary in a Rocky Linux 8 container, which pins glibc 2.28:
|
||||
|
||||
```sh
|
||||
./build.sh linux-x64
|
||||
# = docker build --pull -t vips-dev-linux-x64 platforms/linux-x64
|
||||
# docker run --rm -v "$PWD:/packaging" vips-dev-linux-x64 sh -c /packaging/build/posix.sh
|
||||
```
|
||||
|
||||
`platforms/linux-x64/Dockerfile` installs the toolchain: gcc-toolset-15, cmake, nasm, pkgconf, meson
|
||||
and ninja (from pip), a nightly Rust toolchain through rustup, and cargo-c. `build/posix.sh` then
|
||||
downloads each library's archive, applies the patches, builds every library as a static library into
|
||||
`/target`, and links libvips-cpp. It packs the result as `sharp-libvips-linux-x64.tar.gz`, whose
|
||||
`lib/libvips-cpp.so.8.18.7` is the file the npm package ships. Meson runs with
|
||||
`wrap_mode = 'nofallback'` (`platforms/linux-x64/meson.ini`), so it downloads no subproject. The only
|
||||
forced fallback, gvdb, builds from the copy inside glib's archive.
|
||||
|
||||
## Rebuilding from this source
|
||||
|
||||
You need a Linux x86-64 host with Docker. Building the container image needs network access to fetch
|
||||
the toolchain (Rocky Linux packages, rustup, pip, and crates.io for cargo-c). Source for the library
|
||||
itself comes only from this directory.
|
||||
|
||||
1. **Unpack the build definitions.**
|
||||
|
||||
```sh
|
||||
tar -xzf sources/sharp-libvips-1.3.4.tar.gz # creates sharp-libvips-1.3.4/
|
||||
```
|
||||
|
||||
2. **Point the build at this source.**
|
||||
|
||||
```sh
|
||||
sh build/prepare-offline.sh sharp-libvips-1.3.4
|
||||
```
|
||||
|
||||
It copies the archives and patches into `sharp-libvips-1.3.4/offline/`. It unpacks the crates into
|
||||
a vendored directory source; Cargo checks each against the lockfile's checksum. It then makes three
|
||||
mechanical edits:
|
||||
- in `build/posix.sh`, `CURL` becomes `offline/offline-curl.sh`. That script answers each URL the
|
||||
script would download with its local copy, using `build/url-map.tsv`, and fails for any other URL;
|
||||
- in `build/posix.sh`, `cargo update --workspace` becomes a copy of the resolved `Cargo.lock`. Cargo
|
||||
is set offline and uses the vendored crates;
|
||||
- in `platforms/linux-x64/Dockerfile`, the Rust toolchain is pinned to `nightly-2026-09-27`, the
|
||||
toolchain whose `rustc` built the original. The upstream file installs whatever nightly is
|
||||
current.
|
||||
|
||||
Nothing that is compiled changes: every patch is still applied by `build/posix.sh`, exactly as
|
||||
upstream.
|
||||
|
||||
3. **Build.**
|
||||
|
||||
```sh
|
||||
cd sharp-libvips-1.3.4
|
||||
./build.sh linux-x64
|
||||
```
|
||||
|
||||
The result is `sharp-libvips-1.3.4/sharp-libvips-linux-x64.tar.gz`. Its
|
||||
`lib/libvips-cpp.so.8.18.7` is the rebuilt library. A full build takes from tens of minutes to an
|
||||
hour, depending on the host.
|
||||
|
||||
**Modifying a component.** Before step 3, change the source the build uses. Then `build/posix.sh`
|
||||
compiles your version in place of the original:
|
||||
- for a C or C++ library, replace its archive in `offline/files/` under the same name, or add a patch
|
||||
to the matching step of `build/posix.sh`;
|
||||
- for a Rust crate, edit it under `offline/vendor/`. The `.cargo-checksum.json` files list no
|
||||
per-file checksums, so an edited crate still builds.
|
||||
|
||||
The result is not byte-identical to the redistributed binary: a compiler toolchain, timestamps, and
|
||||
paths that were not recorded can all differ. It is built from the same source, by the same
|
||||
definitions, for the same target.
|
||||
|
||||
## Using a rebuilt or modified library
|
||||
|
||||
sharp's native addon, `@img/sharp-linux-x64/lib/sharp-linux-x64-0.35.5.node`, links
|
||||
`libvips-cpp.so.8.18.7` dynamically. Its RPATH begins with `$ORIGIN/../../sharp-libvips-linux-x64/lib`.
|
||||
In an installed or extracted Ekklesia Module Package, that resolves to:
|
||||
|
||||
```text
|
||||
<module directory>/node_modules/@img/sharp-libvips-linux-x64/lib/libvips-cpp.so.8.18.7
|
||||
```
|
||||
|
||||
To use your library, replace that file with your build's `lib/libvips-cpp.so.8.18.7`. Keep the file
|
||||
name and SONAME, and keep libvips's C++ API compatible with libvips 8.18, which sharp 0.35.5 is
|
||||
compiled against. Then restart the process that loads the module. No other file changes, and the
|
||||
addon does not need rebuilding.
|
||||
|
||||
`ldd <module directory>/node_modules/@img/sharp-linux-x64/lib/sharp-linux-x64-0.35.5.node` shows which
|
||||
`libvips-cpp.so.8.18.7` the addon resolves.
|
||||
|
||||
A module package's own `ekklesia.package.json` records the SHA-256 of each file it shipped, so a
|
||||
modified tree is no longer the package Ekklesia verified at installation. Install it as your own
|
||||
build.
|
||||
Generated
Vendored
+3440
File diff suppressed because it is too large
Load Diff
Vendored
+25
@@ -0,0 +1,25 @@
|
||||
#!/bin/sh
|
||||
# Stands in for curl in sharp-libvips's build/posix.sh, so the build reads only this artifact's
|
||||
# sources. Given the URL posix.sh would download, it prints the recorded local copy, or with -O
|
||||
# writes it to the current directory under the URL's file name. A URL with no recorded copy fails:
|
||||
# the build never reaches the network for source.
|
||||
set -eu
|
||||
write=""
|
||||
url=""
|
||||
for argument in "$@"; do
|
||||
case "$argument" in
|
||||
-O) write=yes ;;
|
||||
-*) ;;
|
||||
*) url="$argument" ;;
|
||||
esac
|
||||
done
|
||||
file="$(awk -F '\t' -v url="$url" '$1 == url { print $2 }' /packaging/offline/url-map.tsv)"
|
||||
if [ -z "$file" ]; then
|
||||
echo "offline-curl: no recorded source for $url" >&2
|
||||
exit 22
|
||||
fi
|
||||
if [ -n "$write" ]; then
|
||||
cp "/packaging/offline/files/$file" "./${url##*/}"
|
||||
else
|
||||
cat "/packaging/offline/files/$file"
|
||||
fi
|
||||
+61
@@ -0,0 +1,61 @@
|
||||
#!/bin/sh
|
||||
# Prepares an extracted sharp-libvips v1.3.4 source tree to build lib/libvips-cpp.so.8.18.7 from this
|
||||
# artifact's sources alone. BUILDING.md describes each step.
|
||||
#
|
||||
# sh build/prepare-offline.sh <extracted sharp-libvips-1.3.4 directory>
|
||||
#
|
||||
# Only the source acquisition changes; nothing that is compiled does:
|
||||
# - every archive and patch build/posix.sh downloads is read from this artifact (offline-curl.sh);
|
||||
# - librsvg's Rust crates come from this artifact as a vendored directory source, and its lockfile
|
||||
# is the resolved one recorded here instead of a fresh `cargo update` against crates.io;
|
||||
# - platforms/linux-x64/Dockerfile installs the exact nightly Rust toolchain the binary records.
|
||||
set -eu
|
||||
here="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
tree="$(cd "$1" && pwd)"
|
||||
[ -f "$tree/build/posix.sh" ] && [ -f "$tree/platforms/linux-x64/Dockerfile" ] || {
|
||||
echo "prepare-offline: $1 is not an extracted sharp-libvips source tree." >&2
|
||||
exit 1
|
||||
}
|
||||
offline="$tree/offline"
|
||||
rm -rf "$offline"
|
||||
mkdir -p "$offline/files" "$offline/vendor"
|
||||
|
||||
cp "$here"/sources/*.tar.* "$here"/patches/*.patch "$offline/files/"
|
||||
cp "$tree/THIRD-PARTY-NOTICES.md" "$offline/files/THIRD-PARTY-NOTICES.md"
|
||||
cp "$here/build/url-map.tsv" "$here/build/Cargo.lock" "$offline/"
|
||||
cp "$here/build/offline-curl.sh" "$offline/offline-curl.sh"
|
||||
chmod 755 "$offline/offline-curl.sh"
|
||||
|
||||
for crate in "$here"/sources/crates/*.crate; do
|
||||
tar -xzf "$crate" -C "$offline/vendor"
|
||||
name="$(basename "$crate" .crate)"
|
||||
checksum="$(sha256sum "$crate" | cut -d ' ' -f 1)"
|
||||
printf '{"files":{},"package":"%s"}\n' "$checksum" > "$offline/vendor/$name/.cargo-checksum.json"
|
||||
done
|
||||
cat > "$offline/cargo-config.toml" <<'CONFIG'
|
||||
[source.crates-io]
|
||||
replace-with = "vendored-sources"
|
||||
|
||||
[source.vendored-sources]
|
||||
directory = "/packaging/offline/vendor"
|
||||
|
||||
[net]
|
||||
offline = true
|
||||
CONFIG
|
||||
|
||||
posix="$tree/build/posix.sh"
|
||||
grep -q '^CURL="curl ' "$posix" && grep -q '^cargo update --workspace$' "$posix" || {
|
||||
echo "prepare-offline: build/posix.sh is not the v1.3.4 script this artifact was prepared for." >&2
|
||||
exit 1
|
||||
}
|
||||
sed -i \
|
||||
-e 's|^CURL="curl .*"$|CURL="/packaging/offline/offline-curl.sh"\nexport CARGO_NET_OFFLINE=true\nmkdir -p "${CARGO_HOME:-$HOME/.cargo}" \&\& cp /packaging/offline/cargo-config.toml "${CARGO_HOME:-$HOME/.cargo}/config.toml"|' \
|
||||
-e 's|^cargo update --workspace$|cp /packaging/offline/Cargo.lock Cargo.lock|' \
|
||||
"$posix"
|
||||
dockerfile="$tree/platforms/linux-x64/Dockerfile"
|
||||
grep -q -- '--default-toolchain nightly ' "$dockerfile" || {
|
||||
echo "prepare-offline: platforms/linux-x64/Dockerfile is not the v1.3.4 one." >&2
|
||||
exit 1
|
||||
}
|
||||
sed -i 's|--default-toolchain nightly |--default-toolchain nightly-2026-09-27 |' "$dockerfile"
|
||||
echo "prepare-offline: $tree builds from this artifact's sources."
|
||||
Vendored
+33
@@ -0,0 +1,33 @@
|
||||
https://cairographics.org/releases/cairo-1.18.6.tar.xz cairo-1.18.6.tar.xz
|
||||
https://cairographics.org/releases/pixman-0.46.4.tar.gz pixman-0.46.4.tar.gz
|
||||
https://download.gnome.org/sources/glib/2.90/glib-2.90.0.tar.xz glib-2.90.0.tar.xz
|
||||
https://download.gnome.org/sources/librsvg/2.63/librsvg-2.63.2.tar.xz librsvg-2.63.2.tar.xz
|
||||
https://download.gnome.org/sources/libxml2/2.15/libxml2-2.15.4.tar.xz libxml2-2.15.4.tar.xz
|
||||
https://download.gnome.org/sources/pango/1.58/pango-1.58.2.tar.xz pango-1.58.2.tar.xz
|
||||
https://gist.github.com/kleisauke/284d685efa00908da99ea6afbaaf39ae/raw/bdad5489a61c217850631571caf57f5db6ea8b2c/glib-without-gregex.patch glib-without-gregex.patch
|
||||
https://gist.githubusercontent.com/lovell/313a6901e9db1bf285f2a1f1180499e4/raw/3988223c7dfa4d22745d9392034b0117abef1446/libvips-cpp-soversion.patch libvips-cpp-soversion.patch
|
||||
https://github.com/GNOME/librsvg/commit/9106011db93d701728ac2c9da50c9ab2c1bb5dc6.patch librsvg-9106011db93d701728ac2c9da50c9ab2c1bb5dc6.patch
|
||||
https://github.com/dloebl/cgif/archive/v0.5.4.tar.gz cgif-0.5.4.tar.gz
|
||||
https://github.com/freetype/freetype/archive/VER-2-14-3.tar.gz freetype-2.14.3.tar.gz
|
||||
https://github.com/fribidi/fribidi/releases/download/v1.0.17/fribidi-1.0.17.tar.xz fribidi-1.0.17.tar.xz
|
||||
https://github.com/google/highway/archive/1.4.0.tar.gz highway-1.4.0.tar.gz
|
||||
https://github.com/google/libultrahdr/archive/v2.0.2.tar.gz libultrahdr-2.0.2.tar.gz
|
||||
https://github.com/harfbuzz/harfbuzz/archive/14.5.0.tar.gz harfbuzz-14.5.0.tar.gz
|
||||
https://github.com/libarchive/libarchive/releases/download/v3.8.9/libarchive-3.8.9.tar.xz libarchive-3.8.9.tar.xz
|
||||
https://github.com/libexif/libexif/releases/download/v0.6.26/libexif-0.6.26.tar.xz libexif-0.6.26.tar.xz
|
||||
https://github.com/libexpat/libexpat/releases/download/R_2_8_5/expat-2.8.5.tar.xz expat-2.8.5.tar.xz
|
||||
https://github.com/libffi/libffi/releases/download/v3.8.0/libffi-3.8.0.tar.gz libffi-3.8.0.tar.gz
|
||||
https://github.com/libvips/libvips/releases/download/v8.18.7/vips-8.18.7.tar.xz vips-8.18.7.tar.xz
|
||||
https://github.com/lovell/libimagequant/archive/v2.4.1.tar.gz libimagequant-2.4.1.tar.gz
|
||||
https://github.com/mm2/Little-CMS/releases/download/lcms2.19.1/lcms2-2.19.1.tar.gz lcms2-2.19.1.tar.gz
|
||||
https://github.com/mozilla/mozjpeg/archive/0826579.tar.gz mozjpeg-0826579.tar.gz
|
||||
https://github.com/mozilla/mozjpeg/commit/f90668e0e4fb79c81e1f24a0ccc0e2090af761bf.patch mozjpeg-f90668e0e4fb79c81e1f24a0ccc0e2090af761bf.patch
|
||||
https://github.com/pnggroup/libpng/archive/v1.6.58.tar.gz libpng-1.6.58.tar.gz
|
||||
https://github.com/strukturag/libheif/releases/download/v1.23.5/libheif-1.23.5.tar.gz libheif-1.23.5.tar.gz
|
||||
https://github.com/zlib-ng/zlib-ng/archive/2.3.3.tar.gz zlib-ng-2.3.3.tar.gz
|
||||
https://gitlab.com/libtiff/libtiff/-/archive/v4.7.2/libtiff-v4.7.2.tar.gz libtiff-v4.7.2.tar.gz
|
||||
https://gitlab.freedesktop.org/fontconfig/fontconfig/-/archive/2.18.3/fontconfig-2.18.3.tar.gz fontconfig-2.18.3.tar.gz
|
||||
https://patch-diff.githubusercontent.com/raw/google/libultrahdr/pull/383.patch libultrahdr-pull-383.patch
|
||||
https://raw.githubusercontent.com/lovell/sharp-libvips/main/THIRD-PARTY-NOTICES.md THIRD-PARTY-NOTICES.md
|
||||
https://storage.googleapis.com/aom-releases/libaom-3.15.1.tar.gz libaom-3.15.1.tar.gz
|
||||
https://storage.googleapis.com/downloads.webmproject.org/releases/webp/libwebp-1.6.0.tar.gz libwebp-1.6.0.tar.gz
|
||||
|
+7538
File diff suppressed because it is too large
Load Diff
@@ -5,11 +5,13 @@
|
||||
"type": "module",
|
||||
"files": [
|
||||
"bin",
|
||||
"corresponding-sources",
|
||||
"dist",
|
||||
"license-supplements",
|
||||
"licenses"
|
||||
],
|
||||
"bin": {
|
||||
"ekklesia-corresponding-source": "./bin/ekklesia-corresponding-source.mjs",
|
||||
"ekklesia-pack-module": "./bin/ekklesia-pack-module.mjs"
|
||||
},
|
||||
"exports": {
|
||||
|
||||
@@ -0,0 +1,155 @@
|
||||
#!/usr/bin/env node
|
||||
import path from "node:path";
|
||||
import process from "node:process";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import {
|
||||
acquireCorrespondingSources,
|
||||
buildCorrespondingSource,
|
||||
CorrespondingSourceError,
|
||||
verifyRedistribution,
|
||||
} from "./corresponding-source.js";
|
||||
|
||||
/**
|
||||
* `ekklesia-corresponding-source` — corresponding-source artifacts for native packages (ADR 0055).
|
||||
*
|
||||
* ekklesia-corresponding-source acquire (--package <module package> | --recipe <name>@<version>) --cache <directory>
|
||||
* Downloads every recorded source file the package's covered native dependencies (or the named
|
||||
* recipe) need into the verified cache. The only subcommand that uses the network.
|
||||
* ekklesia-corresponding-source build --package <module package> --cache <directory> --out <directory>
|
||||
* Offline. Writes <package>-corresponding-source.tar.gz, its .sha256, <package>.redistribution.json,
|
||||
* and <package>.source-notice.txt.
|
||||
* ekklesia-corresponding-source verify --package <module package> [--source <archive> --metadata <json>]
|
||||
* The public-distribution gate. Exits 0 only when the package is approved.
|
||||
*/
|
||||
|
||||
const USAGE = [
|
||||
"Usage:",
|
||||
" ekklesia-corresponding-source acquire (--package <archive> | --recipe <name>@<version>) --cache <directory>",
|
||||
" ekklesia-corresponding-source build --package <archive> --cache <directory> --out <directory>",
|
||||
" ekklesia-corresponding-source verify --package <archive> [--source <archive> --metadata <file>]",
|
||||
].join("\n");
|
||||
|
||||
const OPTIONS: Readonly<Record<string, readonly string[]>> = Object.freeze({
|
||||
acquire: ["--package", "--recipe", "--cache"],
|
||||
build: ["--package", "--cache", "--out"],
|
||||
verify: ["--package", "--source", "--metadata"],
|
||||
});
|
||||
const REQUIRED: Readonly<Record<string, readonly string[]>> = Object.freeze({
|
||||
acquire: ["--cache"],
|
||||
build: ["--package", "--cache", "--out"],
|
||||
verify: ["--package"],
|
||||
});
|
||||
|
||||
class ArgumentsError extends Error {}
|
||||
|
||||
export function parseCorrespondingSourceArguments(argv: readonly string[]): {
|
||||
readonly command: "acquire" | "build" | "verify";
|
||||
readonly values: ReadonlyMap<string, string>;
|
||||
} {
|
||||
const [command, ...rest] = argv;
|
||||
const allowed = command === undefined ? undefined : OPTIONS[command];
|
||||
if (allowed === undefined) {
|
||||
throw new ArgumentsError(USAGE);
|
||||
}
|
||||
const values = new Map<string, string>();
|
||||
for (let index = 0; index < rest.length; index += 2) {
|
||||
const option = rest[index] ?? "";
|
||||
const value = rest[index + 1];
|
||||
if (
|
||||
!allowed.includes(option) ||
|
||||
value === undefined ||
|
||||
value.startsWith("--") ||
|
||||
values.has(option)
|
||||
) {
|
||||
throw new ArgumentsError(USAGE);
|
||||
}
|
||||
values.set(option, option === "--recipe" ? value : path.resolve(value));
|
||||
}
|
||||
if (command === "acquire" && values.has("--package") === values.has("--recipe")) {
|
||||
throw new ArgumentsError(USAGE);
|
||||
}
|
||||
if ((REQUIRED[command as string] ?? []).some((option) => !values.has(option))) {
|
||||
throw new ArgumentsError(USAGE);
|
||||
}
|
||||
return { command: command as "acquire" | "build" | "verify", values };
|
||||
}
|
||||
|
||||
async function main(): Promise<number> {
|
||||
try {
|
||||
const { command, values } = parseCorrespondingSourceArguments(process.argv.slice(2));
|
||||
const packageArchivePath = values.get("--package") ?? "";
|
||||
if (command === "acquire") {
|
||||
const recipe = values.get("--recipe");
|
||||
const at = recipe?.lastIndexOf("@") ?? -1;
|
||||
if (recipe !== undefined && at <= 0) {
|
||||
throw new ArgumentsError(USAGE);
|
||||
}
|
||||
const result = await acquireCorrespondingSources({
|
||||
cacheDirectory: values.get("--cache") ?? "",
|
||||
...(recipe === undefined
|
||||
? { packageArchivePath }
|
||||
: { recipes: [{ name: recipe.slice(0, at), version: recipe.slice(at + 1) }] }),
|
||||
});
|
||||
process.stdout.write(
|
||||
`Source cache holds all ${result.files} recorded files (${result.bytes} bytes); fetched ${result.fetched}.\n`,
|
||||
);
|
||||
} else if (command === "build") {
|
||||
const result = await buildCorrespondingSource({
|
||||
cacheDirectory: values.get("--cache") ?? "",
|
||||
outDirectory: values.get("--out") ?? "",
|
||||
packageArchivePath,
|
||||
});
|
||||
if (!result.required) {
|
||||
process.stdout.write(
|
||||
"The package redistributes no covered native code: no artifact is needed.\n",
|
||||
);
|
||||
} else {
|
||||
process.stdout.write(
|
||||
`Wrote ${path.basename(result.sourceArchivePath ?? "")}, ${result.sourceBytes} bytes\nsha256 ${result.sourceSha256}\n`,
|
||||
);
|
||||
}
|
||||
} else {
|
||||
const metadataPath = values.get("--metadata");
|
||||
const sourceArchivePath = values.get("--source");
|
||||
const approval = await verifyRedistribution({
|
||||
packageArchivePath,
|
||||
...(metadataPath === undefined ? {} : { metadataPath }),
|
||||
...(sourceArchivePath === undefined ? {} : { sourceArchivePath }),
|
||||
});
|
||||
process.stdout.write(
|
||||
approval.correspondingSource === undefined
|
||||
? `Approved: ${approval.package.file} redistributes no covered native code.\n`
|
||||
: `Approved: ${approval.package.file} with ${approval.correspondingSource.file} (sha256 ${approval.correspondingSource.sha256}).\n`,
|
||||
);
|
||||
}
|
||||
return 0;
|
||||
} catch (error) {
|
||||
if (error instanceof ArgumentsError) {
|
||||
process.stderr.write(`${error.message}\n`);
|
||||
return 64;
|
||||
}
|
||||
if (error instanceof CorrespondingSourceError) {
|
||||
process.stderr.write(
|
||||
`${error.code}: ${error.message}${error.subject ? ` (${error.subject})` : ""}\n`,
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
process.stderr.write("ekklesia-corresponding-source failed unexpectedly.\n");
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
function isDirectExecution(): boolean {
|
||||
const entrypoint = process.argv[1];
|
||||
// Run through the committed launcher, which imports this file, or directly.
|
||||
return (
|
||||
entrypoint !== undefined &&
|
||||
(import.meta.url === pathToFileURL(path.resolve(entrypoint)).href ||
|
||||
path.basename(entrypoint) === "ekklesia-corresponding-source.mjs" ||
|
||||
path.basename(entrypoint) === "ekklesia-corresponding-source")
|
||||
);
|
||||
}
|
||||
|
||||
if (isDirectExecution()) {
|
||||
process.exitCode = await main();
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1 +1,2 @@
|
||||
export * from "./corresponding-source.js";
|
||||
export * from "./pack-module.js";
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { gzipSync } from "node:zlib";
|
||||
import { gunzipSync, gzipSync } from "node:zlib";
|
||||
|
||||
/**
|
||||
* The deterministic archive encoding of Ekklesia Module Package v1 (ADR 0049).
|
||||
@@ -107,3 +107,101 @@ export function deterministicTarGz(topLevel: string, files: ReadonlyMap<string,
|
||||
parts.push(Buffer.alloc(BLOCK * 2));
|
||||
return gzipSync(Buffer.concat(parts), { level: GZIP_LEVEL });
|
||||
}
|
||||
|
||||
/** One entry of a ustar archive, as read back. */
|
||||
export interface TarEntry {
|
||||
readonly path: string;
|
||||
readonly directory: boolean;
|
||||
readonly data: Buffer;
|
||||
}
|
||||
|
||||
export class TarFormatError extends Error {
|
||||
constructor() {
|
||||
super("An archive is not in Ekklesia's deterministic ustar encoding.");
|
||||
this.name = "TarFormatError";
|
||||
}
|
||||
}
|
||||
|
||||
function field(block: Buffer, offset: number, length: number): string {
|
||||
const raw = block.subarray(offset, offset + length);
|
||||
const end = raw.indexOf(0);
|
||||
return raw.subarray(0, end === -1 ? length : end).toString("latin1");
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads an archive written by `deterministicTarGz` back into its entries, refusing anything that
|
||||
* encoding never produces: a link or special file, another header type, a bad checksum, or data
|
||||
* after the end blocks. Whether the archive is byte-for-byte canonical is checked separately, by
|
||||
* writing the entries again (`isCanonicalTarGz`).
|
||||
*/
|
||||
export function readTarGz(archive: Buffer): readonly TarEntry[] {
|
||||
let raw: Buffer;
|
||||
try {
|
||||
raw = gunzipSync(archive);
|
||||
} catch {
|
||||
throw new TarFormatError();
|
||||
}
|
||||
const entries: TarEntry[] = [];
|
||||
let offset = 0;
|
||||
for (;;) {
|
||||
if (offset + BLOCK > raw.length) {
|
||||
throw new TarFormatError();
|
||||
}
|
||||
const block = raw.subarray(offset, offset + BLOCK);
|
||||
if (block.every((byte) => byte === 0)) {
|
||||
break;
|
||||
}
|
||||
const stored = Number.parseInt(field(block, 148, 8).trim() || "x", 8);
|
||||
let checksum = 0;
|
||||
for (let index = 0; index < BLOCK; index += 1) {
|
||||
checksum += index >= 148 && index < 156 ? 32 : (block[index] ?? 0);
|
||||
}
|
||||
const typeflag = field(block, 156, 1);
|
||||
const size = Number.parseInt(field(block, 124, 12) || "x", 8);
|
||||
if (
|
||||
stored !== checksum ||
|
||||
(typeflag !== "0" && typeflag !== "5") ||
|
||||
!Number.isSafeInteger(size)
|
||||
) {
|
||||
throw new TarFormatError();
|
||||
}
|
||||
const name = field(block, 0, 100);
|
||||
const prefix = field(block, 345, 155);
|
||||
const joined = prefix === "" ? name : `${prefix}/${name}`;
|
||||
const directory = typeflag === "5";
|
||||
const start = offset + BLOCK;
|
||||
if (start + size > raw.length) {
|
||||
throw new TarFormatError();
|
||||
}
|
||||
entries.push({
|
||||
data: Buffer.from(raw.subarray(start, start + size)),
|
||||
directory,
|
||||
path: directory ? joined.replace(/\/$/u, "") : joined,
|
||||
});
|
||||
offset = start + Math.ceil(size / BLOCK) * BLOCK;
|
||||
}
|
||||
return entries;
|
||||
}
|
||||
|
||||
/** Whether `archive` is exactly what `deterministicTarGz` writes for its own entries: the same
|
||||
* order, headers, and padding. Compared uncompressed, so a different zlib cannot matter. */
|
||||
export function isCanonicalTarGz(archive: Buffer, entries: readonly TarEntry[]): boolean {
|
||||
const topLevel = entries[0];
|
||||
if (topLevel === undefined || !topLevel.directory || topLevel.path.includes("/")) {
|
||||
return false;
|
||||
}
|
||||
const files = new Map<string, Buffer>();
|
||||
for (const entry of entries) {
|
||||
if (!entry.directory) {
|
||||
if (!entry.path.startsWith(`${topLevel.path}/`)) {
|
||||
return false;
|
||||
}
|
||||
files.set(entry.path.slice(topLevel.path.length + 1), entry.data);
|
||||
}
|
||||
}
|
||||
try {
|
||||
return gunzipSync(deterministicTarGz(topLevel.path, files)).equals(gunzipSync(archive));
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,342 @@
|
||||
import { execFile } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import { existsSync } from "node:fs";
|
||||
import { readFile, realpath, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { promisify } from "node:util";
|
||||
import { currentModulePackageHost, verifyModulePackage } from "@ekklesia/core/modules";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import {
|
||||
buildCorrespondingSource,
|
||||
CorrespondingSourceError,
|
||||
loadSourceRecipes,
|
||||
verifyRedistribution,
|
||||
} from "../src/corresponding-source.js";
|
||||
import { packModule } from "../src/index.js";
|
||||
import { NATIVE_TARGETS, nativeClosure } from "../src/native.js";
|
||||
import {
|
||||
contentDigest,
|
||||
DEFAULT_SUPPLEMENTS_DIRECTORY,
|
||||
loadLicenseSupplements,
|
||||
} from "../src/supplements.js";
|
||||
import { readTarGz } from "../src/tar.js";
|
||||
import {
|
||||
cleanupFixtures,
|
||||
installedPackageFile,
|
||||
nativeFixtureModule,
|
||||
SOURCE_REVISION,
|
||||
scratchDirectory,
|
||||
} from "./support/fixture-module.js";
|
||||
|
||||
/**
|
||||
* ADR 0055's first complete proof: the native proof module with sharp 0.35.5, whose libvips 1.3.4
|
||||
* binary contains LGPL and MPL components, and its real corresponding-source artifact.
|
||||
*
|
||||
* The recipe checks run everywhere. Building the artifact needs the recorded sources in a verified
|
||||
* cache, which `pnpm test:corresponding-source` acquires once and passes in as
|
||||
* EKKLESIA_CORRESPONDING_SOURCE_CACHE; without it, that part is skipped rather than downloading
|
||||
* about 200 MB on every run.
|
||||
*/
|
||||
|
||||
const execFileAsync = promisify(execFile);
|
||||
const packageRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const repositoryRoot = path.resolve(packageRoot, "..", "..");
|
||||
const LIBVIPS = "@img/sharp-libvips-linux-x64";
|
||||
const cache = process.env.EKKLESIA_CORRESPONDING_SOURCE_CACHE;
|
||||
const host = currentModulePackageHost();
|
||||
const onTargetHost =
|
||||
host.platform === "linux" && host.arch === "x64" && host.libc?.family === "glibc";
|
||||
|
||||
afterEach(cleanupFixtures);
|
||||
|
||||
// `pnpm test:corresponding-source` asks for the full proof: then a missing cache is a failure, not a
|
||||
// skip.
|
||||
it.runIf(process.env.EKKLESIA_CORRESPONDING_SOURCE_REQUIRED === "1")(
|
||||
"has the verified source cache it was asked to use, on a linux-x64-glibc host",
|
||||
() => {
|
||||
expect(onTargetHost).toBe(true);
|
||||
expect(cache !== undefined && existsSync(cache)).toBe(true);
|
||||
},
|
||||
);
|
||||
|
||||
describe("the libvips 1.3.4 source recipe", () => {
|
||||
it("loads intact, matches its reviewed supplement exactly, and pins the installed package", async () => {
|
||||
const recipes = await loadSourceRecipes();
|
||||
const [loaded] = recipes.get(LIBVIPS) ?? [];
|
||||
expect(loaded?.recipe.package).toMatchObject({
|
||||
license: "LGPL-3.0-or-later",
|
||||
target: "linux-x64-glibc",
|
||||
version: "1.3.4",
|
||||
});
|
||||
|
||||
// Prepared against exactly the reviewed supplement for these package bytes.
|
||||
const supplementJson = await readFile(
|
||||
path.join(DEFAULT_SUPPLEMENTS_DIRECTORY, LIBVIPS, "1.3.4", "supplement.json"),
|
||||
);
|
||||
expect(loaded?.recipe.supplement.sha256).toBe(
|
||||
createHash("sha256").update(supplementJson).digest("hex"),
|
||||
);
|
||||
const supplement = (await loadLicenseSupplements())
|
||||
.get(LIBVIPS)
|
||||
?.find((candidate) => candidate.version === "1.3.4");
|
||||
expect(loaded?.recipe.package.contentDigest).toBe(supplement?.contentDigest);
|
||||
|
||||
// And those are the bytes sharp 0.35.5 installs.
|
||||
const root = await realpath(await installedPackageFile("sharp"));
|
||||
const target = NATIVE_TARGETS.get("linux-x64-glibc");
|
||||
if (target === undefined) throw new Error("target");
|
||||
const installed = (await nativeClosure([{ name: "sharp", root }], target)).find(
|
||||
(entry) => entry.name === LIBVIPS,
|
||||
);
|
||||
expect(installed && contentDigest(installed.files)).toBe(loaded?.recipe.package.contentDigest);
|
||||
|
||||
// Every component the supplement records inside the binary maps to a source component. The one
|
||||
// exception is the package's own entry, which carries the standard texts of its declared licence.
|
||||
const referenced = new Set(
|
||||
(loaded?.recipe.components ?? []).flatMap((component) =>
|
||||
component.licenseMaterial.map((material) => `${material.name} ${material.version}`),
|
||||
),
|
||||
);
|
||||
const unmapped = (supplement?.components ?? [])
|
||||
.map((component) => `${component.name} ${component.version}`)
|
||||
.filter((component) => !referenced.has(component));
|
||||
expect(unmapped).toEqual([`${LIBVIPS} 1.3.4`]);
|
||||
|
||||
const components = loaded?.recipe.components ?? [];
|
||||
const byRole = (role: string) =>
|
||||
components.filter((component) => component.role === role).length;
|
||||
expect({
|
||||
buildDefinition: byRole("build-definition"),
|
||||
buildInput: byRole("build-input"),
|
||||
compiled: byRole("compiled"),
|
||||
}).toEqual({
|
||||
buildDefinition: 1,
|
||||
buildInput: 206,
|
||||
compiled: 176,
|
||||
});
|
||||
// The five patches build/posix.sh applies, and the Rust standard library of the recorded nightly.
|
||||
expect(
|
||||
components.flatMap((component) => component.patches.map((patch) => patch.file)).sort(),
|
||||
).toEqual([
|
||||
"patches/glib-without-gregex.patch",
|
||||
"patches/librsvg-9106011db93d701728ac2c9da50c9ab2c1bb5dc6.patch",
|
||||
"patches/libultrahdr-pull-383.patch",
|
||||
"patches/libvips-cpp-soversion.patch",
|
||||
"patches/mozjpeg-f90668e0e4fb79c81e1f24a0ccc0e2090af761bf.patch",
|
||||
]);
|
||||
expect(
|
||||
components.find((component) => component.name === "Rust standard library")?.sources[0]?.file,
|
||||
).toBe("sources/rust/rust-src-nightly-2026-09-27.tar.xz");
|
||||
// gvdb ships inside glib's archive; proxy-libintl is never built for this target.
|
||||
expect(components.find((component) => component.name === "gvdb")?.includedIn).toEqual({
|
||||
component: "glib",
|
||||
path: "subprojects/gvdb",
|
||||
});
|
||||
expect(components.some((component) => component.name === "proxy-libintl")).toBe(false);
|
||||
expect(loaded?.recipe.exclusions.map((exclusion) => exclusion.name)).toContain("proxy-libintl");
|
||||
});
|
||||
});
|
||||
|
||||
async function packNative(): Promise<string> {
|
||||
const moduleDirectory = await nativeFixtureModule();
|
||||
const outDirectory = await scratchDirectory("ekklesia-native-out-");
|
||||
const result = await packModule({
|
||||
licenseFile: path.join(repositoryRoot, "LICENSE"),
|
||||
moduleDirectory,
|
||||
nodeRange: ">=24.11.0 <25",
|
||||
outDirectory,
|
||||
sourceRevision: SOURCE_REVISION,
|
||||
target: "linux-x64-glibc",
|
||||
});
|
||||
return result.archivePath;
|
||||
}
|
||||
|
||||
describe.runIf(onTargetHost && cache !== undefined && existsSync(cache ?? ""))(
|
||||
"the native proof's corresponding source, from the verified cache",
|
||||
() => {
|
||||
it("builds deterministically, is approved by the gate, and fails closed without it", {
|
||||
timeout: 600_000,
|
||||
}, async () => {
|
||||
const first = await packNative();
|
||||
const second = await packNative();
|
||||
expect((await readFile(first)).equals(await readFile(second))).toBe(true);
|
||||
|
||||
const outA = await scratchDirectory("ekklesia-source-a-");
|
||||
const outB = path.join(await scratchDirectory("ekklesia-unrelated-"), "nested", "out");
|
||||
const a = await buildCorrespondingSource({
|
||||
cacheDirectory: cache ?? "",
|
||||
outDirectory: outA,
|
||||
packageArchivePath: first,
|
||||
});
|
||||
const b = await buildCorrespondingSource({
|
||||
cacheDirectory: cache ?? "",
|
||||
outDirectory: outB,
|
||||
packageArchivePath: second,
|
||||
});
|
||||
expect(a.required).toBe(true);
|
||||
expect(a.sourceSha256).toBe(b.sourceSha256);
|
||||
expect(
|
||||
(await readFile(a.sourceArchivePath ?? "")).equals(
|
||||
await readFile(b.sourceArchivePath ?? ""),
|
||||
),
|
||||
).toBe(true);
|
||||
expect(path.basename(a.sourceArchivePath ?? "")).toBe(
|
||||
"native-proof-0.1.0-linux-x64-glibc-corresponding-source.tar.gz",
|
||||
);
|
||||
|
||||
// What it holds: every recorded source, as bytes, and nothing unrecorded.
|
||||
const archive = await readFile(a.sourceArchivePath ?? "");
|
||||
const entries = readTarGz(archive);
|
||||
const top = "native-proof-0.1.0-linux-x64-glibc-corresponding-source";
|
||||
const files = entries
|
||||
.filter((entry) => !entry.directory)
|
||||
.map((entry) => entry.path.slice(top.length + 1));
|
||||
const manifest = JSON.parse(
|
||||
entries
|
||||
.find((entry) => entry.path === `${top}/corresponding-source.json`)
|
||||
?.data.toString("utf8") ?? "{}",
|
||||
);
|
||||
const dependency = manifest.dependencies[0];
|
||||
expect(dependency).toMatchObject({
|
||||
name: LIBVIPS,
|
||||
version: "1.3.4",
|
||||
binaries: ["lib/libvips-cpp.so.8.18.7"],
|
||||
});
|
||||
const base = `dependencies/${LIBVIPS}/1.3.4`;
|
||||
const sources = files.filter((file) => file.startsWith(`${base}/sources/`));
|
||||
expect(sources.filter((file) => file.startsWith(`${base}/sources/crates/`))).toHaveLength(
|
||||
353,
|
||||
);
|
||||
expect(
|
||||
sources.filter((file) => /\/sources\/[^/]+\.tar\.(?:gz|xz)$/u.test(file)),
|
||||
).toHaveLength(28);
|
||||
expect(files.filter((file) => file.startsWith(`${base}/patches/`))).toHaveLength(5);
|
||||
for (const required of [
|
||||
"BUILDING.md",
|
||||
"PROVENANCE.md",
|
||||
"recipe.json",
|
||||
"licenses/supplement.json",
|
||||
"build/prepare-offline.sh",
|
||||
"build/Cargo.lock",
|
||||
"sources/rust/rust-src-nightly-2026-09-27.tar.xz",
|
||||
]) {
|
||||
expect(files, required).toContain(`${base}/${required}`);
|
||||
}
|
||||
expect(new Set(manifest.files.map((file: { path: string }) => file.path))).toEqual(
|
||||
new Set(files.filter((file) => file !== "corresponding-source.json")),
|
||||
);
|
||||
|
||||
// The binary-source relationship, by digest.
|
||||
const metadata = JSON.parse(await readFile(a.metadataPath ?? "", "utf8"));
|
||||
const packageSha256 = createHash("sha256")
|
||||
.update(await readFile(first))
|
||||
.digest("hex");
|
||||
expect(metadata).toMatchObject({
|
||||
correspondingSource: {
|
||||
file: path.basename(a.sourceArchivePath ?? ""),
|
||||
sha256: a.sourceSha256,
|
||||
size: archive.length,
|
||||
},
|
||||
covered: [
|
||||
{
|
||||
contentDigest:
|
||||
"ekklesia-content-v1:sha256:2927fe11493461fa34847191d0b9b1c6441007c3e99f306e5708cb07fb0a5e59",
|
||||
name: LIBVIPS,
|
||||
version: "1.3.4",
|
||||
},
|
||||
],
|
||||
package: {
|
||||
file: path.basename(first),
|
||||
module: { id: "native-proof", version: "0.1.0" },
|
||||
sha256: packageSha256,
|
||||
target: "linux-x64-glibc",
|
||||
},
|
||||
});
|
||||
expect(metadata.notice).toContain("GNU LGPL and MPL");
|
||||
expect(metadata.notice).toContain(`SHA-256 ${a.sourceSha256}`);
|
||||
|
||||
// The gate approves exactly this, and fails closed when it is removed or corrupted.
|
||||
await expect(
|
||||
verifyRedistribution({
|
||||
metadataPath: a.metadataPath ?? "",
|
||||
packageArchivePath: first,
|
||||
sourceArchivePath: a.sourceArchivePath ?? "",
|
||||
}),
|
||||
).resolves.toMatchObject({ approved: true, correspondingSource: { sha256: a.sourceSha256 } });
|
||||
const missing = verifyRedistribution({
|
||||
metadataPath: a.metadataPath ?? "",
|
||||
packageArchivePath: first,
|
||||
});
|
||||
await expect(missing).rejects.toBeInstanceOf(CorrespondingSourceError);
|
||||
await expect(missing).rejects.toMatchObject({
|
||||
code: "CORRESPONDING_SOURCE_ARTIFACT_MISSING",
|
||||
});
|
||||
const corrupted = Buffer.from(archive);
|
||||
const flip = Math.floor(corrupted.length / 3);
|
||||
corrupted[flip] = (corrupted[flip] ?? 0) ^ 0x01;
|
||||
await writeFile(b.sourceArchivePath ?? "", corrupted);
|
||||
await expect(
|
||||
verifyRedistribution({
|
||||
metadataPath: b.metadataPath ?? "",
|
||||
packageArchivePath: second,
|
||||
sourceArchivePath: b.sourceArchivePath ?? "",
|
||||
}),
|
||||
).rejects.toMatchObject({
|
||||
code: expect.stringMatching(
|
||||
/^CORRESPONDING_SOURCE_(?:ARTIFACT_INVALID|MANIFEST_MISMATCH|METADATA_MISMATCH)$/u,
|
||||
),
|
||||
});
|
||||
await rm(b.sourceArchivePath ?? "");
|
||||
await expect(
|
||||
verifyRedistribution({
|
||||
metadataPath: b.metadataPath ?? "",
|
||||
packageArchivePath: second,
|
||||
sourceArchivePath: b.sourceArchivePath ?? "",
|
||||
}),
|
||||
).rejects.toMatchObject({ code: "CORRESPONDING_SOURCE_ARTIFACT_MISSING" });
|
||||
});
|
||||
|
||||
it("leaves the binary package itself unchanged: it verifies, and sharp runs from it", {
|
||||
timeout: 300_000,
|
||||
}, async () => {
|
||||
const archivePath = await packNative();
|
||||
const archive = await readFile(archivePath);
|
||||
const extractTo = path.join(
|
||||
await scratchDirectory("ekklesia-native-installed-"),
|
||||
"native-proof",
|
||||
);
|
||||
await expect(
|
||||
verifyModulePackage({
|
||||
archivePath,
|
||||
expectedSha256: createHash("sha256").update(archive).digest("hex"),
|
||||
extractTo,
|
||||
platform: {
|
||||
databaseRuntime: { "drizzle-orm": "0.45.2" },
|
||||
moduleSdkVersion: "0.2.1",
|
||||
platformVersion: "0.2.1",
|
||||
},
|
||||
}),
|
||||
).resolves.toMatchObject({ moduleId: "native-proof", target: "linux-x64-glibc" });
|
||||
// The source archive never travels inside the package.
|
||||
expect(readTarGz(archive).some((entry) => /corresponding-source/u.test(entry.path))).toBe(
|
||||
false,
|
||||
);
|
||||
const { stdout } = await execFileAsync(
|
||||
process.execPath,
|
||||
[
|
||||
"--input-type=module",
|
||||
"-e",
|
||||
`const m = await import(${JSON.stringify(`file://${extractTo}/server/index.js`)}); process.stdout.write(JSON.stringify(await m.proof()));`,
|
||||
],
|
||||
{ cwd: tmpdir(), env: {} },
|
||||
);
|
||||
expect(JSON.parse(stdout)).toEqual({
|
||||
pixels: [255, 0, 0, 255, 0, 0, 255, 0, 0, 255, 0, 0],
|
||||
png: { format: "png", height: 3, width: 5 },
|
||||
resized: { channels: 3, height: 2, width: 2 },
|
||||
});
|
||||
});
|
||||
},
|
||||
);
|
||||
@@ -0,0 +1,774 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { mkdir, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { gunzipSync, gzipSync } from "node:zlib";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import {
|
||||
acquireCorrespondingSources,
|
||||
buildCorrespondingSource,
|
||||
CorrespondingSourceError,
|
||||
type CorrespondingSourceErrorCode,
|
||||
hasSourceObligations,
|
||||
loadSourceRecipes,
|
||||
verifyRedistribution,
|
||||
} from "../src/corresponding-source.js";
|
||||
import { contentDigest } from "../src/supplements.js";
|
||||
import { deterministicTarGz, readTarGz } from "../src/tar.js";
|
||||
import { cleanupFixtures, scratchDirectory } from "./support/fixture-module.js";
|
||||
|
||||
/**
|
||||
* Corresponding-source artifacts and the public-distribution gate (ADR 0055), on small synthetic
|
||||
* inputs: a fake LGPL native package, its reviewed supplement, its source recipe, and a source
|
||||
* cache. The real libvips proof is `corresponding-source-native.test.ts`.
|
||||
*/
|
||||
|
||||
afterEach(cleanupFixtures);
|
||||
|
||||
function sha256(data: string | Uint8Array): string {
|
||||
return createHash("sha256").update(data).digest("hex");
|
||||
}
|
||||
|
||||
const PACKAGE_FILE = "example-0.1.0-linux-x64-glibc.tar.gz";
|
||||
const SOURCE_FILE = "example-0.1.0-linux-x64-glibc-corresponding-source.tar.gz";
|
||||
const METADATA_FILE = "example-0.1.0-linux-x64-glibc.redistribution.json";
|
||||
const TOP = "example-0.1.0-linux-x64-glibc-corresponding-source";
|
||||
|
||||
const SOURCES: Readonly<Record<string, string>> = {
|
||||
"sources/fake-build-1.0.tar.gz": "build definitions for libfake\n",
|
||||
"sources/fakelib-2.0.tar.gz": "fakelib 2.0 source\n",
|
||||
"patches/fakelib-fix.patch": "--- a/x\n+++ b/x\n@@ -1 +1 @@\n-a\n+b\n",
|
||||
"sources/helperlib-1.1.tar.gz": "helperlib 1.1 source\n",
|
||||
"sources/codegen-0.3.crate": "codegen 0.3 build-only source\n",
|
||||
};
|
||||
|
||||
interface FixtureOptions {
|
||||
readonly version?: string;
|
||||
readonly libraryBytes?: string;
|
||||
readonly license?: string;
|
||||
readonly notices?: string;
|
||||
readonly recipe?: (recipe: Record<string, unknown>) => void;
|
||||
readonly withRecipe?: boolean;
|
||||
}
|
||||
|
||||
interface Fixture {
|
||||
readonly root: string;
|
||||
readonly packagePath: string;
|
||||
readonly cache: string;
|
||||
readonly recipesDirectory: string;
|
||||
readonly supplementsDirectory: string;
|
||||
}
|
||||
|
||||
function noticeEntry(name: string, version: string, licence: string, uses: string): string {
|
||||
const rule = "-".repeat(80);
|
||||
return `${rule}\n${name} ${version}\nLicense: ${licence}\nUsed in: ${uses}\n${rule}\n\n[LICENSE]\n\ntext\n`;
|
||||
}
|
||||
|
||||
/** Writes a fake native package into a Module Package archive, its supplement, recipe, and cache. */
|
||||
async function fixture(options: FixtureOptions = {}): Promise<Fixture> {
|
||||
const version = options.version ?? "1.0.0";
|
||||
const license = options.license ?? "LGPL-2.1-or-later";
|
||||
const root = await scratchDirectory("ekklesia-corresponding-");
|
||||
const nativeFiles = new Map<string, Buffer>([
|
||||
["lib/libfake.so.1", Buffer.from(options.libraryBytes ?? `\u007fELF fake library ${version}`)],
|
||||
["package.json", Buffer.from(JSON.stringify({ license, name: "fake-native", version }))],
|
||||
]);
|
||||
const digest = contentDigest(nativeFiles);
|
||||
const packageFiles = new Map<string, Buffer>([
|
||||
[
|
||||
"ekklesia.package.json",
|
||||
Buffer.from(
|
||||
JSON.stringify({
|
||||
module: { id: "example", version: "0.1.0" },
|
||||
packageFormat: 1,
|
||||
target: "linux-x64-glibc",
|
||||
}),
|
||||
),
|
||||
],
|
||||
[
|
||||
"THIRD_PARTY_NOTICES",
|
||||
Buffer.from(options.notices ?? noticeEntry("fake-native", version, license, "node_modules")),
|
||||
],
|
||||
...[...nativeFiles].map(([file, data]) => [`node_modules/fake-native/${file}`, data] as const),
|
||||
]);
|
||||
const packageDirectory = path.join(root, "package");
|
||||
await mkdir(packageDirectory);
|
||||
const packagePath = path.join(packageDirectory, PACKAGE_FILE);
|
||||
await writeFile(packagePath, deterministicTarGz("example", packageFiles));
|
||||
|
||||
// The reviewed supplement for exactly these package bytes.
|
||||
const supplementsDirectory = path.join(root, "supplements");
|
||||
const supplementDirectory = path.join(supplementsDirectory, "fake-native", version);
|
||||
await mkdir(path.join(supplementDirectory, "texts"), { recursive: true });
|
||||
const material = async (content: string) => {
|
||||
await writeFile(path.join(supplementDirectory, "texts", `${sha256(content)}.txt`), content);
|
||||
return `texts/${sha256(content)}.txt`;
|
||||
};
|
||||
const supplement = {
|
||||
supplementFormat: 1,
|
||||
package: { name: "fake-native", version, license, contentDigest: digest },
|
||||
statement: "Covers the libraries built into lib/libfake.so.1.",
|
||||
components: [
|
||||
{
|
||||
name: "fakelib",
|
||||
version: "2.0",
|
||||
license: "LGPL-2.1-or-later",
|
||||
source: "https://example.org/fakelib-2.0.tar.gz",
|
||||
material: [
|
||||
{ title: "COPYING", text: await material("GNU LESSER GENERAL PUBLIC LICENSE (fake)\n") },
|
||||
],
|
||||
},
|
||||
{
|
||||
name: "helperlib",
|
||||
version: "1.1",
|
||||
license: "MIT",
|
||||
source: "https://example.org/helperlib-1.1.tar.gz",
|
||||
material: [{ title: "LICENSE", text: await material("MIT License (fake)\n") }],
|
||||
},
|
||||
],
|
||||
};
|
||||
const supplementJson = `${JSON.stringify(supplement, null, 2)}\n`;
|
||||
await writeFile(path.join(supplementDirectory, "supplement.json"), supplementJson);
|
||||
await writeFile(path.join(supplementDirectory, "PROVENANCE.md"), "# How this was assembled\n");
|
||||
|
||||
// The recipe, and a cache holding the recorded bytes.
|
||||
const recipesDirectory = path.join(root, "recipes");
|
||||
const cache = path.join(root, "cache");
|
||||
await mkdir(path.join(cache, "sha256"), { recursive: true });
|
||||
const file = async (relative: string) => {
|
||||
const content = SOURCES[relative] ?? "";
|
||||
await writeFile(path.join(cache, "sha256", sha256(content)), content);
|
||||
return {
|
||||
file: relative,
|
||||
origin: `https://example.org/${path.basename(relative)}`,
|
||||
sha256: sha256(content),
|
||||
size: Buffer.byteLength(content),
|
||||
};
|
||||
};
|
||||
const building = "# Rebuilding libfake\n\nRun `make` in fake-build.\n";
|
||||
const recipe: Record<string, unknown> = {
|
||||
recipeFormat: 1,
|
||||
package: {
|
||||
name: "fake-native",
|
||||
version,
|
||||
license,
|
||||
contentDigest: digest,
|
||||
target: "linux-x64-glibc",
|
||||
binaries: ["lib/libfake.so.1"],
|
||||
},
|
||||
supplement: { version, sha256: sha256(supplementJson) },
|
||||
documents: [
|
||||
{ path: "BUILDING.md", sha256: sha256(building), size: Buffer.byteLength(building) },
|
||||
],
|
||||
components: [
|
||||
{
|
||||
name: "fake-build",
|
||||
version: "1.0",
|
||||
role: "build-definition",
|
||||
license: "MIT",
|
||||
relationship: "The build definitions.",
|
||||
licenseMaterial: [],
|
||||
sources: [await file("sources/fake-build-1.0.tar.gz")],
|
||||
patches: [],
|
||||
},
|
||||
{
|
||||
name: "fakelib",
|
||||
version: "2.0",
|
||||
role: "compiled",
|
||||
license: "LGPL-2.1-or-later",
|
||||
relationship: "Compiled into lib/libfake.so.1.",
|
||||
licenseMaterial: [{ name: "fakelib", version: "2.0" }],
|
||||
sources: [await file("sources/fakelib-2.0.tar.gz")],
|
||||
patches: [await file("patches/fakelib-fix.patch")],
|
||||
},
|
||||
{
|
||||
name: "helperlib",
|
||||
version: "1.1",
|
||||
role: "compiled",
|
||||
license: "MIT",
|
||||
relationship: "Compiled into lib/libfake.so.1.",
|
||||
licenseMaterial: [{ name: "helperlib", version: "1.1" }],
|
||||
sources: [await file("sources/helperlib-1.1.tar.gz")],
|
||||
patches: [],
|
||||
},
|
||||
{
|
||||
name: "codegen",
|
||||
version: "0.3",
|
||||
role: "build-input",
|
||||
license: "MIT",
|
||||
relationship: "Needed only to build.",
|
||||
licenseMaterial: [],
|
||||
sources: [await file("sources/codegen-0.3.crate")],
|
||||
patches: [],
|
||||
},
|
||||
],
|
||||
exclusions: [{ name: "testlib", reason: "Used only by tests." }],
|
||||
};
|
||||
options.recipe?.(recipe);
|
||||
if (options.withRecipe !== false) {
|
||||
const recipeDirectory = path.join(recipesDirectory, "fake-native", version);
|
||||
await mkdir(recipeDirectory, { recursive: true });
|
||||
await writeFile(
|
||||
path.join(recipeDirectory, "recipe.json"),
|
||||
`${JSON.stringify(recipe, null, 2)}\n`,
|
||||
);
|
||||
await writeFile(path.join(recipeDirectory, "BUILDING.md"), building);
|
||||
}
|
||||
return { cache, packagePath, recipesDirectory, root, supplementsDirectory };
|
||||
}
|
||||
|
||||
async function build(fx: Fixture, out = path.join(fx.root, "out")) {
|
||||
return buildCorrespondingSource({
|
||||
cacheDirectory: fx.cache,
|
||||
outDirectory: out,
|
||||
packageArchivePath: fx.packagePath,
|
||||
recipesDirectory: fx.recipesDirectory,
|
||||
supplementsDirectory: fx.supplementsDirectory,
|
||||
});
|
||||
}
|
||||
|
||||
function verify(fx: Fixture, source?: string, metadata?: string) {
|
||||
return verifyRedistribution({
|
||||
packageArchivePath: fx.packagePath,
|
||||
recipesDirectory: fx.recipesDirectory,
|
||||
supplementsDirectory: fx.supplementsDirectory,
|
||||
...(source === undefined ? {} : { sourceArchivePath: source }),
|
||||
...(metadata === undefined ? {} : { metadataPath: metadata }),
|
||||
});
|
||||
}
|
||||
|
||||
async function rejects(promise: Promise<unknown>, code: CorrespondingSourceErrorCode) {
|
||||
const error = await promise.then(
|
||||
() => undefined,
|
||||
(caught: unknown) => caught,
|
||||
);
|
||||
expect(error).toBeInstanceOf(CorrespondingSourceError);
|
||||
expect((error as CorrespondingSourceError).code).toBe(code);
|
||||
return error as CorrespondingSourceError;
|
||||
}
|
||||
|
||||
/** A built release, ready to tamper with: its source archive's files, and a way to write a changed
|
||||
* archive (and metadata naming it) where the gate will look. */
|
||||
async function release() {
|
||||
const fx = await fixture();
|
||||
const out = path.join(fx.root, "out");
|
||||
await build(fx, out);
|
||||
const sourcePath = path.join(out, SOURCE_FILE);
|
||||
const metadataPath = path.join(out, METADATA_FILE);
|
||||
const files = new Map<string, Buffer>();
|
||||
for (const entry of readTarGz(await readFile(sourcePath))) {
|
||||
if (!entry.directory) files.set(entry.path.slice(TOP.length + 1), entry.data);
|
||||
}
|
||||
/** Writes `changed` as the archive and keeps the metadata consistent with its bytes, so only the
|
||||
* change itself can fail the gate. */
|
||||
const write = async (changed: Map<string, Buffer>, archive?: Buffer) => {
|
||||
const bytes = archive ?? deterministicTarGz(TOP, changed);
|
||||
await writeFile(sourcePath, bytes);
|
||||
const metadata = JSON.parse(await readFile(metadataPath, "utf8"));
|
||||
const digest = sha256(bytes);
|
||||
metadata.correspondingSource.sha256 = digest;
|
||||
metadata.correspondingSource.size = bytes.length;
|
||||
metadata.correspondingSource.manifestSha256 = sha256(
|
||||
changed.get("corresponding-source.json") ?? "",
|
||||
);
|
||||
metadata.notice = metadata.notice.replace(/SHA-256 [0-9a-f]{64}/u, `SHA-256 ${digest}`);
|
||||
await writeFile(metadataPath, `${JSON.stringify(metadata, null, 2)}\n`);
|
||||
};
|
||||
/** Rewrites the manifest's file list to match `changed`, as a careful forger would. */
|
||||
const relist = (changed: Map<string, Buffer>) => {
|
||||
const manifest = JSON.parse(changed.get("corresponding-source.json")?.toString("utf8") ?? "{}");
|
||||
manifest.files = [...changed]
|
||||
.filter(([file]) => file !== "corresponding-source.json")
|
||||
.map(([file, data]) => ({ path: file, size: data.length, sha256: sha256(data) }))
|
||||
.sort((left: { path: string }, right: { path: string }) => (left.path < right.path ? -1 : 1));
|
||||
changed.set("corresponding-source.json", Buffer.from(`${JSON.stringify(manifest, null, 2)}\n`));
|
||||
return manifest;
|
||||
};
|
||||
return { files, fx, metadataPath, out, relist, sourcePath, write };
|
||||
}
|
||||
|
||||
describe("building the corresponding-source artifact", () => {
|
||||
it("writes a deterministic archive, its digest, the redistribution metadata, and the release notice", async () => {
|
||||
const fx = await fixture();
|
||||
const first = await build(fx, path.join(fx.root, "out-a"));
|
||||
// A second build into an unrelated directory, from a moved cache and package copy.
|
||||
const second = await build(
|
||||
fx,
|
||||
path.join(await scratchDirectory("ekklesia-elsewhere-"), "out-b"),
|
||||
);
|
||||
expect(first.required).toBe(true);
|
||||
expect(first.sourceSha256).toBe(second.sourceSha256);
|
||||
expect(await readFile(first.sourceArchivePath ?? "")).toEqual(
|
||||
await readFile(second.sourceArchivePath ?? ""),
|
||||
);
|
||||
expect(await readFile(`${first.sourceArchivePath}.sha256`, "utf8")).toBe(
|
||||
`${first.sourceSha256} ${SOURCE_FILE}\n`,
|
||||
);
|
||||
|
||||
const entries = readTarGz(await readFile(first.sourceArchivePath ?? ""));
|
||||
const files = entries
|
||||
.filter((entry) => !entry.directory)
|
||||
.map((entry) => entry.path.slice(TOP.length + 1));
|
||||
expect(files).toEqual([
|
||||
"README.md",
|
||||
"SHA256SUMS",
|
||||
"corresponding-source.json",
|
||||
"dependencies/fake-native/1.0.0/BUILDING.md",
|
||||
"dependencies/fake-native/1.0.0/PROVENANCE.md",
|
||||
"dependencies/fake-native/1.0.0/licenses/supplement.json",
|
||||
expect.stringMatching(
|
||||
/^dependencies\/fake-native\/1\.0\.0\/licenses\/texts\/[0-9a-f]{64}\.txt$/u,
|
||||
),
|
||||
expect.stringMatching(
|
||||
/^dependencies\/fake-native\/1\.0\.0\/licenses\/texts\/[0-9a-f]{64}\.txt$/u,
|
||||
),
|
||||
"dependencies/fake-native/1.0.0/patches/fakelib-fix.patch",
|
||||
"dependencies/fake-native/1.0.0/recipe.json",
|
||||
"dependencies/fake-native/1.0.0/sources/codegen-0.3.crate",
|
||||
"dependencies/fake-native/1.0.0/sources/fake-build-1.0.tar.gz",
|
||||
"dependencies/fake-native/1.0.0/sources/fakelib-2.0.tar.gz",
|
||||
"dependencies/fake-native/1.0.0/sources/helperlib-1.1.tar.gz",
|
||||
]);
|
||||
// The actual bytes, not references to them.
|
||||
const patch = entries.find((entry) => entry.path.endsWith("patches/fakelib-fix.patch"));
|
||||
expect(patch?.data.toString("utf8")).toBe(SOURCES["patches/fakelib-fix.patch"]);
|
||||
|
||||
const manifest = JSON.parse(
|
||||
entries
|
||||
.find((entry) => entry.path.endsWith("/corresponding-source.json"))
|
||||
?.data.toString("utf8") ?? "{}",
|
||||
);
|
||||
expect(manifest.module).toEqual({ id: "example", target: "linux-x64-glibc", version: "0.1.0" });
|
||||
expect(manifest.dependencies[0]).toMatchObject({
|
||||
binaries: ["lib/libfake.so.1"],
|
||||
exclusions: [{ name: "testlib", reason: "Used only by tests." }],
|
||||
name: "fake-native",
|
||||
version: "1.0.0",
|
||||
});
|
||||
expect(
|
||||
manifest.dependencies[0].components.map((component: { role: string }) => component.role),
|
||||
).toEqual(["build-definition", "compiled", "compiled", "build-input"]);
|
||||
|
||||
const metadata = JSON.parse(await readFile(first.metadataPath ?? "", "utf8"));
|
||||
expect(metadata).toMatchObject({
|
||||
correspondingSource: {
|
||||
file: SOURCE_FILE,
|
||||
sha256: first.sourceSha256,
|
||||
manifestSha256: first.manifestSha256,
|
||||
},
|
||||
covered: [{ name: "fake-native", version: "1.0.0" }],
|
||||
package: { file: PACKAGE_FILE, sha256: sha256(await readFile(fx.packagePath)) },
|
||||
});
|
||||
const notice = await readFile(first.noticePath ?? "", "utf8");
|
||||
expect(notice).toBe(metadata.notice);
|
||||
expect(notice).toContain(
|
||||
"This release includes third-party software covered by licenses including the GNU LGPL",
|
||||
);
|
||||
expect(notice).toContain(
|
||||
"Corresponding source and build/relink materials for the covered native dependencies",
|
||||
);
|
||||
expect(notice).toContain(SOURCE_FILE);
|
||||
expect(notice).toContain(`SHA-256 ${first.sourceSha256}`);
|
||||
// Nothing machine-specific in what the build generated.
|
||||
for (const generated of ["README.md", "SHA256SUMS", "corresponding-source.json"]) {
|
||||
const content =
|
||||
entries.find((entry) => entry.path === `${TOP}/${generated}`)?.data.toString("utf8") ?? "";
|
||||
expect(content).not.toContain(fx.root);
|
||||
expect(content).not.toContain(fx.cache);
|
||||
}
|
||||
expect(JSON.stringify(metadata)).not.toContain(fx.root);
|
||||
});
|
||||
|
||||
it("is approved by the gate exactly as built", async () => {
|
||||
const { fx, metadataPath, sourcePath } = await release();
|
||||
await expect(verify(fx, sourcePath, metadataPath)).resolves.toMatchObject({
|
||||
approved: true,
|
||||
correspondingSource: { file: SOURCE_FILE },
|
||||
covered: [{ name: "fake-native", version: "1.0.0" }],
|
||||
});
|
||||
});
|
||||
|
||||
it("refuses to build from cached bytes that differ from the recipe, and writes nothing", async () => {
|
||||
const fx = await fixture();
|
||||
await writeFile(
|
||||
path.join(fx.cache, "sha256", sha256(SOURCES["patches/fakelib-fix.patch"] ?? "")),
|
||||
"changed patch\n",
|
||||
);
|
||||
await rejects(build(fx), "CORRESPONDING_SOURCE_SOURCE_MISMATCH");
|
||||
await rm(path.join(fx.cache, "sha256", sha256(SOURCES["sources/fakelib-2.0.tar.gz"] ?? "")));
|
||||
await expect(readFile(path.join(fx.root, "out", SOURCE_FILE))).rejects.toThrow();
|
||||
});
|
||||
|
||||
it("refuses a missing cached file, and never overwrites an existing release", async () => {
|
||||
const fx = await fixture();
|
||||
await rm(path.join(fx.cache, "sha256", sha256(SOURCES["sources/helperlib-1.1.tar.gz"] ?? "")));
|
||||
await rejects(build(fx), "CORRESPONDING_SOURCE_SOURCE_MISSING");
|
||||
const fresh = await fixture();
|
||||
await build(fresh);
|
||||
await rejects(build(fresh), "CORRESPONDING_SOURCE_OUTPUT_EXISTS");
|
||||
});
|
||||
|
||||
it("builds nothing for a package with no covered native code, and the gate approves it without an artifact", async () => {
|
||||
const fx = await fixture({ license: "MIT", withRecipe: false });
|
||||
// The fixture's supplement records an LGPL component, so remove it: a package whose own files and
|
||||
// licence carry no source obligation needs no corresponding source.
|
||||
await rm(fx.supplementsDirectory, { force: true, recursive: true });
|
||||
await expect(build(fx)).resolves.toEqual({ required: false });
|
||||
await expect(verify(fx)).resolves.toMatchObject({ approved: true, covered: [] });
|
||||
});
|
||||
|
||||
it("treats a dependency as covered when its reviewed supplement records a copyleft component", async () => {
|
||||
// Declared MIT, but the supplement says an LGPL library is built into it.
|
||||
const fx = await fixture({ license: "MIT", withRecipe: false });
|
||||
// The supplement's declared licence must match the package's for the supplement to apply.
|
||||
const supplementPath = path.join(
|
||||
fx.supplementsDirectory,
|
||||
"fake-native",
|
||||
"1.0.0",
|
||||
"supplement.json",
|
||||
);
|
||||
const supplement = JSON.parse(await readFile(supplementPath, "utf8"));
|
||||
expect(supplement.package.license).toBe("MIT");
|
||||
await rejects(build(fx), "CORRESPONDING_SOURCE_RECIPE_MISSING");
|
||||
});
|
||||
});
|
||||
|
||||
describe("the public-distribution gate fails closed", () => {
|
||||
it("without the corresponding-source archive, or without the redistribution metadata", async () => {
|
||||
const { fx, metadataPath, sourcePath } = await release();
|
||||
await rejects(verify(fx), "CORRESPONDING_SOURCE_ARTIFACT_MISSING");
|
||||
await rm(sourcePath);
|
||||
await rejects(verify(fx, sourcePath, metadataPath), "CORRESPONDING_SOURCE_ARTIFACT_MISSING");
|
||||
const second = await release();
|
||||
await rejects(verify(second.fx, second.sourcePath), "CORRESPONDING_SOURCE_METADATA_MISMATCH");
|
||||
});
|
||||
|
||||
it("when the metadata names a different source digest, or the archive's bytes are corrupted", async () => {
|
||||
const { fx, metadataPath, sourcePath } = await release();
|
||||
const metadata = JSON.parse(await readFile(metadataPath, "utf8"));
|
||||
metadata.correspondingSource.sha256 = "0".repeat(64);
|
||||
await writeFile(metadataPath, JSON.stringify(metadata));
|
||||
await rejects(verify(fx, sourcePath, metadataPath), "CORRESPONDING_SOURCE_METADATA_MISMATCH");
|
||||
|
||||
const corrupt = await release();
|
||||
const bytes = await readFile(corrupt.sourcePath);
|
||||
const middle = Math.floor(bytes.length / 2);
|
||||
bytes[middle] = (bytes[middle] ?? 0) ^ 0xff;
|
||||
await writeFile(corrupt.sourcePath, bytes);
|
||||
await rejects(
|
||||
verify(corrupt.fx, corrupt.sourcePath, corrupt.metadataPath),
|
||||
"CORRESPONDING_SOURCE_ARTIFACT_INVALID",
|
||||
);
|
||||
});
|
||||
|
||||
it("when the release notice does not name the exact artifact", async () => {
|
||||
const { fx, metadataPath, sourcePath } = await release();
|
||||
const metadata = JSON.parse(await readFile(metadataPath, "utf8"));
|
||||
metadata.notice = metadata.notice.replace(SOURCE_FILE, "some-other-source.tar.gz");
|
||||
await writeFile(metadataPath, JSON.stringify(metadata));
|
||||
await rejects(verify(fx, sourcePath, metadataPath), "CORRESPONDING_SOURCE_METADATA_MISMATCH");
|
||||
});
|
||||
|
||||
it("when the native package's bytes are not those the recipe was prepared for", async () => {
|
||||
const built = await release();
|
||||
// The same name and version, different library bytes: a different content digest.
|
||||
const other = await fixture({ libraryBytes: "\u007fELF a different build" });
|
||||
await writeFile(built.fx.packagePath, await readFile(other.packagePath));
|
||||
await rejects(
|
||||
verify(built.fx, built.sourcePath, built.metadataPath),
|
||||
"CORRESPONDING_SOURCE_RECIPE_MISMATCH",
|
||||
);
|
||||
});
|
||||
|
||||
it("when a source component's version differs from the recipe", async () => {
|
||||
const { files, fx, metadataPath, relist, sourcePath, write } = await release();
|
||||
const manifest = JSON.parse(files.get("corresponding-source.json")?.toString("utf8") ?? "{}");
|
||||
manifest.dependencies[0].components[1].version = "2.1";
|
||||
files.set("corresponding-source.json", Buffer.from(`${JSON.stringify(manifest, null, 2)}\n`));
|
||||
relist(files);
|
||||
await write(files);
|
||||
const error = await rejects(
|
||||
verify(fx, sourcePath, metadataPath),
|
||||
"CORRESPONDING_SOURCE_MANIFEST_MISMATCH",
|
||||
);
|
||||
expect(error.subject).toBe("[email protected]");
|
||||
});
|
||||
|
||||
it("when a required source component is missing", async () => {
|
||||
const { files, fx, metadataPath, relist, sourcePath, write } = await release();
|
||||
const manifest = JSON.parse(files.get("corresponding-source.json")?.toString("utf8") ?? "{}");
|
||||
manifest.dependencies[0].components.splice(2, 1);
|
||||
files.set("corresponding-source.json", Buffer.from(`${JSON.stringify(manifest, null, 2)}\n`));
|
||||
files.delete("dependencies/fake-native/1.0.0/sources/helperlib-1.1.tar.gz");
|
||||
relist(files);
|
||||
await write(files);
|
||||
await rejects(verify(fx, sourcePath, metadataPath), "CORRESPONDING_SOURCE_MANIFEST_MISMATCH");
|
||||
});
|
||||
|
||||
it("when a patch's bytes changed, even with the file list rewritten to match", async () => {
|
||||
const { files, fx, metadataPath, relist, sourcePath, write } = await release();
|
||||
files.set(
|
||||
"dependencies/fake-native/1.0.0/patches/fakelib-fix.patch",
|
||||
Buffer.from("a different patch\n"),
|
||||
);
|
||||
relist(files);
|
||||
await write(files);
|
||||
await rejects(verify(fx, sourcePath, metadataPath), "CORRESPONDING_SOURCE_MANIFEST_MISMATCH");
|
||||
});
|
||||
|
||||
it("when the manifest records a file the archive does not hold", async () => {
|
||||
const { files, fx, metadataPath, sourcePath, write } = await release();
|
||||
files.delete("dependencies/fake-native/1.0.0/sources/fakelib-2.0.tar.gz");
|
||||
await write(files);
|
||||
const error = await rejects(
|
||||
verify(fx, sourcePath, metadataPath),
|
||||
"CORRESPONDING_SOURCE_MANIFEST_MISMATCH",
|
||||
);
|
||||
expect(error.subject).toBe("dependencies/fake-native/1.0.0/sources/fakelib-2.0.tar.gz");
|
||||
});
|
||||
|
||||
it("when the archive holds source material its manifest does not record", async () => {
|
||||
const { files, fx, metadataPath, sourcePath, write } = await release();
|
||||
files.set(
|
||||
"dependencies/fake-native/1.0.0/sources/unreviewed.tar.gz",
|
||||
Buffer.from("not recorded\n"),
|
||||
);
|
||||
await write(files);
|
||||
const error = await rejects(
|
||||
verify(fx, sourcePath, metadataPath),
|
||||
"CORRESPONDING_SOURCE_UNRECORDED_FILE",
|
||||
);
|
||||
expect(error.subject).toBe("dependencies/fake-native/1.0.0/sources/unreviewed.tar.gz");
|
||||
});
|
||||
|
||||
it("when a generated file names a machine-specific path", async () => {
|
||||
const { files, fx, metadataPath, relist, sourcePath, write } = await release();
|
||||
files.set("README.md", Buffer.from("Built from /home/builder/cache/sha256.\n"));
|
||||
relist(files);
|
||||
await write(files);
|
||||
await rejects(verify(fx, sourcePath, metadataPath), "CORRESPONDING_SOURCE_LOCAL_PATH");
|
||||
});
|
||||
|
||||
it("when the archive's entries are not in the deterministic order", async () => {
|
||||
const { files, fx, metadataPath, sourcePath, write } = await release();
|
||||
// The same entries and bytes, with the last two file entries swapped.
|
||||
const raw = gunzipSync(await readFile(sourcePath));
|
||||
const chunks: Buffer[] = [];
|
||||
let offset = 0;
|
||||
while (offset < raw.length && !raw.subarray(offset, offset + 512).every((byte) => byte === 0)) {
|
||||
const size = Number.parseInt(
|
||||
raw
|
||||
.subarray(offset + 124, offset + 136)
|
||||
.toString("latin1")
|
||||
.replace(/\0.*$/u, "")
|
||||
.trim() || "0",
|
||||
8,
|
||||
);
|
||||
const length = 512 + Math.ceil(size / 512) * 512;
|
||||
chunks.push(raw.subarray(offset, offset + length));
|
||||
offset += length;
|
||||
}
|
||||
const last = chunks.length - 1;
|
||||
[chunks[last - 1], chunks[last]] = [chunks[last] as Buffer, chunks[last - 1] as Buffer];
|
||||
await write(files, gzipSync(Buffer.concat([...chunks, Buffer.alloc(1024)]), { level: 9 }));
|
||||
await rejects(verify(fx, sourcePath, metadataPath), "CORRESPONDING_SOURCE_ARTIFACT_INVALID");
|
||||
});
|
||||
|
||||
it("when the artifact was built for another version of the native dependency", async () => {
|
||||
// The 1.3.3-for-1.3.4 case: an artifact for fake-native 1.0.0 beside a package with 1.0.1.
|
||||
const old = await release();
|
||||
const current = await fixture({ version: "1.0.1" });
|
||||
await mkdir(path.join(current.recipesDirectory, "fake-native", "1.0.0"), { recursive: true });
|
||||
for (const file of ["recipe.json", "BUILDING.md"]) {
|
||||
await writeFile(
|
||||
path.join(current.recipesDirectory, "fake-native", "1.0.0", file),
|
||||
await readFile(path.join(old.fx.recipesDirectory, "fake-native", "1.0.0", file)),
|
||||
);
|
||||
}
|
||||
await rejects(
|
||||
verify(current, old.sourcePath, old.metadataPath),
|
||||
"CORRESPONDING_SOURCE_MANIFEST_MISMATCH",
|
||||
);
|
||||
});
|
||||
|
||||
it("when the package bundles copyleft code, which this mechanism does not cover", async () => {
|
||||
const notices =
|
||||
noticeEntry("fake-native", "1.0.0", "LGPL-2.1-or-later", "node_modules") +
|
||||
noticeEntry("bundled-lgpl", "3.0.0", "LGPL-3.0-only", "backend");
|
||||
const fx = await fixture({ notices });
|
||||
await rejects(build(fx), "CORRESPONDING_SOURCE_BUNDLED_COPYLEFT");
|
||||
await rejects(verify(fx), "CORRESPONDING_SOURCE_BUNDLED_COPYLEFT");
|
||||
});
|
||||
|
||||
it("when a covered dependency has no recipe, or its recipe was prepared against another supplement", async () => {
|
||||
await rejects(
|
||||
build(await fixture({ withRecipe: false })),
|
||||
"CORRESPONDING_SOURCE_RECIPE_MISSING",
|
||||
);
|
||||
const fx = await fixture({
|
||||
recipe: (recipe) => {
|
||||
(recipe.supplement as { sha256: string }).sha256 = "f".repeat(64);
|
||||
},
|
||||
});
|
||||
await rejects(build(fx), "CORRESPONDING_SOURCE_RECIPE_MISMATCH");
|
||||
const unknownMaterial = await fixture({
|
||||
recipe: (recipe) => {
|
||||
const components = recipe.components as {
|
||||
licenseMaterial: { name: string; version: string }[];
|
||||
}[];
|
||||
(components[1] as { licenseMaterial: unknown[] }).licenseMaterial = [
|
||||
{ name: "fakelib", version: "9.9" },
|
||||
];
|
||||
},
|
||||
});
|
||||
await rejects(build(unknownMaterial), "CORRESPONDING_SOURCE_RECIPE_MISMATCH");
|
||||
});
|
||||
});
|
||||
|
||||
describe("source recipes are reviewed inputs", () => {
|
||||
async function loadWith(change: (recipe: Record<string, unknown>) => void) {
|
||||
const fx = await fixture({ recipe: change });
|
||||
return loadSourceRecipes(fx.recipesDirectory);
|
||||
}
|
||||
|
||||
it("refuses an absolute or climbing source path, a non-https origin, or a compiled component without licence material", async () => {
|
||||
const component = (recipe: Record<string, unknown>, index: number) =>
|
||||
(recipe.components as Record<string, unknown>[])[index] as Record<string, unknown>;
|
||||
const first = (recipe: Record<string, unknown>) =>
|
||||
(component(recipe, 1).sources as Record<string, unknown>[])[0] as Record<string, unknown>;
|
||||
for (const change of [
|
||||
(recipe: Record<string, unknown>) => {
|
||||
first(recipe).file = "/home/builder/fakelib-2.0.tar.gz";
|
||||
},
|
||||
(recipe: Record<string, unknown>) => {
|
||||
first(recipe).file = "sources/../../fakelib-2.0.tar.gz";
|
||||
},
|
||||
(recipe: Record<string, unknown>) => {
|
||||
first(recipe).origin = "http://example.org/fakelib-2.0.tar.gz";
|
||||
},
|
||||
(recipe: Record<string, unknown>) => {
|
||||
component(recipe, 1).licenseMaterial = [];
|
||||
},
|
||||
(recipe: Record<string, unknown>) => {
|
||||
component(recipe, 2).version = "2.0";
|
||||
component(recipe, 2).name = "fakelib";
|
||||
},
|
||||
]) {
|
||||
await rejects(loadWith(change), "CORRESPONDING_SOURCE_RECIPE_INVALID");
|
||||
}
|
||||
});
|
||||
|
||||
it("refuses a document whose bytes changed, and an unlisted file beside the recipe", async () => {
|
||||
const fx = await fixture();
|
||||
const directory = path.join(fx.recipesDirectory, "fake-native", "1.0.0");
|
||||
await writeFile(path.join(directory, "BUILDING.md"), "edited without review\n");
|
||||
await rejects(loadSourceRecipes(fx.recipesDirectory), "CORRESPONDING_SOURCE_RECIPE_INVALID");
|
||||
const second = await fixture();
|
||||
await writeFile(
|
||||
path.join(second.recipesDirectory, "fake-native", "1.0.0", "notes.txt"),
|
||||
"stray\n",
|
||||
);
|
||||
await rejects(
|
||||
loadSourceRecipes(second.recipesDirectory),
|
||||
"CORRESPONDING_SOURCE_RECIPE_INVALID",
|
||||
);
|
||||
});
|
||||
|
||||
it("recognizes licences with source obligations, by SPDX identifier or by upstream name", () => {
|
||||
for (const licence of [
|
||||
"LGPL-3.0-or-later",
|
||||
"MPL-2.0",
|
||||
"GPL-2.0-only",
|
||||
"Mozilla Public License 1.1",
|
||||
"MIT AND LGPL-2.1-or-later",
|
||||
"EPL-2.0",
|
||||
]) {
|
||||
expect(hasSourceObligations(licence), licence).toBe(true);
|
||||
}
|
||||
for (const licence of [
|
||||
"MIT",
|
||||
"Apache-2.0",
|
||||
"BSD-3-Clause",
|
||||
"ISC",
|
||||
"Zlib",
|
||||
"MIT OR Apache-2.0",
|
||||
]) {
|
||||
expect(hasSourceObligations(licence), licence).toBe(false);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("acquisition is separate, verified, and the only network step", () => {
|
||||
it("fetches each recorded file once, verifies it, and refuses a response that differs", async () => {
|
||||
const fx = await fixture();
|
||||
const fetched: string[] = [];
|
||||
const served = new Map(
|
||||
Object.entries(SOURCES).map(([file, content]) => [
|
||||
`https://example.org/${path.basename(file)}`,
|
||||
content,
|
||||
]),
|
||||
);
|
||||
const empty = path.join(fx.root, "fresh-cache");
|
||||
const result = await acquireCorrespondingSources({
|
||||
cacheDirectory: empty,
|
||||
fetch: async (url) => {
|
||||
fetched.push(url);
|
||||
return Buffer.from(served.get(url) ?? "");
|
||||
},
|
||||
packageArchivePath: fx.packagePath,
|
||||
recipesDirectory: fx.recipesDirectory,
|
||||
supplementsDirectory: fx.supplementsDirectory,
|
||||
});
|
||||
expect(result).toMatchObject({ fetched: 5, files: 5 });
|
||||
// A second run finds everything verified and fetches nothing.
|
||||
const again = await acquireCorrespondingSources({
|
||||
cacheDirectory: empty,
|
||||
fetch: async () => {
|
||||
throw new Error("no network expected");
|
||||
},
|
||||
recipes: [{ name: "fake-native", version: "1.0.0" }],
|
||||
recipesDirectory: fx.recipesDirectory,
|
||||
});
|
||||
expect(again).toMatchObject({ fetched: 0, files: 5 });
|
||||
// The built artifact from the acquired cache is identical to one built from the original cache.
|
||||
const fromAcquired = await buildCorrespondingSource({
|
||||
cacheDirectory: empty,
|
||||
outDirectory: path.join(fx.root, "acquired-out"),
|
||||
packageArchivePath: fx.packagePath,
|
||||
recipesDirectory: fx.recipesDirectory,
|
||||
supplementsDirectory: fx.supplementsDirectory,
|
||||
});
|
||||
expect(fromAcquired.sourceSha256).toBe((await build(fx)).sourceSha256);
|
||||
|
||||
// A transient upstream failure is retried; only the recorded bytes are ever stored.
|
||||
const flaky = path.join(fx.root, "flaky-cache");
|
||||
const attempts = new Map<string, number>();
|
||||
await expect(
|
||||
acquireCorrespondingSources({
|
||||
cacheDirectory: flaky,
|
||||
fetch: async (url) => {
|
||||
const count = (attempts.get(url) ?? 0) + 1;
|
||||
attempts.set(url, count);
|
||||
if (count === 1) {
|
||||
throw new Error("connection reset");
|
||||
}
|
||||
return Buffer.from(served.get(url) ?? "");
|
||||
},
|
||||
recipes: [{ name: "fake-native", version: "1.0.0" }],
|
||||
recipesDirectory: fx.recipesDirectory,
|
||||
}),
|
||||
).resolves.toMatchObject({ fetched: 5 });
|
||||
|
||||
const tampered = path.join(fx.root, "tampered-cache");
|
||||
await rejects(
|
||||
acquireCorrespondingSources({
|
||||
cacheDirectory: tampered,
|
||||
fetch: async (url) => Buffer.from(`${served.get(url) ?? ""}upstream changed\n`),
|
||||
recipes: [{ name: "fake-native", version: "1.0.0" }],
|
||||
recipesDirectory: fx.recipesDirectory,
|
||||
}),
|
||||
"CORRESPONDING_SOURCE_ACQUISITION_FAILED",
|
||||
);
|
||||
const { readdir } = await import("node:fs/promises");
|
||||
expect(await readdir(path.join(tampered, "sha256"))).toEqual([]);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user