fix: keep pnpm's package store out of the release builder's clean-source check
CI / verify (pull_request) Successful in 1m4s
CI / infrastructure-smoke (pull_request) Successful in 37s
CI / migration-smoke (pull_request) Successful in 1m54s
CI / package-release (pull_request) Successful in 32s
CI / image-build (pull_request) Successful in 30s
CI / deploy-smoke (same-origin) (pull_request) Successful in 50s
CI / deploy-smoke (split) (pull_request) Successful in 1m32s
CI / verify (pull_request) Successful in 1m4s
CI / infrastructure-smoke (pull_request) Successful in 37s
CI / migration-smoke (pull_request) Successful in 1m54s
CI / package-release (pull_request) Successful in 32s
CI / image-build (pull_request) Successful in 30s
CI / deploy-smoke (same-origin) (pull_request) Successful in 50s
CI / deploy-smoke (split) (pull_request) Successful in 1m32s
The package-release job failed its clean-tree assertion in CI. Reproduced as root in a container with a bind-mounted checkout, as the Gitea runner does: after checkout the tree is clean; `pnpm install --frozen-lockfile` then creates `.pnpm-store/` (17,272 untracked files) at the checkout's root, because pnpm keeps its store on the node_modules file system and the checkout is its own mount; `pnpm build` changes nothing. No tracked file is ever modified. - `.pnpm-store/` is ignored, like node_modules: a package cache, never source. The clean-commit requirement is unchanged for every tracked file and every untracked file Git does not ignore. - The clean-tree failure now lists `git status --short` and `git diff --stat`. - Nothing is built before the clean-source assertion: the gate no longer runs a global `pnpm build` first. The builder asserts a clean tree, builds, then fails if its build changed the tree, before packing. Core's verifier is built after both releases, for the consumer proof only. - Every secondary offline install (the second worktree, the consumer module) uses the workspace install's own store explicitly, since pnpm picks a store per file system and those directories are on another. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
9debe4e891
commit
4f1b1d68c7
@@ -1,4 +1,8 @@
|
|||||||
node_modules/
|
node_modules/
|
||||||
|
# pnpm's content-addressed package store. pnpm keeps it beside node_modules on the same file system,
|
||||||
|
# so when the checkout is its own mount (a CI runner's workspace volume, a bind-mounted container),
|
||||||
|
# `pnpm install` creates it here, at the checkout's root. A package cache, never source.
|
||||||
|
.pnpm-store/
|
||||||
.turbo/
|
.turbo/
|
||||||
coverage/
|
coverage/
|
||||||
dist/
|
dist/
|
||||||
|
|||||||
@@ -63,12 +63,18 @@ pnpm packages:release # writes .package-release/ekklesia-packages-<vers
|
|||||||
|
|
||||||
`scripts/package-release.ts`:
|
`scripts/package-release.ts`:
|
||||||
|
|
||||||
1. **Refuses a modified source tree.** A release is built from a commit. `--allow-modified` exists
|
1. **Refuses a modified source tree, before it builds anything.** A release is built from a
|
||||||
for local development only; its manifest then says `"modified": true`, and such a release must
|
commit. A tree is clean when `git status` shows no change to a tracked file and no untracked file
|
||||||
never be published.
|
that Git does not ignore. Ignored files are build output and package-manager state:
|
||||||
|
`node_modules`, `dist`, `.turbo`, and pnpm's `.pnpm-store`, which pnpm creates at the checkout's
|
||||||
|
root when the checkout is its own mount (a CI runner's workspace volume). The failure lists every
|
||||||
|
change. `--allow-modified` exists for local development only; its manifest then says
|
||||||
|
`"modified": true`, and such a release must never be published.
|
||||||
2. **Checks the release set and versions** in the workspace.
|
2. **Checks the release set and versions** in the workspace.
|
||||||
3. **Rebuilds from clean output.** It removes each public package's `dist/` and rebuilds without the
|
3. **Rebuilds from clean output.** It removes each public package's `dist/` and rebuilds without the
|
||||||
build cache, so output from a deleted source file can never ship.
|
build cache, so output from a deleted source file can never ship. It then checks the source tree
|
||||||
|
again: a build that changed a tracked file, or wrote an untracked one Git does not ignore, fails
|
||||||
|
the release, because what is packed must be exactly the commit.
|
||||||
4. **Packs each package with `pnpm pack`.** Pack-only pnpm hooks (`scripts/lib/pack-hooks.cjs`)
|
4. **Packs each package with `pnpm pack`.** Pack-only pnpm hooks (`scripts/lib/pack-hooks.cjs`)
|
||||||
apply, never the workspace's install settings ([Reproducibility](#reproducibility)).
|
apply, never the workspace's install settings ([Reproducibility](#reproducibility)).
|
||||||
5. **Inspects every tarball.** It checks:
|
5. **Inspects every tarball.** It checks:
|
||||||
@@ -127,11 +133,11 @@ depend only on the commit.
|
|||||||
pnpm test:package-release # scripts/test-package-release.sh
|
pnpm test:package-release # scripts/test-package-release.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
1. **Builds the train** from this checkout.
|
1. **Builds the train** from this checkout. Nothing is built before it.
|
||||||
2. **Builds it again** from a second worktree of the same commit at another absolute path, and
|
2. **Builds it again** from a second worktree of the same commit at another absolute path, and
|
||||||
requires both release directories to be byte-identical: every tarball, `SHA256SUMS`, and the
|
requires both release directories to be byte-identical: every tarball, `SHA256SUMS`, and the
|
||||||
manifest.
|
manifest.
|
||||||
3. **Runs the standalone consumer proof** (`scripts/test-standalone-module.sh --release <dir>`) on
|
3. **Builds Core's verifier**, then **runs the standalone consumer proof** (`scripts/test-standalone-module.sh --release <dir>`) on
|
||||||
that release:
|
that release:
|
||||||
- **The release is checked first.** Every artifact must match the manifest's size, SHA-256, and
|
- **The release is checked first.** Every artifact must match the manifest's size, SHA-256, and
|
||||||
integrity, and the release must not be built from a modified tree.
|
integrity, and the release must not be built from a modified tree.
|
||||||
@@ -139,7 +145,8 @@ pnpm test:package-release # scripts/test-package-release.sh
|
|||||||
`tests/fixtures/standalone-module`, copied outside the repository. Every `@ekklesia/*`
|
`tests/fixtures/standalone-module`, copied outside the repository. Every `@ekklesia/*`
|
||||||
package, direct or transitive, resolves to its release tarball. There is no workspace link and
|
package, direct or transitive, resolves to its release tarball. There is no workspace link and
|
||||||
no Ekklesia source checkout. The install runs offline: third-party packages come from the
|
no Ekklesia source checkout. The install runs offline: third-party packages come from the
|
||||||
local pnpm store, so no registry is involved at all.
|
workspace install's own pnpm store, named explicitly because pnpm otherwise picks a store per
|
||||||
|
file system, so no registry is involved at all.
|
||||||
- **The lockfile matches the release.** It must record each artifact's own integrity. The module
|
- **The lockfile matches the release.** It must record each artifact's own integrity. The module
|
||||||
then reinstalls from the lockfile alone with `--frozen-lockfile`.
|
then reinstalls from the lockfile alone with `--frozen-lockfile`.
|
||||||
- **The install has the right shape.** There is no `drizzle-orm`, and exactly one `react`, which
|
- **The install has the right shape.** There is no `drizzle-orm`, and exactly one `react`, which
|
||||||
|
|||||||
+1
-1
@@ -42,7 +42,7 @@
|
|||||||
"test": "turbo run test test:root",
|
"test": "turbo run test test:root",
|
||||||
"test:migrations:integration": "pnpm build && bash scripts/test-migrations-integration.sh",
|
"test:migrations:integration": "pnpm build && bash scripts/test-migrations-integration.sh",
|
||||||
"test:root": "vitest run --config vitest.config.ts",
|
"test:root": "vitest run --config vitest.config.ts",
|
||||||
"test:package-release": "pnpm build && bash scripts/test-package-release.sh",
|
"test:package-release": "bash scripts/test-package-release.sh",
|
||||||
"test:watch": "vitest",
|
"test:watch": "vitest",
|
||||||
"typecheck": "turbo run typecheck typecheck:root",
|
"typecheck": "turbo run typecheck typecheck:root",
|
||||||
"typecheck:root": "tsc --noEmit --project tsconfig.json",
|
"typecheck:root": "tsc --noEmit --project tsconfig.json",
|
||||||
|
|||||||
@@ -433,6 +433,20 @@ async function inspectTarball(
|
|||||||
return [...new Set(problems)].map((problem) => `${entry.name} ${problem}.`);
|
return [...new Set(problems)].map((problem) => `${entry.name} ${problem}.`);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Every uncommitted change in the source tree, as `git status --short` lines followed by
|
||||||
|
* `git diff --stat`, all relative to the repository root; empty for a clean tree. Ignored files
|
||||||
|
* (build output, `node_modules`) are not source and never count.
|
||||||
|
*/
|
||||||
|
async function sourceTreeChanges(): Promise<string[]> {
|
||||||
|
const status = (
|
||||||
|
await run("git", ["status", "--short", "--untracked-files=all"], repositoryRoot)
|
||||||
|
).trimEnd();
|
||||||
|
if (status === "") return [];
|
||||||
|
const stat = (await run("git", ["diff", "--stat"], repositoryRoot)).trimEnd();
|
||||||
|
return [...status.split("\n"), ...(stat === "" ? [] : stat.split("\n"))];
|
||||||
|
}
|
||||||
|
|
||||||
export class PackageReleaseError extends Error {
|
export class PackageReleaseError extends Error {
|
||||||
readonly problems: readonly string[];
|
readonly problems: readonly string[];
|
||||||
|
|
||||||
@@ -462,13 +476,12 @@ export async function buildPackageRelease(
|
|||||||
if (setProblems.length > 0) throw new PackageReleaseError(setProblems);
|
if (setProblems.length > 0) throw new PackageReleaseError(setProblems);
|
||||||
|
|
||||||
const revision = (await run("git", ["rev-parse", "HEAD"], repositoryRoot)).trim();
|
const revision = (await run("git", ["rev-parse", "HEAD"], repositoryRoot)).trim();
|
||||||
const modified =
|
const dirty = await sourceTreeChanges();
|
||||||
(
|
const modified = dirty.length > 0;
|
||||||
await run("git", ["status", "--porcelain", "--untracked-files=all"], repositoryRoot)
|
|
||||||
).trim() !== "";
|
|
||||||
if (modified && options.allowModified !== true) {
|
if (modified && options.allowModified !== true) {
|
||||||
throw new PackageReleaseError([
|
throw new PackageReleaseError([
|
||||||
"The source tree has uncommitted changes. A release is built from a clean commit.",
|
"The source tree has uncommitted changes. A release is built from a clean commit.",
|
||||||
|
...dirty,
|
||||||
]);
|
]);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -498,6 +511,15 @@ export async function buildPackageRelease(
|
|||||||
],
|
],
|
||||||
repositoryRoot,
|
repositoryRoot,
|
||||||
);
|
);
|
||||||
|
// The source was clean before the build; it must still be, so what is packed is the commit. A build
|
||||||
|
// that wrote a tracked file, or an untracked file Git does not ignore, fails the release here.
|
||||||
|
const changedByBuild = (await sourceTreeChanges()).filter((line) => !dirty.includes(line));
|
||||||
|
if (changedByBuild.length > 0) {
|
||||||
|
throw new PackageReleaseError([
|
||||||
|
"The build changed the source tree. A release packs exactly its commit.",
|
||||||
|
...changedByBuild,
|
||||||
|
]);
|
||||||
|
}
|
||||||
|
|
||||||
const scratch = await mkdtemp(path.join(tmpdir(), "ekklesia-package-release-"));
|
const scratch = await mkdtemp(path.join(tmpdir(), "ekklesia-package-release-"));
|
||||||
try {
|
try {
|
||||||
|
|||||||
@@ -9,8 +9,11 @@
|
|||||||
# 2. Builds it again from a separate worktree of the same commit, at another absolute path, and
|
# 2. Builds it again from a separate worktree of the same commit, at another absolute path, and
|
||||||
# requires the two release directories to be byte-identical: every tarball, SHA256SUMS, and the
|
# requires the two release directories to be byte-identical: every tarball, SHA256SUMS, and the
|
||||||
# manifest.
|
# manifest.
|
||||||
# 3. Installs the release into the standalone fixture and proves it builds, packages, and verifies
|
# 3. Builds Core's verifier, then installs the release into the standalone fixture and proves it
|
||||||
# (`scripts/test-standalone-module.sh`).
|
# builds, packages, and verifies (`scripts/test-standalone-module.sh`).
|
||||||
|
#
|
||||||
|
# Nothing is built before step 1: the release builder asserts a clean source tree first, builds, and
|
||||||
|
# asserts the build changed none of it. Every offline install uses the workspace install's own store.
|
||||||
#
|
#
|
||||||
# `--allow-modified` is for local development on a modified source tree. The second build is then
|
# `--allow-modified` is for local development on a modified source tree. The second build is then
|
||||||
# made in this checkout again, because a worktree of the commit would not hold the modifications.
|
# made in this checkout again, because a worktree of the commit would not hold the modifications.
|
||||||
@@ -25,6 +28,11 @@ elif (($#)); then
|
|||||||
exit 64
|
exit 64
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# The workspace install's own pnpm store. pnpm chooses a store per file system, so an install in a
|
||||||
|
# temporary directory would otherwise look in another, empty one; every install here is offline.
|
||||||
|
store="$(dirname "$(cd "$repository_root" && pnpm store path)")"
|
||||||
|
export EKKLESIA_PNPM_STORE="$store"
|
||||||
|
|
||||||
work="$(mktemp -d "${TMPDIR:-/tmp}/ekklesia-package-release-XXXXXX")"
|
work="$(mktemp -d "${TMPDIR:-/tmp}/ekklesia-package-release-XXXXXX")"
|
||||||
worktree="$work/second-checkout"
|
worktree="$work/second-checkout"
|
||||||
cleanup() {
|
cleanup() {
|
||||||
@@ -44,7 +52,7 @@ if ((${#allow_modified[@]})); then
|
|||||||
node "$repository_root/scripts/package-release.ts" --out "$work/second" "${allow_modified[@]}" >/dev/null
|
node "$repository_root/scripts/package-release.ts" --out "$work/second" "${allow_modified[@]}" >/dev/null
|
||||||
else
|
else
|
||||||
git -C "$repository_root" worktree add --detach --quiet "$worktree" HEAD
|
git -C "$repository_root" worktree add --detach --quiet "$worktree" HEAD
|
||||||
(cd "$worktree" && pnpm install --offline --frozen-lockfile --reporter=silent)
|
(cd "$worktree" && pnpm install --offline --frozen-lockfile --store-dir "$store" --reporter=silent)
|
||||||
node "$worktree/scripts/package-release.ts" --out "$work/second" >/dev/null
|
node "$worktree/scripts/package-release.ts" --out "$work/second" >/dev/null
|
||||||
fi
|
fi
|
||||||
if ! diff -r "$release" "$(find "$work/second" -mindepth 1 -maxdepth 1 -type d)"; then
|
if ! diff -r "$release" "$(find "$work/second" -mindepth 1 -maxdepth 1 -type d)"; then
|
||||||
@@ -53,6 +61,11 @@ if ! diff -r "$release" "$(find "$work/second" -mindepth 1 -maxdepth 1 -type d)"
|
|||||||
fi
|
fi
|
||||||
echo " byte-identical: $(ls "$release" | wc -l) files"
|
echo " byte-identical: $(ls "$release" | wc -l) files"
|
||||||
|
|
||||||
|
# The consumer proof verifies with Core's own verifier. Built only now, after both releases: the
|
||||||
|
# release builder asserts a clean source tree before it builds anything.
|
||||||
|
echo "==> Building Core's verifier"
|
||||||
|
(cd "$repository_root" && pnpm exec turbo run build --filter=@ekklesia/core --output-logs=errors-only >/dev/null)
|
||||||
|
|
||||||
echo "==> Consuming the release from a standalone module"
|
echo "==> Consuming the release from a standalone module"
|
||||||
bash "$repository_root/scripts/test-standalone-module.sh" --release "$release" "${allow_modified[@]}"
|
bash "$repository_root/scripts/test-standalone-module.sh" --release "$release" "${allow_modified[@]}"
|
||||||
|
|
||||||
|
|||||||
@@ -12,6 +12,7 @@
|
|||||||
# UI, packages with the release's own `@ekklesia/module-pack`, and verifies with Core's verifier.
|
# UI, packages with the release's own `@ekklesia/module-pack`, and verifies with Core's verifier.
|
||||||
#
|
#
|
||||||
# `--allow-modified` accepts a release built from a modified source tree, for local development.
|
# `--allow-modified` accepts a release built from a modified source tree, for local development.
|
||||||
|
# Core must be built (`core/dist`): the module is verified with Core's own verifier.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
repository_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
repository_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
@@ -37,6 +38,10 @@ while (($#)); do
|
|||||||
done
|
done
|
||||||
[[ -n "$release" && -d "$release" ]] || fail "no release directory given"
|
[[ -n "$release" && -d "$release" ]] || fail "no release directory given"
|
||||||
|
|
||||||
|
# Third-party packages come from the workspace install's own pnpm store, offline. pnpm chooses a
|
||||||
|
# store per file system, so the module, in a temporary directory, is pointed at it explicitly.
|
||||||
|
store="${EKKLESIA_PNPM_STORE:-$(dirname "$(cd "$repository_root" && pnpm store path)")}"
|
||||||
|
|
||||||
work="$(mktemp -d "${TMPDIR:-/tmp}/ekklesia-standalone-XXXXXX")"
|
work="$(mktemp -d "${TMPDIR:-/tmp}/ekklesia-standalone-XXXXXX")"
|
||||||
trap 'rm -rf "$work"' EXIT
|
trap 'rm -rf "$work"' EXIT
|
||||||
|
|
||||||
@@ -96,7 +101,7 @@ pinned="$(node -p 'require(process.argv[1]).devDependencies["@ekklesia/module-pa
|
|||||||
} >"$module/pnpm-workspace.yaml"
|
} >"$module/pnpm-workspace.yaml"
|
||||||
|
|
||||||
echo "==> Installing the module from the release tarballs, offline"
|
echo "==> Installing the module from the release tarballs, offline"
|
||||||
(cd "$module" && pnpm install --offline --reporter=append-only >"$work/install.log" 2>&1) ||
|
(cd "$module" && pnpm install --offline --store-dir "$store" --reporter=append-only >"$work/install.log" 2>&1) ||
|
||||||
{
|
{
|
||||||
tail -40 "$work/install.log" >&2
|
tail -40 "$work/install.log" >&2
|
||||||
fail "install (third-party packages come from the local store: run pnpm install in the workspace first)"
|
fail "install (third-party packages come from the local store: run pnpm install in the workspace first)"
|
||||||
@@ -125,7 +130,7 @@ EOF
|
|||||||
|
|
||||||
echo "==> Reinstalling from the lockfile alone (--frozen-lockfile)"
|
echo "==> Reinstalling from the lockfile alone (--frozen-lockfile)"
|
||||||
rm -rf "$module/node_modules"
|
rm -rf "$module/node_modules"
|
||||||
(cd "$module" && pnpm install --offline --frozen-lockfile --reporter=append-only >"$work/frozen.log" 2>&1) ||
|
(cd "$module" && pnpm install --offline --frozen-lockfile --store-dir "$store" --reporter=append-only >"$work/frozen.log" 2>&1) ||
|
||||||
{ tail -40 "$work/frozen.log" >&2; fail "frozen-lockfile install"; }
|
{ tail -40 "$work/frozen.log" >&2; fail "frozen-lockfile install"; }
|
||||||
|
|
||||||
echo "==> Checking what the install resolved"
|
echo "==> Checking what the install resolved"
|
||||||
|
|||||||
@@ -1,3 +1,4 @@
|
|||||||
|
import { spawnSync } from "node:child_process";
|
||||||
import { readdir, readFile } from "node:fs/promises";
|
import { readdir, readFile } from "node:fs/promises";
|
||||||
import path from "node:path";
|
import path from "node:path";
|
||||||
import { fileURLToPath } from "node:url";
|
import { fileURLToPath } from "node:url";
|
||||||
@@ -361,6 +362,17 @@ describe("the pack hook", () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe("a clean source tree", () => {
|
||||||
|
it("ignores pnpm's package store, which pnpm creates in a checkout that is its own mount", () => {
|
||||||
|
// In CI the checkout is a workspace volume, so `pnpm install` puts `.pnpm-store/` at its root:
|
||||||
|
// a package cache, never source, which must not make the release builder's clean tree dirty.
|
||||||
|
const ignored = spawnSync("git", ["check-ignore", "--quiet", ".pnpm-store/v11/index.db"], {
|
||||||
|
cwd: repositoryRoot,
|
||||||
|
});
|
||||||
|
expect(ignored.status).toBe(0);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
describe("the release directory", () => {
|
describe("the release directory", () => {
|
||||||
const version = "0.1.2";
|
const version = "0.1.2";
|
||||||
function release(mutate?: (files: Map<string, Uint8Array>) => void) {
|
function release(mutate?: (files: Map<string, Uint8Array>) => void) {
|
||||||
|
|||||||
Reference in New Issue
Block a user