ekklesiaandClaude Opus 5
b9ca0d560d
`manifest.storage.buckets` has existed since manifest version 1 and meant nothing — discovery validated it and no code read it. This turns it into a real capability, which is what stands between Announcements and the module system: that product currently fetches a hardcoded LAN S3 endpoint, parses ListObjectsV2 XML by hand, and hands the browser direct object URLs. - A module declares a logical bucket and receives a scoped facade at `context.storage` — list, head, get, put, delete. The contract names no provider: no endpoint, region, credential, or physical bucket is reachable from it, which is what lets the same module run under a future standalone host over its own or a shared S3-compatible service. - Core's first provider speaks S3 through the AWS SDK. SeaweedFS is the deployment target (ADR 0004), never an API name. Probing the pinned image found `CreateBucket` reporting `BucketAlreadyExists` rather than AWS's `BucketAlreadyOwnedByYou`; treating only the AWS spelling as success would have failed every second boot. A live test pins it. - Logical → physical mapping is deterministic and collision-safe: an installation prefix, the module and bucket ids joined by `--` (which cannot occur inside either identifier, so `a-b`+`c` and `a`+`b-c` cannot collide), and a digest over the full untruncated identity. Two modules declaring `assets` always get separate namespaces; neither can see the physical name. - Declaring a bucket is now a requirement: with no provider configured the module is `blocked · storage_unavailable` through ADR 0028's existing resolver, propagating to required dependents. An installation with no object storage stays a complete deployment. - Binding availability is not provider health. Eligibility asks whether storage is configured, never whether it answered; bucket creation is lazy on first use so a storage outage cannot stop the platform booting; a running module whose store blinks stays running and retries. Nothing deletes a bucket. - Credentials are deployment-supplied, never persisted, never routed through module configuration, and never visible to modules, module UI, diagnostics, or API responses. Provider errors translate to a closed code set carrying no endpoint, bucket, request id, or XML. - Retires ADR 0029's transitional `bucket` configuration key. No manifest version change. The template declares no bucket, so it still runs without S3. See ADR 0030. Co-Authored-By: Claude Opus 5 <[email protected]>
263 lines
19 KiB
Markdown
263 lines
19 KiB
Markdown
# Testing
|
||
|
||
Ekklesia separates fast code-level evidence from live infrastructure evidence.
|
||
|
||
## Unit and repository verification
|
||
|
||
```bash
|
||
pnpm test
|
||
pnpm verify
|
||
```
|
||
|
||
These commands exercise package tests, repository boundaries, type checking, offline Core and
|
||
module migration discovery, identity schema drift detection, formatting, linting, and builds.
|
||
Migration-runner unit tests use a narrow database/session port to prove deterministic planning,
|
||
approval, transaction orchestration, cleanup, and sanitized errors. They do not claim to prove
|
||
PostgreSQL behavior. Identity unit tests cover branded opaque-ID contracts, principal shapes,
|
||
configuration validation (secret rules, canonical-origin-only base URL and trusted origins, and the
|
||
full loopback/unspecified classifier production rejects — `.localhost`, all of `127.0.0.0/8`,
|
||
`0.0.0.0`, `::1` compressed and expanded, `::`, and IPv4-mapped forms), bounded/validated
|
||
connection and operation (`query_timeout`/`statement_timeout`) database timeouts, the Core identity
|
||
runtime (explicit construction, exact schema mapping, only email/password and rate limiting
|
||
enabled, an explicit `advanced.disableCSRFCheck`/`disableOriginCheck: false` pair, an explicit
|
||
error-only sanitized logger that strips Better Auth's own message and arguments regardless of what
|
||
they contain, pinned 8–128 password-length bounds, sanitized failures, exact whole-name
|
||
session-cookie detection that never fires on a prefixed, suffixed, or differently-cased lookalike,
|
||
and that a broken session store during principal resolution is never silently reported as anonymous
|
||
while an unrelated cookie performs no identity database work at all), the PostgreSQL transport
|
||
policy (`core/tests/database-transport.test.ts`: mode validation including every loose or aliased
|
||
spelling, production's plaintext refusal with no loopback exception, connection-string
|
||
TLS-parameter rejection, and that no configuration path yields `rejectUnauthorized: false` or leaks
|
||
CA material into an error), and Core migration discovery (deterministic ordering, the reserved
|
||
`core` identity, and collision rejection) — all without a live database. `apps/api`'s own unit tests cover its pure Hono app factory (route shapes, module
|
||
route mounting/enforcement/CORS/response sanitization, CORS allow/deny/preflight, deterministic 404
|
||
and sanitized error responses), its runtime configuration, its idempotent shutdown lifecycle
|
||
(concurrent callers share one server-close-then-module-runtime-stop-then-pool-close promise; a
|
||
module-stop failure never prevents pool cleanup; a startup failure after the pool was already
|
||
created still closes it, stopping the module runtime first if it had already started;
|
||
`SIGINT`/`SIGTERM` wiring is exercised through an injectable exit function rather than the real
|
||
`process.exit`), and that importing any of its modules — including the executable server module
|
||
without calling its exported `main` — never reads the environment, connects to PostgreSQL, imports
|
||
a module entrypoint, or starts a server. Real, working (non-throwing) module fixtures prove
|
||
production module-runtime startup/shutdown ordering and cleanup against a mocked pool/Better
|
||
Auth/server — see [Authorization](authorization.md#tests).
|
||
|
||
`pnpm verify` is intentionally Docker-free. It never plans or applies migrations against a live
|
||
database.
|
||
|
||
## Live PostgreSQL migration evidence
|
||
|
||
Docker and the Docker Compose plugin are required for the live suite:
|
||
|
||
```bash
|
||
pnpm test:migrations:integration
|
||
```
|
||
|
||
The command first builds the workspace through the repository's Turbo build graph, then starts the
|
||
repository's pinned PostgreSQL image in a unique Compose project, waits for bounded health
|
||
readiness, runs the production PostgreSQL adapter and migration services, and removes the
|
||
container, network, and volume. A build failure occurs before Docker provisioning. Each scenario
|
||
also receives a safely named disposable database. Direct host runs use a Docker-assigned random
|
||
port bound only to `127.0.0.1`; a containerized Gitea job joins the isolated Compose network and
|
||
uses the PostgreSQL service name.
|
||
|
||
The live suite proves fresh read-only planning, initial and incremental application, genuine
|
||
no-op behavior, exact applied checksums and history, edited or corrupt history rejection,
|
||
whole-batch DDL rollback, Drizzle breakpoint execution, real advisory-lock concurrency and
|
||
timeouts, lock release after failure, exact destructive approvals, and public-error sanitization.
|
||
It does not make migrations automatic during Core startup.
|
||
|
||
A dedicated set of scenarios exercises the committed Core identity migration against real
|
||
PostgreSQL: read-only planning on a fresh database, applying it and confirming exactly the four
|
||
expected `ekklesia_core_*` tables and their indexes exist, a genuine no-op on a second apply, Core
|
||
executing before a fixture module migration in one batch, previously applied module-only history
|
||
staying byte-for-byte unchanged once Core is introduced, and a later module migration's failure
|
||
rolling back cleanly without damaging the already-committed Core identity migration or leaking the
|
||
advisory lock.
|
||
|
||
A second dedicated suite (`apps/api/tests/api-http.integration.ts`, included in the same
|
||
`pnpm test:migrations:integration` run) starts the real `apps/api` HTTP server against a freshly
|
||
migrated disposable database and exercises it with real HTTP requests: public sign-up (with no
|
||
automatic session, per `autoSignIn: false`), explicit sign-in producing a host-only
|
||
`HttpOnly`/`SameSite=Lax`/secure session cookie, `/api/v1/principal` for both the authenticated and
|
||
anonymous shapes, sign-out actually invalidating the session, a malformed or missing cookie
|
||
resolving as anonymous, duplicate sign-up creating no second identity row, two distinct accounts
|
||
producing identical principal shapes with no elevation and no owner/role/permission data at the
|
||
database level, no plaintext password ever stored, the exact 8–128 password-length boundary (7
|
||
rejected, 8 accepted, 128 accepted, 129 rejected), CORS never returning an allow-origin header for
|
||
an untrusted origin, Better Auth's handler being unreachable outside `/api/auth/*`, a database
|
||
outage during principal resolution producing a sanitized `503` rather than a false anonymous `200`,
|
||
starting the API against an unmigrated database creating no tables at all, and a clean listener/pool
|
||
shutdown.
|
||
|
||
A dedicated group within that suite proves session-store readiness against the real database rather
|
||
than a mock, since that is the only place a broken store looks exactly like an anonymous request: a
|
||
request carrying no cookies and one carrying only an unrelated cookie both resolve as anonymous
|
||
while issuing **no** identity database work at all; a request presenting the configured session
|
||
cookie against a dropped session table, a dropped column, and a revoked `SELECT` privilege each
|
||
produce a sanitized `503 PRINCIPAL_RESOLUTION_FAILED` with no relation name, column name, SQL, or
|
||
driver text in the body or the logs; and a genuinely expired or unknown session against a healthy
|
||
store still resolves as anonymous.
|
||
|
||
The same suite also proves Better Auth's own origin/CSRF protection is production-equivalent under
|
||
the test runner, not just in a plain Node script: a no-cookie first authentication attempt succeeds
|
||
from a trusted browser origin and is rejected (`403`, no permissive CORS header, no leaked detail)
|
||
from an untrusted one; a cookie-bearing state-changing request (sign-out) is rejected the same way
|
||
from an untrusted origin, and the presented session remains valid afterward, proving the rejected
|
||
request never reached sign-out logic. See
|
||
[ADR 0012](../architecture/decisions/0012-identity-origin-csrf-protection-under-test.md) for why
|
||
this previously could not be reproduced reliably under Vitest (Better Auth's own default silently
|
||
skips these checks whenever `isTest()` is true, unless `advanced.disableOriginCheck` is set
|
||
explicitly) and how the fix removes that gap entirely. A further scenario forces a real identity
|
||
database failure and asserts, against actual `console.error`/`warn`/`log` output, that Better
|
||
Auth's own internal error logging — configured explicitly in `core/src/identity/runtime.ts` — never
|
||
emits a raw exception, SQL, connection detail, or the rejected request's origin.
|
||
|
||
A dedicated group in the API suite also drives Pass 3's explicit-grant invariants through a real
|
||
authorized grants-management session: a malformed non-UUID revoke returning a sanitized `400` with
|
||
no database work, a canonical unknown UUID returning `404`, a scoped create body returning `400`
|
||
with nothing written, normalized-impossible/non-canonical/past expirations returning `400`, an
|
||
equivalent creation returning a sanitized `409` that names no existing grant, a bounded paginated
|
||
list traversed across three pages with relationship grants absent from every one, and one HTTP
|
||
revocation clearing every unrevoked legacy equivalent with a per-row audit entry.
|
||
|
||
A third dedicated suite (`apps/api/tests/module-database.integration.ts`, included in the same
|
||
`pnpm test:migrations:integration` run) proves `context.services.database` against a real,
|
||
installed test module (`core/tests/fixtures/module-database-integration/db-facade/` — never the
|
||
repository's live `modules/` directory, and invisible to normal discovery/Doctor/`modules:validate`
|
||
unless a test explicitly points discovery at it): the module's own table does not exist until its
|
||
migration is explicitly applied, and constructing the pool or facade never creates it; `start`,
|
||
`stop`, and an authorized HTTP handler all receive the identical `ModuleRuntimeServices` object,
|
||
proven by reference equality; a real authorized mutation persists and a later read observes it,
|
||
gated by the same permission/trusted-Origin enforcement every other module route uses; a real
|
||
committed transaction persists and a real rollback (a deliberate unique-constraint violation)
|
||
leaves no partial mutation; a module deliberately catching a real PostgreSQL failure returns its
|
||
own response untouched, while an uncaught one sanitizes to the existing
|
||
`MODULE_HANDLER_FAILED` boundary with no leaked SQL, error text, or path; and a `stop` hook's write
|
||
completes and is independently verifiable, through the harness's own administrative connection —
|
||
never the API's own pool — before that pool closes. See
|
||
[ADR 0016](../architecture/decisions/0016-module-database-access.md).
|
||
|
||
A fourth dedicated suite (`core/tests/database-tls.integration.ts`, included in the same
|
||
`pnpm test:migrations:integration` run) proves the production PostgreSQL transport policy against a
|
||
real TLS-enabled PostgreSQL rather than against the shape of an options object: a trusted
|
||
certificate whose identity matches the host connects and is confirmed encrypted through
|
||
`pg_stat_ssl`; a certificate that does not chain to the configured CA is refused
|
||
(`UNABLE_TO_VERIFY_LEAF_SIGNATURE`); the same server reached through an address its certificate does
|
||
not cover is refused (`ERR_TLS_CERT_ALTNAME_INVALID`); the identical connection *without* the
|
||
policy's pinned `checkServerIdentity` succeeds, which is the concrete gap the policy closes; and
|
||
`verify-full` cannot be satisfied by a plaintext-only endpoint. Every CA, certificate, and private
|
||
key is generated at test runtime into a temporary directory and deleted afterwards — **no private
|
||
key material is ever committed to this repository** — and the disposable container is removed in
|
||
`afterAll` even when an assertion fails. See
|
||
[ADR 0018](../architecture/decisions/0018-identity-session-resolution-and-database-transport.md).
|
||
|
||
A fifth dedicated suite (`core/tests/module-lifecycle-store.integration.ts`, included in the same
|
||
`pnpm test:migrations:integration` run) proves the module enablement store against real PostgreSQL:
|
||
that migration `0004` really creates `ekklesia_core_module_state` with exactly the four columns and
|
||
the one declared CHECK constraint the schema states — and, load-bearingly, that `enabled` has **no
|
||
column default**, so the platform's default for a newly discovered module is genuinely the absence of
|
||
a row rather than a materialized decision nobody made; that a decision survives a completely new
|
||
connection; that the upsert is idempotent on `module_id` and bumps `updated_at` in place rather than
|
||
inserting a second row; that a malformed identifier is rejected before it reaches PostgreSQL and the
|
||
empty-string case is refused by the database too; and that a record for a module which is no longer
|
||
deployed is retained and simply ignored by resolution, so redeploying restores the decision.
|
||
|
||
The API suite additionally drives the whole lifecycle end to end against a real database and real
|
||
fixture modules: a disabled module's compiled entrypoint is never imported (proven by its
|
||
`register`/`start`/`stop` markers staying empty), its routes return the platform's ordinary `404`,
|
||
its UI artifact and inventory and navigation entries are all absent, and its neighbours run normally;
|
||
a module blocked by a disabled required dependency is explained over HTTP with a structured reason
|
||
naming that dependency and no filesystem path anywhere in the payload; a change made through
|
||
`PUT /api/v1/module-lifecycle/<id>/enabled` reports `restartRequired: true` while the module
|
||
demonstrably keeps serving, and is applied only by the next start; and a disabled module's migrations
|
||
are still discovered, because deployed schema ownership is not runtime enabled state. See
|
||
[ADR 0028](../architecture/decisions/0028-module-operational-lifecycle.md).
|
||
|
||
A sixth dedicated suite (`core/tests/module-configuration-store.integration.ts`, included in the same
|
||
`pnpm test:migrations:integration` run) proves the module configuration store against real
|
||
PostgreSQL: that migration `0005` really creates `ekklesia_core_module_configuration` with the
|
||
composite `(module_id, key)` primary key, the declared CHECK constraints, and **no column that could
|
||
hold secret material**; that a replace is transactional, whole-document, and scoped to one module so
|
||
another module's rows are untouched; that stored values survive a completely new connection; that an
|
||
oversized value is refused by the application *and* by the database; that a store error never carries
|
||
the offending value; and — the security-relevant one — that a module id or key is only ever data in a
|
||
parameterized column, so an injection attempt is a validation error and the table still exists
|
||
afterwards.
|
||
|
||
The API suite additionally proves the configuration boundary end to end: a secret's value never
|
||
appears in any response (including when a row for that key is planted directly in storage), a
|
||
declared-but-undeclared key, a wrong JSON type, a constraint violation, and a `secret` key each fail
|
||
with a distinct sanitized code and store nothing, a reserved property name such as `__proto__` is
|
||
rejected without touching `Object.prototype`, and a module blocked by its own configuration is absent
|
||
from the mountable UI inventory exactly like a disabled one. See
|
||
[ADR 0029](../architecture/decisions/0029-module-configuration-and-secrets.md).
|
||
|
||
A seventh dedicated suite (`core/tests/module-storage.integration.ts`, included in the same
|
||
`pnpm test:migrations:integration` run) proves the object-storage contract against a **real
|
||
S3-compatible server** — the SeaweedFS service the repository's Compose file already pins, brought up
|
||
alongside PostgreSQL by the same harness, so this adds no new CI infrastructure.
|
||
|
||
It exists because mocking would have hidden a real divergence: probing the pinned SeaweedFS 4.28 image
|
||
directly showed that `CreateBucket` on an existing bucket reports `BucketAlreadyExists`, **not** AWS's
|
||
`BucketAlreadyOwnedByYou`. A provider treating only the AWS spelling as success would have failed on
|
||
every second boot against this repository's own development storage, and no unit test would have
|
||
caught it. The suite pins that idempotency, plus a real round trip with content type and unquoted
|
||
ETag preserved, streaming a body without buffering, a Web stream going straight into a `Response`,
|
||
prefix listing and real pagination, a missing object mapping to a structured not-found error carrying
|
||
no endpoint or XML, two modules declaring the same logical bucket staying genuinely isolated on one
|
||
server, the longest bucket name the mapping can produce being accepted, a real connection failure
|
||
mapping to `STORAGE_PROVIDER_UNAVAILABLE`, and a bucket surviving the deletion of every object in it.
|
||
|
||
Unit suites (`core/tests/module-storage.test.ts`, `core/tests/module-storage-lifecycle.test.ts`) use an
|
||
injected in-memory provider and never touch the network. They prove the parts the protocol cannot:
|
||
binding determinism and collision-safety, scoping and cross-module isolation, key validation, lazy
|
||
provisioning and retry-after-failure, configuration validation and credential redaction, and the
|
||
lifecycle integration including dependency propagation. See
|
||
[ADR 0030](../architecture/decisions/0030-host-provided-module-object-storage.md).
|
||
|
||
### Which kind of timing proof lives where
|
||
|
||
Pass 4 introduced several deadlines, and their tests deliberately differ in kind:
|
||
|
||
- **Unit deadline tests** (`core/tests/module-runtime.test.ts`) drive module stop and rollback
|
||
hooks through a short, test-only `hookTimeoutMs` override with manually controlled
|
||
never-settling promises — no real waiting, no sleeps, and an explicit `unhandledRejection` check
|
||
proving a hook that rejects after its deadline stays handled.
|
||
- **Real raw-HTTP force-close** (`apps/api/tests/api-http.integration.ts`) binds the real listener,
|
||
opens a real socket, writes a deliberately incomplete request, and proves shutdown force-closes
|
||
it at a short `drainTimeoutMsOverride` and still reaches pool closure.
|
||
- **Real TCP connection timeout** (`core/tests/postgres-migration.integration.ts`) binds a loopback
|
||
blackhole listener that accepts the socket and never completes PostgreSQL startup, proving the
|
||
configured connection deadline; the listener and every accepted socket close in `finally`.
|
||
- **Real PostgreSQL operation timeout and SQL batching** (same file) use a bounded server-side
|
||
delay only because the deadline itself is what is under test, and disposable databases to prove
|
||
that marker bytes inside literals, dollar-quoted bodies, comments, and quoted identifiers execute
|
||
intact, and that a failure rolls the whole batch back.
|
||
|
||
Every one of these uses a short **test-only override**; the production defaults (10 s per module
|
||
hook, 30 s HTTP drain, 10 s migration connection/query/statement) are asserted separately as
|
||
constants and are never waited out.
|
||
|
||
CI runs the same command in the dedicated `migration-smoke` job. After this change is merged, add
|
||
the following pull-request status context to `main` branch protection:
|
||
|
||
```text
|
||
CI / migration-smoke (pull_request)
|
||
```
|
||
|
||
## Safe troubleshooting
|
||
|
||
Confirm the prerequisites without printing configuration:
|
||
|
||
```bash
|
||
docker info
|
||
docker compose version
|
||
```
|
||
|
||
Then rerun `pnpm test:migrations:integration`. The harness emits fixed failure messages and does
|
||
not dump PostgreSQL logs, SQL, container inspection output, or the generated database URL. Do not
|
||
enable query logging, print the test environment, run `docker compose config` without redaction,
|
||
or paste Docker inspection output into an issue. If the Docker daemon itself stopped during
|
||
cleanup, restore it before rerunning so the harness can create a fresh uniquely named project.
|