322 lines
24 KiB
Markdown
322 lines
24 KiB
Markdown
# Testing
|
||
|
||
Ekklesia separates fast code-level evidence from live infrastructure evidence.
|
||
|
||
## Unit and repository verification
|
||
|
||
```bash
|
||
pnpm test
|
||
pnpm verify
|
||
```
|
||
|
||
These commands exercise package tests, repository boundaries, type checking, offline Core and
|
||
module migration discovery, identity schema drift detection, formatting, linting, and builds.
|
||
Migration-runner unit tests use a narrow database/session port to prove deterministic planning,
|
||
approval, transaction orchestration, cleanup, and sanitized errors. They do not claim to prove
|
||
PostgreSQL behavior. Identity unit tests cover branded opaque-ID contracts, principal shapes,
|
||
configuration validation (secret rules, canonical-origin-only base URL and trusted origins, and the
|
||
full loopback/unspecified classifier production rejects — `.localhost`, all of `127.0.0.0/8`,
|
||
`0.0.0.0`, `::1` compressed and expanded, `::`, and IPv4-mapped forms), bounded/validated
|
||
connection and operation (`query_timeout`/`statement_timeout`) database timeouts, the Core identity
|
||
runtime (explicit construction, exact schema mapping, only email/password and rate limiting
|
||
enabled, an explicit `advanced.disableCSRFCheck`/`disableOriginCheck: false` pair, an explicit
|
||
error-only sanitized logger that strips Better Auth's own message and arguments regardless of what
|
||
they contain, pinned 8–128 password-length bounds, sanitized failures, exact whole-name
|
||
session-cookie detection that never fires on a prefixed, suffixed, or differently-cased lookalike,
|
||
and that a broken session store during principal resolution is never silently reported as anonymous
|
||
while an unrelated cookie performs no identity database work at all), the PostgreSQL transport
|
||
policy (`core/tests/database-transport.test.ts`: mode validation including every loose or aliased
|
||
spelling, production's plaintext refusal with no loopback exception, connection-string
|
||
TLS-parameter rejection, and that no configuration path yields `rejectUnauthorized: false` or leaks
|
||
CA material into an error), and Core migration discovery (deterministic ordering, the reserved
|
||
`core` identity, and collision rejection) — all without a live database. `apps/api`'s own unit tests cover its pure Hono app factory (route shapes, module
|
||
route mounting/enforcement/CORS/response sanitization, CORS allow/deny/preflight, deterministic 404
|
||
and sanitized error responses), its runtime configuration, its idempotent shutdown lifecycle
|
||
(concurrent callers share one server-close-then-module-runtime-stop-then-pool-close promise; a
|
||
module-stop failure never prevents pool cleanup; a startup failure after the pool was already
|
||
created still closes it, stopping the module runtime first if it had already started;
|
||
`SIGINT`/`SIGTERM` wiring is exercised through an injectable exit function rather than the real
|
||
`process.exit`), and that importing any of its modules — including the executable server module
|
||
without calling its exported `main` — never reads the environment, connects to PostgreSQL, imports
|
||
a module entrypoint, or starts a server. Real, working (non-throwing) module fixtures prove
|
||
production module-runtime startup/shutdown ordering and cleanup against a mocked pool/Better
|
||
Auth/server — see [Authorization](authorization.md#tests).
|
||
|
||
`pnpm verify` is intentionally Docker-free. It never plans or applies migrations against a live
|
||
database.
|
||
|
||
## Live PostgreSQL migration evidence
|
||
|
||
Docker and the Docker Compose plugin are required for the live suite:
|
||
|
||
```bash
|
||
pnpm test:migrations:integration
|
||
```
|
||
|
||
The command first builds the workspace through the repository's Turbo build graph, then starts the
|
||
repository's pinned PostgreSQL image in a unique Compose project, waits for bounded health
|
||
readiness, runs the production PostgreSQL adapter and migration services, and removes the
|
||
container, network, and volume. A build failure occurs before Docker provisioning. Each scenario
|
||
also receives a safely named disposable database. Direct host runs use a Docker-assigned random
|
||
port bound only to `127.0.0.1`; a containerized Gitea job joins the isolated Compose network and
|
||
uses the PostgreSQL service name.
|
||
|
||
The live suite proves fresh read-only planning, initial and incremental application, genuine
|
||
no-op behavior, exact applied checksums and history, edited or corrupt history rejection,
|
||
whole-batch DDL rollback, Drizzle breakpoint execution, real advisory-lock concurrency and
|
||
timeouts, lock release after failure, exact destructive approvals, and public-error sanitization.
|
||
It does not make migrations automatic during Core startup.
|
||
|
||
A dedicated set of scenarios exercises the committed Core identity migration against real
|
||
PostgreSQL: read-only planning on a fresh database, applying it and confirming exactly the four
|
||
expected `ekklesia_core_*` tables and their indexes exist, a genuine no-op on a second apply, Core
|
||
executing before a fixture module migration in one batch, previously applied module-only history
|
||
staying byte-for-byte unchanged once Core is introduced, and a later module migration's failure
|
||
rolling back cleanly without damaging the already-committed Core identity migration or leaking the
|
||
advisory lock.
|
||
|
||
A second dedicated suite (`apps/api/tests/api-http.integration.ts`, included in the same
|
||
`pnpm test:migrations:integration` run) starts the real `apps/api` HTTP server against a freshly
|
||
migrated disposable database and exercises it with real HTTP requests: public sign-up (with no
|
||
automatic session, per `autoSignIn: false`), explicit sign-in producing a host-only
|
||
`HttpOnly`/`SameSite=Lax`/secure session cookie, `/api/v1/principal` for both the authenticated and
|
||
anonymous shapes, sign-out actually invalidating the session, a malformed or missing cookie
|
||
resolving as anonymous, duplicate sign-up creating no second identity row, two distinct accounts
|
||
producing identical principal shapes with no elevation and no owner/role/permission data at the
|
||
database level, no plaintext password ever stored, the exact 8–128 password-length boundary (7
|
||
rejected, 8 accepted, 128 accepted, 129 rejected), CORS never returning an allow-origin header for
|
||
an untrusted origin, Better Auth's handler being unreachable outside `/api/auth/*`, a database
|
||
outage during principal resolution producing a sanitized `503` rather than a false anonymous `200`,
|
||
starting the API against an unmigrated database creating no tables at all, and a clean listener/pool
|
||
shutdown.
|
||
|
||
A dedicated group within that suite proves session-store readiness against the real database rather
|
||
than a mock, since that is the only place a broken store looks exactly like an anonymous request: a
|
||
request carrying no cookies and one carrying only an unrelated cookie both resolve as anonymous
|
||
while issuing **no** identity database work at all; a request presenting the configured session
|
||
cookie against a dropped session table, a dropped column, and a revoked `SELECT` privilege each
|
||
produce a sanitized `503 PRINCIPAL_RESOLUTION_FAILED` with no relation name, column name, SQL, or
|
||
driver text in the body or the logs; and a genuinely expired or unknown session against a healthy
|
||
store still resolves as anonymous.
|
||
|
||
The same suite also proves Better Auth's own origin/CSRF protection is production-equivalent under
|
||
the test runner, not just in a plain Node script: a no-cookie first authentication attempt succeeds
|
||
from a trusted browser origin and is rejected (`403`, no permissive CORS header, no leaked detail)
|
||
from an untrusted one; a cookie-bearing state-changing request (sign-out) is rejected the same way
|
||
from an untrusted origin, and the presented session remains valid afterward, proving the rejected
|
||
request never reached sign-out logic. See
|
||
[ADR 0012](../architecture/decisions/0012-identity-origin-csrf-protection-under-test.md) for why
|
||
this previously could not be reproduced reliably under Vitest (Better Auth's own default silently
|
||
skips these checks whenever `isTest()` is true, unless `advanced.disableOriginCheck` is set
|
||
explicitly) and how the fix removes that gap entirely. A further scenario forces a real identity
|
||
database failure and asserts, against actual `console.error`/`warn`/`log` output, that Better
|
||
Auth's own internal error logging — configured explicitly in `core/src/identity/runtime.ts` — never
|
||
emits a raw exception, SQL, connection detail, or the rejected request's origin.
|
||
|
||
A dedicated group in the API suite also drives Pass 3's explicit-grant invariants through a real
|
||
authorized grants-management session: a malformed non-UUID revoke returning a sanitized `400` with
|
||
no database work, a canonical unknown UUID returning `404`, a scoped create body returning `400`
|
||
with nothing written, normalized-impossible/non-canonical/past expirations returning `400`, an
|
||
equivalent creation returning a sanitized `409` that names no existing grant, a bounded paginated
|
||
list traversed across three pages with relationship grants absent from every one, and one HTTP
|
||
revocation clearing every unrevoked legacy equivalent with a per-row audit entry.
|
||
|
||
A third dedicated suite (`apps/api/tests/module-database.integration.ts`, included in the same
|
||
`pnpm test:migrations:integration` run) proves `context.services.database` against a real,
|
||
installed test module (`core/tests/fixtures/module-database-integration/db-facade/` — never the
|
||
repository's live `modules/` directory, and invisible to normal discovery/Doctor/`modules:validate`
|
||
unless a test explicitly points discovery at it): the module's own table does not exist until its
|
||
migration is explicitly applied, and constructing the pool or facade never creates it; `start`,
|
||
`stop`, and an authorized HTTP handler all receive the identical `ModuleRuntimeServices` object,
|
||
proven by reference equality; a real authorized mutation persists and a later read observes it,
|
||
gated by the same permission/trusted-Origin enforcement every other module route uses; a real
|
||
committed transaction persists and a real rollback (a deliberate unique-constraint violation)
|
||
leaves no partial mutation; a module deliberately catching a real PostgreSQL failure returns its
|
||
own response untouched, while an uncaught one sanitizes to the existing
|
||
`MODULE_HANDLER_FAILED` boundary with no leaked SQL, error text, or path; and a `stop` hook's write
|
||
completes and is independently verifiable, through the harness's own administrative connection —
|
||
never the API's own pool — before that pool closes. See
|
||
[ADR 0016](../architecture/decisions/0016-module-database-access.md).
|
||
|
||
A fourth dedicated suite (`core/tests/database-tls.integration.ts`, included in the same
|
||
`pnpm test:migrations:integration` run) proves the production PostgreSQL transport policy against a
|
||
real TLS-enabled PostgreSQL rather than against the shape of an options object: a trusted
|
||
certificate whose identity matches the host connects and is confirmed encrypted through
|
||
`pg_stat_ssl`; a certificate that does not chain to the configured CA is refused
|
||
(`UNABLE_TO_VERIFY_LEAF_SIGNATURE`); the same server reached through an address its certificate does
|
||
not cover is refused (`ERR_TLS_CERT_ALTNAME_INVALID`); the identical connection *without* the
|
||
policy's pinned `checkServerIdentity` succeeds, which is the concrete gap the policy closes; and
|
||
`verify-full` cannot be satisfied by a plaintext-only endpoint. Every CA, certificate, and private
|
||
key is generated at test runtime into a temporary directory and deleted afterwards — **no private
|
||
key material is ever committed to this repository** — and the disposable container is removed in
|
||
`afterAll` even when an assertion fails. See
|
||
[ADR 0018](../architecture/decisions/0018-identity-session-resolution-and-database-transport.md).
|
||
|
||
A fifth dedicated suite (`core/tests/module-lifecycle-store.integration.ts`, included in the same
|
||
`pnpm test:migrations:integration` run) proves the module enablement store against real PostgreSQL:
|
||
that migration `0004` really creates `ekklesia_core_module_state` with exactly the four columns and
|
||
the one declared CHECK constraint the schema states — and, load-bearingly, that `enabled` has **no
|
||
column default**, so the platform's default for a newly discovered module is genuinely the absence of
|
||
a row rather than a materialized decision nobody made; that a decision survives a completely new
|
||
connection; that the upsert is idempotent on `module_id` and bumps `updated_at` in place rather than
|
||
inserting a second row; that a malformed identifier is rejected before it reaches PostgreSQL and the
|
||
empty-string case is refused by the database too; and that a record for a module which is no longer
|
||
deployed is retained and simply ignored by resolution, so redeploying restores the decision.
|
||
|
||
The API suite additionally drives the whole lifecycle end to end against a real database and real
|
||
fixture modules: a disabled module's compiled entrypoint is never imported (proven by its
|
||
`register`/`start`/`stop` markers staying empty), its routes return the platform's ordinary `404`,
|
||
its UI artifact and inventory and navigation entries are all absent, and its neighbours run normally;
|
||
a module blocked by a disabled required dependency is explained over HTTP with a structured reason
|
||
naming that dependency and no filesystem path anywhere in the payload; a change made through
|
||
`PUT /api/v1/module-lifecycle/<id>/enabled` reports `restartRequired: true` while the module
|
||
demonstrably keeps serving, and is applied only by the next start; and a disabled module's migrations
|
||
are still discovered, because deployed schema ownership is not runtime enabled state. See
|
||
[ADR 0028](../architecture/decisions/0028-module-operational-lifecycle.md).
|
||
|
||
A sixth dedicated suite (`core/tests/module-configuration-store.integration.ts`, included in the same
|
||
`pnpm test:migrations:integration` run) proves the module configuration store against real
|
||
PostgreSQL: that migration `0005` really creates `ekklesia_core_module_configuration` with the
|
||
composite `(module_id, key)` primary key, the declared CHECK constraints, and **no column that could
|
||
hold secret material**; that a replace is transactional, whole-document, and scoped to one module so
|
||
another module's rows are untouched; that stored values survive a completely new connection; that an
|
||
oversized value is refused by the application *and* by the database; that a store error never carries
|
||
the offending value; and — the security-relevant one — that a module id or key is only ever data in a
|
||
parameterized column, so an injection attempt is a validation error and the table still exists
|
||
afterwards.
|
||
|
||
The API suite additionally proves the configuration boundary end to end: a secret's value never
|
||
appears in any response (including when a row for that key is planted directly in storage), a
|
||
declared-but-undeclared key, a wrong JSON type, a constraint violation, and a `secret` key each fail
|
||
with a distinct sanitized code and store nothing, a reserved property name such as `__proto__` is
|
||
rejected without touching `Object.prototype`, and a module blocked by its own configuration is absent
|
||
from the mountable UI inventory exactly like a disabled one. See
|
||
[ADR 0029](../architecture/decisions/0029-module-configuration-and-secrets.md).
|
||
|
||
A seventh dedicated suite (`core/tests/module-storage.integration.ts`, included in the same
|
||
`pnpm test:migrations:integration` run) proves the object-storage contract against a **real
|
||
S3-compatible server** — the SeaweedFS service the repository's Compose file already pins, brought up
|
||
alongside PostgreSQL by the same harness, so this adds no new CI infrastructure.
|
||
|
||
It exists because mocking would have hidden a real divergence: probing the pinned SeaweedFS 4.28 image
|
||
directly showed that `CreateBucket` on an existing bucket reports `BucketAlreadyExists`, **not** AWS's
|
||
`BucketAlreadyOwnedByYou`. A provider treating only the AWS spelling as success would have failed on
|
||
every second boot against this repository's own development storage, and no unit test would have
|
||
caught it. The suite pins that idempotency, plus a real round trip with content type and unquoted
|
||
ETag preserved, streaming a body without buffering, a Web stream going straight into a `Response`,
|
||
prefix listing and real pagination, a missing object mapping to a structured not-found error carrying
|
||
no endpoint or XML, two modules declaring the same logical bucket staying genuinely isolated on one
|
||
server, the longest bucket name the mapping can produce being accepted, a real connection failure
|
||
mapping to `STORAGE_PROVIDER_UNAVAILABLE`, and a bucket surviving the deletion of every object in it.
|
||
|
||
An eighth suite (`core/tests/module-delivery.integration.ts`, same harness) proves direct browser
|
||
delivery against that same live server, because every claim it makes is a property of the agreement
|
||
between the signer and the provider rather than of code in this repository
|
||
([ADR 0031](../architecture/decisions/0031-direct-object-delivery.md)). It pins: a delivery URL
|
||
returning the object with its content type intact to an ordinary credential-free `fetch`; the same
|
||
object being refused **without** the URL's authorization, so delivery can never be quietly "fixed" by
|
||
making a bucket public; tampered signatures, swapped keys, and swapped buckets all rejected; expiry
|
||
actually enforced; a `Range` request answered directly with `206` and a correct `Content-Range`;
|
||
repeated calls inside a window returning a byte-identical URL so browser caches and URL-keyed DOM
|
||
elements hold; `Cache-Control` never outliving the URL; an optional download filename; and two
|
||
modules' identical logical bucket names resolving to different objects.
|
||
|
||
Its most load-bearing tests concern the **browser delivery origin**. They sign for a public origin and
|
||
replay the request against the real backend through `node:http` with an explicit `Host` header —
|
||
which `fetch` cannot do, since `Host` is a forbidden header name there and it would silently send the
|
||
wrong one and prove nothing. Together they establish that a signature survives a reverse proxy that
|
||
preserves `Host` (even across TLS termination, because the scheme is not signed), that it fails when
|
||
a proxy rewrites `Host`, and that a URL signed for the internal endpoint cannot be rewritten to the
|
||
public one afterwards — which is the entire reason the delivery origin is a signing input rather than
|
||
a post-processing step.
|
||
|
||
An Announcements-shaped fixture in the same file models the real slideshow: it lists its own slides,
|
||
filters images, builds a delivery URL for each, and fetches every one directly. It is written to
|
||
demonstrate what it *cannot* say — its whole world is a `ModuleStorage` and two names from its own
|
||
manifest — and it asserts that the module's durable state holds keys while only the URL carries
|
||
host detail. It also proves that forty URLs cost no provider round trips.
|
||
|
||
Unit suites (`core/tests/module-storage.test.ts`, `core/tests/module-storage-lifecycle.test.ts`) use an
|
||
injected in-memory provider and never touch the network. They prove the parts the protocol cannot:
|
||
binding determinism and collision-safety, scoping and cross-module isolation, key validation, lazy
|
||
provisioning and retry-after-failure, configuration validation and credential redaction, and the
|
||
lifecycle integration including dependency propagation. They also own the delivery bounds, which need
|
||
no server at all: every out-of-range lifetime refused rather than clamped, every download filename
|
||
that could inject a response header refused, and the module-facing surface pinned so a new accessor
|
||
cannot appear unnoticed. See
|
||
[ADR 0030](../architecture/decisions/0030-host-provided-module-object-storage.md) and
|
||
[ADR 0031](../architecture/decisions/0031-direct-object-delivery.md).
|
||
|
||
Phase 7A pass 5 adds realtime evidence at every boundary
|
||
([ADR 0032](../architecture/decisions/0032-native-module-realtime-transport.md)). Contract and Core
|
||
unit suites prove channel grammar, module/permission scoping, staged registration, mandatory initial
|
||
snapshots, atomic snapshot-plus-subscribe, sequence ordering, payload/container limits, latest-state
|
||
coalescing, subscriber limits, exact shutdown behavior, per-module publisher deactivation, sanitized
|
||
errors, and that modules declaring no realtime continue unchanged. Browser SDK/shell suites prove
|
||
that the public contract has no server dependency or transport vocabulary, descriptors and local
|
||
channels are strictly contained, credentials are host-owned, reconnect requires a fresh snapshot,
|
||
terminal EventSource failure cleans up listeners, and explicit unmount is idempotent.
|
||
|
||
`apps/api/tests/module-realtime-routes.test.ts` exercises permission-before-allocation, the
|
||
snapshot/update wire envelope, lack of replay ids, stream headers, comment keepalives, forced
|
||
five-minute reauthorization through a short test-only override, shutdown closure, sanitized
|
||
subscriber failures, and the explicit `HEAD` refusal that prevents an infinite discarded body from
|
||
retaining a subscriber. `apps/api/tests/module-realtime-live.test.ts` binds an actual Node
|
||
HTTP server on an OS-assigned loopback port and drives an Announcements-shaped module through real
|
||
discovery, registration, startup, two live SSE clients, fanout, disconnect cleanup, reconnect with a
|
||
fresh latest snapshot, and shutdown. Every bind, fetch, stream read, cancellation, cleanup poll, and
|
||
listener close is bounded; the test also pins an opaque signed delivery URL byte-for-byte. It uses no
|
||
PostgreSQL or object-store double because neither belongs in realtime transport.
|
||
|
||
The live PostgreSQL API suite adds lifecycle composition evidence: an enabled module's realtime route
|
||
is live while disabled and dependency-blocked modules have no route, just like their HTTP/UI
|
||
surfaces. Its real SSE fetch/read/cancel path is bounded and migrations remain independent of runtime
|
||
enablement. During realtime development, run the focused live Node suite repeatedly as well as
|
||
`pnpm verify` and `pnpm test:migrations:integration`; consecutive runs are specifically intended to
|
||
catch retained timers, listeners, subscribers, or loopback servers.
|
||
|
||
### Which kind of timing proof lives where
|
||
|
||
Pass 4 introduced several deadlines, and their tests deliberately differ in kind:
|
||
|
||
- **Unit deadline tests** (`core/tests/module-runtime.test.ts`) drive module stop and rollback
|
||
hooks through a short, test-only `hookTimeoutMs` override with manually controlled
|
||
never-settling promises — no real waiting, no sleeps, and an explicit `unhandledRejection` check
|
||
proving a hook that rejects after its deadline stays handled.
|
||
- **Real raw-HTTP force-close** (`apps/api/tests/api-http.integration.ts`) binds the real listener,
|
||
opens a real socket, writes a deliberately incomplete request, and proves shutdown force-closes
|
||
it at a short `drainTimeoutMsOverride` and still reaches pool closure.
|
||
- **Real TCP connection timeout** (`core/tests/postgres-migration.integration.ts`) binds a loopback
|
||
blackhole listener that accepts the socket and never completes PostgreSQL startup, proving the
|
||
configured connection deadline; the listener and every accepted socket close in `finally`.
|
||
- **Real PostgreSQL operation timeout and SQL batching** (same file) use a bounded server-side
|
||
delay only because the deadline itself is what is under test, and disposable databases to prove
|
||
that marker bytes inside literals, dollar-quoted bodies, comments, and quoted identifiers execute
|
||
intact, and that a failure rolls the whole batch back.
|
||
|
||
Every one of these uses a short **test-only override**; the production defaults (10 s per module
|
||
hook, 30 s HTTP drain, 10 s migration connection/query/statement) are asserted separately as
|
||
constants and are never waited out.
|
||
|
||
CI runs the same command in the dedicated `migration-smoke` job. After this change is merged, add
|
||
the following pull-request status context to `main` branch protection:
|
||
|
||
```text
|
||
CI / migration-smoke (pull_request)
|
||
```
|
||
|
||
## Safe troubleshooting
|
||
|
||
Confirm the prerequisites without printing configuration:
|
||
|
||
```bash
|
||
docker info
|
||
docker compose version
|
||
```
|
||
|
||
Then rerun `pnpm test:migrations:integration`. The harness emits fixed failure messages and does
|
||
not dump PostgreSQL logs, SQL, container inspection output, or the generated database URL. Do not
|
||
enable query logging, print the test environment, run `docker compose config` without redaction,
|
||
or paste Docker inspection output into an issue. If the Docker daemon itself stopped during
|
||
cleanup, restore it before rerunning so the harness can create a fresh uniquely named project.
|