ekklesiaandClaude Opus 5
490b85b07c
Nothing bounded shutdown, and a module that failed to start was never cleaned up. `startModules` appended to its rollback set only after `start()` resolved, so a hook that acquired a connection, registered a listener, or recorded partial initialization and then threw had its own `stop()` skipped, while every previously completed module was rolled back correctly. A module now joins the rollback set the moment its start hook is invoked: its own stop hook runs first, then earlier modules in reverse order. The original start issue stays first in the reported issues, and a failure stopping the failing module is appended without preventing the rest of the rollback. The consequence for module authors is explicit and documented — `stop()` may run after a partially completed `start()` and must be safe and idempotent about resources it may or may not hold. Neither stage of shutdown had a deadline. `shutdown()` awaited `server.close()` with no bound, so one open connection could hold it forever, and each stop or rollback hook was awaited without a bound, so a hook that never settled kept the pool open indefinitely. Both now have fixed, non-configurable defaults: 10 seconds per module hook and 30 seconds for the graceful HTTP drain. A hook that reaches its deadline records MODULE_STOP_TIMEOUT or MODULE_ROLLBACK_TIMEOUT and the sequence continues with the next module, so the whole stop sequence is finite in the started module count rather than one global ten-second bound. Timers that lose the race are cleared, the losing promise gets a permanent rejection handler so a late throw cannot become an unhandled rejection, and a timed-out hook is never awaited again. At the drain deadline the platform closes idle and then active connections through Node's own public closeIdleConnections/closeAllConnections, after server.close(), touching no private field, and stops waiting for the close callback. It then runs the bounded module stop sequence and closes the pool, so a drain timeout or a stop failure never skips later cleanup. Both are reported together as an ApiShutdownError carrying only fixed booleans and the already sanitized module error; a forced drain is a shutdown failure, not a success. Stated honestly rather than overstated: a deadline bounds how long the platform waits, not how long module code runs. An in-process promise cannot be canceled, and closing a socket does not prove a handler promise ended. What is guaranteed is that shutdown stops waiting and the pool closes afterwards, so late database work fails. Co-Authored-By: Claude Opus 5 <[email protected]>
190 lines
13 KiB
Markdown
190 lines
13 KiB
Markdown
# Testing
|
||
|
||
Ekklesia separates fast code-level evidence from live infrastructure evidence.
|
||
|
||
## Unit and repository verification
|
||
|
||
```bash
|
||
pnpm test
|
||
pnpm verify
|
||
```
|
||
|
||
These commands exercise package tests, repository boundaries, type checking, offline Core and
|
||
module migration discovery, identity schema drift detection, formatting, linting, and builds.
|
||
Migration-runner unit tests use a narrow database/session port to prove deterministic planning,
|
||
approval, transaction orchestration, cleanup, and sanitized errors. They do not claim to prove
|
||
PostgreSQL behavior. Identity unit tests cover branded opaque-ID contracts, principal shapes,
|
||
configuration validation (secret rules, canonical-origin-only base URL and trusted origins, and the
|
||
full loopback/unspecified classifier production rejects — `.localhost`, all of `127.0.0.0/8`,
|
||
`0.0.0.0`, `::1` compressed and expanded, `::`, and IPv4-mapped forms), bounded/validated
|
||
connection and operation (`query_timeout`/`statement_timeout`) database timeouts, the Core identity
|
||
runtime (explicit construction, exact schema mapping, only email/password and rate limiting
|
||
enabled, an explicit `advanced.disableCSRFCheck`/`disableOriginCheck: false` pair, an explicit
|
||
error-only sanitized logger that strips Better Auth's own message and arguments regardless of what
|
||
they contain, pinned 8–128 password-length bounds, sanitized failures, exact whole-name
|
||
session-cookie detection that never fires on a prefixed, suffixed, or differently-cased lookalike,
|
||
and that a broken session store during principal resolution is never silently reported as anonymous
|
||
while an unrelated cookie performs no identity database work at all), the PostgreSQL transport
|
||
policy (`core/tests/database-transport.test.ts`: mode validation including every loose or aliased
|
||
spelling, production's plaintext refusal with no loopback exception, connection-string
|
||
TLS-parameter rejection, and that no configuration path yields `rejectUnauthorized: false` or leaks
|
||
CA material into an error), and Core migration discovery (deterministic ordering, the reserved
|
||
`core` identity, and collision rejection) — all without a live database. `apps/api`'s own unit tests cover its pure Hono app factory (route shapes, module
|
||
route mounting/enforcement/CORS/response sanitization, CORS allow/deny/preflight, deterministic 404
|
||
and sanitized error responses), its runtime configuration, its idempotent shutdown lifecycle
|
||
(concurrent callers share one server-close-then-module-runtime-stop-then-pool-close promise; a
|
||
module-stop failure never prevents pool cleanup; a startup failure after the pool was already
|
||
created still closes it, stopping the module runtime first if it had already started;
|
||
`SIGINT`/`SIGTERM` wiring is exercised through an injectable exit function rather than the real
|
||
`process.exit`), and that importing any of its modules — including the executable server module
|
||
without calling its exported `main` — never reads the environment, connects to PostgreSQL, imports
|
||
a module entrypoint, or starts a server. Real, working (non-throwing) module fixtures prove
|
||
production module-runtime startup/shutdown ordering and cleanup against a mocked pool/Better
|
||
Auth/server — see [Authorization](authorization.md#tests).
|
||
|
||
`pnpm verify` is intentionally Docker-free. It never plans or applies migrations against a live
|
||
database.
|
||
|
||
## Live PostgreSQL migration evidence
|
||
|
||
Docker and the Docker Compose plugin are required for the live suite:
|
||
|
||
```bash
|
||
pnpm test:migrations:integration
|
||
```
|
||
|
||
The command first builds the workspace through the repository's Turbo build graph, then starts the
|
||
repository's pinned PostgreSQL image in a unique Compose project, waits for bounded health
|
||
readiness, runs the production PostgreSQL adapter and migration services, and removes the
|
||
container, network, and volume. A build failure occurs before Docker provisioning. Each scenario
|
||
also receives a safely named disposable database. Direct host runs use a Docker-assigned random
|
||
port bound only to `127.0.0.1`; a containerized Gitea job joins the isolated Compose network and
|
||
uses the PostgreSQL service name.
|
||
|
||
The live suite proves fresh read-only planning, initial and incremental application, genuine
|
||
no-op behavior, exact applied checksums and history, edited or corrupt history rejection,
|
||
whole-batch DDL rollback, Drizzle breakpoint execution, real advisory-lock concurrency and
|
||
timeouts, lock release after failure, exact destructive approvals, and public-error sanitization.
|
||
It does not make migrations automatic during Core startup.
|
||
|
||
A dedicated set of scenarios exercises the committed Core identity migration against real
|
||
PostgreSQL: read-only planning on a fresh database, applying it and confirming exactly the four
|
||
expected `ekklesia_core_*` tables and their indexes exist, a genuine no-op on a second apply, Core
|
||
executing before a fixture module migration in one batch, previously applied module-only history
|
||
staying byte-for-byte unchanged once Core is introduced, and a later module migration's failure
|
||
rolling back cleanly without damaging the already-committed Core identity migration or leaking the
|
||
advisory lock.
|
||
|
||
A second dedicated suite (`apps/api/tests/api-http.integration.ts`, included in the same
|
||
`pnpm test:migrations:integration` run) starts the real `apps/api` HTTP server against a freshly
|
||
migrated disposable database and exercises it with real HTTP requests: public sign-up (with no
|
||
automatic session, per `autoSignIn: false`), explicit sign-in producing a host-only
|
||
`HttpOnly`/`SameSite=Lax`/secure session cookie, `/api/v1/principal` for both the authenticated and
|
||
anonymous shapes, sign-out actually invalidating the session, a malformed or missing cookie
|
||
resolving as anonymous, duplicate sign-up creating no second identity row, two distinct accounts
|
||
producing identical principal shapes with no elevation and no owner/role/permission data at the
|
||
database level, no plaintext password ever stored, the exact 8–128 password-length boundary (7
|
||
rejected, 8 accepted, 128 accepted, 129 rejected), CORS never returning an allow-origin header for
|
||
an untrusted origin, Better Auth's handler being unreachable outside `/api/auth/*`, a database
|
||
outage during principal resolution producing a sanitized `503` rather than a false anonymous `200`,
|
||
starting the API against an unmigrated database creating no tables at all, and a clean listener/pool
|
||
shutdown.
|
||
|
||
A dedicated group within that suite proves session-store readiness against the real database rather
|
||
than a mock, since that is the only place a broken store looks exactly like an anonymous request: a
|
||
request carrying no cookies and one carrying only an unrelated cookie both resolve as anonymous
|
||
while issuing **no** identity database work at all; a request presenting the configured session
|
||
cookie against a dropped session table, a dropped column, and a revoked `SELECT` privilege each
|
||
produce a sanitized `503 PRINCIPAL_RESOLUTION_FAILED` with no relation name, column name, SQL, or
|
||
driver text in the body or the logs; and a genuinely expired or unknown session against a healthy
|
||
store still resolves as anonymous.
|
||
|
||
The same suite also proves Better Auth's own origin/CSRF protection is production-equivalent under
|
||
the test runner, not just in a plain Node script: a no-cookie first authentication attempt succeeds
|
||
from a trusted browser origin and is rejected (`403`, no permissive CORS header, no leaked detail)
|
||
from an untrusted one; a cookie-bearing state-changing request (sign-out) is rejected the same way
|
||
from an untrusted origin, and the presented session remains valid afterward, proving the rejected
|
||
request never reached sign-out logic. See
|
||
[ADR 0012](../architecture/decisions/0012-identity-origin-csrf-protection-under-test.md) for why
|
||
this previously could not be reproduced reliably under Vitest (Better Auth's own default silently
|
||
skips these checks whenever `isTest()` is true, unless `advanced.disableOriginCheck` is set
|
||
explicitly) and how the fix removes that gap entirely. A further scenario forces a real identity
|
||
database failure and asserts, against actual `console.error`/`warn`/`log` output, that Better
|
||
Auth's own internal error logging — configured explicitly in `core/src/identity/runtime.ts` — never
|
||
emits a raw exception, SQL, connection detail, or the rejected request's origin.
|
||
|
||
A dedicated group in the API suite also drives Pass 3's explicit-grant invariants through a real
|
||
authorized grants-management session: a malformed non-UUID revoke returning a sanitized `400` with
|
||
no database work, a canonical unknown UUID returning `404`, a scoped create body returning `400`
|
||
with nothing written, normalized-impossible/non-canonical/past expirations returning `400`, an
|
||
equivalent creation returning a sanitized `409` that names no existing grant, a bounded paginated
|
||
list traversed across three pages with relationship grants absent from every one, and one HTTP
|
||
revocation clearing every unrevoked legacy equivalent with a per-row audit entry.
|
||
|
||
A third dedicated suite (`apps/api/tests/module-database.integration.ts`, included in the same
|
||
`pnpm test:migrations:integration` run) proves `context.services.database` against a real,
|
||
installed test module (`core/tests/fixtures/module-database-integration/db-facade/` — never the
|
||
repository's live `modules/` directory, and invisible to normal discovery/Doctor/`modules:validate`
|
||
unless a test explicitly points discovery at it): the module's own table does not exist until its
|
||
migration is explicitly applied, and constructing the pool or facade never creates it; `start`,
|
||
`stop`, and an authorized HTTP handler all receive the identical `ModuleRuntimeServices` object,
|
||
proven by reference equality; a real authorized mutation persists and a later read observes it,
|
||
gated by the same permission/trusted-Origin enforcement every other module route uses; a real
|
||
committed transaction persists and a real rollback (a deliberate unique-constraint violation)
|
||
leaves no partial mutation; a module deliberately catching a real PostgreSQL failure returns its
|
||
own response untouched, while an uncaught one sanitizes to the existing
|
||
`MODULE_HANDLER_FAILED` boundary with no leaked SQL, error text, or path; and a `stop` hook's write
|
||
completes and is independently verifiable, through the harness's own administrative connection —
|
||
never the API's own pool — before that pool closes. See
|
||
[ADR 0016](../architecture/decisions/0016-module-database-access.md).
|
||
|
||
A fourth dedicated suite (`core/tests/database-tls.integration.ts`, included in the same
|
||
`pnpm test:migrations:integration` run) proves the production PostgreSQL transport policy against a
|
||
real TLS-enabled PostgreSQL rather than against the shape of an options object: a trusted
|
||
certificate whose identity matches the host connects and is confirmed encrypted through
|
||
`pg_stat_ssl`; a certificate that does not chain to the configured CA is refused
|
||
(`UNABLE_TO_VERIFY_LEAF_SIGNATURE`); the same server reached through an address its certificate does
|
||
not cover is refused (`ERR_TLS_CERT_ALTNAME_INVALID`); the identical connection *without* the
|
||
policy's pinned `checkServerIdentity` succeeds, which is the concrete gap the policy closes; and
|
||
`verify-full` cannot be satisfied by a plaintext-only endpoint. Every CA, certificate, and private
|
||
key is generated at test runtime into a temporary directory and deleted afterwards — **no private
|
||
key material is ever committed to this repository** — and the disposable container is removed in
|
||
`afterAll` even when an assertion fails. See
|
||
[ADR 0018](../architecture/decisions/0018-identity-session-resolution-and-database-transport.md).
|
||
|
||
### Which kind of timing proof lives where
|
||
|
||
Pass 4 introduced several deadlines, and their tests deliberately differ in kind:
|
||
|
||
- **Unit deadline tests** (`core/tests/module-runtime.test.ts`) drive module stop and rollback
|
||
hooks through a short, test-only `hookTimeoutMs` override with manually controlled
|
||
never-settling promises — no real waiting, no sleeps, and an explicit `unhandledRejection` check
|
||
proving a hook that rejects after its deadline stays handled.
|
||
- **Real raw-HTTP force-close** (`apps/api/tests/api-http.integration.ts`) binds the real listener,
|
||
opens a real socket, writes a deliberately incomplete request, and proves shutdown force-closes
|
||
it at a short `drainTimeoutMsOverride` and still reaches pool closure.
|
||
Both use a short **test-only override**; the production defaults (10 s per module hook, 30 s HTTP
|
||
drain) are asserted separately as constants and are never waited out.
|
||
|
||
CI runs the same command in the dedicated `migration-smoke` job. After this change is merged, add
|
||
the following pull-request status context to `main` branch protection:
|
||
|
||
```text
|
||
CI / migration-smoke (pull_request)
|
||
```
|
||
|
||
## Safe troubleshooting
|
||
|
||
Confirm the prerequisites without printing configuration:
|
||
|
||
```bash
|
||
docker info
|
||
docker compose version
|
||
```
|
||
|
||
Then rerun `pnpm test:migrations:integration`. The harness emits fixed failure messages and does
|
||
not dump PostgreSQL logs, SQL, container inspection output, or the generated database URL. Do not
|
||
enable query logging, print the test environment, run `docker compose config` without redaction,
|
||
or paste Docker inspection output into an issue. If the Docker daemon itself stopped during
|
||
cleanup, restore it before rerunning so the harness can create a fresh uniquely named project.
|