Files

T

ekklesiaandClaude Opus 5 490b85b07c fix: bound module lifecycle shutdown

Nothing bounded shutdown, and a module that failed to start was never cleaned up.

`startModules` appended to its rollback set only after `start()` resolved, so a
hook that acquired a connection, registered a listener, or recorded partial
initialization and then threw had its own `stop()` skipped, while every
previously completed module was rolled back correctly. A module now joins the
rollback set the moment its start hook is invoked: its own stop hook runs first,
then earlier modules in reverse order. The original start issue stays first in
the reported issues, and a failure stopping the failing module is appended
without preventing the rest of the rollback. The consequence for module authors
is explicit and documented — `stop()` may run after a partially completed
`start()` and must be safe and idempotent about resources it may or may not
hold.

Neither stage of shutdown had a deadline. `shutdown()` awaited `server.close()`
with no bound, so one open connection could hold it forever, and each stop or
rollback hook was awaited without a bound, so a hook that never settled kept the
pool open indefinitely. Both now have fixed, non-configurable defaults: 10
seconds per module hook and 30 seconds for the graceful HTTP drain. A hook that
reaches its deadline records MODULE_STOP_TIMEOUT or MODULE_ROLLBACK_TIMEOUT and
the sequence continues with the next module, so the whole stop sequence is
finite in the started module count rather than one global ten-second bound.
Timers that lose the race are cleared, the losing promise gets a permanent
rejection handler so a late throw cannot become an unhandled rejection, and a
timed-out hook is never awaited again.

At the drain deadline the platform closes idle and then active connections
through Node's own public closeIdleConnections/closeAllConnections, after
server.close(), touching no private field, and stops waiting for the close
callback. It then runs the bounded module stop sequence and closes the pool, so
a drain timeout or a stop failure never skips later cleanup. Both are reported
together as an ApiShutdownError carrying only fixed booleans and the already
sanitized module error; a forced drain is a shutdown failure, not a success.

Stated honestly rather than overstated: a deadline bounds how long the platform
waits, not how long module code runs. An in-process promise cannot be canceled,
and closing a socket does not prove a handler promise ended. What is guaranteed
is that shutdown stops waiting and the pool closes afterwards, so late database
work fails.

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-07-26 23:12:18 -05:00

13 KiB

Raw Blame History

Testing

Ekklesia separates fast code-level evidence from live infrastructure evidence.

Unit and repository verification

pnpm test
pnpm verify

These commands exercise package tests, repository boundaries, type checking, offline Core and module migration discovery, identity schema drift detection, formatting, linting, and builds. Migration-runner unit tests use a narrow database/session port to prove deterministic planning, approval, transaction orchestration, cleanup, and sanitized errors. They do not claim to prove PostgreSQL behavior. Identity unit tests cover branded opaque-ID contracts, principal shapes, configuration validation (secret rules, canonical-origin-only base URL and trusted origins, and the full loopback/unspecified classifier production rejects — .localhost, all of 127.0.0.0/8, 0.0.0.0, ::1 compressed and expanded, ::, and IPv4-mapped forms), bounded/validated connection and operation (query_timeout/statement_timeout) database timeouts, the Core identity runtime (explicit construction, exact schema mapping, only email/password and rate limiting enabled, an explicit advanced.disableCSRFCheck/disableOriginCheck: false pair, an explicit error-only sanitized logger that strips Better Auth's own message and arguments regardless of what they contain, pinned 8–128 password-length bounds, sanitized failures, exact whole-name session-cookie detection that never fires on a prefixed, suffixed, or differently-cased lookalike, and that a broken session store during principal resolution is never silently reported as anonymous while an unrelated cookie performs no identity database work at all), the PostgreSQL transport policy (core/tests/database-transport.test.ts: mode validation including every loose or aliased spelling, production's plaintext refusal with no loopback exception, connection-string TLS-parameter rejection, and that no configuration path yields rejectUnauthorized: false or leaks CA material into an error), and Core migration discovery (deterministic ordering, the reserved core identity, and collision rejection) — all without a live database. apps/api's own unit tests cover its pure Hono app factory (route shapes, module route mounting/enforcement/CORS/response sanitization, CORS allow/deny/preflight, deterministic 404 and sanitized error responses), its runtime configuration, its idempotent shutdown lifecycle (concurrent callers share one server-close-then-module-runtime-stop-then-pool-close promise; a module-stop failure never prevents pool cleanup; a startup failure after the pool was already created still closes it, stopping the module runtime first if it had already started; SIGINT/SIGTERM wiring is exercised through an injectable exit function rather than the real process.exit), and that importing any of its modules — including the executable server module without calling its exported main — never reads the environment, connects to PostgreSQL, imports a module entrypoint, or starts a server. Real, working (non-throwing) module fixtures prove production module-runtime startup/shutdown ordering and cleanup against a mocked pool/Better Auth/server — see Authorization.

pnpm verify is intentionally Docker-free. It never plans or applies migrations against a live database.

Live PostgreSQL migration evidence

Docker and the Docker Compose plugin are required for the live suite:

pnpm test:migrations:integration

The command first builds the workspace through the repository's Turbo build graph, then starts the repository's pinned PostgreSQL image in a unique Compose project, waits for bounded health readiness, runs the production PostgreSQL adapter and migration services, and removes the container, network, and volume. A build failure occurs before Docker provisioning. Each scenario also receives a safely named disposable database. Direct host runs use a Docker-assigned random port bound only to 127.0.0.1; a containerized Gitea job joins the isolated Compose network and uses the PostgreSQL service name.

The live suite proves fresh read-only planning, initial and incremental application, genuine no-op behavior, exact applied checksums and history, edited or corrupt history rejection, whole-batch DDL rollback, Drizzle breakpoint execution, real advisory-lock concurrency and timeouts, lock release after failure, exact destructive approvals, and public-error sanitization. It does not make migrations automatic during Core startup.

A dedicated set of scenarios exercises the committed Core identity migration against real PostgreSQL: read-only planning on a fresh database, applying it and confirming exactly the four expected ekklesia_core_* tables and their indexes exist, a genuine no-op on a second apply, Core executing before a fixture module migration in one batch, previously applied module-only history staying byte-for-byte unchanged once Core is introduced, and a later module migration's failure rolling back cleanly without damaging the already-committed Core identity migration or leaking the advisory lock.

A second dedicated suite (apps/api/tests/api-http.integration.ts, included in the same pnpm test:migrations:integration run) starts the real apps/api HTTP server against a freshly migrated disposable database and exercises it with real HTTP requests: public sign-up (with no automatic session, per autoSignIn: false), explicit sign-in producing a host-only HttpOnly/SameSite=Lax/secure session cookie, /api/v1/principal for both the authenticated and anonymous shapes, sign-out actually invalidating the session, a malformed or missing cookie resolving as anonymous, duplicate sign-up creating no second identity row, two distinct accounts producing identical principal shapes with no elevation and no owner/role/permission data at the database level, no plaintext password ever stored, the exact 8–128 password-length boundary (7 rejected, 8 accepted, 128 accepted, 129 rejected), CORS never returning an allow-origin header for an untrusted origin, Better Auth's handler being unreachable outside /api/auth/*, a database outage during principal resolution producing a sanitized 503 rather than a false anonymous 200, starting the API against an unmigrated database creating no tables at all, and a clean listener/pool shutdown.

A dedicated group within that suite proves session-store readiness against the real database rather than a mock, since that is the only place a broken store looks exactly like an anonymous request: a request carrying no cookies and one carrying only an unrelated cookie both resolve as anonymous while issuing no identity database work at all; a request presenting the configured session cookie against a dropped session table, a dropped column, and a revoked SELECT privilege each produce a sanitized 503 PRINCIPAL_RESOLUTION_FAILED with no relation name, column name, SQL, or driver text in the body or the logs; and a genuinely expired or unknown session against a healthy store still resolves as anonymous.

The same suite also proves Better Auth's own origin/CSRF protection is production-equivalent under the test runner, not just in a plain Node script: a no-cookie first authentication attempt succeeds from a trusted browser origin and is rejected (403, no permissive CORS header, no leaked detail) from an untrusted one; a cookie-bearing state-changing request (sign-out) is rejected the same way from an untrusted origin, and the presented session remains valid afterward, proving the rejected request never reached sign-out logic. See ADR 0012 for why this previously could not be reproduced reliably under Vitest (Better Auth's own default silently skips these checks whenever isTest() is true, unless advanced.disableOriginCheck is set explicitly) and how the fix removes that gap entirely. A further scenario forces a real identity database failure and asserts, against actual console.error/warn/log output, that Better Auth's own internal error logging — configured explicitly in core/src/identity/runtime.ts — never emits a raw exception, SQL, connection detail, or the rejected request's origin.

A dedicated group in the API suite also drives Pass 3's explicit-grant invariants through a real authorized grants-management session: a malformed non-UUID revoke returning a sanitized 400 with no database work, a canonical unknown UUID returning 404, a scoped create body returning 400 with nothing written, normalized-impossible/non-canonical/past expirations returning 400, an equivalent creation returning a sanitized 409 that names no existing grant, a bounded paginated list traversed across three pages with relationship grants absent from every one, and one HTTP revocation clearing every unrevoked legacy equivalent with a per-row audit entry.

A third dedicated suite (apps/api/tests/module-database.integration.ts, included in the same pnpm test:migrations:integration run) proves context.services.database against a real, installed test module (core/tests/fixtures/module-database-integration/db-facade/ — never the repository's live modules/ directory, and invisible to normal discovery/Doctor/modules:validate unless a test explicitly points discovery at it): the module's own table does not exist until its migration is explicitly applied, and constructing the pool or facade never creates it; start, stop, and an authorized HTTP handler all receive the identical ModuleRuntimeServices object, proven by reference equality; a real authorized mutation persists and a later read observes it, gated by the same permission/trusted-Origin enforcement every other module route uses; a real committed transaction persists and a real rollback (a deliberate unique-constraint violation) leaves no partial mutation; a module deliberately catching a real PostgreSQL failure returns its own response untouched, while an uncaught one sanitizes to the existing MODULE_HANDLER_FAILED boundary with no leaked SQL, error text, or path; and a stop hook's write completes and is independently verifiable, through the harness's own administrative connection — never the API's own pool — before that pool closes. See ADR 0016.

A fourth dedicated suite (core/tests/database-tls.integration.ts, included in the same pnpm test:migrations:integration run) proves the production PostgreSQL transport policy against a real TLS-enabled PostgreSQL rather than against the shape of an options object: a trusted certificate whose identity matches the host connects and is confirmed encrypted through pg_stat_ssl; a certificate that does not chain to the configured CA is refused (UNABLE_TO_VERIFY_LEAF_SIGNATURE); the same server reached through an address its certificate does not cover is refused (ERR_TLS_CERT_ALTNAME_INVALID); the identical connection without the policy's pinned checkServerIdentity succeeds, which is the concrete gap the policy closes; and verify-full cannot be satisfied by a plaintext-only endpoint. Every CA, certificate, and private key is generated at test runtime into a temporary directory and deleted afterwards — no private key material is ever committed to this repository — and the disposable container is removed in afterAll even when an assertion fails. See ADR 0018.

Which kind of timing proof lives where

Pass 4 introduced several deadlines, and their tests deliberately differ in kind:

Unit deadline tests (core/tests/module-runtime.test.ts) drive module stop and rollback hooks through a short, test-only hookTimeoutMs override with manually controlled never-settling promises — no real waiting, no sleeps, and an explicit unhandledRejection check proving a hook that rejects after its deadline stays handled.
Real raw-HTTP force-close (apps/api/tests/api-http.integration.ts) binds the real listener, opens a real socket, writes a deliberately incomplete request, and proves shutdown force-closes it at a short drainTimeoutMsOverride and still reaches pool closure. Both use a short test-only override; the production defaults (10 s per module hook, 30 s HTTP drain) are asserted separately as constants and are never waited out.

CI runs the same command in the dedicated migration-smoke job. After this change is merged, add the following pull-request status context to main branch protection:

CI / migration-smoke (pull_request)

Safe troubleshooting

Confirm the prerequisites without printing configuration:

docker info
docker compose version

Then rerun pnpm test:migrations:integration. The harness emits fixed failure messages and does not dump PostgreSQL logs, SQL, container inspection output, or the generated database URL. Do not enable query logging, print the test environment, run docker compose config without redaction, or paste Docker inspection output into an issue. If the Docker daemon itself stopped during cleanup, restore it before rerunning so the harness can create a fresh uniquely named project.

13 KiB Raw Blame History Unescape Escape