fix: harden migration command execution
Two problems in the migration commands: no timeouts, and SQL split on literal bytes. `migrations:plan` and `migrations:apply` constructed their connection with neither a connection deadline nor an operation deadline — a real server reported `statement_timeout = 0` on the migration connection. Both now default to 10 seconds for the connection deadline, the client query timeout, and PostgreSQL's own statement_timeout, applied at the one shared construction boundary so a future call site cannot silently omit them. History reads, advisory-lock queries, transaction control, migration SQL, and history writes are all covered. The existing lock-wait deadline, the Pass 2 TLS policy, and Doctor's and the bootstrap CLI's own timeouts are untouched, and no environment variable or flag was added. Explicit client fields do not win under the pinned driver. [email protected].0 builds parameters as `Object.assign({}, config, parse(config.connectionString))`, so the URL overrides passed options — verified directly, where an explicit statement_timeout of 10000 alongside `?statement_timeout=0` produced a server-reported 0, while a clean URL produced 10s. A DATABASE_URL carrying connect_timeout, query_timeout, statement_timeout, or an `options` value setting statement_timeout is therefore refused with the existing fixed configuration error, before any client is constructed and without echoing the URL, parameter, or value. Unrelated parameters are left alone. The runner split each migration on the literal `--> statement-breakpoint` bytes. Those bytes inside a string literal, a dollar-quoted function body, a quoted identifier, or a comment tore the migration into fragments that were no longer valid SQL — each case verified to produce two fragments where one was required. Each migration's exact SQL is now passed to the client once, with no values and no prepared-statement name, inside the existing transaction, and its history row inserted only after that batch succeeds. One transaction still spans all pending migrations; advisory locking, checksum validation, approval preflight, ordering, rollback, cleanup, and sanitized error identity are unchanged. The port method is renamed executeStatement to executeMigrationSql so the contract says what it receives. PostgreSQL parses the batch because it owns the only correct parser. A homegrown splitter would have to track single quotes, dollar quoting with arbitrary tags, quoted identifiers, line comments, nested block comments, and escape syntax to answer a question the server already answers, and a parser dependency would add supply-chain surface for the same reason. Drizzle's marker lines are ordinary `--` comments when the batch is sent whole. No custom SQL parser exists here. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
490b85b07c
commit
2983789779
@@ -163,6 +163,32 @@ database or cryptographic guarantee. Phase 5D is still not complete** — passes
|
|||||||
migration correctness) and 5 (deployment/supply chain plus a final independent re-audit) remain
|
migration correctness) and 5 (deployment/supply chain plus a final independent re-audit) remain
|
||||||
outstanding, and Phase 6 remains blocked.
|
outstanding, and Phase 6 remains blocked.
|
||||||
|
|
||||||
|
Phase 5D pass 4 of 5 hardens the module runtime and the migration commands. **Two routes of one
|
||||||
|
module that could both match the same request path are now rejected at registration**
|
||||||
|
(`MODULE_ROUTE_OVERLAP`) instead of being resolved by declaration order — with different permissions
|
||||||
|
on `/events/:id` and `/events/new`, registration order alone previously decided which permission
|
||||||
|
guarded `/events/new`. Decoded route parameters are documented as untrusted, module-owned input that
|
||||||
|
handlers must validate against their own domain grammar; no generic validator or route field was
|
||||||
|
added. A module whose `start` hook fails is now rolled back through **its own** `stop` hook first,
|
||||||
|
so partially acquired resources are released, and every stop/rollback hook runs under a fixed
|
||||||
|
10-second deadline while HTTP shutdown drains for at most 30 seconds before force-closing
|
||||||
|
connections through Node's supported public APIs — shutdown is finite at every stage, reports a
|
||||||
|
forced drain or a hook failure as an unsuccessful shutdown, and always still reaches pool closure.
|
||||||
|
`migrations:plan`/`migrations:apply` now connect with fixed 10-second connection, query, and
|
||||||
|
`statement_timeout` defaults that a `DATABASE_URL` cannot weaken (the pinned driver lets a
|
||||||
|
connection string override explicit options — verified), and each migration's SQL is sent to
|
||||||
|
PostgreSQL **intact, as one parsed batch** inside the existing transaction rather than split on the
|
||||||
|
literal `--> statement-breakpoint` bytes, which corrupted any migration containing those bytes in a
|
||||||
|
string literal, dollar-quoted body, quoted identifier, or comment. See
|
||||||
|
[Module development](docs/module-development/README.md#http-routes),
|
||||||
|
[API](docs/development/api.md#graceful-shutdown), and
|
||||||
|
[ADR 0020](docs/architecture/decisions/0020-module-runtime-and-migration-execution-hardening.md).
|
||||||
|
**Deadlines bound how long the platform waits; they do not cancel or sandbox trusted in-process
|
||||||
|
module code, and closing a socket does not prove a handler promise ended. No custom SQL parser
|
||||||
|
exists. Modules remain trusted compiled backend code with no process, role, schema, or tenant
|
||||||
|
isolation. Phase 5D is still not complete** — pass 5 (deployment/supply chain plus an independent
|
||||||
|
whole-repository re-audit) remains outstanding, and Phase 6 remains blocked.
|
||||||
|
|
||||||
## Quick start
|
## Quick start
|
||||||
|
|
||||||
Prerequisites: Node.js 24 LTS, pnpm 11.12, and Docker with Compose.
|
Prerequisites: Node.js 24 LTS, pnpm 11.12, and Docker with Compose.
|
||||||
|
|||||||
@@ -77,8 +77,72 @@ export class PostgresMigrationDatabaseError extends Error {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The platform's fixed migration connection deadline, applied when a caller does not narrow it.
|
||||||
|
* Lives here, at the one construction boundary every migration command shares, so a future CLI
|
||||||
|
* call site cannot silently omit it.
|
||||||
|
*/
|
||||||
|
export const MIGRATION_CONNECTION_TIMEOUT_MS = 10_000;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The platform's fixed migration operation deadline: both `pg`'s client-side `query_timeout` and
|
||||||
|
* PostgreSQL's own server-side `statement_timeout`. The client-side value bounds a query whose
|
||||||
|
* response never arrives; the server-side value bounds work the server is actually doing. Applied
|
||||||
|
* to every statement the session issues — history reads, advisory-lock queries, transaction
|
||||||
|
* control, migration SQL, and history writes.
|
||||||
|
*/
|
||||||
|
export const MIGRATION_OPERATION_TIMEOUT_MS = 10_000;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Connection-string parameters the pinned `[email protected]` would let silently override the timeouts
|
||||||
|
* selected above. Its `ConnectionParameters` merges as
|
||||||
|
* `Object.assign({}, config, parse(config.connectionString))`, so the **URL wins** over explicitly
|
||||||
|
* passed options — verified directly against the pinned driver, where an explicit
|
||||||
|
* `statement_timeout: 10000` alongside `?statement_timeout=0` produced a server-reported
|
||||||
|
* `statement_timeout` of `0`. Rejecting these is therefore not stylistic: without it the policy
|
||||||
|
* above is advisory.
|
||||||
|
*
|
||||||
|
* Deliberately narrow — only settings that actually conflict with the timeouts this boundary
|
||||||
|
* selects. `options` is included solely because it can carry `-c statement_timeout=...`, and is
|
||||||
|
* rejected only when it does.
|
||||||
|
*/
|
||||||
|
const CONFLICTING_TIMEOUT_PARAMETERS = Object.freeze([
|
||||||
|
"connect_timeout",
|
||||||
|
"query_timeout",
|
||||||
|
"statement_timeout",
|
||||||
|
]);
|
||||||
|
|
||||||
|
function assertNoConflictingTimeoutParameters(connectionString: string): void {
|
||||||
|
let search: URLSearchParams;
|
||||||
|
try {
|
||||||
|
search = new URL(connectionString).searchParams;
|
||||||
|
} catch {
|
||||||
|
throw databaseError(
|
||||||
|
"POSTGRES_MIGRATION_CONFIGURATION_INVALID",
|
||||||
|
"The PostgreSQL database URL is invalid.",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
for (const [rawKey, rawValue] of search.entries()) {
|
||||||
|
const key = rawKey.toLowerCase();
|
||||||
|
if (CONFLICTING_TIMEOUT_PARAMETERS.includes(key)) {
|
||||||
|
throw databaseError(
|
||||||
|
"POSTGRES_MIGRATION_CONFIGURATION_INVALID",
|
||||||
|
"The PostgreSQL database URL must not set a connection or statement timeout.",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (key === "options" && rawValue.toLowerCase().includes("statement_timeout")) {
|
||||||
|
throw databaseError(
|
||||||
|
"POSTGRES_MIGRATION_CONFIGURATION_INVALID",
|
||||||
|
"The PostgreSQL database URL must not set a connection or statement timeout.",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
export interface PostgresMigrationDatabaseOptions {
|
export interface PostgresMigrationDatabaseOptions {
|
||||||
|
/** Omitted means `MIGRATION_CONNECTION_TIMEOUT_MS`. Tests may narrow it; nothing disables it. */
|
||||||
readonly connectionTimeoutMs?: number;
|
readonly connectionTimeoutMs?: number;
|
||||||
|
/** Omitted means `MIGRATION_OPERATION_TIMEOUT_MS`. Tests may narrow it; nothing disables it. */
|
||||||
readonly operationTimeoutMs?: number;
|
readonly operationTimeoutMs?: number;
|
||||||
/** The validated transport policy (`core/src/database/transport.js`). Every production executable
|
/** The validated transport policy (`core/src/database/transport.js`). Every production executable
|
||||||
* that plans or applies migrations passes one; omitting it keeps the pre-existing plaintext
|
* that plans or applies migrations passes one; omitting it keeps the pre-existing plaintext
|
||||||
@@ -177,14 +241,17 @@ function createClientConfig(
|
|||||||
throw invalidConfiguration("The PostgreSQL migration database options are invalid.");
|
throw invalidConfiguration("The PostgreSQL migration database options are invalid.");
|
||||||
}
|
}
|
||||||
|
|
||||||
const validatedConnectionTimeoutMs = validateTimeout(
|
// Defaults are applied here, at the shared construction boundary, so every migration command
|
||||||
connectionTimeoutMs,
|
// inherits a finite deadline without having to remember to ask for one.
|
||||||
"The PostgreSQL connection timeout is invalid.",
|
const validatedConnectionTimeoutMs =
|
||||||
);
|
validateTimeout(connectionTimeoutMs, "The PostgreSQL connection timeout is invalid.") ??
|
||||||
const validatedOperationTimeoutMs = validateTimeout(
|
MIGRATION_CONNECTION_TIMEOUT_MS;
|
||||||
operationTimeoutMs,
|
const validatedOperationTimeoutMs =
|
||||||
"The PostgreSQL operation timeout is invalid.",
|
validateTimeout(operationTimeoutMs, "The PostgreSQL operation timeout is invalid.") ??
|
||||||
);
|
MIGRATION_OPERATION_TIMEOUT_MS;
|
||||||
|
|
||||||
|
// The URL must not be able to weaken what was just selected.
|
||||||
|
assertNoConflictingTimeoutParameters(connectionString);
|
||||||
|
|
||||||
// Also rejects a URL carrying its own TLS parameters, so `pg`'s connection-string precedence
|
// Also rejects a URL carrying its own TLS parameters, so `pg`'s connection-string precedence
|
||||||
// can never silently replace this policy with a weaker one.
|
// can never silently replace this policy with a weaker one.
|
||||||
@@ -195,16 +262,10 @@ function createClientConfig(
|
|||||||
|
|
||||||
return {
|
return {
|
||||||
connectionString,
|
connectionString,
|
||||||
|
connectionTimeoutMillis: validatedConnectionTimeoutMs,
|
||||||
|
query_timeout: validatedOperationTimeoutMs,
|
||||||
...(ssl === undefined ? {} : { ssl }),
|
...(ssl === undefined ? {} : { ssl }),
|
||||||
...(validatedConnectionTimeoutMs === undefined
|
statement_timeout: validatedOperationTimeoutMs,
|
||||||
? {}
|
|
||||||
: { connectionTimeoutMillis: validatedConnectionTimeoutMs }),
|
|
||||||
...(validatedOperationTimeoutMs === undefined
|
|
||||||
? {}
|
|
||||||
: {
|
|
||||||
query_timeout: validatedOperationTimeoutMs,
|
|
||||||
statement_timeout: validatedOperationTimeoutMs,
|
|
||||||
}),
|
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -402,7 +463,7 @@ class PostgresMigrationDatabaseSession implements MigrationDatabaseSession {
|
|||||||
|
|
||||||
const transaction = Object.freeze<MigrationTransaction>({
|
const transaction = Object.freeze<MigrationTransaction>({
|
||||||
ensureHistoryTable: async () => this.#ensureHistoryTable(),
|
ensureHistoryTable: async () => this.#ensureHistoryTable(),
|
||||||
executeStatement: async (sql) => this.#executeStatement(sql),
|
executeMigrationSql: async (sql) => this.#executeMigrationSql(sql),
|
||||||
insertAppliedMigration: async (migration) => this.#insertAppliedMigration(migration),
|
insertAppliedMigration: async (migration) => this.#insertAppliedMigration(migration),
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -479,15 +540,17 @@ class PostgresMigrationDatabaseSession implements MigrationDatabaseSession {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
async #executeStatement(sql: string): Promise<void> {
|
/**
|
||||||
|
* Sends one migration's complete SQL to PostgreSQL in a single simple-query call — no values, no
|
||||||
|
* prepared-statement name, no rewriting — so the server performs all SQL parsing and the packaged
|
||||||
|
* bytes are what actually execute.
|
||||||
|
*/
|
||||||
|
async #executeMigrationSql(sql: string): Promise<void> {
|
||||||
this.#assertTransactionActive("POSTGRES_MIGRATION_EXECUTION_FAILED");
|
this.#assertTransactionActive("POSTGRES_MIGRATION_EXECUTION_FAILED");
|
||||||
try {
|
try {
|
||||||
await this.#client.query(sql);
|
await this.#client.query(sql);
|
||||||
} catch {
|
} catch {
|
||||||
throw databaseError(
|
throw databaseError("POSTGRES_MIGRATION_EXECUTION_FAILED", "A module migration failed.");
|
||||||
"POSTGRES_MIGRATION_EXECUTION_FAILED",
|
|
||||||
"A module migration statement failed.",
|
|
||||||
);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -4,7 +4,6 @@ import type {
|
|||||||
} from "../modules/migration-discovery.js";
|
} from "../modules/migration-discovery.js";
|
||||||
|
|
||||||
const DEFAULT_MIGRATION_LOCK_TIMEOUT_MS = 10_000;
|
const DEFAULT_MIGRATION_LOCK_TIMEOUT_MS = 10_000;
|
||||||
const DRIZZLE_STATEMENT_BREAKPOINT = "--> statement-breakpoint";
|
|
||||||
|
|
||||||
export type MigrationRunnerErrorCode =
|
export type MigrationRunnerErrorCode =
|
||||||
| "MIGRATION_APPROVAL_LOOKUP_FAILED"
|
| "MIGRATION_APPROVAL_LOOKUP_FAILED"
|
||||||
@@ -40,7 +39,12 @@ export interface MigrationHistoryInsert {
|
|||||||
|
|
||||||
export interface MigrationTransaction {
|
export interface MigrationTransaction {
|
||||||
ensureHistoryTable(): Promise<void>;
|
ensureHistoryTable(): Promise<void>;
|
||||||
executeStatement(sql: string): Promise<void>;
|
/**
|
||||||
|
* Executes one migration's **complete** SQL exactly as packaged, in a single call, inside the
|
||||||
|
* caller's transaction. Never a fragment: the runner does not split migration SQL, so the
|
||||||
|
* implementation must pass the string through to PostgreSQL unmodified.
|
||||||
|
*/
|
||||||
|
executeMigrationSql(sql: string): Promise<void>;
|
||||||
insertAppliedMigration(migration: MigrationHistoryInsert): Promise<void>;
|
insertAppliedMigration(migration: MigrationHistoryInsert): Promise<void>;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -484,9 +488,14 @@ export async function applyModuleMigrations(
|
|||||||
await transaction.ensureHistoryTable();
|
await transaction.ensureHistoryTable();
|
||||||
for (const migration of pending) {
|
for (const migration of pending) {
|
||||||
executingIdentity = migration.identity;
|
executingIdentity = migration.identity;
|
||||||
for (const statement of migration.sql.split(DRIZZLE_STATEMENT_BREAKPOINT)) {
|
// PostgreSQL parses the batch, not this runner. Drizzle's `--> statement-breakpoint`
|
||||||
await transaction.executeStatement(statement);
|
// lines are ordinary `--` comments to the server, so sending the migration whole keeps
|
||||||
}
|
// the marker's bytes meaningful wherever they legitimately appear — inside a string
|
||||||
|
// literal, a dollar-quoted function body, a quoted identifier, or a comment — while a
|
||||||
|
// literal `split` on those bytes tore such a migration into fragments that were no
|
||||||
|
// longer valid SQL. Deliberately not a homegrown SQL lexer and not a new parser
|
||||||
|
// dependency: the database already has the only correct parser.
|
||||||
|
await transaction.executeMigrationSql(migration.sql);
|
||||||
await transaction.insertAppliedMigration(freezeHistoryInsert(migration));
|
await transaction.insertAppliedMigration(freezeHistoryInsert(migration));
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -200,7 +200,7 @@ class FakeMigrationDatabaseSession implements MigrationDatabaseSession {
|
|||||||
}
|
}
|
||||||
stagedTableExists = true;
|
stagedTableExists = true;
|
||||||
},
|
},
|
||||||
executeStatement: async (sql) => {
|
executeMigrationSql: async (sql) => {
|
||||||
this.#database.events.push("sql:execute");
|
this.#database.events.push("sql:execute");
|
||||||
this.#database.executedStatements.push(sql);
|
this.#database.executedStatements.push(sql);
|
||||||
const statementIndex = this.#statementCount;
|
const statementIndex = this.#statementCount;
|
||||||
@@ -655,7 +655,7 @@ describe("module migration apply", () => {
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("preserves exact Drizzle breakpoints without naively splitting semicolons", async () => {
|
it("sends each migration's complete SQL to PostgreSQL in exactly one call, byte for byte", async () => {
|
||||||
const sql =
|
const sql =
|
||||||
'CREATE TABLE "people_records" ("id" integer);\n--> statement-breakpoint\nALTER TABLE "people_records" ADD COLUMN "name" text; SELECT 2;';
|
'CREATE TABLE "people_records" ("id" integer);\n--> statement-breakpoint\nALTER TABLE "people_records" ADD COLUMN "name" text; SELECT 2;';
|
||||||
const initial = migration("people", 0, { sql });
|
const initial = migration("people", 0, { sql });
|
||||||
@@ -666,15 +666,54 @@ describe("module migration apply", () => {
|
|||||||
database,
|
database,
|
||||||
});
|
});
|
||||||
|
|
||||||
expect(database.executedStatements).toEqual([
|
// One call, unmodified: the runner never splits, trims, normalizes, or rewrites migration SQL.
|
||||||
'CREATE TABLE "people_records" ("id" integer);\n',
|
// PostgreSQL parses the batch, and Drizzle's marker lines are ordinary `--` comments to it.
|
||||||
'\nALTER TABLE "people_records" ADD COLUMN "name" text; SELECT 2;',
|
expect(database.executedStatements).toEqual([sql]);
|
||||||
]);
|
|
||||||
expect(database.events.indexOf(`history:insert:${initial.identity}`)).toBeGreaterThan(
|
expect(database.events.indexOf(`history:insert:${initial.identity}`)).toBeGreaterThan(
|
||||||
database.events.lastIndexOf("sql:execute"),
|
database.events.lastIndexOf("sql:execute"),
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it.each([
|
||||||
|
[
|
||||||
|
"a string literal containing the marker bytes",
|
||||||
|
`INSERT INTO "notes" ("body") VALUES ('keep this: --> statement-breakpoint intact');`,
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"a dollar-quoted function body containing the marker and semicolons",
|
||||||
|
`CREATE FUNCTION touch() RETURNS void AS $$\nBEGIN\n -- --> statement-breakpoint\n PERFORM 1;\n PERFORM 2;\nEND;\n$$ LANGUAGE plpgsql;`,
|
||||||
|
],
|
||||||
|
["a line comment containing the marker", `-- note: --> statement-breakpoint\nSELECT 1;`],
|
||||||
|
["a block comment containing the marker", `/* --> statement-breakpoint */ SELECT 1;`],
|
||||||
|
[
|
||||||
|
"a quoted identifier containing the marker",
|
||||||
|
`CREATE TABLE "odd--> statement-breakpoint name" ("id" integer);`,
|
||||||
|
],
|
||||||
|
])("never fragments %s", async (_label, sql) => {
|
||||||
|
const initial = migration("people", 0, { sql });
|
||||||
|
const database = new FakeMigrationDatabase();
|
||||||
|
|
||||||
|
await applyModuleMigrations({
|
||||||
|
aggregate: aggregate({ id: "people", migrations: [initial] }),
|
||||||
|
database,
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(database.executedStatements).toEqual([sql]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("sends several generated migrations as one call each, in order", async () => {
|
||||||
|
const first = migration("people", 0, { sql: `SELECT 1;\n--> statement-breakpoint\nSELECT 2;` });
|
||||||
|
const second = migration("people", 1, { sql: `SELECT 3;` });
|
||||||
|
const database = new FakeMigrationDatabase();
|
||||||
|
|
||||||
|
await applyModuleMigrations({
|
||||||
|
aggregate: aggregate({ id: "people", migrations: [first, second] }),
|
||||||
|
database,
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(database.executedStatements).toEqual([first.sql, second.sql]);
|
||||||
|
});
|
||||||
|
|
||||||
it("rolls back the whole pending batch on a later SQL failure and preserves prior history", async () => {
|
it("rolls back the whole pending batch on a later SQL failure and preserves prior history", async () => {
|
||||||
const prior = migration("people", 0);
|
const prior = migration("people", 0);
|
||||||
const pendingFirst = migration("people", 1);
|
const pendingFirst = migration("people", 1);
|
||||||
|
|||||||
@@ -6,6 +6,8 @@ vi.mock("pg", () => ({ Client: clientConstructor }));
|
|||||||
|
|
||||||
import {
|
import {
|
||||||
createPostgresMigrationDatabase,
|
createPostgresMigrationDatabase,
|
||||||
|
MIGRATION_CONNECTION_TIMEOUT_MS,
|
||||||
|
MIGRATION_OPERATION_TIMEOUT_MS,
|
||||||
PostgresMigrationDatabaseError,
|
PostgresMigrationDatabaseError,
|
||||||
} from "../src/migrations/postgres-database.js";
|
} from "../src/migrations/postgres-database.js";
|
||||||
|
|
||||||
@@ -110,3 +112,110 @@ describe("PostgreSQL migration database composition", () => {
|
|||||||
expect(client.end).toHaveBeenCalledOnce();
|
expect(client.end).toHaveBeenCalledOnce();
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Pass 4: finite migration timeouts live at this one construction boundary, and the pinned driver's
|
||||||
|
* connection-string precedence cannot weaken them.
|
||||||
|
*/
|
||||||
|
describe("PostgreSQL migration timeout policy", () => {
|
||||||
|
function mockClient() {
|
||||||
|
const client = {
|
||||||
|
connect: vi.fn().mockResolvedValue(undefined),
|
||||||
|
end: vi.fn().mockResolvedValue(undefined),
|
||||||
|
on: vi.fn(),
|
||||||
|
};
|
||||||
|
clientConstructor.mockImplementation(
|
||||||
|
class {
|
||||||
|
readonly connect = client.connect;
|
||||||
|
readonly end = client.end;
|
||||||
|
readonly on = client.on;
|
||||||
|
} as unknown as () => typeof client,
|
||||||
|
);
|
||||||
|
return client;
|
||||||
|
}
|
||||||
|
|
||||||
|
it("applies the fixed 10-second defaults when a caller omits them", async () => {
|
||||||
|
mockClient();
|
||||||
|
|
||||||
|
const database = createPostgresMigrationDatabase("postgresql://example.test/ekklesia");
|
||||||
|
const session = await database.openSession();
|
||||||
|
|
||||||
|
// This is exactly what `migrations:plan` and `migrations:apply` construct: no timeout options.
|
||||||
|
expect(clientConstructor).toHaveBeenCalledWith({
|
||||||
|
connectionString: "postgresql://example.test/ekklesia",
|
||||||
|
connectionTimeoutMillis: MIGRATION_CONNECTION_TIMEOUT_MS,
|
||||||
|
query_timeout: MIGRATION_OPERATION_TIMEOUT_MS,
|
||||||
|
statement_timeout: MIGRATION_OPERATION_TIMEOUT_MS,
|
||||||
|
});
|
||||||
|
expect(MIGRATION_CONNECTION_TIMEOUT_MS).toBe(10_000);
|
||||||
|
expect(MIGRATION_OPERATION_TIMEOUT_MS).toBe(10_000);
|
||||||
|
|
||||||
|
await session.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("still honours an explicit narrower test override", async () => {
|
||||||
|
mockClient();
|
||||||
|
|
||||||
|
const database = createPostgresMigrationDatabase("postgresql://example.test/ekklesia", {
|
||||||
|
connectionTimeoutMs: 250,
|
||||||
|
operationTimeoutMs: 500,
|
||||||
|
});
|
||||||
|
const session = await database.openSession();
|
||||||
|
|
||||||
|
expect(clientConstructor).toHaveBeenCalledWith({
|
||||||
|
connectionString: "postgresql://example.test/ekklesia",
|
||||||
|
connectionTimeoutMillis: 250,
|
||||||
|
query_timeout: 500,
|
||||||
|
statement_timeout: 500,
|
||||||
|
});
|
||||||
|
|
||||||
|
await session.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it.each([
|
||||||
|
"postgresql://example.test/ekklesia?statement_timeout=0",
|
||||||
|
"postgresql://example.test/ekklesia?STATEMENT_TIMEOUT=0",
|
||||||
|
"postgresql://example.test/ekklesia?query_timeout=0",
|
||||||
|
"postgresql://example.test/ekklesia?connect_timeout=0",
|
||||||
|
"postgresql://example.test/ekklesia?options=-c%20statement_timeout%3D0",
|
||||||
|
"postgresql://example.test/ekklesia?statement_timeout=900000",
|
||||||
|
])("refuses a connection string that would override the policy (%s)", (databaseUrl) => {
|
||||||
|
// The pinned [email protected] merges as `Object.assign({}, config, parse(connectionString))`, so the
|
||||||
|
// URL wins over the explicit values above — verified against the real driver, where an explicit
|
||||||
|
// `statement_timeout: 10000` alongside `?statement_timeout=0` produced a server-reported `0`.
|
||||||
|
const constructionsBefore = clientConstructor.mock.calls.length;
|
||||||
|
let captured: unknown;
|
||||||
|
try {
|
||||||
|
createPostgresMigrationDatabase(databaseUrl);
|
||||||
|
} catch (error) {
|
||||||
|
captured = error;
|
||||||
|
}
|
||||||
|
|
||||||
|
expect(captured).toBeInstanceOf(PostgresMigrationDatabaseError);
|
||||||
|
expect(captured).toMatchObject({ code: "POSTGRES_MIGRATION_CONFIGURATION_INVALID" });
|
||||||
|
// The rejection never echoes the URL, the offending parameter, or its value.
|
||||||
|
const publicText = `${String(captured)}\n${
|
||||||
|
captured instanceof Error ? (captured.stack ?? "") : ""
|
||||||
|
}`;
|
||||||
|
expect(publicText).not.toContain("example.test");
|
||||||
|
expect(publicText).not.toContain("statement_timeout=");
|
||||||
|
expect(publicText).not.toContain("900000");
|
||||||
|
// Rejected before any client is constructed, so no connection is ever attempted.
|
||||||
|
expect(clientConstructor.mock.calls.length).toBe(constructionsBefore);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("leaves unrelated connection-string parameters alone", async () => {
|
||||||
|
mockClient();
|
||||||
|
|
||||||
|
const database = createPostgresMigrationDatabase(
|
||||||
|
"postgresql://example.test/ekklesia?application_name=ekklesia&lock_timeout=5000",
|
||||||
|
);
|
||||||
|
const session = await database.openSession();
|
||||||
|
|
||||||
|
expect(clientConstructor).toHaveBeenCalledWith(
|
||||||
|
expect.objectContaining({ statement_timeout: MIGRATION_OPERATION_TIMEOUT_MS }),
|
||||||
|
);
|
||||||
|
|
||||||
|
await session.close();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
import { createHash, randomBytes } from "node:crypto";
|
import { createHash, randomBytes } from "node:crypto";
|
||||||
|
import net from "node:net";
|
||||||
import path from "node:path";
|
import path from "node:path";
|
||||||
import { fileURLToPath } from "node:url";
|
import { fileURLToPath } from "node:url";
|
||||||
import { Client, type QueryResult, type QueryResultRow } from "pg";
|
import { Client, type QueryResult, type QueryResultRow } from "pg";
|
||||||
@@ -13,6 +14,7 @@ import {
|
|||||||
MigrationRunnerError,
|
MigrationRunnerError,
|
||||||
type MigrationRunnerErrorCode,
|
type MigrationRunnerErrorCode,
|
||||||
planModuleMigrations,
|
planModuleMigrations,
|
||||||
|
PostgresMigrationDatabaseError,
|
||||||
} from "../src/migrations/index.js";
|
} from "../src/migrations/index.js";
|
||||||
import type {
|
import type {
|
||||||
DiscoveredModuleMigration,
|
DiscoveredModuleMigration,
|
||||||
@@ -1457,3 +1459,179 @@ describe.sequential("live PostgreSQL Core identity migration execution", () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Pass 4: the migration commands' finite timeout defaults, and whole-batch SQL execution, proven
|
||||||
|
* against real PostgreSQL and a real TCP endpoint rather than a fake.
|
||||||
|
*
|
||||||
|
* Every listener, client, and disposable database here is closed in `finally`/`afterAll`, including
|
||||||
|
* after a failed assertion.
|
||||||
|
*/
|
||||||
|
describe.sequential("live PostgreSQL migration command hardening", () => {
|
||||||
|
it("bounds a connection to an endpoint that accepts TCP but never completes PostgreSQL startup", async () => {
|
||||||
|
// A real blackhole listener bound only to loopback: it accepts the socket and then says
|
||||||
|
// nothing, which is exactly the case `connectionTimeoutMillis` exists for.
|
||||||
|
const sockets: net.Socket[] = [];
|
||||||
|
const listener = net.createServer((socket) => {
|
||||||
|
sockets.push(socket);
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
await new Promise<void>((resolve, reject) => {
|
||||||
|
listener.once("error", reject);
|
||||||
|
listener.listen(0, "127.0.0.1", () => resolve());
|
||||||
|
});
|
||||||
|
const address = listener.address();
|
||||||
|
if (!address || typeof address !== "object") {
|
||||||
|
throw harnessError("The blackhole listener has no bound address.");
|
||||||
|
}
|
||||||
|
|
||||||
|
// A short, test-only override; production keeps MIGRATION_CONNECTION_TIMEOUT_MS unchanged.
|
||||||
|
const blackholeDatabase = createPostgresMigrationDatabase(
|
||||||
|
`postgresql://ekklesia:[email protected]:${address.port}/ekklesia`,
|
||||||
|
{ connectionTimeoutMs: 300, operationTimeoutMs: 300 },
|
||||||
|
);
|
||||||
|
|
||||||
|
const startedAt = Date.now();
|
||||||
|
let captured: unknown;
|
||||||
|
try {
|
||||||
|
await blackholeDatabase.openSession();
|
||||||
|
} catch (error) {
|
||||||
|
captured = error;
|
||||||
|
}
|
||||||
|
|
||||||
|
expect(captured).toBeInstanceOf(PostgresMigrationDatabaseError);
|
||||||
|
expect(captured).toMatchObject({ code: "POSTGRES_MIGRATION_CONNECTION_FAILED" });
|
||||||
|
// Finite, and nowhere near an unbounded wait.
|
||||||
|
expect(Date.now() - startedAt).toBeLessThan(10_000);
|
||||||
|
const publicText = `${String(captured)}\n${(captured as Error).stack ?? ""}`;
|
||||||
|
expect(publicText).not.toContain("127.0.0.1");
|
||||||
|
expect(publicText).not.toContain("ekklesia:ekklesia");
|
||||||
|
expect(publicText).not.toContain(String(address.port));
|
||||||
|
} finally {
|
||||||
|
for (const socket of sockets) {
|
||||||
|
socket.destroy();
|
||||||
|
}
|
||||||
|
await new Promise<void>((resolve) => {
|
||||||
|
listener.close(() => resolve());
|
||||||
|
});
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it("aborts a migration that exceeds the operation deadline and commits no history row", async () => {
|
||||||
|
await withIsolatedDatabase("operation_timeout", async (database) => {
|
||||||
|
const slow = migration("people", 0, {
|
||||||
|
sql: "SELECT pg_sleep(5);",
|
||||||
|
tag: "0000_slow",
|
||||||
|
});
|
||||||
|
// The operation deadline is the behavior under test, so a bounded server-side delay is the
|
||||||
|
// only way to exercise it.
|
||||||
|
const migrationDatabase = createPostgresMigrationDatabase(database.url, {
|
||||||
|
connectionTimeoutMs: 5_000,
|
||||||
|
operationTimeoutMs: 250,
|
||||||
|
});
|
||||||
|
|
||||||
|
const error = await captureRunnerError(
|
||||||
|
applyModuleMigrations({
|
||||||
|
aggregate: aggregate({ id: "people", migrations: [slow] }),
|
||||||
|
database: migrationDatabase,
|
||||||
|
}),
|
||||||
|
"MIGRATION_EXECUTION_FAILED",
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(error.migrationIdentity).toBe(slow.identity);
|
||||||
|
const publicText = `${String(error)}\n${error.stack ?? ""}`;
|
||||||
|
expect(publicText).not.toContain("pg_sleep");
|
||||||
|
expect(publicText).not.toContain(database.name);
|
||||||
|
|
||||||
|
// Nothing was committed: the transaction rolled back with its history insert.
|
||||||
|
const history = await safeQuery<{ identity: string }>(
|
||||||
|
database.client,
|
||||||
|
`select identity from public.ekklesia_core_migrations`,
|
||||||
|
).catch(() => undefined);
|
||||||
|
expect(history?.rows ?? []).toHaveLength(0);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("executes marker bytes inside literals, dollar-quoted bodies, and comments intact", async () => {
|
||||||
|
await withIsolatedDatabase("intact_sql", async (database) => {
|
||||||
|
const marker = "--> statement-breakpoint";
|
||||||
|
const sql = [
|
||||||
|
`CREATE TABLE "people_notes" ("id" integer primary key, "body" text not null);`,
|
||||||
|
marker,
|
||||||
|
`INSERT INTO "people_notes" ("id", "body") VALUES (1, 'keep this: ${marker} intact');`,
|
||||||
|
marker,
|
||||||
|
`CREATE FUNCTION people_touch() RETURNS integer AS $$`,
|
||||||
|
`BEGIN`,
|
||||||
|
` -- ${marker}`,
|
||||||
|
` PERFORM 1;`,
|
||||||
|
` RETURN 42;`,
|
||||||
|
`END;`,
|
||||||
|
`$$ LANGUAGE plpgsql;`,
|
||||||
|
marker,
|
||||||
|
`/* block comment with ${marker} */`,
|
||||||
|
`CREATE TABLE "odd${marker}name" ("id" integer);`,
|
||||||
|
].join("\n");
|
||||||
|
const intact = migration("people", 0, { sql, tag: "0000_intact" });
|
||||||
|
|
||||||
|
const result = await applyModuleMigrations({
|
||||||
|
aggregate: aggregate({ id: "people", migrations: [intact] }),
|
||||||
|
database: createPostgresMigrationDatabase(database.url),
|
||||||
|
});
|
||||||
|
expect(result.appliedCount).toBe(1);
|
||||||
|
|
||||||
|
// The literal survived byte-for-byte, so the marker was never treated as a split point.
|
||||||
|
const stored = await safeQuery<{ body: string }>(
|
||||||
|
database.client,
|
||||||
|
`select body from "people_notes" where id = 1`,
|
||||||
|
);
|
||||||
|
expect(stored.rows[0]?.body).toBe(`keep this: ${marker} intact`);
|
||||||
|
|
||||||
|
// The dollar-quoted body executed as one function, not as fragments.
|
||||||
|
const called = await safeQuery<{ result: number }>(
|
||||||
|
database.client,
|
||||||
|
`select people_touch() as result`,
|
||||||
|
);
|
||||||
|
expect(Number(called.rows[0]?.result)).toBe(42);
|
||||||
|
|
||||||
|
// The quoted identifier kept the marker in its name.
|
||||||
|
const relations = await safeQuery<{ table_name: string }>(
|
||||||
|
database.client,
|
||||||
|
`select table_name from information_schema.tables
|
||||||
|
where table_schema = 'public' and table_name like 'odd%'`,
|
||||||
|
);
|
||||||
|
expect(relations.rows.map((row) => row.table_name)).toEqual([`odd${marker}name`]);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("rolls back the whole multi-statement migration when a later statement fails", async () => {
|
||||||
|
await withIsolatedDatabase("intact_rollback", async (database) => {
|
||||||
|
const failing = migration("people", 0, {
|
||||||
|
sql: [
|
||||||
|
`CREATE TABLE "people_partial" ("id" integer primary key);`,
|
||||||
|
`--> statement-breakpoint`,
|
||||||
|
`INSERT INTO "people_partial" ("id") VALUES (1);`,
|
||||||
|
`--> statement-breakpoint`,
|
||||||
|
`INSERT INTO "people_partial" ("id") VALUES (1);`,
|
||||||
|
].join("\n"),
|
||||||
|
tag: "0000_rollback",
|
||||||
|
});
|
||||||
|
|
||||||
|
const error = await captureRunnerError(
|
||||||
|
applyModuleMigrations({
|
||||||
|
aggregate: aggregate({ id: "people", migrations: [failing] }),
|
||||||
|
database: createPostgresMigrationDatabase(database.url),
|
||||||
|
}),
|
||||||
|
"MIGRATION_EXECUTION_FAILED",
|
||||||
|
);
|
||||||
|
expect(error.migrationIdentity).toBe(failing.identity);
|
||||||
|
|
||||||
|
// The earlier statements in the same batch rolled back with it.
|
||||||
|
const relations = await safeQuery(
|
||||||
|
database.client,
|
||||||
|
`select table_name from information_schema.tables
|
||||||
|
where table_schema = 'public' and table_name = 'people_partial'`,
|
||||||
|
);
|
||||||
|
expect(relations.rows).toHaveLength(0);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|||||||
@@ -90,6 +90,16 @@ aggregate order, and its history row is inserted only after that migration succe
|
|||||||
failure rolls back every schema change and history insert from the current invocation. Migrations
|
failure rolls back every schema change and history insert from the current invocation. Migrations
|
||||||
committed by earlier invocations remain applied.
|
committed by earlier invocations remain applied.
|
||||||
|
|
||||||
|
> **Partial correction — see [ADR 0020](0020-module-runtime-and-migration-execution-hardening.md).**
|
||||||
|
> Splitting on the literal marker, described in the paragraph immediately below, split a migration
|
||||||
|
> whose SQL legitimately contained those bytes inside a string literal, dollar-quoted body, quoted
|
||||||
|
> identifier, or comment. Core now sends each migration's complete SQL to PostgreSQL in one call
|
||||||
|
> inside the same transaction, so the server does all SQL parsing and the marker lines are ordinary
|
||||||
|
> `--` comments. Everything else recorded here — one transaction across the pending batch, history
|
||||||
|
> rows inserted only after their migration succeeds, no semicolon splitting, no parsing,
|
||||||
|
> classifying, reordering, or repairing SQL, and no non-transactional fallback — is unchanged and
|
||||||
|
> still accepted. This section is left as originally accepted, as the historical record.
|
||||||
|
|
||||||
Core matches Drizzle's generated statement-breakpoint behavior: it splits only on the exact
|
Core matches Drizzle's generated statement-breakpoint behavior: it splits only on the exact
|
||||||
literal `--> statement-breakpoint` marker. It never splits SQL on semicolons or tries to parse,
|
literal `--> statement-breakpoint` marker. It never splits SQL on semicolons or tries to parse,
|
||||||
classify, reorder, or repair arbitrary SQL. Transaction-incompatible SQL fails the transaction;
|
classify, reorder, or repair arbitrary SQL. Transaction-incompatible SQL fails the transaction;
|
||||||
|
|||||||
+9
-1
@@ -127,7 +127,15 @@ decides this mapping, and a module has no way to choose or influence its own mou
|
|||||||
ordering is load-bearing, not cosmetic: Hono's router resolves two routes whose patterns overlap at
|
ordering is load-bearing, not cosmetic: Hono's router resolves two routes whose patterns overlap at
|
||||||
the same depth (a static segment and a same-position `:parameter`) by registration order, not by
|
the same depth (a static segment and a same-position `:parameter`) by registration order, not by
|
||||||
static-path specificity, so preserving the runtime's exact publication order is what lets a module
|
static-path specificity, so preserving the runtime's exact publication order is what lets a module
|
||||||
author's more-specific-first declaration continue to take precedence after mounting. Mounting uses
|
author's more-specific-first declaration continue to take precedence after mounting.
|
||||||
|
|
||||||
|
> **Partial correction — see [ADR 0020](0020-module-runtime-and-migration-execution-hardening.md).**
|
||||||
|
> Relying on registration order to resolve an overlap let the *permission* guarding an ambiguous
|
||||||
|
> path depend on declaration sequence. Core now rejects any two same-module, same-method routes
|
||||||
|
> that can both match one request path (`MODULE_ROUTE_OVERLAP`) during staged registration, so no
|
||||||
|
> such pair ever reaches mounting and no ordering decides which permission applies. Preserving the
|
||||||
|
> publication order of *disjoint* routes, and everything else in this section, is unchanged and
|
||||||
|
> still accepted. Mounting uses
|
||||||
one fixed-arity `app.on(method, path, ...middleware, handler)` call per route rather than spreading
|
one fixed-arity `app.on(method, path, ...middleware, handler)` call per route rather than spreading
|
||||||
a variable-length handler array — Hono's overload resolution is arity-sensitive, and a spread call
|
a variable-length handler array — Hono's overload resolution is arity-sensitive, and a spread call
|
||||||
silently resolves to an unrelated, incompatible overload.
|
silently resolves to an unrelated, incompatible overload.
|
||||||
|
|||||||
@@ -0,0 +1,216 @@
|
|||||||
|
# 0020: Module runtime and migration execution hardening
|
||||||
|
|
||||||
|
- Status: Accepted
|
||||||
|
- Date: 2026-07-27
|
||||||
|
- Amends: [0009](0009-transactional-module-migrations.md), [0015](0015-static-module-http-routing-and-runtime-lifecycle.md)
|
||||||
|
- Extends: [0017](0017-public-request-perimeter-hardening.md), [0019](0019-durable-authorization-history-and-grant-management.md)
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Phase 5D passes 1–3 hardened the public perimeter, the identity/database transport, and
|
||||||
|
authorization integrity. Pass 4 addresses the module runtime and the migration commands underneath
|
||||||
|
them. Seven defects were reproduced against the accepted pass-3 tree before anything changed.
|
||||||
|
|
||||||
|
**Registration order decided which permission governed a request.** A module may register both
|
||||||
|
`GET /events/:id` and `GET /events/new`. Driven through the pinned router, `/events/new` reached the
|
||||||
|
parameter handler with `demo.events.read` when the parameter route registered first, and the static
|
||||||
|
handler with `demo.events.create` when it registered second. Nothing rejected the pair. A permission
|
||||||
|
policy was being decided by array order.
|
||||||
|
|
||||||
|
**Decoded parameters looked safer than they are.** The declaration grammar admits only lowercase
|
||||||
|
kebab-case literals and `:name` segments, but that constrains declarations, not requests. Verified
|
||||||
|
through real requests: `%2F..%2Fadmin` arrives as `/../admin`, `a%5Cb` as `a\b`, `a%00b` with an
|
||||||
|
embedded NUL, `a%0D%0Ab` with CR/LF, `1%3B%20drop` as `1; drop`, and `%C3%A9v%C3%A9nement` as
|
||||||
|
`événement`.
|
||||||
|
|
||||||
|
**A module that failed to start was never cleaned up.** `startModules` appended to its rollback set
|
||||||
|
only *after* `start()` resolved, so a hook that acquired a resource and then threw had its own
|
||||||
|
`stop()` skipped while every previously completed module was rolled back.
|
||||||
|
|
||||||
|
**Nothing bounded shutdown.** `RunningApi.shutdown()` awaited `server.close()` with no deadline, so
|
||||||
|
one open connection could hold it forever, and each module `stop()`/rollback hook was awaited
|
||||||
|
without a deadline, so a hook that never settled prevented pool closure indefinitely.
|
||||||
|
|
||||||
|
**Migration commands had no timeouts.** `migrations:plan` and `migrations:apply` constructed their
|
||||||
|
connection with neither a connection deadline nor an operation deadline; a real server confirmed
|
||||||
|
`statement_timeout = 0` on the migration connection.
|
||||||
|
|
||||||
|
**Migration SQL was split on literal bytes.** The runner did
|
||||||
|
`migration.sql.split("--> statement-breakpoint")`. Those bytes inside a string literal, a
|
||||||
|
dollar-quoted body, a quoted identifier, or a comment split the migration into fragments that were
|
||||||
|
no longer valid SQL — each verified to produce two fragments where one was required.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
### Overlapping same-module routes are rejected, not ranked
|
||||||
|
|
||||||
|
Two routes of one module overlap when they share an HTTP method and a segment count, and at every
|
||||||
|
position either both segments are the same static literal or at least one is a parameter. Under
|
||||||
|
this grammar — static literals or `:name` parameters, with no wildcard, regex, optional, or
|
||||||
|
catch-all segment — that condition is exactly "some concrete request path matches both".
|
||||||
|
|
||||||
|
`moduleRoutesOverlap` (`packages/contracts/src/module-http.ts`) implements it, kept framework-neutral
|
||||||
|
because path semantics already live in that package and Hono stays confined to `apps/api`. Core's
|
||||||
|
staged registration checks it after the existing duplicate check, so identical shapes — including
|
||||||
|
paths differing only in parameter name — keep returning `MODULE_ROUTE_DUPLICATE`, and a genuine
|
||||||
|
non-duplicate intersection returns the new `MODULE_ROUTE_OVERLAP`. Detection happens during the
|
||||||
|
registering module's staged registration, so the module publishes **none** of its staged routes or
|
||||||
|
capabilities, no start hook runs, and the HTTP listener never binds.
|
||||||
|
|
||||||
|
Rejection rather than specificity ordering is the point. A specificity rule ("static beats
|
||||||
|
parameter") would silently pick a winner, and the thing being picked is which permission guards the
|
||||||
|
path — that is a policy decision, not a routing preference, and it must not be inferred. Disjoint
|
||||||
|
routes keep their registration and publication order untouched; different methods, different
|
||||||
|
modules (each has its own platform-owned mount prefix), and different segment counts remain legal.
|
||||||
|
No priority, specificity, wildcard, regex, catch-all, optional-segment, mount-prefix, or manifest
|
||||||
|
field was added.
|
||||||
|
|
||||||
|
### Decoded route parameters are untrusted, module-owned input
|
||||||
|
|
||||||
|
`context.params` continues to carry the router's decoded string values in a frozen context.
|
||||||
|
Decoding is not validation, and validating a *declaration* says nothing about what a *request* may
|
||||||
|
send. Every handler must validate each parameter it uses against its own exact domain grammar —
|
||||||
|
UUID, slug, date, numeric id, or whatever that module owns — before any database, filesystem,
|
||||||
|
command, identifier, or other side-effecting use, and reject missing, repeated, malformed, or
|
||||||
|
semantically invalid values with its own sanitized response. Re-encoding or a decode/re-encode round
|
||||||
|
trip is not a substitute.
|
||||||
|
|
||||||
|
No generic parameter grammar, validator service, path-normalization layer, or new route field was
|
||||||
|
added: module parameters are module-owned domain types, and one platform-wide pattern would either
|
||||||
|
reject legitimate values or approve meaningless ones. The platform keeps owning authentication,
|
||||||
|
authorization, `Origin`, CORS, body limits, cache behavior, and uncaught-handler sanitization, and
|
||||||
|
permission enforcement always completes before a handler — and therefore any parameter validation —
|
||||||
|
can run.
|
||||||
|
|
||||||
|
### A module that fails to start rolls itself back first
|
||||||
|
|
||||||
|
A module joins the rollback set the moment its `start` hook is *invoked*. Its own `stop` hook runs
|
||||||
|
first, then previously completed modules in reverse order. The original start issue stays first in
|
||||||
|
`ModuleRuntimeError.issues`; a failure or timeout stopping the failing module is appended and never
|
||||||
|
prevents earlier modules from being rolled back. Every applicable hook is invoked at most once, a
|
||||||
|
runtime that fails to start is never returned, and every hook receives the same frozen
|
||||||
|
`ModuleRuntimeServices` identity its start hook received. A module with no stop hook gets no
|
||||||
|
synthetic callback.
|
||||||
|
|
||||||
|
The consequence for module authors is explicit: **`stop()` may run after a partially completed
|
||||||
|
`start()`** and must be safe and idempotent about resources it may or may not have acquired. The
|
||||||
|
platform cannot undo arbitrary side effects a trusted module performed outside its stop hook, and
|
||||||
|
does not claim to.
|
||||||
|
|
||||||
|
### Fixed, finite deadlines for hooks and for HTTP drain
|
||||||
|
|
||||||
|
- Per-module `stop`/rollback hook: `MODULE_STOP_HOOK_TIMEOUT_MS` = **10 000 ms**.
|
||||||
|
- Graceful HTTP drain: `HTTP_DRAIN_TIMEOUT_MS` = **30 000 ms**.
|
||||||
|
|
||||||
|
Neither is configurable in this pass — no environment variable, manifest field, runtime service, or
|
||||||
|
flag. Narrow test-only overrides exist so tests need not wait out production durations.
|
||||||
|
|
||||||
|
A hook that reaches its deadline records `MODULE_STOP_TIMEOUT` or `MODULE_ROLLBACK_TIMEOUT` and the
|
||||||
|
sequence continues with the next module in reverse order. The whole module-stop sequence is finite
|
||||||
|
because the module count is fixed at discovery: its worst case is
|
||||||
|
`startedModuleCount × 10 000 ms`, **not** one global ten-second bound. Timers that lose the race are
|
||||||
|
cleared, the losing promise gets a permanent rejection handler so a late throw can never become an
|
||||||
|
unhandled rejection, and a timed-out hook is never awaited again. `stop()` remains at-most-once and
|
||||||
|
memoizes one shared promise for concurrent and later callers.
|
||||||
|
|
||||||
|
Shutdown stops accepting connections with `server.close()`, waits up to the drain deadline, and at
|
||||||
|
the deadline closes idle and then active connections through Node's own public
|
||||||
|
`closeIdleConnections`/`closeAllConnections` — after `server.close()`, never touching a private
|
||||||
|
socket or server field — and stops waiting for the close callback. It then runs the bounded module
|
||||||
|
stop sequence and closes the one shared pool. The accepted order
|
||||||
|
`HTTP drain/forced close → bounded module stop → pool close` is unchanged, and a drain timeout or
|
||||||
|
module-stop failure never skips later cleanup. Both outcomes are reported together as
|
||||||
|
`ApiShutdownError`, carrying fixed booleans and the already-sanitized module error — never a
|
||||||
|
request, socket, URL, credential, SQL, or stack detail. A forced drain is a shutdown *failure*, not
|
||||||
|
a success; signal shutdown exits `0` only after a clean shutdown and `1` otherwise, after cleanup
|
||||||
|
has been attempted.
|
||||||
|
|
||||||
|
**A deadline bounds waiting, not execution.** An in-process promise cannot be canceled: a timed-out
|
||||||
|
hook keeps running and may settle later, and force-closing a connection ends the transport without
|
||||||
|
proving that arbitrary handler JavaScript was canceled. What is guaranteed is that shutdown no
|
||||||
|
longer waits, and that the pool closes afterwards, so late database work fails. Graceful ordering
|
||||||
|
holds until the drain deadline; after it, the platform closes connections and proceeds so process
|
||||||
|
shutdown stays bounded.
|
||||||
|
|
||||||
|
### Finite migration connection and operation defaults, enforced against driver precedence
|
||||||
|
|
||||||
|
`createPostgresMigrationDatabase` — the one construction boundary every migration command shares —
|
||||||
|
now defaults to `MIGRATION_CONNECTION_TIMEOUT_MS` = **10 000 ms** (`connectionTimeoutMillis`) and
|
||||||
|
`MIGRATION_OPERATION_TIMEOUT_MS` = **10 000 ms** for both `query_timeout` (client-side) and
|
||||||
|
`statement_timeout` (server-side). Defaults live there so a future CLI call site cannot silently
|
||||||
|
omit them; explicit test overrides may still narrow them. History reads, advisory-lock queries,
|
||||||
|
transaction control, migration SQL, and history writes are all covered. The existing migration
|
||||||
|
lock-wait deadline and error codes, and the Pass 2 TLS/hostname/certificate policy, are unchanged,
|
||||||
|
as are Doctor's and the bootstrap CLI's own timeouts. No timeout environment variable or flag was
|
||||||
|
added.
|
||||||
|
|
||||||
|
Explicit constructor fields do **not** win under the pinned driver. `[email protected]` builds parameters as
|
||||||
|
`Object.assign({}, config, parse(config.connectionString))`, so the connection string overrides
|
||||||
|
passed options — verified directly: an explicit `statement_timeout: 10000` alongside
|
||||||
|
`?statement_timeout=0` produced a server-reported `statement_timeout` of `0`, while a clean URL
|
||||||
|
produced `10s`. A `DATABASE_URL` carrying `connect_timeout`, `query_timeout`, `statement_timeout`,
|
||||||
|
or an `options` value that sets `statement_timeout` is therefore refused with the existing fixed
|
||||||
|
`POSTGRES_MIGRATION_CONFIGURATION_INVALID`, before any client is constructed and without echoing the
|
||||||
|
URL, parameter, or value. The rejection is deliberately narrow — only settings that actually conflict
|
||||||
|
with the selected policy; unrelated parameters are untouched.
|
||||||
|
|
||||||
|
### Each migration is one PostgreSQL-parsed batch
|
||||||
|
|
||||||
|
The runner no longer splits migration SQL. Each pending migration's exact string is passed once to
|
||||||
|
the client — no values, no prepared-statement name, no trimming, normalizing, or rewriting — inside
|
||||||
|
the existing transaction, and its history row is inserted only after that batch succeeds. One
|
||||||
|
transaction still spans all pending migrations and their history records, and advisory locking,
|
||||||
|
checksum validation, approval preflight, ordering, rollback, cleanup, and sanitized error identity
|
||||||
|
are unchanged. The transaction port method was renamed `executeStatement` → `executeMigrationSql` so
|
||||||
|
the contract states what it actually receives.
|
||||||
|
|
||||||
|
PostgreSQL was chosen as the parser because it is the only correct one. A homegrown splitter would
|
||||||
|
need to track single quotes, dollar quoting with arbitrary tags, quoted identifiers, line comments,
|
||||||
|
block-comment nesting, and escape syntax — reimplementing a SQL lexer to answer a question the
|
||||||
|
server already answers — and a parser dependency would add supply-chain surface for the same
|
||||||
|
reason. Drizzle's `--> statement-breakpoint` lines are ordinary `--` comments when the batch is sent
|
||||||
|
whole, so the same bytes inside a literal, dollar-quoted body, quoted identifier, or comment retain
|
||||||
|
their ordinary PostgreSQL meaning. **No custom SQL parser exists in this repository.**
|
||||||
|
|
||||||
|
## Alternatives
|
||||||
|
|
||||||
|
- **Ordering static routes ahead of parameter routes** was rejected: it silently resolves an
|
||||||
|
ambiguity whose resolution is a permission decision, and it would make the governing permission a
|
||||||
|
function of an implementation detail rather than of a declaration.
|
||||||
|
- **A platform-wide route-parameter grammar** was rejected: parameters are module-owned domain
|
||||||
|
types, so any single pattern is simultaneously too strict and too permissive.
|
||||||
|
- **Normalizing or re-encoding decoded parameters** was rejected: it produces a value the module
|
||||||
|
never declared and still does not answer whether the value is a valid domain identifier.
|
||||||
|
- **A second, dedicated rollback hook** was rejected: it would double the lifecycle surface to
|
||||||
|
express something `stop` already means, and every module would have to implement both.
|
||||||
|
- **Making the deadlines configurable** was rejected for this pass: a configurable shutdown bound
|
||||||
|
invites setting it to infinity, which is the defect being fixed.
|
||||||
|
- **Claiming a timed-out hook is terminated** was rejected as untrue; the honest guarantee is that
|
||||||
|
the platform stops waiting.
|
||||||
|
- **Trusting explicit `pg` client fields over the connection string** was rejected once the pinned
|
||||||
|
driver was shown to do the opposite.
|
||||||
|
- **A timeout environment variable** was rejected as out of scope; a fixed default that cannot be
|
||||||
|
weakened is the safer starting point.
|
||||||
|
- **Writing a SQL statement splitter, or adding a SQL parser dependency** — see above.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
An ambiguous same-method route pair fails module registration with a stable code instead of
|
||||||
|
resolving by array order, and no request can reach a match whose permission depended on registration
|
||||||
|
sequence. Decoded parameters are documented as untrusted input that handlers must validate, with no
|
||||||
|
platform grammar implied. A module whose `start` fails is cleaned up through its own `stop` hook
|
||||||
|
before earlier modules. Shutdown is finite in every stage, observable when it is not clean, and
|
||||||
|
always reaches pool closure. Migration commands have finite connection, query, and statement
|
||||||
|
deadlines that a connection string cannot weaken, and each migration reaches PostgreSQL byte-for-byte
|
||||||
|
in a single parsed batch.
|
||||||
|
|
||||||
|
This pass adds no production schema, migration, manifest field, `manifestVersion` change,
|
||||||
|
`ModuleRuntimeServices` field, `ModuleDatabase` operation, route-declaration field, handler-context
|
||||||
|
field, second runtime, route registry, or runtime pool; the three accepted Core migrations and their
|
||||||
|
checksums are byte-identical, one API runtime still owns one physical pool, migrations remain
|
||||||
|
explicit operator work that never runs at application startup, and Hono remains confined to
|
||||||
|
`apps/api`. Modules remain trusted compiled backend code: they are **not** isolated by process,
|
||||||
|
PostgreSQL role, schema, or tenant, and these deadlines do not sandbox them.
|
||||||
|
|
||||||
|
**Phase 5D is not complete.** Pass 5 — deployment/supply-chain work and an independent
|
||||||
|
whole-repository re-audit — remains outstanding, and Phase 6 remains blocked.
|
||||||
@@ -28,3 +28,4 @@ ADRs capture durable choices that affect more than one package or phase.
|
|||||||
- [0017: Public request perimeter hardening — client address, signup gating, body limits, timeouts](0017-public-request-perimeter-hardening.md) — Accepted
|
- [0017: Public request perimeter hardening — client address, signup gating, body limits, timeouts](0017-public-request-perimeter-hardening.md) — Accepted
|
||||||
- [0018: Identity session resolution, canonical public origins, and PostgreSQL transport policy](0018-identity-session-resolution-and-database-transport.md) — Accepted
|
- [0018: Identity session resolution, canonical public origins, and PostgreSQL transport policy](0018-identity-session-resolution-and-database-transport.md) — Accepted
|
||||||
- [0019: Durable authorization history and grant management](0019-durable-authorization-history-and-grant-management.md) — Accepted
|
- [0019: Durable authorization history and grant management](0019-durable-authorization-history-and-grant-management.md) — Accepted
|
||||||
|
- [0020: Module runtime and migration execution hardening](0020-module-runtime-and-migration-execution-hardening.md) — Accepted
|
||||||
|
|||||||
@@ -68,7 +68,7 @@ under the reserved source identifier `"core"`, and
|
|||||||
aggregate, **Core first**. An installed module manifest that ever declares `id: "core"` is rejected
|
aggregate, **Core first**. An installed module manifest that ever declares `id: "core"` is rejected
|
||||||
at discovery time so Core and module migration identities can never collide.
|
at discovery time so Core and module migration identities can never collide.
|
||||||
|
|
||||||
The transactional runner, advisory lock, breakpoint splitter, and the fixed
|
The transactional runner, advisory lock, whole-batch SQL execution, and the fixed
|
||||||
`public.ekklesia_core_migrations` history table (ADR 0009) are unchanged: Core's migration honestly
|
`public.ekklesia_core_migrations` history table (ADR 0009) are unchanged: Core's migration honestly
|
||||||
populates the existing `module_id` column with the literal value `"core"`, exactly like a module
|
populates the existing `module_id` column with the literal value `"core"`, exactly like a module
|
||||||
populates it with its own module ID. There is exactly one migration system and one executor.
|
populates it with its own module ID. There is exactly one migration system and one executor.
|
||||||
@@ -82,6 +82,21 @@ pnpm migrations:apply # locked, transactional application of the pending bat
|
|||||||
pnpm doctor # includes Core's migration in its discovery/plan checks
|
pnpm doctor # includes Core's migration in its discovery/plan checks
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Both `migrations:plan` and `migrations:apply` connect with fixed, finite defaults — a 10-second
|
||||||
|
connection deadline, a 10-second client query timeout, and a 10-second PostgreSQL
|
||||||
|
`statement_timeout` — applied at the one shared construction boundary so no command can omit them.
|
||||||
|
A `DATABASE_URL` that tries to set `connect_timeout`, `query_timeout`, `statement_timeout`, or an
|
||||||
|
`options` value carrying `statement_timeout` is refused with a sanitized configuration error,
|
||||||
|
because the pinned driver lets the connection string override explicitly passed options. This bounds
|
||||||
|
each individual operation; it does **not** promise that a migration completes within ten seconds
|
||||||
|
across retries or cleanup. Migrations remain explicit operator work and never run at application
|
||||||
|
startup.
|
||||||
|
|
||||||
|
Each pending migration's SQL is sent to PostgreSQL as **one complete batch**, inside the existing
|
||||||
|
transaction, so the server does all SQL parsing — Drizzle's `--> statement-breakpoint` lines are
|
||||||
|
ordinary `--` comments to it. No SQL parser exists in this repository, and the marker's bytes keep
|
||||||
|
their ordinary meaning inside a string literal, dollar-quoted body, quoted identifier, or comment.
|
||||||
|
|
||||||
No new commands are required to include identity: it flows through the existing migration
|
No new commands are required to include identity: it flows through the existing migration
|
||||||
commands automatically. A fresh database applies the Core identity migration through
|
commands automatically. A fresh database applies the Core identity migration through
|
||||||
`pnpm migrations:apply` like any other pending migration; a second application is a genuine no-op.
|
`pnpm migrations:apply` like any other pending migration; a second application is a genuine no-op.
|
||||||
|
|||||||
@@ -163,8 +163,17 @@ Pass 4 introduced several deadlines, and their tests deliberately differ in kind
|
|||||||
- **Real raw-HTTP force-close** (`apps/api/tests/api-http.integration.ts`) binds the real listener,
|
- **Real raw-HTTP force-close** (`apps/api/tests/api-http.integration.ts`) binds the real listener,
|
||||||
opens a real socket, writes a deliberately incomplete request, and proves shutdown force-closes
|
opens a real socket, writes a deliberately incomplete request, and proves shutdown force-closes
|
||||||
it at a short `drainTimeoutMsOverride` and still reaches pool closure.
|
it at a short `drainTimeoutMsOverride` and still reaches pool closure.
|
||||||
Both use a short **test-only override**; the production defaults (10 s per module hook, 30 s HTTP
|
- **Real TCP connection timeout** (`core/tests/postgres-migration.integration.ts`) binds a loopback
|
||||||
drain) are asserted separately as constants and are never waited out.
|
blackhole listener that accepts the socket and never completes PostgreSQL startup, proving the
|
||||||
|
configured connection deadline; the listener and every accepted socket close in `finally`.
|
||||||
|
- **Real PostgreSQL operation timeout and SQL batching** (same file) use a bounded server-side
|
||||||
|
delay only because the deadline itself is what is under test, and disposable databases to prove
|
||||||
|
that marker bytes inside literals, dollar-quoted bodies, comments, and quoted identifiers execute
|
||||||
|
intact, and that a failure rolls the whole batch back.
|
||||||
|
|
||||||
|
Every one of these uses a short **test-only override**; the production defaults (10 s per module
|
||||||
|
hook, 30 s HTTP drain, 10 s migration connection/query/statement) are asserted separately as
|
||||||
|
constants and are never waited out.
|
||||||
|
|
||||||
CI runs the same command in the dedicated `migration-smoke` job. After this change is merged, add
|
CI runs the same command in the dedicated `migration-smoke` job. After this change is merged, add
|
||||||
the following pull-request status context to `main` branch protection:
|
the following pull-request status context to `main` branch protection:
|
||||||
|
|||||||
Reference in New Issue
Block a user