fix: bound module lifecycle shutdown

Nothing bounded shutdown, and a module that failed to start was never cleaned up.

`startModules` appended to its rollback set only after `start()` resolved, so a
hook that acquired a connection, registered a listener, or recorded partial
initialization and then threw had its own `stop()` skipped, while every
previously completed module was rolled back correctly. A module now joins the
rollback set the moment its start hook is invoked: its own stop hook runs first,
then earlier modules in reverse order. The original start issue stays first in
the reported issues, and a failure stopping the failing module is appended
without preventing the rest of the rollback. The consequence for module authors
is explicit and documented — `stop()` may run after a partially completed
`start()` and must be safe and idempotent about resources it may or may not
hold.

Neither stage of shutdown had a deadline. `shutdown()` awaited `server.close()`
with no bound, so one open connection could hold it forever, and each stop or
rollback hook was awaited without a bound, so a hook that never settled kept the
pool open indefinitely. Both now have fixed, non-configurable defaults: 10
seconds per module hook and 30 seconds for the graceful HTTP drain. A hook that
reaches its deadline records MODULE_STOP_TIMEOUT or MODULE_ROLLBACK_TIMEOUT and
the sequence continues with the next module, so the whole stop sequence is
finite in the started module count rather than one global ten-second bound.
Timers that lose the race are cleared, the losing promise gets a permanent
rejection handler so a late throw cannot become an unhandled rejection, and a
timed-out hook is never awaited again.

At the drain deadline the platform closes idle and then active connections
through Node's own public closeIdleConnections/closeAllConnections, after
server.close(), touching no private field, and stops waiting for the close
callback. It then runs the bounded module stop sequence and closes the pool, so
a drain timeout or a stop failure never skips later cleanup. Both are reported
together as an ApiShutdownError carrying only fixed booleans and the already
sanitized module error; a forced drain is a shutdown failure, not a success.

Stated honestly rather than overstated: a deadline bounds how long the platform
waits, not how long module code runs. An in-process promise cannot be canceled,
and closing a socket does not prove a handler promise ended. What is guaranteed
is that shutdown stops waiting and the pool closes afterwards, so late database
work fails.

Co-Authored-By: Claude Opus 5 <[email protected]>

apps/api/src/server.ts

@@ -130,6 +130,13 @@ export interface StartApiOptions {
    * test can prove real timeout behavior deterministically without waiting out the production
    * values. Never set by the executable entry point. */
   readonly nodeServerTimeoutOverride?: Partial<NodeServerTimeouts>;
+  /** Test-only escape hatch: narrows the graceful HTTP drain deadline
+   * (`HTTP_DRAIN_TIMEOUT_MS`) so a forced close can be proven in milliseconds. Never set by the
+   * executable entry point. */
+  readonly drainTimeoutMsOverride?: number;
+  /** Test-only escape hatch: narrows the per-module stop/rollback hook deadline
+   * (`MODULE_STOP_HOOK_TIMEOUT_MS`). Never set by the executable entry point. */
+  readonly moduleHookTimeoutMsOverride?: number;
 }
 
 async function safeCloseDatabase(handle: ApiDatabaseHandle): Promise<void> {
@@ -152,6 +159,94 @@ async function safeStopModuleRuntime(moduleRuntime: StartedModuleRuntime): Promi
   }
 }
 
+/**
+ * The platform's fixed graceful-drain deadline. In-flight HTTP work has this long to finish after
+ * the listener stops accepting new connections; after it, connections are closed so process
+ * shutdown stays bounded. Not configurable in this slice.
+ */
+export const HTTP_DRAIN_TIMEOUT_MS = 30_000;
+
+/**
+ * A shutdown that did not complete cleanly. Carries only fixed booleans and the already-sanitized
+ * module-runtime error — never a request, socket, URL, credential, SQL, or stack detail — so an
+ * operator can see *which* stages failed without any of it becoming a disclosure channel.
+ */
+export class ApiShutdownError extends Error {
+  readonly drainTimedOut: boolean;
+  readonly moduleStopFailed: boolean;
+  readonly moduleStopError: unknown;
+
+  constructor(drainTimedOut: boolean, moduleStopError: unknown) {
+    super("The API did not shut down cleanly.");
+    this.name = "ApiShutdownError";
+    this.drainTimedOut = drainTimedOut;
+    this.moduleStopFailed = moduleStopError !== undefined;
+    this.moduleStopError = moduleStopError;
+    this.stack = `${this.name}: ${this.message}`;
+    Object.freeze(this);
+  }
+}
+
+/**
+ * Stops accepting new connections, then waits up to `timeoutMs` for existing HTTP work to finish.
+ *
+ * At the deadline it closes idle and then still-active connections using Node's own supported
+ * public `Server` APIs (`closeIdleConnections`/`closeAllConnections`, both called only after
+ * `server.close()`), and stops waiting for the close callback. No private socket or server field is
+ * touched.
+ *
+ * Force-closing a connection ends the transport. It does **not** prove that arbitrary in-process
+ * handler JavaScript was canceled — a handler promise may still be running and may still settle.
+ * What is guaranteed is that shutdown no longer waits for it, and that the pool closes afterwards,
+ * so its late database work fails.
+ */
+/** `ServerType` is a union that also covers HTTP/2 servers, which do not expose these two methods.
+ * `serve()` here always creates a plain `http.Server`, which does; narrowing with a real capability
+ * check keeps that honest instead of asserting a cast. */
+interface ConnectionClosingServer {
+  closeAllConnections(): void;
+  closeIdleConnections(): void;
+}
+
+function supportsConnectionClosing(
+  server: ServerType,
+): server is ServerType & ConnectionClosingServer {
+  const candidate = server as Partial<ConnectionClosingServer>;
+  return (
+    typeof candidate.closeIdleConnections === "function" &&
+    typeof candidate.closeAllConnections === "function"
+  );
+}
+
+async function drainHttpServer(
+  server: ServerType,
+  timeoutMs: number,
+): Promise<"drained" | "forced"> {
+  let timer: ReturnType<typeof setTimeout> | undefined;
+  const closed = new Promise<void>((resolve) => {
+    server.close(() => resolve());
+  });
+  try {
+    const outcome = await Promise.race([
+      closed.then(() => "drained" as const),
+      new Promise<"forced">((resolve) => {
+        timer = setTimeout(() => resolve("forced"), timeoutMs);
+      }),
+    ]);
+    if (outcome === "drained") {
+      return "drained";
+    }
+    if (supportsConnectionClosing(server)) {
+      server.closeIdleConnections();
+      server.closeAllConnections();
+    }
+    return "forced";
+  } finally {
+    clearTimeout(timer);
+    // The close callback may still fire later; nothing awaits it, and it carries no rejection.
+  }
+}
+
 /**
  * Binds the Node HTTP server. Two platform-owned transport concerns are applied at this exact
  * boundary, below Hono entirely: `wrapFetchWithClientAddress` resolves and sanitizes the caller's
@@ -292,7 +387,11 @@ export async function startApi(
   // handle to stop here, only the pool this failure leaves open.
   let moduleRuntime: StartedModuleRuntime;
   try {
-    moduleRuntime = await startDiscoveredModuleRuntime(inventory.modules, services);
+    moduleRuntime = await startDiscoveredModuleRuntime(inventory.modules, services, {
+      ...(options.moduleHookTimeoutMsOverride === undefined
+        ? {}
+        : { hookTimeoutMs: options.moduleHookTimeoutMsOverride }),
+    });
   } catch (error) {
     await safeCloseDatabase(databaseHandle);
     throw error;
@@ -345,13 +444,12 @@ export async function startApi(
   // code can run against it afterward. Unlike the startup-cleanup paths above, a module-stop
   // failure here is not swallowed — it is a genuine shutdown result a caller should observe — but
   // database cleanup still always runs first, so a stop failure never leaves the pool open.
+  const drainTimeoutMs = options.drainTimeoutMsOverride ?? HTTP_DRAIN_TIMEOUT_MS;
   let shutdownPromise: Promise<void> | undefined;
   const shutdown = (): Promise<void> => {
     if (!shutdownPromise) {
       shutdownPromise = (async () => {
-        await new Promise<void>((resolve) => {
-          server.close(() => resolve());
-        });
+        const drainOutcome = await drainHttpServer(server, drainTimeoutMs);
         let moduleStopError: unknown;
         try {
           await moduleRuntime.stop();
@@ -359,8 +457,14 @@ export async function startApi(
           moduleStopError = error;
         }
         await safeCloseDatabase(databaseHandle);
-        if (moduleStopError) {
-          throw moduleStopError;
+        // Every stage above ran regardless of what the earlier ones did, so a drain timeout or a
+        // module-stop failure can never leave the pool open. Both failures are reported together,
+        // as fixed metadata: a forced drain is a genuine shutdown failure, not a success.
+        if (drainOutcome === "forced" || moduleStopError) {
+          throw new ApiShutdownError(
+            drainOutcome === "forced",
+            moduleStopError === undefined ? undefined : moduleStopError,
+          );
         }
       })();
     }