fix: bound module lifecycle shutdown

Nothing bounded shutdown, and a module that failed to start was never cleaned up.

`startModules` appended to its rollback set only after `start()` resolved, so a
hook that acquired a connection, registered a listener, or recorded partial
initialization and then threw had its own `stop()` skipped, while every
previously completed module was rolled back correctly. A module now joins the
rollback set the moment its start hook is invoked: its own stop hook runs first,
then earlier modules in reverse order. The original start issue stays first in
the reported issues, and a failure stopping the failing module is appended
without preventing the rest of the rollback. The consequence for module authors
is explicit and documented — `stop()` may run after a partially completed
`start()` and must be safe and idempotent about resources it may or may not
hold.

Neither stage of shutdown had a deadline. `shutdown()` awaited `server.close()`
with no bound, so one open connection could hold it forever, and each stop or
rollback hook was awaited without a bound, so a hook that never settled kept the
pool open indefinitely. Both now have fixed, non-configurable defaults: 10
seconds per module hook and 30 seconds for the graceful HTTP drain. A hook that
reaches its deadline records MODULE_STOP_TIMEOUT or MODULE_ROLLBACK_TIMEOUT and
the sequence continues with the next module, so the whole stop sequence is
finite in the started module count rather than one global ten-second bound.
Timers that lose the race are cleared, the losing promise gets a permanent
rejection handler so a late throw cannot become an unhandled rejection, and a
timed-out hook is never awaited again.

At the drain deadline the platform closes idle and then active connections
through Node's own public closeIdleConnections/closeAllConnections, after
server.close(), touching no private field, and stops waiting for the close
callback. It then runs the bounded module stop sequence and closes the pool, so
a drain timeout or a stop failure never skips later cleanup. Both are reported
together as an ApiShutdownError carrying only fixed booleans and the already
sanitized module error; a forced drain is a shutdown failure, not a success.

Stated honestly rather than overstated: a deadline bounds how long the platform
waits, not how long module code runs. An in-process promise cannot be canceled,
and closing a socket does not prove a handler promise ended. What is guaranteed
is that shutdown stops waiting and the pool closes afterwards, so late database
work fails.

Co-Authored-By: Claude Opus 5 <[email protected]>

core/src/modules/runtime.ts

@@ -34,6 +34,7 @@ export type ModuleRuntimeIssueCode =
   | "MODULE_CAPABILITY_UNREGISTERED"
   | "MODULE_REGISTER_FAILED"
   | "MODULE_ROLLBACK_FAILED"
+  | "MODULE_ROLLBACK_TIMEOUT"
   | "MODULE_ROUTE_DEFINITION_INVALID"
   | "MODULE_ROUTE_DUPLICATE"
   | "MODULE_ROUTE_MAX_BODY_BYTES_INVALID"
@@ -47,7 +48,8 @@ export type ModuleRuntimeIssueCode =
   | "MODULE_RUNTIME_IMPORT_FAILED"
   | "MODULE_RUNTIME_VERSION_MISMATCH"
   | "MODULE_START_FAILED"
-  | "MODULE_STOP_FAILED";
+  | "MODULE_STOP_FAILED"
+  | "MODULE_STOP_TIMEOUT";
 
 export interface ModuleRuntimeIssue {
   readonly code: ModuleRuntimeIssueCode;
@@ -74,7 +76,16 @@ export interface StartedModuleRuntime {
   stop(): Promise<void>;
 }
 
-export interface StartModuleRuntimeOptions extends ModuleDiscoveryOptions {
+export interface ModuleRuntimeTimeoutOptions {
+  /** Test-only narrowing of `MODULE_STOP_HOOK_TIMEOUT_MS`, applied to both normal stop and startup
+   * rollback, so a deadline can be proven in milliseconds instead of ten seconds. Production
+   * composition never sets this. */
+  readonly hookTimeoutMs?: number;
+}
+
+export interface StartModuleRuntimeOptions
+  extends ModuleDiscoveryOptions,
+    ModuleRuntimeTimeoutOptions {
   /** The same frozen services object every started module's `start`/`stop` hooks and (once
    * `apps/api` mounts routes) HTTP handlers receive for this runtime — a required production
    * dependency, never an optional test-only global. See `ModuleRuntimeServices`. */
@@ -489,18 +500,93 @@ async function registerModules(
   }
 }
 
+/**
+ * The platform's fixed wait for one module's `stop` hook, during either a normal stop or a startup
+ * rollback. Not configurable in this slice — no environment variable, manifest field, or runtime
+ * service — so every deployment bounds cleanup identically.
+ *
+ * This bounds *waiting*, not execution. An in-process promise cannot be canceled: a hook that
+ * exceeds this deadline keeps running and may settle later, but it no longer delays the rest of
+ * shutdown, and the pool closes after the bounded sequence, so its late database work fails. The
+ * platform does not sandbox, terminate, or undo a trusted module's JavaScript.
+ */
+export const MODULE_STOP_HOOK_TIMEOUT_MS = 10_000;
+
+interface StopModulesOptions {
+  /** Test-only narrowing of `MODULE_STOP_HOOK_TIMEOUT_MS` so a deadline can be proven without
+   * waiting out the production value. Production composition never sets this. */
+  readonly hookTimeoutMs?: number;
+}
+
+/**
+ * Awaits one hook under a deadline. The losing branch is always handled: the timer is cleared when
+ * the hook wins, and a hook that loses gets a permanent no-op rejection handler attached so a late
+ * throw can never surface as an unhandled rejection. A timed-out hook is never awaited again.
+ */
+async function awaitHookWithinDeadline(
+  hook: Promise<void>,
+  timeoutMs: number,
+): Promise<"completed" | "timed-out"> {
+  let timer: ReturnType<typeof setTimeout> | undefined;
+  // Attached before the race so the rejection is considered handled even if the timeout wins.
+  const observed = hook.then(
+    () => "completed" as const,
+    (error: unknown) => {
+      throw error;
+    },
+  );
+  observed.catch(() => undefined);
+  try {
+    return await Promise.race([
+      observed,
+      new Promise<"timed-out">((resolve) => {
+        timer = setTimeout(() => resolve("timed-out"), timeoutMs);
+      }),
+    ]);
+  } finally {
+    clearTimeout(timer);
+  }
+}
+
+/**
+ * Stops modules in reverse order, giving each hook a bounded slice of time and continuing to the
+ * next module whatever the previous one did. Every applicable hook is invoked at most once.
+ *
+ * The whole sequence is finite because the module count is finite and fixed at discovery: its
+ * worst case is `startedModuleCount * MODULE_STOP_HOOK_TIMEOUT_MS`, not one global ten-second
+ * bound.
+ */
 async function stopModules(
   modules: readonly LoadedModule[],
   registry: StagedCapabilityRegistry,
   phase: "rollback" | "stop",
   services: ModuleRuntimeServices,
+  options: StopModulesOptions = {},
 ): Promise<readonly ModuleRuntimeIssue[]> {
+  const timeoutMs = options.hookTimeoutMs ?? MODULE_STOP_HOOK_TIMEOUT_MS;
   const issues: ModuleRuntimeIssue[] = [];
   for (const module of [...modules].reverse()) {
+    const moduleId = module.discovered.manifest.id;
+    const stop = module.definition.stop;
+    if (!stop) {
+      // No synthetic callback and no timer for a module that declares no stop hook.
+      continue;
+    }
     try {
-      await module.definition.stop?.(
-        registry.createRuntimeContext(module.discovered, phase, services),
+      const outcome = await awaitHookWithinDeadline(
+        Promise.resolve(stop(registry.createRuntimeContext(module.discovered, phase, services))),
+        timeoutMs,
       );
+      if (outcome === "timed-out") {
+        issues.push(
+          runtimeIssue(
+            phase === "rollback" ? "MODULE_ROLLBACK_TIMEOUT" : "MODULE_STOP_TIMEOUT",
+            phase,
+            moduleId,
+            `Module ${moduleId} ${phase} did not complete within the platform deadline.`,
+          ),
+        );
+      }
     } catch (error) {
       if (error instanceof ModuleRuntimeSignal) {
         issues.push(error.issue);
@@ -509,8 +595,8 @@ async function stopModules(
           runtimeIssue(
             phase === "rollback" ? "MODULE_ROLLBACK_FAILED" : "MODULE_STOP_FAILED",
             phase,
-            module.discovered.manifest.id,
-            `Module ${module.discovered.manifest.id} ${phase} failed.`,
+            moduleId,
+            `Module ${moduleId} ${phase} failed.`,
           ),
         );
       }
@@ -523,14 +609,21 @@ async function startModules(
   modules: readonly LoadedModule[],
   registry: StagedCapabilityRegistry,
   services: ModuleRuntimeServices,
+  options: StopModulesOptions = {},
 ): Promise<void> {
-  const started: LoadedModule[] = [];
+  // A module joins the rollback set the moment its `start` hook is *invoked*, not when it
+  // completes. A hook that acquired a connection, registered a listener, or recorded partial
+  // initialization before throwing still has cleanup to do, and previously only the modules that
+  // had already finished were rolled back — leaving the failing module's own resources behind. Its
+  // `stop` hook therefore runs first, ahead of the earlier modules in reverse order, which is why
+  // module authors must make `stop` safe after a partially completed `start`.
+  const rollbackCandidates: LoadedModule[] = [];
   for (const module of modules) {
+    rollbackCandidates.push(module);
     try {
       await module.definition.start?.(
         registry.createRuntimeContext(module.discovered, "start", services),
       );
-      started.push(module);
     } catch (error) {
       const startIssue =
         error instanceof ModuleRuntimeSignal
@@ -541,7 +634,13 @@ async function startModules(
               module.discovered.manifest.id,
               `Module ${module.discovered.manifest.id} startup failed.`,
             );
-      const rollbackIssues = await stopModules(started, registry, "rollback", services);
+      const rollbackIssues = await stopModules(
+        rollbackCandidates,
+        registry,
+        "rollback",
+        services,
+        options,
+      );
       throw new ModuleRuntimeError([startIssue, ...rollbackIssues]);
     }
   }
@@ -558,12 +657,13 @@ async function startModules(
 export async function startDiscoveredModuleRuntime(
   discoveredModules: readonly DiscoveredModule[],
   services: ModuleRuntimeServices,
+  timeouts: ModuleRuntimeTimeoutOptions = {},
 ): Promise<StartedModuleRuntime> {
   const modules = await loadRuntimeDefinitions(discoveredModules);
   const capabilityRegistry = new StagedCapabilityRegistry();
   const httpRoutes = new StagedModuleHttpRoutes();
   await registerModules(modules, capabilityRegistry, httpRoutes);
-  await startModules(modules, capabilityRegistry, services);
+  await startModules(modules, capabilityRegistry, services, timeouts);
 
   const capabilities = capabilityRegistry.publish();
   const routes = httpRoutes.publish();
@@ -583,7 +683,7 @@ export async function startDiscoveredModuleRuntime(
     routes,
     stop: (): Promise<void> => {
       stopPromise ??= (async () => {
-        const issues = await stopModules(modules, capabilityRegistry, "stop", services);
+        const issues = await stopModules(modules, capabilityRegistry, "stop", services, timeouts);
         if (issues.length > 0) {
           throw new ModuleRuntimeError(issues);
         }
@@ -597,5 +697,7 @@ export async function startModuleRuntime(
   options: StartModuleRuntimeOptions,
 ): Promise<StartedModuleRuntime> {
   const discovered = await discoverModules(options);
-  return startDiscoveredModuleRuntime(discovered, options.services);
+  return startDiscoveredModuleRuntime(discovered, options.services, {
+    ...(options.hookTimeoutMs === undefined ? {} : { hookTimeoutMs: options.hookTimeoutMs }),
+  });
 }

core/tests/module-runtime.test.ts

@@ -369,10 +369,13 @@ describe("module runtime", () => {
     const error = await captureRuntimeError(modulesDirectory);
 
     expect(runtimeIssueCodes(error)).toEqual(["MODULE_START_FAILED", "MODULE_ROLLBACK_FAILED"]);
+    // The module whose `start` threw is rolled back *first*, then the previously started modules in
+    // reverse order — its own hook may have partial initialization to release before it threw.
     expect(recorder.events).toEqual([
       "one:start",
       "two:start",
       "three:start",
+      "three:stop",
       "two:stop",
       "one:stop",
     ]);
@@ -1151,3 +1154,200 @@ describe("module runtime services", () => {
     expect(vi.mocked(services.database.transaction)).not.toHaveBeenCalled();
   });
 });
+
+/**
+ * Pass 4: a module whose `start` hook fails owns cleanup of whatever it managed to acquire, and
+ * every stop/rollback hook runs under a finite platform deadline.
+ */
+describe("module startup rollback and bounded stop hooks", () => {
+  it("releases a partially initialized resource through the failing module's own stop hook", async () => {
+    const modulesDirectory = await createModulesDirectory();
+    const recorder = createEventRecorder();
+    await createRuntimeModule(modulesDirectory, "partial", {
+      serverSource: `let acquired = false;
+      export default {
+        id: "partial", version: "0.1.0",
+        start() {
+          acquired = true;
+          ${recorder.expression}.push("partial:acquired");
+          throw new Error("secret-token at /private/path");
+        },
+        stop() {
+          if (acquired) { acquired = false; ${recorder.expression}.push("partial:released"); }
+        }
+      };`,
+    });
+
+    const error = await captureRuntimeError(modulesDirectory);
+
+    expect(runtimeIssueCodes(error)).toEqual(["MODULE_START_FAILED"]);
+    expect(recorder.events).toEqual(["partial:acquired", "partial:released"]);
+    expect(JSON.stringify(error.issues)).not.toContain("secret-token");
+    expect(JSON.stringify(error.issues)).not.toContain("/private/path");
+  });
+
+  it("reports the failing module's own rollback failure and still rolls back earlier modules", async () => {
+    const modulesDirectory = await createModulesDirectory();
+    const recorder = createEventRecorder();
+    await createRuntimeModule(modulesDirectory, "earlier", {
+      serverSource: `export default {
+        id: "earlier", version: "0.1.0",
+        stop() { ${recorder.expression}.push("earlier:stop"); }
+      };`,
+    });
+    await createRuntimeModule(modulesDirectory, "failing", {
+      manifest: manifestFor("failing", {
+        dependencies: { modules: { required: { earlier: "^0.1.0" } } },
+      }),
+      serverSource: `export default {
+        id: "failing", version: "0.1.0",
+        start() { throw new Error("start blew up"); },
+        stop() { ${recorder.expression}.push("failing:stop"); throw new Error("rollback blew up"); }
+      };`,
+    });
+
+    const error = await captureRuntimeError(modulesDirectory);
+
+    expect(runtimeIssueCodes(error)).toEqual(["MODULE_START_FAILED", "MODULE_ROLLBACK_FAILED"]);
+    expect(recorder.events).toEqual(["failing:stop", "earlier:stop"]);
+  });
+
+  it("gives a ModuleRuntimeSignal thrown during start the same rollback treatment", async () => {
+    const modulesDirectory = await createModulesDirectory();
+    const recorder = createEventRecorder();
+    // `capabilities.require` for a capability this module never declared throws a
+    // `ModuleRuntimeSignal` from inside `start`, rather than an ordinary Error.
+    await createRuntimeModule(modulesDirectory, "signal", {
+      serverSource: `export default {
+        id: "signal", version: "0.1.0",
+        start({ capabilities }) { capabilities.require("undeclared.thing"); },
+        stop() { ${recorder.expression}.push("signal:stop"); }
+      };`,
+    });
+
+    const error = await captureRuntimeError(modulesDirectory);
+
+    expect(error.issues[0]?.phase).toBe("start");
+    expect(recorder.events).toEqual(["signal:stop"]);
+  });
+
+  it("invokes each stop hook at most once across rollback", async () => {
+    const modulesDirectory = await createModulesDirectory();
+    const recorder = createEventRecorder();
+    await createRuntimeModule(modulesDirectory, "counted", {
+      serverSource: `export default {
+        id: "counted", version: "0.1.0",
+        start() { throw new Error("nope"); },
+        stop() { ${recorder.expression}.push("counted:stop"); }
+      };`,
+    });
+
+    await captureRuntimeError(modulesDirectory);
+
+    expect(recorder.events.filter((event) => event === "counted:stop")).toHaveLength(1);
+  });
+
+  it("bounds a never-settling rollback hook and continues with the earlier module", async () => {
+    const modulesDirectory = await createModulesDirectory();
+    const recorder = createEventRecorder();
+    await createRuntimeModule(modulesDirectory, "earlier", {
+      serverSource: `export default {
+        id: "earlier", version: "0.1.0",
+        stop() { ${recorder.expression}.push("earlier:stop"); }
+      };`,
+    });
+    await createRuntimeModule(modulesDirectory, "hanging", {
+      manifest: manifestFor("hanging", {
+        dependencies: { modules: { required: { earlier: "^0.1.0" } } },
+      }),
+      serverSource: `export default {
+        id: "hanging", version: "0.1.0",
+        start() { throw new Error("nope"); },
+        stop() { ${recorder.expression}.push("hanging:stop"); return new Promise(() => {}); }
+      };`,
+    });
+
+    // A short test-only deadline; production uses MODULE_STOP_HOOK_TIMEOUT_MS unchanged.
+    let captured: ModuleRuntimeError | undefined;
+    try {
+      await startModuleRuntime({ ...runtimeOptions(modulesDirectory), hookTimeoutMs: 25 });
+    } catch (error) {
+      captured = error as ModuleRuntimeError;
+    }
+
+    expect(captured).toBeInstanceOf(ModuleRuntimeError);
+    expect(runtimeIssueCodes(captured as ModuleRuntimeError)).toEqual([
+      "MODULE_START_FAILED",
+      "MODULE_ROLLBACK_TIMEOUT",
+    ]);
+    // The timed-out hook did not block the earlier module's rollback.
+    expect(recorder.events).toEqual(["hanging:stop", "earlier:stop"]);
+  });
+
+  it("bounds a never-settling normal stop hook and continues with the earlier module", async () => {
+    const modulesDirectory = await createModulesDirectory();
+    const recorder = createEventRecorder();
+    await createRuntimeModule(modulesDirectory, "earlier", {
+      serverSource: `export default {
+        id: "earlier", version: "0.1.0",
+        stop() { ${recorder.expression}.push("earlier:stop"); }
+      };`,
+    });
+    await createRuntimeModule(modulesDirectory, "hanging", {
+      manifest: manifestFor("hanging", {
+        dependencies: { modules: { required: { earlier: "^0.1.0" } } },
+      }),
+      serverSource: `export default {
+        id: "hanging", version: "0.1.0",
+        stop() { ${recorder.expression}.push("hanging:stop"); return new Promise(() => {}); }
+      };`,
+    });
+
+    const runtime = await startModuleRuntime({
+      ...runtimeOptions(modulesDirectory),
+      hookTimeoutMs: 25,
+    });
+    let captured: ModuleRuntimeError | undefined;
+    try {
+      await runtime.stop();
+    } catch (error) {
+      captured = error as ModuleRuntimeError;
+    }
+
+    expect(runtimeIssueCodes(captured as ModuleRuntimeError)).toEqual(["MODULE_STOP_TIMEOUT"]);
+    expect(recorder.events).toEqual(["hanging:stop", "earlier:stop"]);
+    // Repeated and concurrent callers share the memoized promise and never restart cleanup.
+    const [first, second] = await Promise.allSettled([runtime.stop(), runtime.stop()]);
+    expect(first.status).toBe("rejected");
+    expect(second.status).toBe("rejected");
+    expect(recorder.events.filter((event) => event === "hanging:stop")).toHaveLength(1);
+  });
+
+  it("never lets a hook that rejects after its deadline become an unhandled rejection", async () => {
+    const modulesDirectory = await createModulesDirectory();
+    await createRuntimeModule(modulesDirectory, "late", {
+      serverSource: `export default {
+        id: "late", version: "0.1.0",
+        stop() { return new Promise((_resolve, reject) => { setTimeout(() => reject(new Error("late")), 40); }); }
+      };`,
+    });
+
+    const unhandled: unknown[] = [];
+    const onUnhandled = (reason: unknown): void => {
+      unhandled.push(reason);
+    };
+    process.on("unhandledRejection", onUnhandled);
+    try {
+      const runtime = await startModuleRuntime({
+        ...runtimeOptions(modulesDirectory),
+        hookTimeoutMs: 10,
+      });
+      await expect(runtime.stop()).rejects.toBeInstanceOf(ModuleRuntimeError);
+      // Give the late rejection time to fire, then confirm nothing escaped.
+      await new Promise((resolve) => setTimeout(resolve, 80));
+      expect(unhandled).toEqual([]);
+    } finally {
+      process.off("unhandledRejection", onUnhandled);
+    }
+  });
+});