fix: bound module lifecycle shutdown

Nothing bounded shutdown, and a module that failed to start was never cleaned up. `startModules` appended to its rollback set only after `start()` resolved, so a hook that acquired a connection, registered a listener, or recorded partial initialization and then threw had its own `stop()` skipped, while every previously completed module was rolled back correctly. A module now joins the rollback set the moment its start hook is invoked: its own stop hook runs first, then earlier modules in reverse order. The original start issue stays first in the reported issues, and a failure stopping the failing module is appended without preventing the rest of the rollback. The consequence for module authors is explicit and documented — `stop()` may run after a partially completed `start()` and must be safe and idempotent about resources it may or may not hold. Neither stage of shutdown had a deadline. `shutdown()` awaited `server.close()` with no bound, so one open connection could hold it forever, and each stop or rollback hook was awaited without a bound, so a hook that never settled kept the pool open indefinitely. Both now have fixed, non-configurable defaults: 10 seconds per module hook and 30 seconds for the graceful HTTP drain. A hook that reaches its deadline records MODULE_STOP_TIMEOUT or MODULE_ROLLBACK_TIMEOUT and the sequence continues with the next module, so the whole stop sequence is finite in the started module count rather than one global ten-second bound. Timers that lose the race are cleared, the losing promise gets a permanent rejection handler so a late throw cannot become an unhandled rejection, and a timed-out hook is never awaited again. At the drain deadline the platform closes idle and then active connections through Node's own public closeIdleConnections/closeAllConnections, after server.close(), touching no private field, and stops waiting for the close callback. It then runs the bounded module stop sequence and closes the pool, so a drain timeout or a stop failure never skips later cleanup. Both are reported together as an ApiShutdownError carrying only fixed booleans and the already sanitized module error; a forced drain is a shutdown failure, not a success. Stated honestly rather than overstated: a deadline bounds how long the platform waits, not how long module code runs. An in-process promise cannot be canceled, and closing a socket does not prove a handler promise ended. What is guaranteed is that shutdown stops waiting and the pool closes afterwards, so late database work fails. Co-Authored-By: Claude Opus 5 <[email protected]>
2026-07-26 23:12:18 -05:00
parent 800ecfdb92
commit 490b85b07c
9 changed files with 590 additions and 22 deletions
@@ -369,10 +369,13 @@ describe("module runtime", () => {
    const error = await captureRuntimeError(modulesDirectory);

    expect(runtimeIssueCodes(error)).toEqual(["MODULE_START_FAILED", "MODULE_ROLLBACK_FAILED"]);
+    // The module whose `start` threw is rolled back *first*, then the previously started modules in
+    // reverse order — its own hook may have partial initialization to release before it threw.
    expect(recorder.events).toEqual([
      "one:start",
      "two:start",
      "three:start",
+      "three:stop",
      "two:stop",
      "one:stop",
    ]);
@@ -1151,3 +1154,200 @@ describe("module runtime services", () => {
    expect(vi.mocked(services.database.transaction)).not.toHaveBeenCalled();
  });
 });
+
+/**
+ * Pass 4: a module whose `start` hook fails owns cleanup of whatever it managed to acquire, and
+ * every stop/rollback hook runs under a finite platform deadline.
+ */
+describe("module startup rollback and bounded stop hooks", () => {
+  it("releases a partially initialized resource through the failing module's own stop hook", async () => {
+    const modulesDirectory = await createModulesDirectory();
+    const recorder = createEventRecorder();
+    await createRuntimeModule(modulesDirectory, "partial", {
+      serverSource: `let acquired = false;
+      export default {
+        id: "partial", version: "0.1.0",
+        start() {
+          acquired = true;
+          ${recorder.expression}.push("partial:acquired");
+          throw new Error("secret-token at /private/path");
+        },
+        stop() {
+          if (acquired) { acquired = false; ${recorder.expression}.push("partial:released"); }
+        }
+      };`,
+    });
+
+    const error = await captureRuntimeError(modulesDirectory);
+
+    expect(runtimeIssueCodes(error)).toEqual(["MODULE_START_FAILED"]);
+    expect(recorder.events).toEqual(["partial:acquired", "partial:released"]);
+    expect(JSON.stringify(error.issues)).not.toContain("secret-token");
+    expect(JSON.stringify(error.issues)).not.toContain("/private/path");
+  });
+
+  it("reports the failing module's own rollback failure and still rolls back earlier modules", async () => {
+    const modulesDirectory = await createModulesDirectory();
+    const recorder = createEventRecorder();
+    await createRuntimeModule(modulesDirectory, "earlier", {
+      serverSource: `export default {
+        id: "earlier", version: "0.1.0",
+        stop() { ${recorder.expression}.push("earlier:stop"); }
+      };`,
+    });
+    await createRuntimeModule(modulesDirectory, "failing", {
+      manifest: manifestFor("failing", {
+        dependencies: { modules: { required: { earlier: "^0.1.0" } } },
+      }),
+      serverSource: `export default {
+        id: "failing", version: "0.1.0",
+        start() { throw new Error("start blew up"); },
+        stop() { ${recorder.expression}.push("failing:stop"); throw new Error("rollback blew up"); }
+      };`,
+    });
+
+    const error = await captureRuntimeError(modulesDirectory);
+
+    expect(runtimeIssueCodes(error)).toEqual(["MODULE_START_FAILED", "MODULE_ROLLBACK_FAILED"]);
+    expect(recorder.events).toEqual(["failing:stop", "earlier:stop"]);
+  });
+
+  it("gives a ModuleRuntimeSignal thrown during start the same rollback treatment", async () => {
+    const modulesDirectory = await createModulesDirectory();
+    const recorder = createEventRecorder();
+    // `capabilities.require` for a capability this module never declared throws a
+    // `ModuleRuntimeSignal` from inside `start`, rather than an ordinary Error.
+    await createRuntimeModule(modulesDirectory, "signal", {
+      serverSource: `export default {
+        id: "signal", version: "0.1.0",
+        start({ capabilities }) { capabilities.require("undeclared.thing"); },
+        stop() { ${recorder.expression}.push("signal:stop"); }
+      };`,
+    });
+
+    const error = await captureRuntimeError(modulesDirectory);
+
+    expect(error.issues[0]?.phase).toBe("start");
+    expect(recorder.events).toEqual(["signal:stop"]);
+  });
+
+  it("invokes each stop hook at most once across rollback", async () => {
+    const modulesDirectory = await createModulesDirectory();
+    const recorder = createEventRecorder();
+    await createRuntimeModule(modulesDirectory, "counted", {
+      serverSource: `export default {
+        id: "counted", version: "0.1.0",
+        start() { throw new Error("nope"); },
+        stop() { ${recorder.expression}.push("counted:stop"); }
+      };`,
+    });
+
+    await captureRuntimeError(modulesDirectory);
+
+    expect(recorder.events.filter((event) => event === "counted:stop")).toHaveLength(1);
+  });
+
+  it("bounds a never-settling rollback hook and continues with the earlier module", async () => {
+    const modulesDirectory = await createModulesDirectory();
+    const recorder = createEventRecorder();
+    await createRuntimeModule(modulesDirectory, "earlier", {
+      serverSource: `export default {
+        id: "earlier", version: "0.1.0",
+        stop() { ${recorder.expression}.push("earlier:stop"); }
+      };`,
+    });
+    await createRuntimeModule(modulesDirectory, "hanging", {
+      manifest: manifestFor("hanging", {
+        dependencies: { modules: { required: { earlier: "^0.1.0" } } },
+      }),
+      serverSource: `export default {
+        id: "hanging", version: "0.1.0",
+        start() { throw new Error("nope"); },
+        stop() { ${recorder.expression}.push("hanging:stop"); return new Promise(() => {}); }
+      };`,
+    });
+
+    // A short test-only deadline; production uses MODULE_STOP_HOOK_TIMEOUT_MS unchanged.
+    let captured: ModuleRuntimeError | undefined;
+    try {
+      await startModuleRuntime({ ...runtimeOptions(modulesDirectory), hookTimeoutMs: 25 });
+    } catch (error) {
+      captured = error as ModuleRuntimeError;
+    }
+
+    expect(captured).toBeInstanceOf(ModuleRuntimeError);
+    expect(runtimeIssueCodes(captured as ModuleRuntimeError)).toEqual([
+      "MODULE_START_FAILED",
+      "MODULE_ROLLBACK_TIMEOUT",
+    ]);
+    // The timed-out hook did not block the earlier module's rollback.
+    expect(recorder.events).toEqual(["hanging:stop", "earlier:stop"]);
+  });
+
+  it("bounds a never-settling normal stop hook and continues with the earlier module", async () => {
+    const modulesDirectory = await createModulesDirectory();
+    const recorder = createEventRecorder();
+    await createRuntimeModule(modulesDirectory, "earlier", {
+      serverSource: `export default {
+        id: "earlier", version: "0.1.0",
+        stop() { ${recorder.expression}.push("earlier:stop"); }
+      };`,
+    });
+    await createRuntimeModule(modulesDirectory, "hanging", {
+      manifest: manifestFor("hanging", {
+        dependencies: { modules: { required: { earlier: "^0.1.0" } } },
+      }),
+      serverSource: `export default {
+        id: "hanging", version: "0.1.0",
+        stop() { ${recorder.expression}.push("hanging:stop"); return new Promise(() => {}); }
+      };`,
+    });
+
+    const runtime = await startModuleRuntime({
+      ...runtimeOptions(modulesDirectory),
+      hookTimeoutMs: 25,
+    });
+    let captured: ModuleRuntimeError | undefined;
+    try {
+      await runtime.stop();
+    } catch (error) {
+      captured = error as ModuleRuntimeError;
+    }
+
+    expect(runtimeIssueCodes(captured as ModuleRuntimeError)).toEqual(["MODULE_STOP_TIMEOUT"]);
+    expect(recorder.events).toEqual(["hanging:stop", "earlier:stop"]);
+    // Repeated and concurrent callers share the memoized promise and never restart cleanup.
+    const [first, second] = await Promise.allSettled([runtime.stop(), runtime.stop()]);
+    expect(first.status).toBe("rejected");
+    expect(second.status).toBe("rejected");
+    expect(recorder.events.filter((event) => event === "hanging:stop")).toHaveLength(1);
+  });
+
+  it("never lets a hook that rejects after its deadline become an unhandled rejection", async () => {
+    const modulesDirectory = await createModulesDirectory();
+    await createRuntimeModule(modulesDirectory, "late", {
+      serverSource: `export default {
+        id: "late", version: "0.1.0",
+        stop() { return new Promise((_resolve, reject) => { setTimeout(() => reject(new Error("late")), 40); }); }
+      };`,
+    });
+
+    const unhandled: unknown[] = [];
+    const onUnhandled = (reason: unknown): void => {
+      unhandled.push(reason);
+    };
+    process.on("unhandledRejection", onUnhandled);
+    try {
+      const runtime = await startModuleRuntime({
+        ...runtimeOptions(modulesDirectory),
+        hookTimeoutMs: 10,
+      });
+      await expect(runtime.stop()).rejects.toBeInstanceOf(ModuleRuntimeError);
+      // Give the late rejection time to fire, then confirm nothing escaped.
+      await new Promise((resolve) => setTimeout(resolve, 80));
+      expect(unhandled).toEqual([]);
+    } finally {
+      process.off("unhandledRejection", onUnhandled);
+    }
+  });
+});