fix: bound provider exchanges and stop the storage suite starving its server

The live storage suite failed in CI with three failures and a 30s runner
timeout while passing locally in under three seconds. Two independent
defects produced that, one of them a real runtime defect.

Provider calls had no deadline. `S3Client` was built with no
`requestHandler` and no `maxAttempts`, and the SDK's `connectionTimeout`
and `socketTimeout` both default to 0 — disabled. A provider that accepts
a connection and then goes silent therefore held the caller open forever,
bounded in the suite only by vitest and in production by nothing at all.
ADR 0030 promises such a failure reaches a module as the transient
`STORAGE_PROVIDER_UNAVAILABLE` it retries through; `translateStorageError`
already mapped the SDK's `TimeoutError` onto that code, but the branch was
unreachable because nothing ever timed out. The bound is on socket
idleness rather than total elapsed time so a module streaming a large
object is never cut off for being big, only for going quiet.

The suite provisioned a fresh bucket per test. Every S3 bucket becomes its
own SeaweedFS collection claiming a block of volumes from a budget the
server derives from free disk, and `mini` auto-configures a
production-shaped ~1GB volume — so the ceiling is small and varies with
the runner. Twelve buckets per run exhausted it, and an exhausted budget
is not a failed write: the master retries allocation internally and never
replies, which is the hang CI saw. Tests now share one bucket and isolate
by key namespace, which exercises prefix scoping at the same time;
bucket identity is provisioned only where it is the thing under test.
Pinning a small volume size makes the budget depend far less sharply on
the host's disk.

Verified by starving a server on purpose: the same exhaustion that used to
hang until the runner killed it now fails in ~15s with a structured error
and no runner timeout, so the failure is bounded rather than hidden. Ten
consecutive runs pass in ~2.1s each, using 28 volumes per run against a
5212 budget where the old suite used 84 against 161.

Co-Authored-By: Claude Opus 5 <[email protected]>

core/src/storage/s3-provider.ts

@@ -12,8 +12,8 @@ import {
 import {
   type ModuleStorageBody,
   ModuleStorageError,
-  type ModuleStorageListOptions,
   type ModuleStorageListing,
+  type ModuleStorageListOptions,
   type ModuleStorageObject,
   type ModuleStorageObjectMetadata,
   type ModuleStoragePutOptions,
@@ -135,12 +135,66 @@ function toRequestBody(body: ModuleStorageBody): Uint8Array | string | Readable
   return body;
 }
 
-export function createS3StorageProvider(config: StorageProviderConfig): StorageProvider {
+/**
+ * How long a single exchange with the provider may take before it is abandoned.
+ *
+ * These exist because the SDK's own defaults are *no deadline at all*: `connectionTimeout` and
+ * `socketTimeout` both default to `0`, so a provider that accepts a connection and then goes silent
+ * holds the calling module's request handler open forever. That is not a hypothetical failure —
+ * an S3-compatible server which has exhausted its storage budget can answer `PutObject` by retrying
+ * allocation internally and never replying at all, which is exactly this shape.
+ *
+ * ADR 0030 promises that a provider failure reaches a module as the transient
+ * `STORAGE_PROVIDER_UNAVAILABLE` it is expected to retry through. That promise is unkeepable
+ * without a deadline: `translateStorageError` already maps the SDK's `TimeoutError` onto that code,
+ * but the branch was unreachable while nothing ever timed out.
+ *
+ * The bound is deliberately on socket *idleness* rather than on total elapsed time. A module
+ * streaming a large object must never be cut off for being big, only for going quiet — an idle
+ * timer resets whenever bytes move, whereas a total-elapsed bound would truncate a healthy large
+ * transfer. A healthy exchange settles in milliseconds against host-local storage and in well under
+ * a second against a remote one, so five seconds without a single byte moving already means
+ * something is wrong. Three attempts keep the SDK's resilience to a genuine blip while capping a
+ * wholly unresponsive provider at roughly fifteen seconds — long enough never to fire spuriously,
+ * short enough that a module's request handler still answers its caller.
+ */
+export interface S3TransportBounds {
+  /** Milliseconds allowed to establish the connection before the attempt is abandoned. */
+  readonly connectionTimeoutMs: number;
+  /** Milliseconds the socket may sit idle mid-exchange; resets whenever bytes move. */
+  readonly socketIdleTimeoutMs: number;
+  /** Total attempts, including the first, for a failure the SDK classifies as retryable. */
+  readonly maxAttempts: number;
+}
+
+const DEFAULT_TRANSPORT_BOUNDS: S3TransportBounds = Object.freeze({
+  connectionTimeoutMs: 3_000,
+  maxAttempts: 3,
+  socketIdleTimeoutMs: 5_000,
+});
+
+export function createS3StorageProvider(
+  config: StorageProviderConfig,
+  /** Tightened only by the live provider suite, so it can prove the deadline actually fires without
+   * spending the production budget waiting for it. Hosts always take the defaults. */
+  bounds: Partial<S3TransportBounds> = {},
+): StorageProvider {
+  const { connectionTimeoutMs, maxAttempts, socketIdleTimeoutMs } = {
+    ...DEFAULT_TRANSPORT_BOUNDS,
+    ...bounds,
+  };
   const client = new S3Client({
     credentials: { accessKeyId: config.accessKey, secretAccessKey: config.secretKey },
     endpoint: config.endpoint,
     forcePathStyle: config.forcePathStyle,
+    maxAttempts,
     region: config.region,
+    // Not `requestTimeout`: that bounds total elapsed request-and-response time, which would cap a
+    // legitimately large upload or download by size rather than by health.
+    requestHandler: {
+      connectionTimeout: connectionTimeoutMs,
+      socketTimeout: socketIdleTimeoutMs,
+    },
   });
 
   return Object.freeze({