feat: add host-provided module object storage
CI / verify (pull_request) Successful in 36s
CI / infrastructure-smoke (pull_request) Successful in 13s
CI / migration-smoke (pull_request) Failing after 2m20s

`manifest.storage.buckets` has existed since manifest version 1 and meant
nothing — discovery validated it and no code read it. This turns it into a
real capability, which is what stands between Announcements and the module
system: that product currently fetches a hardcoded LAN S3 endpoint, parses
ListObjectsV2 XML by hand, and hands the browser direct object URLs.

- A module declares a logical bucket and receives a scoped facade at
  `context.storage` — list, head, get, put, delete. The contract names no
  provider: no endpoint, region, credential, or physical bucket is reachable
  from it, which is what lets the same module run under a future standalone
  host over its own or a shared S3-compatible service.
- Core's first provider speaks S3 through the AWS SDK. SeaweedFS is the
  deployment target (ADR 0004), never an API name. Probing the pinned image
  found `CreateBucket` reporting `BucketAlreadyExists` rather than AWS's
  `BucketAlreadyOwnedByYou`; treating only the AWS spelling as success would
  have failed every second boot. A live test pins it.
- Logical → physical mapping is deterministic and collision-safe: an
  installation prefix, the module and bucket ids joined by `--` (which cannot
  occur inside either identifier, so `a-b`+`c` and `a`+`b-c` cannot collide),
  and a digest over the full untruncated identity. Two modules declaring
  `assets` always get separate namespaces; neither can see the physical name.
- Declaring a bucket is now a requirement: with no provider configured the
  module is `blocked · storage_unavailable` through ADR 0028's existing
  resolver, propagating to required dependents. An installation with no object
  storage stays a complete deployment.
- Binding availability is not provider health. Eligibility asks whether
  storage is configured, never whether it answered; bucket creation is lazy on
  first use so a storage outage cannot stop the platform booting; a running
  module whose store blinks stays running and retries. Nothing deletes a
  bucket.
- Credentials are deployment-supplied, never persisted, never routed through
  module configuration, and never visible to modules, module UI, diagnostics,
  or API responses. Provider errors translate to a closed code set carrying no
  endpoint, bucket, request id, or XML.
- Retires ADR 0029's transitional `bucket` configuration key. No manifest
  version change. The template declares no bucket, so it still runs without S3.

See ADR 0030.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
ekklesia
2026-08-23 08:45:53 -05:00
co-authored by Claude Opus 5
parent 527239f6de
commit b9ca0d560d
55 changed files with 4420 additions and 74 deletions
+185
View File
@@ -0,0 +1,185 @@
import {
EMPTY_MODULE_STORAGE,
isValidModuleStorageKey,
type ModuleStorage,
type ModuleStorageBody,
type ModuleStorageBucket,
ModuleStorageError,
type ModuleStorageListOptions,
type ModuleStorageListing,
type ModuleStorageObject,
type ModuleStorageObjectMetadata,
type ModuleStoragePutOptions,
} from "@ekklesia/contracts";
import { type ModuleStorageBinding, resolveModuleStorageBindings } from "./binding.js";
import type { StorageProvider } from "./provider.js";
/**
* Builds the scoped storage facade one module receives.
*
* **This is where isolation is enforced, and it is enforced by construction.** The facade closes over
* exactly the bindings this module's own manifest declared; there is no lookup table spanning
* modules, no name that resolves outside those bindings, and no accessor taking a physical bucket.
* Module A cannot reach module B's storage by knowing its bucket name, its module id, or its
* physical bucket name, because none of those is an input to anything here (ADR 0030).
*
* Two modules that both declare `assets` receive two facades over two different physical buckets —
* see `physicalBucketName` for why that separation is collision-safe rather than merely likely.
*/
/** Ensures a binding's physical bucket exists, at most once per process per binding.
*
* Lazy rather than eager at startup, deliberately: a storage outage must never stop the platform
* from booting. The promise is memoized so concurrent first uses share one round trip, and a
* *failed* attempt is discarded so a transient outage does not poison the binding for the life of
* the process — the next call retries, which is exactly the resilience a module like a slideshow
* depends on (ADR 0030). */
function createBucketEnsurer(
provider: StorageProvider,
physicalBucket: string,
): () => Promise<void> {
let pending: Promise<void> | undefined;
return () => {
pending ??= provider.ensureBucket(physicalBucket).catch((error: unknown) => {
pending = undefined;
throw error;
});
return pending;
};
}
/** Echoes the offending key only when it is a short string — a module's own input is safe to hand
* back, but a non-string or an enormous value is not something to put into an error message. */
function assertValidKey(key: unknown): asserts key is string {
if (isValidModuleStorageKey(key)) {
return;
}
const echo = typeof key === "string" && key.length <= 256 ? key : undefined;
throw new ModuleStorageError("STORAGE_KEY_INVALID", echo);
}
function createBucketHandle(
provider: StorageProvider,
binding: ModuleStorageBinding,
): ModuleStorageBucket {
const ensure = createBucketEnsurer(provider, binding.physicalBucket);
return Object.freeze({
// The **logical** id, never the physical bucket. A module that logs this leaks nothing about
// where the host keeps its data.
id: binding.logicalBucketId,
async delete(key: string): Promise<void> {
assertValidKey(key);
await ensure();
await provider.delete(binding.physicalBucket, key);
},
async get(key: string): Promise<ModuleStorageObject> {
assertValidKey(key);
await ensure();
return provider.get(binding.physicalBucket, key);
},
async head(key: string): Promise<ModuleStorageObjectMetadata> {
assertValidKey(key);
await ensure();
return provider.head(binding.physicalBucket, key);
},
async list(options: ModuleStorageListOptions = {}): Promise<ModuleStorageListing> {
await ensure();
return provider.list(binding.physicalBucket, options);
},
async put(
key: string,
body: ModuleStorageBody,
options: ModuleStoragePutOptions = {},
): Promise<ModuleStorageObjectMetadata> {
assertValidKey(key);
if (body instanceof ReadableStream && options.contentLength === undefined) {
throw new ModuleStorageError("STORAGE_CONTENT_LENGTH_REQUIRED", key);
}
await ensure();
return provider.put(binding.physicalBucket, key, body, options);
},
});
}
export interface CreateModuleStorageOptions {
readonly provider: StorageProvider;
readonly bucketPrefix: string;
readonly moduleId: string;
/** The logical bucket ids from this module's own manifest, in declaration order. */
readonly logicalBucketIds: readonly string[];
}
/**
* The scoped `ModuleStorage` for one module.
*
* A module declaring no buckets gets `EMPTY_MODULE_STORAGE` — present, frozen, and refusing every
* bucket — so `context.storage` is never `undefined` and module code never branches on whether the
* host supplied storage.
*/
export function createModuleStorage(options: CreateModuleStorageOptions): ModuleStorage {
if (options.logicalBucketIds.length === 0) {
return EMPTY_MODULE_STORAGE;
}
const bindings = resolveModuleStorageBindings(
options.bucketPrefix,
options.moduleId,
options.logicalBucketIds,
);
// A `Map` rather than a plain object: every lookup key here comes from module code, and indexing a
// plain object with such a key is the shape of a prototype-pollution bug. With a `Map` there is no
// prototype to reach, so `constructor` or `__proto__` are ordinary absent keys.
const handles = new Map<string, ModuleStorageBucket>(
bindings.map((binding) => [
binding.logicalBucketId,
createBucketHandle(options.provider, binding),
]),
);
const declared = Object.freeze([...handles.keys()].sort());
return Object.freeze({
bucket: (logicalBucketId: string): ModuleStorageBucket => {
const handle = typeof logicalBucketId === "string" ? handles.get(logicalBucketId) : undefined;
if (!handle) {
// Including another module's bucket id, which is simply not a key in this map.
throw new ModuleStorageError("STORAGE_BUCKET_UNDECLARED", logicalBucketId);
}
return handle;
},
buckets: (): readonly string[] => declared,
});
}
/**
* Every module's storage, resolved once at startup.
*
* Returns an empty map when the host has no storage provider. That is not an error: an installation
* with no object storage is a complete, valid deployment, and only modules that *declared* a bucket
* are affected — those are reported blocked by the lifecycle resolver before any of this is reached
* (ADR 0030).
*/
export function createModuleStorageLookup(
provider: StorageProvider | undefined,
bucketPrefix: string,
modules: readonly { readonly moduleId: string; readonly logicalBucketIds: readonly string[] }[],
): ReadonlyMap<string, ModuleStorage> {
if (!provider) {
return new Map();
}
return new Map(
modules.map((module) => [
module.moduleId,
createModuleStorage({
bucketPrefix,
logicalBucketIds: module.logicalBucketIds,
moduleId: module.moduleId,
provider,
}),
]),
);
}