feat: add host-provided module object storage
CI / verify (pull_request) Successful in 36s
CI / infrastructure-smoke (pull_request) Successful in 13s
CI / migration-smoke (pull_request) Failing after 2m20s

`manifest.storage.buckets` has existed since manifest version 1 and meant
nothing — discovery validated it and no code read it. This turns it into a
real capability, which is what stands between Announcements and the module
system: that product currently fetches a hardcoded LAN S3 endpoint, parses
ListObjectsV2 XML by hand, and hands the browser direct object URLs.

- A module declares a logical bucket and receives a scoped facade at
  `context.storage` — list, head, get, put, delete. The contract names no
  provider: no endpoint, region, credential, or physical bucket is reachable
  from it, which is what lets the same module run under a future standalone
  host over its own or a shared S3-compatible service.
- Core's first provider speaks S3 through the AWS SDK. SeaweedFS is the
  deployment target (ADR 0004), never an API name. Probing the pinned image
  found `CreateBucket` reporting `BucketAlreadyExists` rather than AWS's
  `BucketAlreadyOwnedByYou`; treating only the AWS spelling as success would
  have failed every second boot. A live test pins it.
- Logical → physical mapping is deterministic and collision-safe: an
  installation prefix, the module and bucket ids joined by `--` (which cannot
  occur inside either identifier, so `a-b`+`c` and `a`+`b-c` cannot collide),
  and a digest over the full untruncated identity. Two modules declaring
  `assets` always get separate namespaces; neither can see the physical name.
- Declaring a bucket is now a requirement: with no provider configured the
  module is `blocked · storage_unavailable` through ADR 0028's existing
  resolver, propagating to required dependents. An installation with no object
  storage stays a complete deployment.
- Binding availability is not provider health. Eligibility asks whether
  storage is configured, never whether it answered; bucket creation is lazy on
  first use so a storage outage cannot stop the platform booting; a running
  module whose store blinks stays running and retries. Nothing deletes a
  bucket.
- Credentials are deployment-supplied, never persisted, never routed through
  module configuration, and never visible to modules, module UI, diagnostics,
  or API responses. Provider errors translate to a closed code set carrying no
  endpoint, bucket, request id, or XML.
- Retires ADR 0029's transitional `bucket` configuration key. No manifest
  version change. The template declares no bucket, so it still runs without S3.

See ADR 0030.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
ekklesia
2026-08-23 08:45:53 -05:00
co-authored by Claude Opus 5
parent 527239f6de
commit b9ca0d560d
55 changed files with 4420 additions and 74 deletions
@@ -0,0 +1,195 @@
import {
type ModuleStorage,
type ModuleStorageBody,
ModuleStorageError,
type ModuleStorageListOptions,
type ModuleStorageListing,
type ModuleStorageObject,
type ModuleStorageObjectMetadata,
type ModuleStoragePutOptions,
} from "@ekklesia/contracts";
import type { DiscoveredModule, ModuleStorageLookup } from "@ekklesia/core/modules";
import { createModuleStorageLookup, type StorageProvider } from "@ekklesia/core/storage";
/** Builds and drains Web streams without depending on DOM-only helper types, which this package's
* TypeScript configuration deliberately does not load. */
function streamOf(bytes: Uint8Array): ReadableStream<Uint8Array> {
return new ReadableStream<Uint8Array>({
start(controller) {
controller.enqueue(bytes);
controller.close();
},
});
}
async function collect(stream: ReadableStream<Uint8Array>): Promise<Uint8Array> {
const reader = stream.getReader();
const chunks: Uint8Array[] = [];
let total = 0;
for (;;) {
const { done, value } = await reader.read();
if (done) {
break;
}
chunks.push(value);
total += value.byteLength;
}
const merged = new Uint8Array(total);
let offset = 0;
for (const chunk of chunks) {
merged.set(chunk, offset);
offset += chunk.byteLength;
}
return merged;
}
interface StoredObject {
readonly bytes: Uint8Array;
readonly contentType?: string;
readonly lastModified: Date;
}
/**
* An in-memory `StorageProvider` for tests that exercise **wiring, scoping, and routing** rather
* than the S3 protocol.
*
* It implements the same interface the real S3 provider does, so everything above the provider seam
* — binding resolution, the scoped facade, lifecycle eligibility, route wiring — is exercised
* unchanged. What it deliberately does *not* reproduce is the protocol itself: real SeaweedFS
* semantics (non-idempotent `CreateBucket`, `NoSuchKey` vs `NoSuchBucket`, ETag quoting, streaming
* bodies) are proven against a live server in `core/tests/module-storage.integration.ts`, so this
* double can never quietly stand in for a provider guarantee nobody checked.
*
* It keys everything by **physical** bucket, exactly as the real provider does, which is what lets a
* test assert that two modules declaring the same logical name really did land in different
* namespaces.
*/
export function createInMemoryStorageProvider(): StorageProvider & {
readonly buckets: ReadonlyMap<string, Map<string, StoredObject>>;
ensureBucketCalls: number;
} {
const buckets = new Map<string, Map<string, StoredObject>>();
const state = { ensureBucketCalls: 0 };
const objectsIn = (physicalBucket: string): Map<string, StoredObject> => {
const bucket = buckets.get(physicalBucket);
if (!bucket) {
throw new ModuleStorageError("STORAGE_BUCKET_UNAVAILABLE");
}
return bucket;
};
const metadataFor = (key: string, stored: StoredObject): ModuleStorageObjectMetadata =>
Object.freeze({
contentLength: stored.bytes.byteLength,
key,
lastModified: stored.lastModified,
...(stored.contentType === undefined ? {} : { contentType: stored.contentType }),
});
const provider: StorageProvider = {
async close(): Promise<void> {
// Nothing to release.
},
async delete(physicalBucket: string, key: string): Promise<void> {
objectsIn(physicalBucket).delete(key);
},
async ensureBucket(physicalBucket: string): Promise<void> {
state.ensureBucketCalls += 1;
if (!buckets.has(physicalBucket)) {
buckets.set(physicalBucket, new Map());
}
},
async get(physicalBucket: string, key: string): Promise<ModuleStorageObject> {
const stored = objectsIn(physicalBucket).get(key);
if (!stored) {
throw new ModuleStorageError("STORAGE_OBJECT_NOT_FOUND", key);
}
return Object.freeze({
...metadataFor(key, stored),
body: streamOf(stored.bytes),
});
},
async head(physicalBucket: string, key: string): Promise<ModuleStorageObjectMetadata> {
const stored = objectsIn(physicalBucket).get(key);
if (!stored) {
throw new ModuleStorageError("STORAGE_OBJECT_NOT_FOUND", key);
}
return metadataFor(key, stored);
},
async list(
physicalBucket: string,
options: ModuleStorageListOptions,
): Promise<ModuleStorageListing> {
const entries = [...objectsIn(physicalBucket).entries()]
.filter(([key]) => options.prefix === undefined || key.startsWith(options.prefix))
.sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0));
return Object.freeze({
objects: Object.freeze(entries.map(([key, stored]) => metadataFor(key, stored))),
});
},
async probe(): Promise<void> {
// Always reachable.
},
async put(
physicalBucket: string,
key: string,
body: ModuleStorageBody,
options: ModuleStoragePutOptions,
): Promise<ModuleStorageObjectMetadata> {
const bytes =
typeof body === "string"
? new TextEncoder().encode(body)
: body instanceof Uint8Array
? body
: await collect(body as ReadableStream<Uint8Array>);
const stored: StoredObject = {
bytes,
lastModified: new Date(0),
...(options.contentType === undefined ? {} : { contentType: options.contentType }),
};
objectsIn(physicalBucket).set(key, stored);
return metadataFor(key, stored);
},
};
return Object.assign(provider, {
buckets: buckets as ReadonlyMap<string, Map<string, StoredObject>>,
get ensureBucketCalls(): number {
return state.ensureBucketCalls;
},
});
}
export interface ModuleStorageOption {
readonly snapshots: ModuleStorageLookup;
readonly providerConfigured: boolean;
}
/**
* Builds the `createApiApp` storage option the same way `startApi` does — resolve every module's
* declared buckets against one provider and one prefix — rather than hand-writing a map, so a test
* can never assert against a shape the real composition would not produce.
*/
export function moduleStorageOptionFor(
discoveredModules: readonly DiscoveredModule[],
provider?: StorageProvider,
): ModuleStorageOption {
return {
providerConfigured: provider !== undefined,
snapshots: createModuleStorageLookup(
provider,
"ekklesia",
discoveredModules.map((module) => ({
logicalBucketIds: module.manifest.storage.buckets.map((bucket) => bucket.id),
moduleId: module.manifest.id,
})),
),
};
}
/** The storage option for a `createApiApp` caller that exercises no storage at all — the explicit
* "this installation has no object storage" statement the required option asks for. */
export function emptyModuleStorage(): ModuleStorageOption {
return { providerConfigured: false, snapshots: new Map<string, ModuleStorage>() };
}