feat: add host-provided module object storage
CI / verify (pull_request) Successful in 36s
CI / infrastructure-smoke (pull_request) Successful in 13s
CI / migration-smoke (pull_request) Failing after 2m20s

`manifest.storage.buckets` has existed since manifest version 1 and meant
nothing — discovery validated it and no code read it. This turns it into a
real capability, which is what stands between Announcements and the module
system: that product currently fetches a hardcoded LAN S3 endpoint, parses
ListObjectsV2 XML by hand, and hands the browser direct object URLs.

- A module declares a logical bucket and receives a scoped facade at
  `context.storage` — list, head, get, put, delete. The contract names no
  provider: no endpoint, region, credential, or physical bucket is reachable
  from it, which is what lets the same module run under a future standalone
  host over its own or a shared S3-compatible service.
- Core's first provider speaks S3 through the AWS SDK. SeaweedFS is the
  deployment target (ADR 0004), never an API name. Probing the pinned image
  found `CreateBucket` reporting `BucketAlreadyExists` rather than AWS's
  `BucketAlreadyOwnedByYou`; treating only the AWS spelling as success would
  have failed every second boot. A live test pins it.
- Logical → physical mapping is deterministic and collision-safe: an
  installation prefix, the module and bucket ids joined by `--` (which cannot
  occur inside either identifier, so `a-b`+`c` and `a`+`b-c` cannot collide),
  and a digest over the full untruncated identity. Two modules declaring
  `assets` always get separate namespaces; neither can see the physical name.
- Declaring a bucket is now a requirement: with no provider configured the
  module is `blocked · storage_unavailable` through ADR 0028's existing
  resolver, propagating to required dependents. An installation with no object
  storage stays a complete deployment.
- Binding availability is not provider health. Eligibility asks whether
  storage is configured, never whether it answered; bucket creation is lazy on
  first use so a storage outage cannot stop the platform booting; a running
  module whose store blinks stays running and retries. Nothing deletes a
  bucket.
- Credentials are deployment-supplied, never persisted, never routed through
  module configuration, and never visible to modules, module UI, diagnostics,
  or API responses. Provider errors translate to a closed code set carrying no
  endpoint, bucket, request id, or XML.
- Retires ADR 0029's transitional `bucket` configuration key. No manifest
  version change. The template declares no bucket, so it still runs without S3.

See ADR 0030.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
ekklesia
2026-08-23 08:45:53 -05:00
co-authored by Claude Opus 5
parent 527239f6de
commit b9ca0d560d
55 changed files with 4420 additions and 74 deletions
+223
View File
@@ -0,0 +1,223 @@
/**
* The one module-facing object-storage contract.
*
* **Nothing in this file names a provider.** There is no endpoint, no region, no credential, no
* bucket policy, no signing, no `S3`, and no `SeaweedFS` — a module asks for the *logical* bucket it
* declared in its manifest and reads and writes objects in it. Which physical service satisfies that
* requirement is entirely the host's business (ADR 0030).
*
* That is what makes the same module code runnable under Ekklesia and under a future standalone
* host: both construct this interface, one over a shared S3-compatible service and the other over
* whatever it has, and the module cannot tell the difference.
*/
/** Every failure a module can observe from storage, as a closed set of codes rather than a provider
* library's exception hierarchy. */
export const MODULE_STORAGE_ERROR_CODES = [
/** The module asked for a logical bucket its own manifest does not declare. A programming error,
* not a runtime condition — the platform cannot invent a binding a module never asked for. */
"STORAGE_BUCKET_UNDECLARED",
/** The key does not exist in this bucket. */
"STORAGE_OBJECT_NOT_FOUND",
/** The bucket this binding names is not usable — absent, or not creatable with the host's
* credentials. Distinct from a provider-wide outage. */
"STORAGE_BUCKET_UNAVAILABLE",
/** The storage service could not be reached at all. Transient by nature: a module may retry, and
* this never changes the module's lifecycle state (ADR 0030). */
"STORAGE_PROVIDER_UNAVAILABLE",
/** The host's credentials do not permit this operation. */
"STORAGE_OPERATION_DENIED",
/** The object key is not one this contract accepts. See `isValidModuleStorageKey`. */
"STORAGE_KEY_INVALID",
/** A `ReadableStream` body was supplied without `contentLength`. Refused rather than buffered:
* buffering would defeat the only reason to pass a stream, and discovering that through memory
* pressure in production is worse than discovering it at the call site. */
"STORAGE_CONTENT_LENGTH_REQUIRED",
/** Anything else the provider reported, reduced to a code. */
"STORAGE_OPERATION_FAILED",
] as const;
export type ModuleStorageErrorCode = (typeof MODULE_STORAGE_ERROR_CODES)[number];
const storageErrorMessages: Readonly<Record<ModuleStorageErrorCode, string>> = Object.freeze({
STORAGE_BUCKET_UNAVAILABLE: "The requested storage bucket is not available.",
STORAGE_BUCKET_UNDECLARED: "This module did not declare the requested storage bucket.",
STORAGE_CONTENT_LENGTH_REQUIRED: "A streamed storage body requires an explicit content length.",
STORAGE_KEY_INVALID: "The storage object key is invalid.",
STORAGE_OBJECT_NOT_FOUND: "The storage object was not found.",
STORAGE_OPERATION_DENIED: "The storage operation was denied.",
STORAGE_OPERATION_FAILED: "The storage operation failed.",
STORAGE_PROVIDER_UNAVAILABLE: "The storage service could not be reached.",
});
/**
* A sanitized storage failure.
*
* Carries a code and a fixed message, and deliberately **nothing else** — no endpoint, no region, no
* access key, no bucket name (physical or logical), no request id, no provider XML body, and no
* caught cause. A module may catch this, log it, and return it to a browser; every one of those
* paths would otherwise be a way for host infrastructure detail to escape. It follows the same
* discipline as `ModuleLifecycleStoreError` and `ModuleConfigurationStoreError`.
*
* `key` echoes a value the **module itself supplied** — an object key for the object errors, or the
* logical bucket id for `STORAGE_BUCKET_UNDECLARED`. It is safe precisely because it never came from
* the host: it is the module's own input handed back so a caught error is actionable.
*/
export class ModuleStorageError extends Error {
readonly code: ModuleStorageErrorCode;
readonly key?: string;
constructor(code: ModuleStorageErrorCode, key?: string) {
super(storageErrorMessages[code]);
this.name = "ModuleStorageError";
this.code = code;
if (key !== undefined) {
this.key = key;
}
this.stack = `${this.name}: ${this.message}`;
Object.freeze(this);
}
}
/** The S3 object-key ceiling, which every S3-compatible provider shares. Measured in UTF-8 bytes,
* not characters, because that is what the wire format bounds. */
export const MODULE_STORAGE_KEY_MAX_BYTES = 1024;
/**
* Whether a value is an object key this contract accepts.
*
* An object key is **not a filesystem path** — `..` has no traversal meaning to an object store, and
* the physical bucket comes from the host's binding rather than from anything in the key, so a key
* structurally cannot escape its bucket. What is rejected here is what would be *ambiguous* or
* unrepresentable rather than what would be dangerous: an empty key, one that exceeds the wire
* limit, control characters (which break header and XML encodings), lone surrogates (unencodable as
* UTF-8), a backslash (whose meaning differs between providers), and a leading `/` (which some
* providers normalize away and others do not, so one key would name two objects).
*
* `/` inside a key is allowed and ordinary: object stores use it as a display and prefix convention,
* and modules genuinely need `slides/welcome.png`.
*/
export function isValidModuleStorageKey(value: unknown): value is string {
if (typeof value !== "string" || value.length === 0) {
return false;
}
if (value.startsWith("/") || value.includes("\\")) {
return false;
}
for (const character of value) {
const codePoint = character.codePointAt(0);
if (codePoint === undefined) {
return false;
}
// C0 controls, DEL, and C1 controls.
if (codePoint <= 0x1f || (codePoint >= 0x7f && codePoint <= 0x9f)) {
return false;
}
// A lone surrogate survives in a JavaScript string but cannot be encoded as UTF-8.
if (codePoint >= 0xd800 && codePoint <= 0xdfff) {
return false;
}
}
return new TextEncoder().encode(value).length <= MODULE_STORAGE_KEY_MAX_BYTES;
}
/** What a module may hand to `put`. A `ReadableStream` is accepted so a large object never has to be
* buffered in memory, but it must be accompanied by `contentLength` — every S3-compatible provider
* needs the length up front for a single-part upload, and guessing it would mean buffering the very
* thing the stream exists to avoid. */
export type ModuleStorageBody = Uint8Array | string | Blob | ReadableStream<Uint8Array>;
/** What the store knows about an object besides its bytes. Deliberately the four fields every
* S3-compatible provider actually returns, rather than a mirror of S3's full metadata surface. */
export interface ModuleStorageObjectMetadata {
readonly key: string;
/** Preserved exactly as it was written. A module serving images through its own route needs this
* to set a correct `Content-Type` without sniffing bytes. */
readonly contentType?: string;
readonly contentLength?: number;
readonly etag?: string;
readonly lastModified?: Date;
}
/** An object's metadata plus its bytes as a Web stream, so a module can pass it straight into a
* `Response` without ever holding the whole object in memory. */
export interface ModuleStorageObject extends ModuleStorageObjectMetadata {
readonly body: ReadableStream<Uint8Array>;
}
export interface ModuleStoragePutOptions {
readonly contentType?: string;
/** Required when `body` is a `ReadableStream`, ignored otherwise. */
readonly contentLength?: number;
}
export interface ModuleStorageListOptions {
readonly prefix?: string;
/** Provider-bounded; a provider may return fewer. */
readonly limit?: number;
/** The `cursor` from a previous listing. Opaque — a module passes it back and never parses it. */
readonly cursor?: string;
}
export interface ModuleStorageListing {
readonly objects: readonly ModuleStorageObjectMetadata[];
/** Present when more objects remain. Absent means the listing is complete — which is why this
* exists at all: every S3-compatible provider caps a single listing, and a contract without a
* cursor would silently truncate a bucket with more objects than that cap. */
readonly cursor?: string;
}
/**
* One logical bucket a module declared, already bound to wherever the host keeps it.
*
* `id` is the **logical** id from the module's own manifest. The physical bucket, its naming scheme,
* and the service holding it are not reachable from here by design — there is no accessor for them,
* so a module cannot come to depend on the host's implementation and cannot be ported away from it
* later.
*/
export interface ModuleStorageBucket {
readonly id: string;
list(options?: ModuleStorageListOptions): Promise<ModuleStorageListing>;
head(key: string): Promise<ModuleStorageObjectMetadata>;
get(key: string): Promise<ModuleStorageObject>;
put(
key: string,
body: ModuleStorageBody,
options?: ModuleStoragePutOptions,
): Promise<ModuleStorageObjectMetadata>;
delete(key: string): Promise<void>;
}
/**
* A module's own scoped storage.
*
* Scoped by construction: `bucket()` resolves only against the logical buckets **this** module
* declared, so knowing another module's bucket name buys nothing — there is no name that maps
* outside this module's own bindings, and no accessor that takes a physical bucket, an endpoint, or
* a credential.
*
* There is deliberately no `createBucket`, `deleteBucket`, or bucket-listing operation. Which
* buckets exist is decided by the manifest an operator installed, not by running module code, and
* nothing a module does can destroy a bucket (ADR 0030).
*/
export interface ModuleStorage {
/** The logical bucket ids this module declared, sorted. */
buckets(): readonly string[];
/** Throws `ModuleStorageError("STORAGE_BUCKET_UNDECLARED")` for anything this module did not
* declare — including another module's bucket. It throws rather than returning `undefined` because
* asking for an undeclared bucket is a manifest/code mismatch, and a silent no-op would hide it
* until data went missing. */
bucket(logicalBucketId: string): ModuleStorageBucket;
}
/**
* The storage a module with no declared buckets receives. `context.storage` is therefore always
* present and never `undefined`, so module code never branches on whether the host supplied one —
* the same reasoning `EMPTY_MODULE_CONFIGURATION` follows.
*/
export const EMPTY_MODULE_STORAGE: ModuleStorage = Object.freeze({
bucket: (logicalBucketId: string): ModuleStorageBucket => {
throw new ModuleStorageError("STORAGE_BUCKET_UNDECLARED", logicalBucketId);
},
buckets: (): readonly string[] => Object.freeze([]),
});