Ekklesia/docs/architecture/decisions/0030-host-provided-module-object-storage.md

# 0030: Modules declare logical buckets; hosts provide object storage

- Status: Accepted
- Date: 2026-08-23
- Extends: [0004](0004-replace-unmaintained-minio.md),
  [0016](0016-module-database-access.md),
  [0021](0021-modules-as-hostable-products.md),
  [0028](0028-module-operational-lifecycle.md),
  [0029](0029-module-configuration-and-secrets.md)

## Context

`manifest.storage.buckets` has existed since manifest version 1 and meant nothing. A module could
declare `{ "id": "records" }`, discovery would validate the identifier and reject duplicates, and
then nothing anywhere read the field. Doctor said so plainly: *"Core storage integration has not
been implemented yet."*

Meanwhile the product that will become Ekklesia's first storage consumer does this
(`The_River/Announcements`, commit `9aab486`):

```js
const response = await fetch("http://10.40.0.2:9000/announcements?list-type=2");
const data = new XMLParser().parse(await response.text());
slides = (data.ListBucketResult?.Contents ?? [])
  .map((file) => file.Key)
  .filter((name) => /\.(png|jpg|jpeg|webp)$/i.test(name))
  .map((name) => `http://10.40.0.2:9000/announcements/${encode(name)}`);
```

A hardcoded LAN address, a hand-rolled `ListObjectsV2` XML parse, an anonymously readable bucket,
and direct S3 URLs handed to the browser. Every one of those has to disappear, and none of them can
be replaced with "let the module configure its own S3 endpoint" — that would make every module a
separate S3 deployment decision and reintroduce the coupling in a tidier costume.

Two facts from the repository shaped what was possible.

**The provider question was already decided.** ADR 0004 replaced the archived MinIO baseline with
SeaweedFS 4.28 in the development Compose stack, deliberately using `S3_*` configuration names
rather than product-specific ones, and recorded that *"provider-specific Core APIs were rejected
because they would couple every module to this choice."* The service is already pinned, already
provisioned, and already has credentials in `.env.example`.

**ADR 0029 left a loose end and named it.** It accepted a `bucket` *configuration key* as a
transitional answer while observing that `manifest.storage.buckets` already existed for exactly this
purpose, and asked for the overlap to be resolved rather than left to drift. This ADR resolves it.

## Decision

### The logical/physical split, and which side owns what

```text
a module owns          I require logical bucket "announcement-assets"
a host owns            endpoint · region · credentials · path style · TLS · physical bucket naming
```

That line is the whole design. A module names a bucket; the host decides what a bucket *is*. The
module-facing contract (`packages/contracts/src/module-storage.ts`) contains no endpoint, no region,
no credential, no signing, no `S3`, and no `SeaweedFS` — not as a field, not as a type, not as a
string. A future standalone host constructs the same interface over its own SeaweedFS, a shared one,
MinIO, or AWS S3, and the module cannot tell.

### `manifest.storage.buckets` becomes a real requirement, and the manifest does not change

Declaring a bucket now means "I require this". No new field, no new version — **manifest version 5
is unchanged**. The existing declaration already expressed the requirement precisely; it simply had
no consumer.

Every declared bucket is required. A separate `required: false` would have meant a manifest version
bump (`moduleBucketDeclarationSchema` is `.strict()`) for a case no module has, and "declare it if
you need it" is a simpler rule than one with an escape hatch.

**This retires ADR 0029's transitional `bucket` configuration key.** A logical bucket is a static
declaration the platform can validate before any module code runs, not a value an operator types.
Module configuration is for things an operator decides; storage requirements are for things a module
knows about itself.

The template consequently declares **no** bucket any more. A starting point that could not run
without S3 configured would be a poor starting point, so storage is something a module author opts
into deliberately.

### Physical naming: deterministic, collision-safe, and never module-visible

```text
ekklesia-announcements--announcement-assets-3f0c1d9a6b21
└─prefix─┘└──── module--bucket (truncated) ────┘└─digest─┘
```

Three properties, each load-bearing:

**`--` as the separator, because it cannot occur inside either identifier.** Both `moduleIdSchema`
and `logicalBucketIdSchema` require `[a-z0-9]` after every hyphen, so a doubled hyphen is
unrepresentable in each half. A single hyphen would have been ambiguous in precisely the way that
matters: module `a-b` + bucket `c` and module `a` + bucket `b-c` would both render `a-b-c`, silently
handing two unrelated modules one namespace.

**A digest over the full, untruncated identity.** S3 caps bucket names at 63 characters while
Ekklesia identifiers allow 64 + 48, so the readable middle is truncated. The digest is taken *before*
truncation, so two modules whose names truncate identically still get different buckets.

**A configurable prefix** (`S3_BUCKET_PREFIX`, default `ekklesia`), so several Ekklesia-compatible
deployments can share one S3-compatible service — which is exactly the standalone story: one
SeaweedFS holding Announcements, Video, Sermons, and Signage, isolated by binding rather than by
running four storage servers.

The result is never exposed to module code. `ModuleStorageBucket.id` is the **logical** id, and there
is no accessor for anything else.

### The module-facing API, kept small

```ts
const assets = context.storage.bucket("announcement-assets");
await assets.list({ prefix: "slides/" });
await assets.get("slides/welcome.png");   // → { body: ReadableStream, contentType, ... }
await assets.put("slides/welcome.png", bytes, { contentType: "image/png" });
await assets.head("slides/welcome.png");
await assets.delete("slides/welcome.png");
```

Five operations, chosen against what Announcements actually does — list a bucket, read images — plus
the writes any real product needs. There is deliberately no multipart upload, no lifecycle rule, no
replication, no ACL or bucket-policy management, and **no bucket creation or deletion**: which
buckets exist follows from an installed manifest, not from running module code.

`get` returns a Web `ReadableStream`, so an object goes straight into a `Response` without this
process holding it in memory. `put` accepts a stream too, but requires `contentLength` alongside one
— every S3-compatible provider needs the length up front for a single-part upload, and inferring it
would mean buffering the very thing the stream exists to avoid. Refusing loudly beats discovering it
through production memory pressure.

Metadata is `contentType`, `contentLength`, `etag`, `lastModified` — the four fields every
S3-compatible provider actually returns, not a mirror of S3's metadata surface. Content type matters
because Announcements serves images and must set a correct header without sniffing bytes.

Object keys allow `/` (modules genuinely use `slides/welcome.png`) and reject what would be
*ambiguous* rather than dangerous: empty, over 1024 UTF-8 bytes, control characters, lone surrogates,
backslashes, and a leading `/`. A key cannot escape its bucket in any case — the physical bucket
comes from the binding, never from the key.

### Errors are a closed set, and nothing from the provider survives

`ModuleStorageError` with eight codes (`STORAGE_OBJECT_NOT_FOUND`, `STORAGE_PROVIDER_UNAVAILABLE`,
`STORAGE_OPERATION_DENIED`, …). No message, request id, XML body, bucket name, endpoint, or caught
cause crosses the translation boundary. Provider errors routinely carry the host and sometimes the
signed request, and a module is entitled to log whatever it catches, so this is the only place that
boundary can be held. The one echoed value is the module's own object key.

### `context.storage`, beside `services` rather than inside it

Storage sits on `ModuleRuntimeContext` and `ModuleRouteHandlerContext` — **not** on
`ModuleRegistrationContext`, exactly like `services`, because it performs network I/O and
registration stays declarative. (`config` is on registration precisely because it is the opposite: a
resolved in-memory snapshot.)

It is emphatically not a field on `ModuleRuntimeServices`. That object is deliberately **one shared
identity every module receives**, a property the runtime has a test for; a storage facade is scoped
to one module's buckets. Putting it there would have forced one of those two properties to give, and
would have handed every module every other module's storage.

### Startup performs no storage I/O, and bucket creation is lazy

Constructing the provider opens no connection. Buckets are ensured on **first use** of a binding,
memoized per binding so it happens once per process — and a *failed* attempt is discarded, so a
transient outage does not poison the binding for the life of the process.

Eager provisioning at startup was rejected, and this was the sharpest decision in the pass. If
Ekklesia ensured buckets during boot, a storage service that was briefly down would stop the whole
platform from starting — strictly worse than the product being replaced, whose slideshow survives its
object store vanishing. The platform must not become less resilient because storage got abstracted.

Creation is idempotent and destruction does not exist. Nothing deletes a bucket: not disabling a
module, not removing one, not emptying it.

### Lifecycle: binding availability, never provider health

One new blocked reason, `storage_unavailable`, meaning *this module declares a logical bucket and
this host has no storage provider configured*. It names the logical bucket and nothing else.

```text
enabled → storage bindable? → configuration valid? → dependencies eligible? → runs
```

Resolution stays **pure and synchronous**. It asks whether storage is *configured*, never whether the
service answered — so eligibility can never depend on a probe that happened to time out during boot.
A storage-blocked module propagates to its required dependents through ADR 0028's existing mechanism
rather than a parallel one.

The distinction this preserves is the one the real product already proved matters:

| situation | classification |
| --- | --- |
| no provider configured | lifecycle problem — module blocked |
| required bucket cannot be bound | lifecycle problem — module blocked |
| configured provider times out at 10:15 | operational error the module catches and retries |

`running` still means the `start` hook completed, never that every external resource stays healthy.
A transient outage writes nothing, anywhere.

### Host configuration, using the names ADR 0004 already chose

`S3_ENDPOINT`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`, `S3_REGION`, `S3_FORCE_PATH_STYLE`, and the new
`S3_BUCKET_PREFIX` — read only by the executable entry point, validated into a typed config, and
never seen by a module, a module UI, a browser bundle, a diagnostic, or an API response.

Storage is **optional**. No `S3_ENDPOINT` means no provider, which is a complete deployment: modules
that declare no bucket are unaffected. Once an endpoint *is* set, credentials become required — a
half-configured provider silently attempting anonymous access would be a worse failure than a loud
one. Production requires `https` on the same reasoning the database transport requires verified TLS;
an endpoint embedding userinfo is rejected outright rather than stripped.

Credentials are never persisted in PostgreSQL and never routed through module configuration
(ADR 0029), which has no encryption at rest to offer them.

### The provider: S3-compatible, via the AWS SDK, not hand-rolled

`@aws-sdk/client-s3` addressing an arbitrary endpoint with path-style URLs. Hand-rolling SigV4 for a
security-sensitive path was rejected; the SDK is the mature, maintained implementation and works
against any S3-compatible service.

Probing the pinned SeaweedFS image directly, rather than trusting AWS documentation, found a real
divergence: **`CreateBucket` on an existing bucket reports `BucketAlreadyExists`, not AWS's
`BucketAlreadyOwnedByYou`.** A provider treating only the AWS spelling as success would have failed
on every second boot against this repository's own development storage. Both are accepted, and a live
test pins it. This is exactly the provider-quirk risk ADR 0004 anticipated.

### Browser delivery goes through the module's own routes

```text
browser → module API route → context.storage → object bytes
```

No S3 credentials reach the browser, no provider client is bundled for it, and `ModuleUiHostContext`
is unchanged. A module streams an object into a `Response` from its own authorized route.

This is a genuine behavioural change for Announcements, which today hands the browser direct S3 URLs.
Proxying costs a hop and makes the module API the media path; signed URLs and public-object routing
are the follow-up design, deliberately not built here without a real requirement to shape them.

### Standalone → Ekklesia migration: the seam, not the engine

Core provides list, read, and write. **A module owns semantic migration**, because only the module
knows which objects are canonical, which are generated, whether database references need updating,
and how to verify success. Core will not inspect Announcements filenames and guess.

```text
copy → verify → switch binding/state → verify → optionally remove old data later
```

Nothing in this pass deletes source data, and nothing ever should implicitly.

### Storage unavailable never means "start your own"

A module running inside Ekklesia that finds storage unconfigured is **blocked with a clear
diagnostic**, never permitted to launch its own storage server. A future standalone deployment
wrapper may bundle SeaweedFS as a convenience — that is deployment composition, not module runtime
behaviour, and the distinction must not blur.

## Alternatives

- **Per-module `storageEndpoint` / `storageAccessKey` configuration.** Rejected — the mistake this
  ADR exists to prevent. Every module would become a separate S3 deployment decision.
- **A new manifest field or version.** Rejected: `manifest.storage.buckets` already expressed the
  requirement exactly; it only lacked a consumer.
- **Keeping ADR 0029's `bucket` configuration key.** Retired: two mechanisms for one fact.
- **Handing modules a raw bucket name or provider client.** Rejected: it would let any module open
  any bucket and would weld every module to S3.
- **A single hyphen between module and bucket.** Rejected: demonstrably ambiguous, and the ambiguity
  silently merges two modules' namespaces.
- **Hashing without a readable portion.** Rejected: an operator debugging a permissions problem needs
  to recognize the bucket.
- **Eager bucket provisioning at startup.** Rejected: it would make a storage outage a platform
  outage, less resilient than the product being replaced.
- **Treating provider unreachability as a lifecycle state.** Rejected explicitly: it would mark
  running modules blocked over a network blip and destroy the distinction ADR 0028 established.
- **Hand-rolled SigV4.** Rejected for a security-sensitive path with a mature alternative.
- **Storage on `ModuleRuntimeServices`.** Rejected: `services` is one shared object; storage is
  per-module.
- **Signed URLs / public object routing now.** Deferred: no requirement yet exists to shape them, and
  guessing would produce a contract we would have to break.
- **Range requests in `get`.** Deferred: an additive option, cheap to add when video needs seeking,
  and outside "keep the first contract small".
- **Naming the contract after SeaweedFS.** Rejected by ADR 0004 before this pass began.

## Consequences

`manifest.storage.buckets` means something for the first time since it was introduced. A module asks
for a logical bucket and gets isolated, host-managed storage; the host decides what backs it; and
Announcements has a path away from `http://10.40.0.2:9000` that does not replace one hardcoded
address with a configurable one.

Core gains one dependency (`@aws-sdk/client-s3`, server-only), one new export path
(`@ekklesia/core/storage`), one blocked reason, one Doctor check, and a required `moduleStorage`
option on `createApiApp`. The template no longer declares a bucket. The live integration harness now
brings up the Compose file's already-pinned SeaweedFS alongside PostgreSQL.

Costs worth naming. Object delivery to browsers now goes through module routes, which is a real
change in shape for Announcements and will want signed URLs or caching before large media. The
provider is **not** "production-ready" by ADR 0004's own bar — that list includes ranges, multipart,
and signed URLs, none of which exist here. And bucket bindings are platform-global, needing a scope
column whenever an organization model arrives, like every other Core table.

## Deferred

Not implemented or promised here: signed URLs, public object routing, or a CDN; range requests;
multipart upload; object lifecycle rules, replication, or bucket policies; a storage settings
Dashboard; per-organization storage; provider administration or SeaweedFS process management; the
standalone host and its Compose stack; automatic standalone → Ekklesia migration and any migration
UI; and the Announcements migration itself.