feat: add host-provided module object storage
CI / verify (pull_request) Successful in 36s
CI / infrastructure-smoke (pull_request) Successful in 13s
CI / migration-smoke (pull_request) Failing after 2m20s

`manifest.storage.buckets` has existed since manifest version 1 and meant
nothing — discovery validated it and no code read it. This turns it into a
real capability, which is what stands between Announcements and the module
system: that product currently fetches a hardcoded LAN S3 endpoint, parses
ListObjectsV2 XML by hand, and hands the browser direct object URLs.

- A module declares a logical bucket and receives a scoped facade at
  `context.storage` — list, head, get, put, delete. The contract names no
  provider: no endpoint, region, credential, or physical bucket is reachable
  from it, which is what lets the same module run under a future standalone
  host over its own or a shared S3-compatible service.
- Core's first provider speaks S3 through the AWS SDK. SeaweedFS is the
  deployment target (ADR 0004), never an API name. Probing the pinned image
  found `CreateBucket` reporting `BucketAlreadyExists` rather than AWS's
  `BucketAlreadyOwnedByYou`; treating only the AWS spelling as success would
  have failed every second boot. A live test pins it.
- Logical → physical mapping is deterministic and collision-safe: an
  installation prefix, the module and bucket ids joined by `--` (which cannot
  occur inside either identifier, so `a-b`+`c` and `a`+`b-c` cannot collide),
  and a digest over the full untruncated identity. Two modules declaring
  `assets` always get separate namespaces; neither can see the physical name.
- Declaring a bucket is now a requirement: with no provider configured the
  module is `blocked · storage_unavailable` through ADR 0028's existing
  resolver, propagating to required dependents. An installation with no object
  storage stays a complete deployment.
- Binding availability is not provider health. Eligibility asks whether
  storage is configured, never whether it answered; bucket creation is lazy on
  first use so a storage outage cannot stop the platform booting; a running
  module whose store blinks stays running and retries. Nothing deletes a
  bucket.
- Credentials are deployment-supplied, never persisted, never routed through
  module configuration, and never visible to modules, module UI, diagnostics,
  or API responses. Provider errors translate to a closed code set carrying no
  endpoint, bucket, request id, or XML.
- Retires ADR 0029's transitional `bucket` configuration key. No manifest
  version change. The template declares no bucket, so it still runs without S3.

See ADR 0030.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
ekklesia
2026-08-23 08:45:53 -05:00
co-authored by Claude Opus 5
parent 527239f6de
commit b9ca0d560d
55 changed files with 4420 additions and 74 deletions
+69 -6
View File
@@ -12,8 +12,8 @@ transport), and Phase 5C (module database access) have landed. **Phase 5D (secur
against an independent security audit) is complete: all five passes have landed and the final
independent whole-repository re-audit has closed, under the current trusted-contributor,
not-yet-deployed threat model — and Phase 6 is unblocked and underway.** Phase 6 has since
completed through 6D, and **Phase 7A pass 1 (the minimal operational module lifecycle) and
pass 2 (the module configuration and secrets contract) have landed.** The operational
completed through 6D, and **Phase 7A passes 1 (the minimal operational module lifecycle), 2 (the module configuration and
secrets contract), and 3 (host-provided module object storage) have landed.** The operational
requirements that remain are prerequisites for two specific *future* activities — executing
untrusted CI contributions, and a public production launch — and neither gates Phase 6; see the
pass 5 closure below and
@@ -506,10 +506,71 @@ and credentials for shared infrastructure stay platform concerns: a module decla
needs, not an S3 endpoint, leaving room for the object storage service Core still lacks. See
[ADR 0029](docs/architecture/decisions/0029-module-configuration-and-secrets.md).
Phase 7A pass 3 turns `manifest.storage.buckets` — declarable since manifest version 1 and inert ever
since — into a real **host-provided object-storage capability**. This is what stands between
Announcements and the module system: it currently fetches `http://10.40.0.2:9000/announcements`, parses
S3 XML by hand, and hands the browser direct S3 URLs.
A module declares a **logical** bucket and gets a scoped facade:
```json
"storage": { "buckets": [{ "id": "announcement-assets" }] }
```
```ts
const assets = context.storage.bucket("announcement-assets");
const object = await assets.get("slides/welcome.png");
return new Response(object.body, { headers: { "content-type": object.contentType } });
```
**The contract names no provider.** No endpoint, region, credential, or physical bucket is reachable
from it — not as a field, not as a type, not as a string. That is what lets the same module run under
Ekklesia and under a future standalone host backed by its own SeaweedFS, a shared one, MinIO, or AWS
S3. A module that configures its own S3 endpoint is a module welded to one deployment, which is the
mistake this pass exists to prevent.
Core's first provider speaks the S3-compatible protocol through the AWS SDK. SeaweedFS is the intended
target ([ADR 0004](docs/architecture/decisions/0004-replace-unmaintained-minio.md)) and already runs in
the development Compose stack — but it is a deployment choice, never an API name. Probing it directly
found a real divergence worth having caught: `CreateBucket` on an existing bucket reports
`BucketAlreadyExists`, **not** AWS's `BucketAlreadyOwnedByYou`, so treating only the AWS spelling as
success would have failed on every second boot. A live test against the real server pins it.
**Logical → physical mapping is deterministic and collision-safe.**
`ekklesia-announcements--announcement-assets-3f0c1d9a6b21`: an installation prefix (so several
deployments can share one storage service), the module and bucket ids joined by `--` — which cannot
occur inside either identifier, so `a-b`+`c` and `a`+`b-c` can never collide — and a digest over the
full untruncated identity. Two modules both declaring `assets` always get separate namespaces, and a
module can never see or choose the physical name.
**Declaring a bucket is now a real requirement.** With no provider configured, such a module is
reported `blocked · storage_unavailable` naming the logical bucket, and its required dependents block
behind it through the existing resolver — no second eligibility system. An installation with *no*
object storage remains a complete deployment; modules declaring no bucket are unaffected, which is why
the template now declares none.
**A storage outage is not a lifecycle event.** Eligibility asks whether storage is *configured*, never
whether it answered. Bucket creation is lazy on first use rather than at startup, so a storage service
being down can never stop the platform from booting — and a running module whose object store blinks
stays `running`, catches a structured error, and retries, exactly as Announcements does today. Nothing
ever deletes a bucket, on disable or otherwise.
Credentials are deployment-supplied (`S3_ENDPOINT`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`,
`S3_BUCKET_PREFIX`), never persisted, never routed through module configuration, and never visible to
modules, module UI, diagnostics, or any API response — errors are translated to a closed code set that
carries no endpoint, bucket, request id, or XML. This also retires the transitional `bucket`
*configuration* key ADR 0029 flagged: the manifest declaration is the requirement.
Browsers reach objects through a module's own routes (`browser → module API → scoped storage`), so no
S3 knowledge or credential goes near the browser. **Signed URLs, public object delivery, range
requests, and multipart upload are deliberately deferred**, and the provider is not yet
"production-ready" by ADR 0004's own bar. See
[ADR 0030](docs/architecture/decisions/0030-host-provided-module-object-storage.md).
**Still not the finished Dashboard.** There is no navigation editor or drag and drop, no nested
folders, no icons or colors, no Dashboard Home or widgets, no design system, no client-side routing,
and no module-management or settings UI — module enablement and configuration are an API and a CLI,
not a screen. Sign-in UI, a production CSP and deployment for the built shell, asset publication beyond
and no module-management, settings, or storage UI — module enablement and configuration are an API and
a CLI, not a screen, and object storage is deployment configuration. Sign-in UI, a production CSP and deployment for the built shell, asset publication beyond
the UI entrypoint, React runtime sharing, the standalone host, and every real feature module also do
not exist.
@@ -601,8 +662,10 @@ Identity
Trusts only the actual immediate socket peer for rate limiting; no forwarded header is trusted.
Object storage
○ Check unavailable
Core storage integration has not been implemented yet.
○ Object storage not configured
No S3_ENDPOINT is set. This is a complete deployment unless a module requires storage.
✓ No module requires object storage
0 modules declare a logical storage bucket.
Result: READY WITH WARNINGS
```