Zayin/Ekklesia
3
0
Fork 0
Code Issues 1 Pull Requests 3 Actions Packages Projects Releases 8 Wiki Activity

feat: add host-provided module object storage #29

Merged
Zayin_Admin merged 2 commits from phase-7a/module-storage into main 2026-08-23 21:40:12 +00:00
Conversation 0 Commits 2 Files Changed 56
+4546 -75
Zayin_Admin commented 2026-08-23 19:07:58 +00:00
Owner
Copy Link
Copy Source

manifest.storage.buckets has existed since manifest version 1 and meant
nothing — discovery validated it and no code read it. This turns it into a
real capability, which is what stands between Announcements and the module
system: that product currently fetches a hardcoded LAN S3 endpoint, parses
ListObjectsV2 XML by hand, and hands the browser direct object URLs.

  • A module declares a logical bucket and receives a scoped facade at
    context.storage — list, head, get, put, delete. The contract names no
    provider: no endpoint, region, credential, or physical bucket is reachable
    from it, which is what lets the same module run under a future standalone
    host over its own or a shared S3-compatible service.
  • Core's first provider speaks S3 through the AWS SDK. SeaweedFS is the
    deployment target (ADR 0004), never an API name. Probing the pinned image
    found CreateBucket reporting BucketAlreadyExists rather than AWS's
    BucketAlreadyOwnedByYou; treating only the AWS spelling as success would
    have failed every second boot. A live test pins it.
  • Logical → physical mapping is deterministic and collision-safe: an
    installation prefix, the module and bucket ids joined by -- (which cannot
    occur inside either identifier, so a-b+c and a+b-c cannot collide),
    and a digest over the full untruncated identity. Two modules declaring
    assets always get separate namespaces; neither can see the physical name.
  • Declaring a bucket is now a requirement: with no provider configured the
    module is blocked · storage_unavailable through ADR 0028's existing
    resolver, propagating to required dependents. An installation with no object
    storage stays a complete deployment.
  • Binding availability is not provider health. Eligibility asks whether
    storage is configured, never whether it answered; bucket creation is lazy on
    first use so a storage outage cannot stop the platform booting; a running
    module whose store blinks stays running and retries. Nothing deletes a
    bucket.
  • Credentials are deployment-supplied, never persisted, never routed through
    module configuration, and never visible to modules, module UI, diagnostics,
    or API responses. Provider errors translate to a closed code set carrying no
    endpoint, bucket, request id, or XML.
  • Retires ADR 0029's transitional bucket configuration key. No manifest
    version change. The template declares no bucket, so it still runs without S3.

See ADR 0030.

Co-Authored-By: Claude Opus 5 [email protected]

`manifest.storage.buckets` has existed since manifest version 1 and meant nothing — discovery validated it and no code read it. This turns it into a real capability, which is what stands between Announcements and the module system: that product currently fetches a hardcoded LAN S3 endpoint, parses ListObjectsV2 XML by hand, and hands the browser direct object URLs. - A module declares a logical bucket and receives a scoped facade at `context.storage` — list, head, get, put, delete. The contract names no provider: no endpoint, region, credential, or physical bucket is reachable from it, which is what lets the same module run under a future standalone host over its own or a shared S3-compatible service. - Core's first provider speaks S3 through the AWS SDK. SeaweedFS is the deployment target (ADR 0004), never an API name. Probing the pinned image found `CreateBucket` reporting `BucketAlreadyExists` rather than AWS's `BucketAlreadyOwnedByYou`; treating only the AWS spelling as success would have failed every second boot. A live test pins it. - Logical → physical mapping is deterministic and collision-safe: an installation prefix, the module and bucket ids joined by `--` (which cannot occur inside either identifier, so `a-b`+`c` and `a`+`b-c` cannot collide), and a digest over the full untruncated identity. Two modules declaring `assets` always get separate namespaces; neither can see the physical name. - Declaring a bucket is now a requirement: with no provider configured the module is `blocked · storage_unavailable` through ADR 0028's existing resolver, propagating to required dependents. An installation with no object storage stays a complete deployment. - Binding availability is not provider health. Eligibility asks whether storage is configured, never whether it answered; bucket creation is lazy on first use so a storage outage cannot stop the platform booting; a running module whose store blinks stays running and retries. Nothing deletes a bucket. - Credentials are deployment-supplied, never persisted, never routed through module configuration, and never visible to modules, module UI, diagnostics, or API responses. Provider errors translate to a closed code set carrying no endpoint, bucket, request id, or XML. - Retires ADR 0029's transitional `bucket` configuration key. No manifest version change. The template declares no bucket, so it still runs without S3. See ADR 0030. Co-Authored-By: Claude Opus 5 <[email protected]>
Zayin_Admin added 1 commit 2026-08-23 19:07:58 +00:00
feat: add host-provided module object storage
CI / verify (pull_request) Successful in 36s
Details
CI / infrastructure-smoke (pull_request) Successful in 13s
Details
CI / migration-smoke (pull_request) Failing after 2m20s
Details
b9ca0d560d 
`manifest.storage.buckets` has existed since manifest version 1 and meant
nothing — discovery validated it and no code read it. This turns it into a
real capability, which is what stands between Announcements and the module
system: that product currently fetches a hardcoded LAN S3 endpoint, parses
ListObjectsV2 XML by hand, and hands the browser direct object URLs.

- A module declares a logical bucket and receives a scoped facade at
  `context.storage` — list, head, get, put, delete. The contract names no
  provider: no endpoint, region, credential, or physical bucket is reachable
  from it, which is what lets the same module run under a future standalone
  host over its own or a shared S3-compatible service.
- Core's first provider speaks S3 through the AWS SDK. SeaweedFS is the
  deployment target (ADR 0004), never an API name. Probing the pinned image
  found `CreateBucket` reporting `BucketAlreadyExists` rather than AWS's
  `BucketAlreadyOwnedByYou`; treating only the AWS spelling as success would
  have failed every second boot. A live test pins it.
- Logical → physical mapping is deterministic and collision-safe: an
  installation prefix, the module and bucket ids joined by `--` (which cannot
  occur inside either identifier, so `a-b`+`c` and `a`+`b-c` cannot collide),
  and a digest over the full untruncated identity. Two modules declaring
  `assets` always get separate namespaces; neither can see the physical name.
- Declaring a bucket is now a requirement: with no provider configured the
  module is `blocked · storage_unavailable` through ADR 0028's existing
  resolver, propagating to required dependents. An installation with no object
  storage stays a complete deployment.
- Binding availability is not provider health. Eligibility asks whether
  storage is configured, never whether it answered; bucket creation is lazy on
  first use so a storage outage cannot stop the platform booting; a running
  module whose store blinks stays running and retries. Nothing deletes a
  bucket.
- Credentials are deployment-supplied, never persisted, never routed through
  module configuration, and never visible to modules, module UI, diagnostics,
  or API responses. Provider errors translate to a closed code set carrying no
  endpoint, bucket, request id, or XML.
- Retires ADR 0029's transitional `bucket` configuration key. No manifest
  version change. The template declares no bucket, so it still runs without S3.

See ADR 0030.

Co-Authored-By: Claude Opus 5 <[email protected]>
Zayin_Admin added 1 commit 2026-08-23 21:36:30 +00:00
fix: bound provider exchanges and stop the storage suite starving its server
CI / verify (pull_request) Successful in 36s
Details
CI / infrastructure-smoke (pull_request) Successful in 21s
Details
CI / migration-smoke (pull_request) Successful in 51s
Details
af4a8009fb 
The live storage suite failed in CI with three failures and a 30s runner
timeout while passing locally in under three seconds. Two independent
defects produced that, one of them a real runtime defect.

Provider calls had no deadline. `S3Client` was built with no
`requestHandler` and no `maxAttempts`, and the SDK's `connectionTimeout`
and `socketTimeout` both default to 0 — disabled. A provider that accepts
a connection and then goes silent therefore held the caller open forever,
bounded in the suite only by vitest and in production by nothing at all.
ADR 0030 promises such a failure reaches a module as the transient
`STORAGE_PROVIDER_UNAVAILABLE` it retries through; `translateStorageError`
already mapped the SDK's `TimeoutError` onto that code, but the branch was
unreachable because nothing ever timed out. The bound is on socket
idleness rather than total elapsed time so a module streaming a large
object is never cut off for being big, only for going quiet.

The suite provisioned a fresh bucket per test. Every S3 bucket becomes its
own SeaweedFS collection claiming a block of volumes from a budget the
server derives from free disk, and `mini` auto-configures a
production-shaped ~1GB volume — so the ceiling is small and varies with
the runner. Twelve buckets per run exhausted it, and an exhausted budget
is not a failed write: the master retries allocation internally and never
replies, which is the hang CI saw. Tests now share one bucket and isolate
by key namespace, which exercises prefix scoping at the same time;
bucket identity is provisioned only where it is the thing under test.
Pinning a small volume size makes the budget depend far less sharply on
the host's disk.

Verified by starving a server on purpose: the same exhaustion that used to
hang until the runner killed it now fails in ~15s with a structured error
and no runner timeout, so the failure is bounded rather than hidden. Ten
consecutive runs pass in ~2.1s each, using 28 volumes per run against a
5212 budget where the old suite used 84 against 161.

Co-Authored-By: Claude Opus 5 <[email protected]>
Zayin_Admin merged commit 9a3b8dab8f into main 2026-08-23 21:40:12 +00:00
Zayin_Admin deleted branch phase-7a/module-storage 2026-08-23 21:40:13 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
Joey Zayin_Admin ekklesia-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Zayin/Ekklesia#29