feat: deliver module objects directly to browsers through opaque read URLs
ADR 0030 left exactly one way to get an image to a browser: read the
object into the module's process and write it back out of a module HTTP
route. Every byte of every asset travelled storage -> Ekklesia -> browser.
For the slideshow this platform exists to replace that means Ekklesia in
the path of every image on every display, permanently, for bytes that
never change — and it would make Ekklesia responsible for re-implementing
byte ranges, without which video seeking cannot work at all.
A module can now ask a logical bucket for a temporary browser URL:
const { url, expiresAt } = await assets.createReadUrl("slides/welcome.png");
The contract names intent, not mechanism. There is no signing, expiry
format, endpoint, or bucket in it — a host with no S3 anywhere could
satisfy the same method, which is what keeps a future single-module
standalone host able to implement it. The URL is opaque to module code and
a bearer capability to whoever holds it, so a module authorizes the
request in its own route before asking: storage bounds reach, never
identity. Lifetimes run one minute to six hours, fifteen by default, and
out-of-range values are refused rather than clamped, so "expires forever"
is unreachable. Logical keys stay canonical state; URLs are generated on
demand and never persisted or logged.
The provider endpoint and the browser delivery origin are separated
because measurement showed they cannot be collapsed: an S3-compatible
signature covers the request Host, so a URL signed for one origin and
rewritten to another is rejected. The browser-facing origin is therefore a
signing input (S3_DELIVERY_ENDPOINT), defaulting to S3_ENDPOINT — so the
standard deployment configures nothing and delivery just works. The scheme
is not signed, so a reverse proxy may terminate TLS provided it passes the
original Host through; that is pinned by a test rather than described and
hoped for.
Read URLs are stable within a window of one third of their lifetime.
Signing is a pure function of the clock, so naive re-signing yields a new
URL per second — a guaranteed browser cache miss, and the reference client
keys its <img> by URL, so an unchanged image would be torn down and
re-downloaded continuously. Quantizing the signing timestamp makes
repeated calls byte-identical while leaving at least two-thirds of the
lifetime; Cache-Control is set at signing time to the guaranteed
remainder, because the pinned SeaweedFS image sends none of its own.
Verified against live SeaweedFS: byte ranges answered directly with 206
and a correct Content-Range, so no range proxy is needed in Core; signing
is local (200 URLs in 66ms), so no batch API is needed; unsigned requests
for the same object are refused, so delivery never depends on a public
bucket; tampered signatures, swapped keys, and swapped buckets all
rejected. Signed write and delete URLs are deliberately not built.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
9a3b8dab8f
commit
7ff063b87a
+34
-4
@@ -682,6 +682,7 @@ and no route returns a provider endpoint or credential.
|
||||
| `S3_REGION` | Defaults to `us-east-1` |
|
||||
| `S3_FORCE_PATH_STYLE` | Defaults to `true`; self-hosted services need it |
|
||||
| `S3_BUCKET_PREFIX` | Defaults to `ekklesia`; namespaces this installation on a shared service |
|
||||
| `S3_DELIVERY_ENDPOINT` | The origin a **browser** reaches storage on. Defaults to `S3_ENDPOINT`; set it only when that is not browser-reachable |
|
||||
|
||||
Production requires `https`, exactly as the database transport requires verified TLS. An endpoint
|
||||
embedding credentials is rejected rather than stripped. These values reach Core's provider and
|
||||
@@ -690,6 +691,16 @@ nothing else — never a module, module UI, browser bundle, diagnostic, or API r
|
||||
The repository's development Compose stack already runs SeaweedFS for this, and `.env.example`
|
||||
already carries matching values.
|
||||
|
||||
`S3_DELIVERY_ENDPOINT` exists because an S3-compatible signature covers the request's `Host`, so a
|
||||
delivery URL cannot be signed for one origin and rewritten to another — the browser-facing origin has
|
||||
to be known when the URL is signed. Leave it unset whenever `S3_ENDPOINT` is itself reachable from a
|
||||
browser, which is true of the development stack (Compose publishes storage on loopback) and of any
|
||||
deployment where Core and browsers use the same name. Set it when Core reaches storage privately —
|
||||
`http://object-storage:8333` — and browsers reach it publicly. The scheme is *not* signed, so a
|
||||
reverse proxy may terminate TLS and forward plaintext, **provided it passes the original `Host`
|
||||
header through unchanged**; a proxy that rewrites `Host` to its backend makes every delivery URL fail
|
||||
with `403`.
|
||||
|
||||
### Logical buckets, and what modules get
|
||||
|
||||
A module declares logical buckets in its manifest (`storage.buckets`) and receives a scoped facade at
|
||||
@@ -717,14 +728,33 @@ never prevents the platform from booting, and nothing ever deletes a bucket.
|
||||
|
||||
### Serving objects to browsers
|
||||
|
||||
There are no signed URLs and no public object routing. A module streams objects through its own
|
||||
authorized route:
|
||||
A module can hand a browser a temporary URL and let it fetch the object from storage directly
|
||||
([ADR 0031](../architecture/decisions/0031-direct-object-delivery.md)):
|
||||
|
||||
```text
|
||||
browser → GET /api/v1/modules/<id>/assets/<key> → context.storage → object bytes
|
||||
browser → GET /api/v1/modules/<id>/slides → module route → { key, url, expiresAt }
|
||||
browser → GET <url> ─────────────────────────────────────→ storage service
|
||||
```
|
||||
|
||||
which keeps every S3 concern server-side and gives module UI no credentials.
|
||||
The module authorizes the request in its own route, asks its logical bucket for a URL, and returns
|
||||
it. Ekklesia is not in the data path, so byte ranges, caching, and revalidation are the storage
|
||||
service's job — a `Range` request against a delivery URL is answered directly with `206`, which is
|
||||
what makes media seeking work without a range proxy in Core.
|
||||
|
||||
```ts
|
||||
const assets = context.storage.bucket("announcement-assets");
|
||||
const { url, expiresAt } = await assets.createReadUrl("slides/welcome.png");
|
||||
```
|
||||
|
||||
A delivery URL is a **bearer capability**: whoever holds it can read that one object until it
|
||||
expires. Deciding who may have one is the module's authorization decision, made before it asks;
|
||||
the storage contract only bounds how far the URL reaches. Treat a complete URL as a secret — never
|
||||
log one (see [Security](security.md#object-storage-is-scoped-by-binding-and-modules-never-hold-credentials)).
|
||||
|
||||
Streaming through the module's own route with `context.storage.get(...)` remains available and is
|
||||
still the right choice when the response must be computed, transformed, or authorized per byte.
|
||||
There are no signed **write** URLs: uploads go through a module route, where the module can validate
|
||||
what it is accepting.
|
||||
|
||||
## Module HTTP routes
|
||||
|
||||
|
||||
@@ -283,6 +283,39 @@ correctness and portability boundary like every other module-facing mechanism he
|
||||
What it does guarantee is that Ekklesia never hands a module another module's data, never gives module
|
||||
UI or a browser a credential, and never prints one in diagnostics.
|
||||
|
||||
### Delivery URLs are bearer capabilities, and are treated as secrets
|
||||
|
||||
A module can ask a bucket for a temporary URL that lets a browser fetch one object directly
|
||||
([ADR 0031](../architecture/decisions/0031-direct-object-delivery.md)). Whoever holds that URL can
|
||||
read that object until it expires, with no further check of who they are.
|
||||
|
||||
- **Authorization happens before issuance, not inside the URL.** A module's route authorizes the
|
||||
request and only then asks for a URL. The storage contract has no notion of a principal and offers
|
||||
no hook that would let it make a permission decision; it bounds *reach*, not *who*.
|
||||
- **Lifetimes are bounded and cannot be waived.** One minute at the shortest, six hours at the
|
||||
longest, fifteen minutes by default. Out-of-range requests are refused rather than clamped, so
|
||||
"expires forever" is unreachable rather than merely discouraged.
|
||||
- **One object, one bucket.** The signature covers the URL's own path, so editing the key or the
|
||||
bucket in a delivery URL yields a rejection rather than a different object — including another
|
||||
module's object.
|
||||
- **Never logged, never printed.** A complete delivery URL is treated exactly like a token: it is not
|
||||
logged, and Doctor never prints one. Where a URL must be identified in a log, identify it by module
|
||||
id, logical bucket id, key, and expiry — never by its query parameters.
|
||||
- **Reads only.** There are no signed write or delete URLs. Uploads go through a module route where
|
||||
the module can validate what it accepts.
|
||||
- **Never canonical state.** Modules persist logical object keys and generate URLs on demand. A
|
||||
stored URL is a stored credential with an expiry date in it.
|
||||
|
||||
**What a browser sees.** A delivery URL reveals the storage origin, the physical bucket name, the
|
||||
object key, and signing parameters. Physical bucket names are implementation metadata, not secrets —
|
||||
nothing is authorized by knowing one, since the bucket is not publicly readable and a tampered URL is
|
||||
rejected. The guarantee is that *module code* cannot reach those details, not that a browser cannot
|
||||
see the URL it is fetching.
|
||||
|
||||
**Buckets stay private.** Direct delivery deliberately does **not** work by making buckets anonymous;
|
||||
an unsigned request for the same object is refused, and an integration test asserts it so that a
|
||||
future change cannot "fix" delivery by opening a bucket up.
|
||||
|
||||
### Client-side permission checks are not a security boundary
|
||||
|
||||
A host may tell a module UI what permissions it believes the viewer holds
|
||||
|
||||
@@ -209,12 +209,43 @@ no endpoint or XML, two modules declaring the same logical bucket staying genuin
|
||||
server, the longest bucket name the mapping can produce being accepted, a real connection failure
|
||||
mapping to `STORAGE_PROVIDER_UNAVAILABLE`, and a bucket surviving the deletion of every object in it.
|
||||
|
||||
An eighth suite (`core/tests/module-delivery.integration.ts`, same harness) proves direct browser
|
||||
delivery against that same live server, because every claim it makes is a property of the agreement
|
||||
between the signer and the provider rather than of code in this repository
|
||||
([ADR 0031](../architecture/decisions/0031-direct-object-delivery.md)). It pins: a delivery URL
|
||||
returning the object with its content type intact to an ordinary credential-free `fetch`; the same
|
||||
object being refused **without** the URL's authorization, so delivery can never be quietly "fixed" by
|
||||
making a bucket public; tampered signatures, swapped keys, and swapped buckets all rejected; expiry
|
||||
actually enforced; a `Range` request answered directly with `206` and a correct `Content-Range`;
|
||||
repeated calls inside a window returning a byte-identical URL so browser caches and URL-keyed DOM
|
||||
elements hold; `Cache-Control` never outliving the URL; an optional download filename; and two
|
||||
modules' identical logical bucket names resolving to different objects.
|
||||
|
||||
Its most load-bearing tests concern the **browser delivery origin**. They sign for a public origin and
|
||||
replay the request against the real backend through `node:http` with an explicit `Host` header —
|
||||
which `fetch` cannot do, since `Host` is a forbidden header name there and it would silently send the
|
||||
wrong one and prove nothing. Together they establish that a signature survives a reverse proxy that
|
||||
preserves `Host` (even across TLS termination, because the scheme is not signed), that it fails when
|
||||
a proxy rewrites `Host`, and that a URL signed for the internal endpoint cannot be rewritten to the
|
||||
public one afterwards — which is the entire reason the delivery origin is a signing input rather than
|
||||
a post-processing step.
|
||||
|
||||
An Announcements-shaped fixture in the same file models the real slideshow: it lists its own slides,
|
||||
filters images, builds a delivery URL for each, and fetches every one directly. It is written to
|
||||
demonstrate what it *cannot* say — its whole world is a `ModuleStorage` and two names from its own
|
||||
manifest — and it asserts that the module's durable state holds keys while only the URL carries
|
||||
host detail. It also proves that forty URLs cost no provider round trips.
|
||||
|
||||
Unit suites (`core/tests/module-storage.test.ts`, `core/tests/module-storage-lifecycle.test.ts`) use an
|
||||
injected in-memory provider and never touch the network. They prove the parts the protocol cannot:
|
||||
binding determinism and collision-safety, scoping and cross-module isolation, key validation, lazy
|
||||
provisioning and retry-after-failure, configuration validation and credential redaction, and the
|
||||
lifecycle integration including dependency propagation. See
|
||||
[ADR 0030](../architecture/decisions/0030-host-provided-module-object-storage.md).
|
||||
lifecycle integration including dependency propagation. They also own the delivery bounds, which need
|
||||
no server at all: every out-of-range lifetime refused rather than clamped, every download filename
|
||||
that could inject a response header refused, and the module-facing surface pinned so a new accessor
|
||||
cannot appear unnoticed. See
|
||||
[ADR 0030](../architecture/decisions/0030-host-provided-module-object-storage.md) and
|
||||
[ADR 0031](../architecture/decisions/0031-direct-object-delivery.md).
|
||||
|
||||
### Which kind of timing proof lives where
|
||||
|
||||
|
||||
Reference in New Issue
Block a user