feat: let a module route stream an image-sized upload, explicitly and bounded
ADR 0017 capped a module route at 1 MiB and said large media belonged to a future storage design. That design arrived — ADR 0030 gave modules a bucket, ADR 0031 took Ekklesia out of the read path — and it did not solve the write path: signed write URLs were deliberately declined, so the only way bytes reach a bucket is a module's own route, and a photograph is not 1 MiB. Extend the route contract rather than adding a second HTTP system. A route declares `bodyMode`: `buffered` (the default, and what every existing route already meant) keeps the 1 MiB ceiling and materializes the body before the handler runs; `streamed` raises the ceiling to 32 MiB and counts bytes as the handler consumes them, holding nothing. The mode is explicit rather than inferred from the size, because inferring it would mean an author who raised a limit silently changed how their own handler receives its body and when a 413 can reach their own code. 32 MiB holds one 25 MiB image plus bounded overhead and is also the platform's absolute ceiling: no route in any mode may declare more, and no unbounded option exists in the contract. The unit is one file per request — twenty images are twenty individually bounded requests, and nothing here can express a 500 MiB aggregate whose failure would waste everything already sent. Replace the buffering dependency with the platform's own limiter. A declared Content-Length over the limit is refused synchronously, before authorization and before a byte is read. Buffered mode now counts what actually arrives rather than trusting the header, which closes a gap the previous implementation had. Streamed mode wraps the body in a counting transform; on overflow the stream errors, the source is cancelled, and the response is replaced with the same sanitized 413 even if the handler swallowed the error — a limit a handler can swallow is not a limit. An unread body is cancelled so an abandoned upload releases its socket, and an abort stays an ordinary failed request rather than becoming a 413. Ordering is unchanged and now strictly better: in streamed mode no byte is read until the handler reads it, and the handler runs only after authorization succeeds, so an unauthorized caller cannot push an upload through at all. The live suite uploads a real 25 MiB body over a real socket end to end. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
c444ff570b
commit
e36485bf99
@@ -0,0 +1,160 @@
|
||||
# 0042: Explicit, bounded streaming request bodies
|
||||
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-27
|
||||
- Supersedes: none
|
||||
- Related: [ADR 0015](0015-static-module-http-routing-and-runtime-lifecycle.md),
|
||||
[ADR 0017](0017-public-request-perimeter-hardening.md),
|
||||
[ADR 0030](0030-host-provided-module-object-storage.md),
|
||||
[ADR 0031](0031-direct-object-delivery.md)
|
||||
|
||||
## Context
|
||||
|
||||
ADR 0017 gave every public route an enforced request-body ceiling and capped a module route at
|
||||
1 MiB, with an explicit note that "large media transfer belongs to a future dedicated storage
|
||||
design, never to widening this ceiling."
|
||||
|
||||
The storage design arrived — ADR 0030 gave modules a scoped object store, ADR 0031 took Ekklesia out
|
||||
of the *read* path with signed delivery URLs — and it did not solve the *write* path. ADR 0031
|
||||
deliberately declined to offer signed write URLs, so the only way bytes reach a module's bucket is
|
||||
through that module's own HTTP route. That route is capped at 1 MiB, and an ordinary photograph is
|
||||
not 1 MiB. A church uploading a slide for a lobby screen cannot, today, get the file in.
|
||||
|
||||
Two properties of the existing implementation also need naming, because they are what the change has
|
||||
to preserve and what it has to fix:
|
||||
|
||||
- The pinned body-limit middleware **fully buffers** any body with no declared `Content-Length`
|
||||
before deciding. At 1 MiB that is a bounded, acceptable cost. At image sizes it would mean the
|
||||
platform holding every concurrent upload in memory.
|
||||
- With a declared `Content-Length` within the limit it *counts nothing at all*, trusting the header.
|
||||
Node's HTTP framing makes that safe today, but it is trust the platform does not need to extend.
|
||||
|
||||
## Decision
|
||||
|
||||
Extend the existing route contract. There is no second HTTP system, no upload endpoint, no signed
|
||||
write URL, and no new middleware stack — one route declaration, one limiter, one `413`.
|
||||
|
||||
### A route declares how its body reaches it
|
||||
|
||||
`ModuleRouteDefinitionInput` gains one optional field:
|
||||
|
||||
```ts
|
||||
http.route({
|
||||
method: "POST",
|
||||
path: "/uploads",
|
||||
permission: "signage.slides.write",
|
||||
maxBodyBytes: 33_554_432,
|
||||
bodyMode: "streamed",
|
||||
handler,
|
||||
});
|
||||
```
|
||||
|
||||
- **`buffered`** (the default, and what every route written before this meant) — the platform reads
|
||||
and counts the whole body **before** the handler runs, so the handler is promised a fully
|
||||
available, bounded body and an oversized request is rejected before any authorization or handler
|
||||
work happens. Ceiling: 1 MiB, unchanged.
|
||||
- **`streamed`** — the platform counts bytes **as the handler consumes them** and never materializes
|
||||
the body. Ceiling: 32 MiB.
|
||||
|
||||
The mode is **explicit rather than inferred from the size**. Inferring it — "over 1 MiB means
|
||||
streamed" — would mean an author who raised a limit silently changed how their own handler receives
|
||||
its body, and when a `413` can arrive relative to their own code. That is exactly the class of
|
||||
invisible change this repository refuses.
|
||||
|
||||
Omitting the field means `buffered`, so no existing route changed behaviour, and the canonical parsed
|
||||
form always carries a mode so nothing downstream re-applies the default or disagrees about it.
|
||||
|
||||
### The ceilings, and why 32 MiB
|
||||
|
||||
`MODULE_ROUTE_MAX_BODY_BYTES_CEILING` stays at 1 MiB for buffered bodies: a body the platform
|
||||
materializes is a body the platform is holding, and 1 MiB is the most a JSON request has any business
|
||||
being.
|
||||
|
||||
`MODULE_ROUTE_STREAMED_MAX_BODY_BYTES_CEILING` is 32 MiB, sized to hold one 25 MiB image plus bounded
|
||||
overhead — a multipart envelope, its boundaries and part headers, and room for a client that pads. It
|
||||
is also `MODULE_ROUTE_ABSOLUTE_MAX_BODY_BYTES`: **no route, in any mode, may declare or accept a
|
||||
larger single request body, and no unbounded option exists anywhere in the contract.** A route
|
||||
declaring more than its mode's ceiling fails registration, exactly as an over-ceiling value always
|
||||
has.
|
||||
|
||||
The unit is deliberately **one file per request**. A UI offering twenty images issues twenty
|
||||
individually bounded requests, each of which the platform can reject cheaply and each of which fails
|
||||
alone; a 500 MiB aggregate request would make one bad byte waste everything already sent, and nothing
|
||||
in this contract can express one.
|
||||
|
||||
### Enforcement, in one limiter
|
||||
|
||||
The platform's own limiter replaces the buffering dependency. It is the same code for both modes and
|
||||
differs only in *when* it counts:
|
||||
|
||||
1. A declared `Content-Length` above the limit is refused synchronously — before authorization,
|
||||
before the handler, before a single body byte is read. That is the cheap rejection an ordinary
|
||||
too-large upload takes. A malformed or contradictory header is treated as *absent* rather than as
|
||||
a rejection, because the byte counting bounds the request either way and refusing could only turn
|
||||
a legitimate request away.
|
||||
2. `buffered`: bytes are counted as they arrive and the request is rejected the moment the running
|
||||
total passes the limit — so a lying or absent `Content-Length` is bounded by what actually
|
||||
arrived rather than by what was claimed, which the previous implementation did not do. At most
|
||||
the route's own declared allowance is ever held.
|
||||
3. `streamed`: the body is wrapped in a counting transform and handed to the handler. The moment the
|
||||
total passes the limit the stream errors, the source is cancelled, and the response is replaced
|
||||
with the same sanitized `413` — **even if the handler caught the error and answered anyway**. A
|
||||
limit a handler can swallow is not a limit.
|
||||
|
||||
A body no handler read is cancelled after the request, so an abandoned upload releases its socket
|
||||
rather than draining. An aborted upload surfaces as an ordinary failed request, never a `413`:
|
||||
nothing exceeded a limit.
|
||||
|
||||
Ordering is unchanged and remains deliberate: trusted-origin check, then the body contract, then
|
||||
permission evaluation, then the handler. In `streamed` mode no body byte is read until the handler
|
||||
reads it and the handler runs only after authorization succeeds, so an unauthorized caller can never
|
||||
push an upload through the platform at all — strictly better than the buffered path.
|
||||
|
||||
Timeouts stay where they were: the server's own `headersTimeout`/`requestTimeout`. A slow upload is a
|
||||
slow *request*, and the platform already has one answer for that rather than a second per-route one.
|
||||
|
||||
### What a module author must know
|
||||
|
||||
In `streamed` mode a `413` can arrive **after** the handler has started. A handler that persists as
|
||||
it reads must be prepared for a partially consumed upload — write to a temporary key and commit only
|
||||
after the read completes. The platform cannot do that for the module, because only the module knows
|
||||
what a partial write means to its own data.
|
||||
|
||||
## Consequences
|
||||
|
||||
- An ordinary image upload works, through the route contract a module already had, with the platform
|
||||
holding none of it.
|
||||
- The buffered path now counts bytes it previously trusted a header for, closing a gap rather than
|
||||
opening one.
|
||||
- Every existing route and every existing test is unchanged: the default is the old behaviour, and
|
||||
the `413` contract, ordering, and sanitization are identical.
|
||||
- `apps/api` owns ~60 lines of stream plumbing it previously delegated. That is the price of a
|
||||
limiter that neither buffers a large body nor can be swallowed by a handler, and it is enforced by
|
||||
tests rather than by reading the dependency's source.
|
||||
- Media larger than 32 MiB — a video — still has no path in. That is deliberate and named below.
|
||||
|
||||
## Alternatives
|
||||
|
||||
- **Raise the single ceiling to 32 MiB.** Rejected: every existing route would silently gain the
|
||||
ability to have 32 MiB buffered in memory on its behalf, and `await request.json()` in a module
|
||||
handler would stop being a bounded operation.
|
||||
- **Infer streaming from `maxBodyBytes`.** Rejected: it makes a size change silently change delivery
|
||||
semantics and error timing for the author's own code.
|
||||
- **A separate upload endpoint or second HTTP system.** Rejected: a second route table, a second
|
||||
authorization path, a second CORS and origin policy, and a second failure vocabulary, to solve
|
||||
something the existing route contract can express with one field.
|
||||
- **Signed write URLs to object storage.** Deliberately still deferred (ADR 0031). A signed write URL
|
||||
is a bearer capability to *create* an object, which needs quota, content-type, and lifecycle
|
||||
answers this pass does not have — and a module that wants to validate, transform, or authorize per
|
||||
byte cannot use one anyway.
|
||||
- **Let the handler enforce its own limit.** Rejected: a limit enforced by the party being protected
|
||||
against is not a limit, and every module would reimplement it slightly differently.
|
||||
- **Trust `Content-Length` in streamed mode.** Rejected: the running byte count costs nothing and
|
||||
removes the need to reason about framing at all.
|
||||
|
||||
## Deferred
|
||||
|
||||
Signed write URLs, resumable or chunked multi-request uploads, per-module or per-installation upload
|
||||
quotas, content-type or magic-byte validation, virus scanning, image transformation, progress
|
||||
reporting, and any body larger than the absolute ceiling — a video pipeline is its own design, not a
|
||||
larger number here.
|
||||
@@ -50,3 +50,4 @@ ADRs capture durable choices that affect more than one package or phase.
|
||||
- [0039: Shared design tokens and foundational elements](0039-shared-design-tokens-and-elements.md) — Accepted
|
||||
- [0040: Module stylesheets, and every browser artifact gets an isolated root](0040-module-stylesheets-and-artifact-isolation.md) — Accepted
|
||||
- [0041: One module may contribute a browser renderer to another module's capability](0041-module-browser-contributions.md) — Accepted
|
||||
- [0042: Explicit, bounded streaming request bodies](0042-bounded-streaming-request-bodies.md) — Accepted
|
||||
|
||||
+31
-9
@@ -408,16 +408,28 @@ protection across restarts or replicas.
|
||||
|
||||
## Request body limits
|
||||
|
||||
Every public body-consuming route has an explicit, platform-enforced byte ceiling
|
||||
(`apps/api/src/body-limits.ts`, built on Hono's pinned `hono/body-limit` middleware — verified
|
||||
directly against its source to reject both an oversized declared `Content-Length` and an oversized
|
||||
declared-length-free streamed body without ever buffering the whole payload first):
|
||||
Every public body-consuming route has an explicit, platform-enforced byte ceiling, enforced by the
|
||||
platform's own limiter (`apps/api/src/body-limits.ts`):
|
||||
|
||||
| Surface | Limit |
|
||||
| --- | --- |
|
||||
| `/api/auth/*` (every Better Auth route) | 8 KiB |
|
||||
| `POST /api/v1/grants`, `POST /api/v1/grants/:grantId/revoke` | 16 KiB |
|
||||
| A module route (`http.route(...)`'s own `maxBodyBytes`) | Up to 1 MiB, the platform ceiling (`MODULE_ROUTE_MAX_BODY_BYTES_CEILING`, `@ekklesia/contracts`) |
|
||||
| A module route declaring `bodyMode: "buffered"` (the default) | Up to 1 MiB (`MODULE_ROUTE_MAX_BODY_BYTES_CEILING`, `@ekklesia/contracts`) |
|
||||
| A module route declaring `bodyMode: "streamed"` | Up to 32 MiB (`MODULE_ROUTE_STREAMED_MAX_BODY_BYTES_CEILING`), which is also `MODULE_ROUTE_ABSOLUTE_MAX_BODY_BYTES` |
|
||||
|
||||
A declared `Content-Length` over the limit is refused synchronously, before authorization and before
|
||||
a single body byte is read. Beyond that the two module modes differ only in *when* the platform
|
||||
counts ([ADR 0042](../architecture/decisions/0042-bounded-streaming-request-bodies.md)):
|
||||
|
||||
- **`buffered`** materializes the body before the handler runs, counting as it arrives, so a lying or
|
||||
absent `Content-Length` is bounded by what actually arrived. The handler is promised a fully
|
||||
available body within its declared allowance, and the platform holds at most that allowance.
|
||||
- **`streamed`** never materializes the body: it is counted as the handler consumes it, and the
|
||||
moment the total passes the limit the stream errors, the source is cancelled, and the response
|
||||
becomes the same `413` — even if the handler caught the error and answered anyway. A `413` can
|
||||
therefore arrive *after* a streamed handler has started, which is the one consequence a module
|
||||
author must design for.
|
||||
|
||||
An oversized request always receives the identical stable, sanitized contract —
|
||||
`413 {"code":"REQUEST_BODY_TOO_LARGE"}` with `Cache-Control: no-store` — and never invokes Better
|
||||
@@ -425,10 +437,20 @@ Auth's business logic, a Core mutation, permission evaluation (where avoidable),
|
||||
or module database behavior. For a state-changing route the body limit is enforced after
|
||||
`requireTrustedOrigin` but before permission evaluation, so a rejected cross-origin mutation's body
|
||||
is never read for size at all, while an oversized same-origin request still fails before any
|
||||
authorization work runs. See [Module HTTP routes](#module-http-routes) for the module-route body
|
||||
contract itself, and [Module development](../module-development/README.md#http-routes) for the
|
||||
module-authoring side of it. Large media transfer is explicitly out of scope for ordinary module
|
||||
HTTP bodies — that belongs to a future dedicated storage design, never to widening this ceiling.
|
||||
authorization work runs. In `streamed` mode no body byte is read until the handler reads it, and the
|
||||
handler runs only after authorization succeeds, so an unauthorized caller can never push an upload
|
||||
through the platform at all.
|
||||
|
||||
Body reading is bounded in time by the server's own `headersTimeout`/`requestTimeout` below, not by a
|
||||
second per-route timeout. A body no handler read is cancelled after the request, so an abandoned
|
||||
upload releases its socket rather than draining; an aborted upload surfaces as an ordinary failed
|
||||
request rather than a `413`, because nothing exceeded a limit.
|
||||
|
||||
See [Module HTTP routes](#module-http-routes) for the module-route body contract itself, and
|
||||
[Module development](../module-development/README.md#http-routes) for the module-authoring side of
|
||||
it. The absolute ceiling is one image-sized file per request: aggregate uploads and anything larger
|
||||
than 32 MiB remain out of scope, and signed write URLs are still deliberately not offered
|
||||
([ADR 0031](../architecture/decisions/0031-direct-object-delivery.md)).
|
||||
|
||||
## Node request and header timeouts
|
||||
|
||||
|
||||
@@ -1005,8 +1005,8 @@ export default defineModuleRuntime({
|
||||
});
|
||||
```
|
||||
|
||||
A route definition has exactly five fields — `method`, `path`, `permission`, `maxBodyBytes`,
|
||||
`handler` — and nothing else; an unknown field (a baseline, a mount prefix, CORS/origin policy,
|
||||
A route definition has exactly five required fields — `method`, `path`, `permission`,
|
||||
`maxBodyBytes`, `handler` — plus one optional `bodyMode`, and nothing else; an unknown field (a baseline, a mount prefix, CORS/origin policy,
|
||||
middleware, audit metadata, authentication behavior, or another module's namespace) fails
|
||||
registration outright rather than being ignored. `method` is one of `GET`, `POST`, `PUT`, `PATCH`,
|
||||
`DELETE`, always uppercase; there is no `OPTIONS` (the platform owns CORS preflight) and no
|
||||
@@ -1071,15 +1071,61 @@ ones. The platform continues to own authentication, authorization, `Origin`, COR
|
||||
cache behavior, and uncaught-handler sanitization, and permission enforcement always completes
|
||||
before your handler runs — an unauthorized caller can never reach your validation at all.
|
||||
|
||||
### Request bodies: `maxBodyBytes` and `bodyMode`
|
||||
|
||||
`maxBodyBytes` is a conscious, required request-body contract, not an optional or defaulted field:
|
||||
`0` means the route accepts no body at all, and a non-empty declared (`Content-Length`) or streamed
|
||||
body is rejected before the handler ever runs; any other value is the exact maximum number of bytes
|
||||
accepted. It must be a non-negative integer no greater than the platform ceiling,
|
||||
`MODULE_ROUTE_MAX_BODY_BYTES_CEILING` (`@ekklesia/contracts`, currently 1 MiB) — a negative,
|
||||
non-integer, non-finite, or over-ceiling value fails registration with
|
||||
`MODULE_ROUTE_MAX_BODY_BYTES_INVALID`. The platform enforces this limit itself, ahead of permission
|
||||
evaluation and the handler, using the identical sanitized `413 {"code":"REQUEST_BODY_TOO_LARGE"}`
|
||||
contract every other body-consuming route in the platform uses — see
|
||||
accepted. A negative, non-integer, non-finite, or over-ceiling value fails registration with
|
||||
`MODULE_ROUTE_MAX_BODY_BYTES_INVALID`.
|
||||
|
||||
`bodyMode` chooses how that body reaches you, and therefore which ceiling applies
|
||||
([ADR 0042](../architecture/decisions/0042-bounded-streaming-request-bodies.md)):
|
||||
|
||||
| `bodyMode` | Ceiling | What the platform does |
|
||||
| --- | --- | --- |
|
||||
| `"buffered"` (default) | 1 MiB | Reads and counts the whole body **before** your handler runs |
|
||||
| `"streamed"` | 32 MiB | Counts bytes **as you consume them**, and never materializes the body |
|
||||
|
||||
```ts
|
||||
http.route({
|
||||
method: "POST",
|
||||
path: "/slides",
|
||||
permission: "signage.slides.write",
|
||||
maxBodyBytes: 33_554_432,
|
||||
bodyMode: "streamed",
|
||||
async handler(request, context) { /* read request.body as a stream */ },
|
||||
});
|
||||
```
|
||||
|
||||
Omitting `bodyMode` means `buffered`, which is what every route written before this existed meant.
|
||||
|
||||
**Memory.** In `buffered` mode the platform holds at most your declared allowance, which is why that
|
||||
ceiling is small and why `await request.json()` is safe there. In `streamed` mode the platform holds
|
||||
nothing: read `request.body` as a stream. Calling `request.arrayBuffer()` on a streamed route works
|
||||
but throws away the entire point — you have just buffered 32 MiB yourself.
|
||||
|
||||
**Streaming.** A declared `Content-Length` over your limit is refused synchronously, before
|
||||
authorization and before a byte is read. Otherwise the platform counts what actually arrives, so a
|
||||
misleading or absent `Content-Length` bounds nothing.
|
||||
|
||||
**A `413` can arrive after your handler starts.** This is the one consequence of `streamed` mode you
|
||||
must design for. When the running total passes your limit the stream errors, the source is cancelled,
|
||||
and the platform replaces your response with the sanitized `413` — even if you caught the error and
|
||||
answered anyway. If you persist as you read, write to a temporary key and commit only once the read
|
||||
completes; the platform cannot do that for you, because only your module knows what a partial write
|
||||
means to its own data.
|
||||
|
||||
**Timeouts.** Body reading is bounded by the server's own header and request timeouts, not by a
|
||||
per-route setting. A slow upload is a slow request, and the platform has one answer for that.
|
||||
|
||||
**One file per request.** 32 MiB is the platform's absolute ceiling in every mode — there is no
|
||||
unbounded option anywhere in the contract. A UI uploading twenty images issues twenty individually
|
||||
bounded requests; nothing here can express a 500 MiB aggregate upload, deliberately, because one bad
|
||||
byte in one would waste everything already sent.
|
||||
|
||||
Every rejection uses the identical sanitized `413 {"code":"REQUEST_BODY_TOO_LARGE"}` contract every
|
||||
other body-consuming route in the platform uses — see
|
||||
[API](../development/api.md#request-body-limits).
|
||||
|
||||
`path` is local to the module, never the path a caller actually sends. The platform alone owns the
|
||||
|
||||
Reference in New Issue
Block a user