feat: add native module realtime transport
CI / verify (pull_request) Successful in 37s
CI / infrastructure-smoke (pull_request) Successful in 13s
CI / migration-smoke (pull_request) Successful in 1m7s

This commit is contained in:
ekklesia
2026-08-23 18:47:18 -05:00
parent fcc531a39c
commit fc6c8045c4
53 changed files with 3959 additions and 108 deletions
@@ -74,6 +74,12 @@ administrator owns rather than something a module declares. A display field adde
become module-owned navigation before that design exists, so the temporary selector renders module
ids instead.
> **Cross-reference — see [ADR 0032](0032-native-module-realtime-transport.md).**
> Phase 7A pass 5 extends this original descriptor with a sixth,
> `realtime: { baseUrl: "/api/v1/module-realtime/example" }`, so the browser host can construct the
> transport-neutral `context.realtime` facade without inventing a platform URL. The five-field shape
> above remains the historical decision this ADR recorded; ADR 0032 owns the extension.
It is not a module management API. There is no create, update, enable, disable, install, or
configure, and no module that declares no UI. Ekklesia has no enable/disable state to report —
`ModuleLifecycleStage` remains vocabulary with no runtime state machine behind it — and reporting one
@@ -210,5 +210,6 @@ value it could branch on.
Signed **write** and **delete** URLs are not built. Uploads still go through a module route, where
the module can validate what it is accepting; handing a browser a capability to write into a
module's bucket needs its own decision about size, type, and quota, and does not belong in the pass
that established reads. Realtime URL refresh is not built either — ordinary API reloads are
sufficient, and a future realtime pass can refresh state and URLs together when it exists.
that established reads. [ADR 0032](0032-native-module-realtime-transport.md) now supplies the native
state channel on which a module can publish a replacement delivery URL. Core still does not inspect,
renew, or automatically publish expiring URLs; that refresh policy remains explicitly module-owned.
@@ -0,0 +1,204 @@
# 0032: Native module realtime is scoped snapshot state over SSE
- Status: Accepted
- Date: 2026-08-23
- Extends: [0006](0006-versioned-capabilities-and-compiled-module-runtime.md),
[0013](0013-module-permission-contract.md),
[0015](0015-module-http-transport.md),
[0021](0021-modules-as-hostable-products.md),
[0022](0022-module-ui-authoring-and-host-contract.md),
[0028](0028-module-operational-lifecycle.md),
[0031](0031-direct-object-delivery.md)
## Context
Ekklesia has a host-neutral module HTTP contract, but no way for a running module to deliver a state
change to its browser UI without polling or owning a second server. The first real consumer is
`The_River/Announcements` at commit `9aab486`: its server sends one complete `slideshow:state`
snapshot on connection and broadcasts the same complete shape after a slide tick, slide-list reload,
or mode change. The browser sends no application messages back, uses no acknowledgements, and on
reconnection needs current state rather than a history of missed transitions.
That application currently uses Socket.IO, hard-coded LAN origins, and MQTT mode control. Those are
properties of the old deployment, not requirements for a module contract. Browser commands already
have the authenticated, CSRF-protected module HTTP path from ADR 0015. MQTT is a legacy external
control integration that will be removed; it is not an Ekklesia internal transport.
The existing Node adapter could be extended with a WebSocket server and upgrade handling, but the API
does not have any of that wiring today. Its public module vocabulary is Fetch `Request`/`Response`,
and it already preserves streaming response bodies. Server-Sent Events fit that boundary without a
new bidirectional protocol, dependency, upgrade/authentication path, reconnect implementation, or
shutdown stack.
## Decision
### The public contract names state, not a network protocol
A module declares a small number of local realtime channels during `register`, each protected by one
permission its own manifest declares:
```ts
context.realtime.channel({
channel: "state",
permission: "announcements.slideshow.view",
});
```
Its `start` hook, running work, and HTTP handlers receive the same module-scoped publisher:
```ts
context.realtime.publish("state", slideshowState);
```
The module does not pass a module id, origin, URL, transport, credential, or permission when it
publishes. It cannot name another module's channel through this facade. `channel` is one validated
lowercase kebab-case identifier, not a hierarchical topic: there are no slashes, wildcards, global
namespace, arbitrary cross-module subscription, or broker semantics.
Registration remains declarative and I/O-free. Publication is a live operation and is therefore
absent from registration, exactly like database and storage access. HTTP handlers receive the
publisher because an ordinary authenticated command frequently changes the state that should be
shown immediately; modules must not capture a late-bound lifecycle context to achieve that.
`stop` receives the same context shape for lifecycle consistency, but its publisher has already been
made inert: stopped code cannot emit new live state.
The first Ekklesia host implementation is an authenticated `GET` SSE stream. SSE is private host
machinery: neither the server module contract nor `ModuleUiHostContext` names SSE, EventSource,
WebSocket, or Socket.IO. A standalone host may implement the exact contracts using another transport
without changing a module.
WebSocket is rejected for this slice. The real consumer has no persistent client-to-server messages,
while commands already fit normal module HTTP. Choosing WebSocket would add an upgrade-specific auth
path, ping/pong, reconnect wrapper, dependency, server wiring, and shutdown behavior without buying a
required capability. It can be reconsidered if a real module demonstrates genuine persistent
bidirectional traffic; that would be a new transport decision, not an invisible change here.
### Snapshot first, then latest-state updates
Every declared channel must publish one authoritative state value before its module's `start` hook
completes. If it does not, startup fails and the existing reverse-order rollback runs. The API is not
bound until every eligible module starts, so every reachable channel necessarily has a snapshot.
Core validates and serializes a publication once, assigns the channel's next monotonically increasing
process-local sequence, and stores that exact serialized value as its latest snapshot. Subscription
atomically registers the consumer and captures that snapshot, eliminating a separate `GET state` then
`subscribe` race. A new connection always receives `snapshot` first and then later `update` values.
The sequence detects ordering, duplicates, and gaps; it is not a durable cursor. It may reset when the
process restarts. The transport deliberately does not promise replay, history, exactly-once delivery,
or recovery through `Last-Event-ID`. A reconnect is a new authorized subscription and receives the
fresh latest snapshot, followed by future updates. The module remains the semantic authority and
reconstructs its state from its own persistence/runtime after restart; Core's in-memory copy is only
ephemeral transport state.
### JSON and resource bounds
Payloads are JSON-compatible data only. Core rejects functions, symbols, bigints, cycles, non-finite
numbers, unsupported object values, and anything whose UTF-8 serialized representation exceeds the
fixed platform ceiling. The byte budget is consumed while traversing strings, keys, containers, and
values, before building or stringifying an arbitrarily large clone. The first ceiling is 64 KiB per
publication: far beyond an Announcements state document, while making it unambiguous that media bytes
belong in object storage.
The channel count, subscribers per channel, and per-client pending work are all finite. Publishing is
synchronous and never awaits a browser. A consumer receives publications in increasing sequence while
it keeps up; if it is slow, it retains at most the newest pending state and intermediate updates are
coalesced. Sequence gaps make that behavior visible. This is correct for authoritative state and
prevents one display from blocking a module or creating an unbounded queue.
### Browser host contract
`ModuleUiHostContext` exposes a transport-neutral, module-scoped subscription method. Module UI names
only its local channel and receives snapshot/update/status callbacks plus an idempotent close handle.
The browser does not supply a module id, permission, origin, URL, cookie, or bearer token.
Ekklesia's server adds a root-relative realtime base to the existing module UI mount descriptor, just
as it already supplies the module HTTP base. The shell strictly validates that root-relative value,
resolves it against the configured API origin, appends only the validated local channel, and opens
native `EventSource` with credentials enabled. A standalone host supplies its own root and transport
adapter behind the same browser contract.
The browser reports only stable transport-neutral states (`connecting`, `connected`, `reconnecting`,
`closed`) and fixed safe errors. A recoverable EventSource transport error reports `reconnecting` and
uses the browser's native retry; a terminal `CLOSED` state removes every listener and reports
`closed` instead of promising a retry that will never occur. It does not expose EventSource objects,
raw server responses, socket exceptions, or URLs to module UI. This is a portability and correctness
boundary, not a sandbox: installed UI is trusted same-realm JavaScript and could use browser globals
directly. Server-side permission enforcement remains the security authority.
### Authentication and authorization
The SSE request uses the existing session cookie. Ekklesia evaluates the normal principal and the
channel's registered permission before allocating a subscriber, using the same permission registry
and evaluator as module HTTP. There is no query-string token and no second realtime auth system.
Unknown, disabled, blocked, stopped, or undeclared channels have no mounted route.
Authorization is checked when a stream is established. Ekklesia closes every stream after five
minutes so EventSource reconnects and authorization is re-evaluated; changing a grant does not
otherwise reach backward into an already-established connection. A deliberate public
display/device principal remains deferred. This transport is proven only with existing authenticated
principals and permission checks, and no channel becomes public as a shortcut.
### Lifecycle and cleanup
Only the exact eligible module set that successfully starts publishes channels. Disablement and
dependency blocking therefore remove realtime the same structural way they remove HTTP and UI: the
module is never imported or mounted. There is no hot unload; lifecycle changes remain
restart-required.
Before normal stop or rollback invokes a module's stop hook, Core deactivates that module's publisher.
Deactivation happens per module in reverse dependency order: a dependency remains operational while
its dependents finish stopping, but a stale captured publisher fails with a fixed runtime error once
its own stop begins. Browser cancellation, transport failure, module stop, startup rollback, server
shutdown, and connection-age expiry all remove subscriptions idempotently.
Server shutdown closes realtime streams before draining the HTTP listener. Otherwise a healthy
long-lived stream would keep the drain open until its deadline and turn ordinary shutdown into a
forced failure. Publishers remain active for finite requests already in flight; module stop hooks run
only after the listener has drained, and Core then deactivates each publisher immediately before its
own hook. This preserves both a command's publish-after-commit path and the guarantee that no ordinary
request handler runs concurrently with stop.
### Proxy and keepalive behavior
Ekklesia sends `text/event-stream`, `Cache-Control: no-store`, disables common reverse-proxy response
buffering where supported, and emits host-owned comment keepalives while a stream is idle. Keepalives
do not enter module code, consume a module sequence, or mutate state. Deployments and reverse proxies
must pass streaming responses incrementally and allow the configured idle interval; there is no
Caddy- or Cloudflare-specific API in the contract.
### Object delivery is unchanged
Realtime carries small state, logical identifiers, metadata, and — when a module intentionally
includes one — an opaque delivery URL. It never carries image, video, or audio bytes. Core transports
the module's serialized payload unchanged; it does not inspect object keys or call `createReadUrl` for
each publication, subscriber, reconnect, or keepalive.
That preserves ADR 0031's byte-stable/quantized URLs and browser cache keys. A module deliberately
refreshes an expiring URL by creating it once and publishing new state. Realtime never re-signs an
unchanged URL behind the module's back.
### Realtime is not Companion's global bus
A future Companion module may observe deliberately exported module capabilities and invoke explicit
module actions, and may bridge supported external protocols. It does not gain privileged access to
every module's browser stream. Browser realtime and external automation/control are separate
concerns; this contract has no cross-module subscription seam to repurpose.
## Consequences
The first real state consumer gets immediate useful state, ordered live updates, automatic browser
reconnect, permission enforcement, bounded memory, and deterministic cleanup without owning a port or
transport server. Modules with no realtime declarations behave exactly as before.
The contract deliberately loses intermediate states for a slow or disconnected client. Modules must
publish authoritative state, not use this channel as an audit log, work queue, payment ledger, or sole
record of a domain transition. A future event-oriented or durable facility would be a separate design.
SSE is one-way. Browser commands remain ordinary authenticated module HTTP, retaining its existing
body limits, trusted-origin checks, and request/response semantics. A demonstrated bidirectional need
may justify WebSocket later, without renaming the module-facing API.
This slice does not add the Announcements module, MQTT, Companion, a kiosk/display host, a public or
device principal, a standalone host, hot lifecycle changes, multi-node fanout, durable history, or a
message broker.
+1
View File
@@ -40,3 +40,4 @@ ADRs capture durable choices that affect more than one package or phase.
- [0029: Module configuration is manifest-declared, host-resolved, and secret-free at rest](0029-module-configuration-and-secrets.md) — Accepted
- [0030: Modules declare logical buckets; hosts provide object storage](0030-host-provided-module-object-storage.md) — Accepted
- [0031: Browsers fetch objects directly, through opaque temporary read URLs](0031-direct-object-delivery.md) — Accepted
- [0032: Native module realtime is scoped snapshot state over SSE](0032-native-module-realtime-transport.md) — Accepted
+44 -11
View File
@@ -70,7 +70,7 @@ Every module-facing contract in this platform is written so that **full Ekklesia
thing that can satisfy it**. The intended second implementation is a reusable single-module
standalone host — conceptually `@ekklesia/standalone-host` — which loads exactly one module and
supplies the same host contracts Ekklesia does: configuration, secrets, object storage, HTTP, module
lifecycle, and later realtime and display/UI hosting.
lifecycle, and realtime, with display/kiosk hosting still later.
**No such package exists yet, and this pass does not create one.** It is recorded here because it is
what several contract decisions are *for*, and because a change that quietly makes it unimplementable
@@ -84,6 +84,12 @@ the scoped facade, binding resolution, delivery URLs and their bounds, lifecycle
host-neutral, and a standalone host implements one interface to get all of it, including direct
browser delivery ([ADR 0031](decisions/0031-direct-object-delivery.md)).
Realtime is the second worked example. A module declares local state channels and publishes opaque
JSON through scoped contracts; its browser UI subscribes through another scoped contract. Neither
side names SSE, EventSource, a platform URL, an origin, a cookie, or a module namespace. Ekklesia's
host adapter uses permission-gated SSE, while a standalone host may choose another concrete
transport without changing the module. See [ADR 0032](decisions/0032-native-module-realtime-transport.md).
### Storage servers are shared infrastructure, never per-module
- A module hosted under Ekklesia uses **Ekklesia's** storage service, through its own bindings.
@@ -280,17 +286,17 @@ by its own esbuild script, that discovers installed module UIs and mounts them.
The missing piece was discovery. `GET /api/v1/module-ui` — the collection root of the same prefix the
artifact route already owns — returns, per publishable module, its id and version, the server-produced
artifact URL, the server-produced root of the module's own API, and the subset of that module's
declared permissions the caller actually holds. Nothing else: no filesystem paths, and no display or
navigation metadata, since ADR 0021 keeps Dashboard grouping an administrator's presentation
configuration. The module set is the publication registry, never a rescan, so the inventory and the
artifact route cannot disagree. It requires `core.modules.ui.read` (baseline `authenticated`) because
it is precisely the enumeration ADR 0025 withheld from the anonymous artifact route.
artifact URL, the server-produced roots of the module's own HTTP and realtime APIs, and the subset of
that module's declared permissions the caller actually holds. Nothing else: no filesystem paths, and
no display or navigation metadata, since ADR 0021 keeps Dashboard grouping an administrator's
presentation configuration. The module set is the publication registry, never a rescan, so the
inventory and artifact route cannot disagree. It requires `core.modules.ui.read` (baseline
`authenticated`) because it is precisely the enumeration ADR 0025 withheld from the anonymous
artifact route. ADR 0032 later extended ADR 0026's original descriptor with the realtime root.
Both URLs being server-produced is the load-bearing part. `moduleUiArtifactUrl` and the newly exported
`moduleApiBaseUrl` are each the single construction of their URL shape, so the browser is a consumer
of both — it never rebuilds `/api/v1/modules/<id>`, the prefix ADR 0022 forbids a module UI from
spelling, and neither prefix is pinned by having a second author.
All three URLs being server-produced is load-bearing. `moduleUiArtifactUrl`, `moduleApiBaseUrl`, and
`moduleRealtimeBaseUrl` are each the single construction of their URL shape, so the browser consumes
them rather than rebuilding a platform module prefix, and none is pinned by having a second author.
The shell builds a `ModuleUiHostContext` entirely from server-derived data, resolves a module's
relative API calls onto its own API root (refusing anything that escapes it), and hands the artifact
@@ -438,3 +444,30 @@ configures nothing. A reverse proxy in front of storage must pass the original `
unchanged; it may terminate TLS, because the scheme is not signed. Streaming through a module's own
route remains available for responses that must be computed or authorized per byte, and signed
**write** URLs are deliberately not offered.
Phase 7A pass 5 adds native module realtime
([ADR 0032](decisions/0032-native-module-realtime-transport.md)). Registration declares a bounded,
module-local channel and one permission owned by that module; live runtime and HTTP contexts receive
only that module's publisher. Every channel must publish an authoritative initial state before
startup completes. Core serializes opaque JSON once, assigns a process-local per-channel sequence,
and atomically installs each subscriber with the latest snapshot, eliminating a GET-then-subscribe
race. It retains only the latest channel state and one coalescing pending update per client, so a
slow browser never blocks module work or creates an unbounded queue. There is no history, replay,
broker namespace, or durable state.
Ekklesia's concrete host transport is one-way Server-Sent Events because the demonstrated
Announcements consumer sends complete state server-to-browser while browser commands already fit
authenticated module HTTP. The channel permission is evaluated through the existing cookie
principal and authorization system before allocation; the host closes streams after five minutes so
reconnect re-evaluates that authority. Native EventSource owns recoverable reconnects, each of which
starts with a fresh snapshot. The wire sends 15-second comment keepalives, disables common proxy
buffering, carries no query token or replay id, and never exposes transport objects through the
module UI SDK.
Shutdown closes long-lived subscriptions before finite HTTP drain, while leaving publishers usable
for requests already in flight. Reverse-order stop then deactivates each module's publisher
immediately before that module's own stop hook, so dependencies remain operational while dependents
clean up and captured publishers cannot outlive their module. Direct object-delivery URLs remain
opaque payload strings and are never re-signed per publication or subscriber. MQTT remains an
external-control concern for a future Companion boundary, not an internal module bus. Display/kiosk
hosting and its public/device principal are explicitly not part of this pass.