fix(display): retry every failed request with backoff, and never give up #5

2026-10-05T23:36:00Z

ekklesia-bot commented

2026-10-05 23:36:00 +00:00

fix(display): retry every failed request with backoff, and never give up

The production screen symptom.

After a kiosk restart the display mounted and showed "Offline — continuing from the last update" indefinitely.
No heartbeat reached the server, and nothing retried.
Cause: start() tried POST viewers/session exactly once. When that request failed (a boot-time network blip, an API restart), the display never subscribed, refreshed, heartbeated or retried until it was reloaded.

Reproduced in a real browser by refusing that one request (Ekklesia draft PR #56, scenario session-blip). The display stayed mounted, data-offline="true", with zero heartbeats and zero state fetches for the full 90 s.

Every network path had the same dead end

Path	On `main`	Now
`POST viewers/session`	tried once	retried until it works, then live exactly as a first-time success
`GET …/display-state`	a failed refresh scheduled no further refresh, so with realtime also down the display never fetched again	schedules its own retry, no realtime needed
realtime `display-state`	a subscription the host reports `closed` (a refused stream) was never replaced	resubscribed

Backoff (src/browser/backoff.ts):

about 1 s, doubling to a one-minute ceiling;
jittered between half and all of each step, so a building's screens do not come back in lockstep;
no attempt limit, because a television has nobody to press Retry;
each path resets once its request works;
stop() clears every retry.

The platform's own host will also retry closed realtime streams (Ekklesia PR 7). This module-side resubscribe is still correct after that, because it acts only on a terminal closed.

Commits

Failing test first: tests/browser/display-recovery.test.ts, on a virtual clock. On main, all three paths fail: one request each, never retried.
The fix, plus tests/browser/backoff.test.ts. The recovery test's virtual window shrinks from an hour to five minutes: eight failures back off for at most about 183 s, and an hour of offline-continuation timers only made the test slow.

Verification

Signage has no CI. Run locally in an Ekklesia main (= v0.1.1) workspace: test 206 passed, plus typecheck and lint, all clean.

Rebased onto `main` after #4

The one conflict in start() is resolved by keeping register(), with #4's parsed session response carried into it. A registration the display can't read is treated like any failed attempt and retried.

A third commit adds session-blip and stream-failure to ci/display-scenarios. Both failed on main, so CI now proves recovery end to end in a real browser.

Local checks, in the CI layout: test 228, typecheck, lint, all pass.

## fix(display): retry every failed request with backoff, and never give up **The production screen symptom.** - After a kiosk restart the display mounted and showed "Offline — continuing from the last update" indefinitely. - No heartbeat reached the server, and nothing retried. - Cause: `start()` tried `POST viewers/session` exactly once. When that request failed (a boot-time network blip, an API restart), the display never subscribed, refreshed, heartbeated or retried until it was reloaded. Reproduced in a real browser by refusing that one request (Ekklesia draft PR #56, scenario `session-blip`). The display stayed mounted, `data-offline="true"`, with zero heartbeats and zero state fetches for the full 90 s. ### Every network path had the same dead end | Path | On `main` | Now | |---|---|---| | `POST viewers/session` | tried once | retried until it works, then live exactly as a first-time success | | `GET …/display-state` | a failed refresh scheduled no further refresh, so with realtime also down the display never fetched again | schedules its own retry, no realtime needed | | realtime `display-state` | a subscription the host reports `closed` (a refused stream) was never replaced | resubscribed | **Backoff** (`src/browser/backoff.ts`): - about 1 s, doubling to a one-minute ceiling; - jittered between half and all of each step, so a building's screens do not come back in lockstep; - **no attempt limit**, because a television has nobody to press Retry; - each path resets once its request works; - `stop()` clears every retry. The platform's own host will also retry closed realtime streams (Ekklesia PR 7). This module-side resubscribe is still correct after that, because it acts only on a terminal `closed`. ### Commits 1. **Failing test first:** `tests/browser/display-recovery.test.ts`, on a virtual clock. On `main`, all three paths fail: one request each, never retried. 2. **The fix**, plus `tests/browser/backoff.test.ts`. The recovery test's virtual window shrinks from an hour to five minutes: eight failures back off for at most about 183 s, and an hour of offline-continuation timers only made the test slow. ### Verification Signage has no CI. Run locally in an Ekklesia `main` (= v0.1.1) workspace: `test` 206 passed, plus `typecheck` and `lint`, all clean. ### Rebased onto `main` after #4 The one conflict in `start()` is resolved by keeping `register()`, with #4's **parsed** session response carried into it. A registration the display can't read is treated like any failed attempt and retried. A third commit adds `session-blip` and `stream-failure` to `ci/display-scenarios`. Both failed on `main`, so CI now proves recovery end to end in a real browser. Local checks, in the CI layout: test 228, typecheck, lint, all pass.

ekklesia-bot referenced this pull request

2026-10-05 23:37:47 +00:00

fix(display): keep one object URL per cached slide, and give them back #6

ekklesia-bot referenced this pull request

2026-10-06 00:12:11 +00:00

ci: build, test, and run Signage inside a pinned Ekklesia on every change #7

Zayin_Admin added 3 commits 2026-10-06 02:32:25 +00:00

test(display): a display recovers from every failed request without a reload 518fd101a2

Pins what a kiosk needs from each network path, on a virtual clock: a
registration that fails once is retried with backoff until it works, a
state refresh that fails is retried without a realtime nudge, and a
realtime stream the host reports closed is subscribed again. Nothing is
scheduled after stop.

Fails today on all three paths: each is tried once. The first matches
the production screen exactly: mounted, showing "Offline — continuing
from the last update", with no heartbeat and no retry until someone
reloads it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(display): retry every failed request with backoff, and never give up fbc8af7510

A display tried each of its network paths once:
- A registration that failed left it continuing from its persisted
  state, behind the offline banner, with no realtime, heartbeat or
  retry, until someone reloaded it. This is what the production kiosk
  showed after a restart.
- A failed state refresh scheduled no further refresh, so a display
  whose realtime was also down never fetched state again.
- A realtime stream the host reported closed was never subscribed
  again.

Each path now retries on its own backoff: about 1 s, doubling to a
one-minute ceiling, with jitter so a building's screens do not return
in lockstep. There is no attempt limit, because a television has
nobody to press Retry. Each resets when its request works again.
- Registration retries until it works, then brings the display live
  exactly as a first-time success does.
- A failed refresh schedules its own retry, with no realtime needed.
- A closed realtime subscription is replaced.
- Stopping clears every retry.

The recovery test's virtual window shrinks from an hour to five
minutes: eight failures back off for at most about 183 s, and an hour
of offline-continuation timers made the test slow without proving more.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

ci: run session-blip and stream-failure on every change

CI / lint (pull_request) Successful in 12s

Details

CI / typecheck (pull_request) Successful in 14s

Details

CI / test (pull_request) Successful in 22s

Details

CI / build (pull_request) Successful in 18s

Details

CI / integration (pull_request) Successful in 3m40s

Details

bf4e98fa30

Both failed on main before this PR: a display that failed its first
viewer-session request, or its first realtime connection, never
recovered without a reload. With the retries in place they pass, and
CI now holds them there.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Zayin_Admin force-pushed fix/display-recovery from d01ffb13e6 to bf4e98fa30

2026-10-06 02:32:25 +00:00

Compare

Zayin_Admin approved these changes

2026-10-06 02:44:16 +00:00

Zayin_Admin merged commit ab4142fb95 into main

2026-10-06 02:44:20 +00:00

Zayin_Admin deleted branch fix/display-recovery

2026-10-06 02:44:20 +00:00

Zayin_Admin referenced this issue from a commit

2026-10-06 03:08:34 +00:00

chore: release signage 0.1.2