fix(display): retry every failed request with backoff, and never give up

A display tried each of its network paths once:
- A registration that failed left it continuing from its persisted
  state, behind the offline banner, with no realtime, heartbeat or
  retry, until someone reloaded it. This is what the production kiosk
  showed after a restart.
- A failed state refresh scheduled no further refresh, so a display
  whose realtime was also down never fetched state again.
- A realtime stream the host reported closed was never subscribed
  again.

Each path now retries on its own backoff: about 1 s, doubling to a
one-minute ceiling, with jitter so a building's screens do not return
in lockstep. There is no attempt limit, because a television has
nobody to press Retry. Each resets when its request works again.
- Registration retries until it works, then brings the display live
  exactly as a first-time success does.
- A failed refresh schedules its own retry, with no realtime needed.
- A closed realtime subscription is replaced.
- Stopping clears every retry.

The recovery test's virtual window shrinks from an hour to five
minutes: eight failures back off for at most about 183 s, and an hour
of offline-continuation timers made the test slow without proving more.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
ekklesia
2026-10-05 21:31:55 -05:00
co-authored by Claude Opus 5.5
parent 518fd101a2
commit fbc8af7510
4 changed files with 184 additions and 44 deletions
+21
View File
@@ -0,0 +1,21 @@
/**
* How long a display waits before trying a failed request again.
*
* Exponential from one second to a one-minute ceiling, so a screen whose server is restarting is
* back within seconds, and one whose server is down for an evening asks once a minute rather than
* hammering it. The wait is jittered between half and all of that step, so a building's worth of
* screens that lost the server at the same moment do not all come back at the same moment.
*
* There is no attempt limit. A television on a wall has nobody to press Retry, so giving up is the
* one thing it must never do.
*/
export const DISPLAY_RETRY_BASE_MS = 1_000;
export const DISPLAY_RETRY_MAX_MS = 60_000;
export function displayRetryDelayMs(attempt: number, random: () => number = Math.random): number {
const step = Math.min(
DISPLAY_RETRY_MAX_MS,
DISPLAY_RETRY_BASE_MS * 2 ** Math.min(Math.max(0, attempt), 16),
);
return Math.round(step / 2 + (step / 2) * Math.min(1, Math.max(0, random())));
}