Complete Phase 5B offline display resilience
Make the /display kiosk survive losing its infrastructure, then surrender authority cleanly when it returns. The Phase 5A renderer is unchanged: this builds on its two permanent scene layers, explicit Cache Storage, IndexedDB, Service Worker, timing anchors, and structured errors. While connected the server is authoritative. While disconnected the display becomes a deterministic local continuation of the last authoritative state it persisted, and the moment the server speaks again the local cursor is abandoned rather than reconciled or reported back. A small client evaluator resolves offline playback from the persisted manifest and clock anchor. Catch-up is arithmetic rather than iterative: whole rotations are skipped between eligibility boundaries, so a month offline resolves in milliseconds. Mixed durations, wraparound, [start, end) schedules, and expirations are evaluated locally, and a null expiration stays permanent. Mode off keeps the hidden cursor advancing, pause holds its slide and remainder, static holds until the cached metadata says the asset genuinely expired, and an activation queued for an unknown playlist holds the last good frame instead of continuing into a playlist that is being replaced. Contract tests run this evaluator and the real Phase 3B engine over identical scenarios and assert the same slide, start, and deadline, so the client tracks the server without refactoring it. Cold start restores signage from the Service Worker shell, IndexedDB, and Cache Storage with the entire stack down, keeping the same persistent display identity. Validated display error codes now become Home Assistant incidents through the Phase 4 machinery, keyed per display and condition so a persisting error reports once and recovers once. Arbitrary browser text stays telemetry. Corrects the deployment assumption: the kiosk runs on a separate machine, so production requires a trusted HTTPS origin and localhost is development only. Verifying on a real trusted certificate exposed two production bugs that localhost had hidden — behind a reverse proxy an outage arrives as a 502 rather than a refused connection, and an unbounded shell fetch left a restarting kiosk blank for the proxy's whole connect timeout. The worker now treats a gateway error as an outage and bounds that wait. Verified with 200/200 backend, 20 client unit, and 30 Playwright tests on a trusted HTTPS origin with no browser security flags weakened. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Media
@@ -369,18 +369,74 @@ direct navigation and refresh through the existing Nginx SPA fallback. The
|
||||
display stays a Chromium kiosk pointed at a web application: v2 makes that
|
||||
application reliable rather than replacing it with a native client.
|
||||
|
||||
Chromium should open `/display` in kiosk mode against an origin the browser
|
||||
treats as secure — `http://localhost`, a private hostname over HTTPS, or the
|
||||
equivalent. Cache Storage, Service Workers, and therefore offline behaviour are
|
||||
unavailable on an insecure origin, and the display degrades to a network-only
|
||||
renderer without them.
|
||||
### Deployment topology
|
||||
|
||||
The kiosk runs on a different computer from the Announcements host, so the
|
||||
localhost secure-context exception is not available in production. A remote
|
||||
display **must** use HTTPS:
|
||||
|
||||
```
|
||||
Remote Chromium kiosk
|
||||
|
|
||||
HTTPS
|
||||
|
|
||||
v
|
||||
Announcements web / reverse proxy
|
||||
|
|
||||
+-- React /display
|
||||
+-- /api
|
||||
+-- /socket.io
|
||||
+-- /media
|
||||
```
|
||||
|
||||
The reverse proxy terminates TLS; the API and web containers may keep speaking
|
||||
plain HTTP behind it. The kiosk browser must trust the certificate — a public
|
||||
certificate or an internal CA installed on the kiosk machine both work.
|
||||
|
||||
Secure-context rules decide what the display can do:
|
||||
|
||||
| Origin | Status |
|
||||
| --- | --- |
|
||||
| `https://display.example.org/display` | Production. Full offline capability. |
|
||||
| `http://localhost:5175/display` | Development and automated testing only. Secure context, so offline works. |
|
||||
| `http://10.0.0.5:5175/display` | Degraded, online only. **Not a valid kiosk configuration.** |
|
||||
|
||||
Plain LAN HTTP has no Service Worker, no Cache Storage, and no offline startup.
|
||||
The display still shows slides while the server is reachable, but it cannot
|
||||
survive an outage. It detects this at startup and reports
|
||||
`offline_capability_unavailable` through its heartbeat so the limitation is
|
||||
visible in `/api/displays` and Home Assistant rather than on screen.
|
||||
|
||||
Do not work around this with `--ignore-certificate-errors`,
|
||||
`--disable-web-security`, or an insecure-origin-as-secure flag. They disable
|
||||
the protections the offline design relies on and do not represent a supportable
|
||||
deployment.
|
||||
|
||||
### Kiosk browser requirements
|
||||
|
||||
The kiosk must use a **persistent** Chromium profile. The Service Worker, Cache
|
||||
Storage, and IndexedDB that make offline operation possible all live in that
|
||||
profile, and an incognito window, a guest profile, or a throwaway
|
||||
`--user-data-dir` discards them on every launch.
|
||||
|
||||
```sh
|
||||
chromium \
|
||||
--kiosk \
|
||||
--user-data-dir=/var/lib/announcements-kiosk \
|
||||
https://display.example.org/display
|
||||
```
|
||||
|
||||
The old v1 kiosk flags that disabled the browser cache are no longer required
|
||||
and should be removed at cutover. They existed because v1 reused mutable slide
|
||||
and must be removed at cutover. They existed because v1 reused mutable slide
|
||||
URLs; v2 media URLs are content-addressed, so a changed image always arrives
|
||||
under a new URL and stale bytes are impossible. Those flags now actively
|
||||
interfere with the Service Worker and Cache Storage the display depends on.
|
||||
|
||||
Clearing the browser's site data deletes the cached slides, the last-known-good
|
||||
state, and the display's identity. Nothing is lost permanently: on its next
|
||||
connection the display registers as a new client, re-caches the active playlist,
|
||||
and resumes. No routine cache management is expected of an operator.
|
||||
|
||||
### Transition guarantee
|
||||
|
||||
The renderer owns two permanent full-screen scene layers. Each layer is a
|
||||
@@ -483,6 +539,95 @@ was required; `/api/displays` parses them back into an object. An error clears
|
||||
when the failing asset renders successfully or leaves the manifest — never
|
||||
merely because time passed.
|
||||
|
||||
### Offline resilience
|
||||
|
||||
While connected the server is authoritative. While disconnected the display
|
||||
becomes a deterministic local continuation of the last authoritative state it
|
||||
persisted. The moment the server speaks again it is authoritative once more:
|
||||
the local cursor is abandoned, never reconciled and never reported back.
|
||||
Offline playback is resilience, not distributed consensus.
|
||||
|
||||
Local continuation uses the persisted manifest and timing anchor. Elapsed time
|
||||
is derived from the anchor — the authoritative `serverTime` and the local clock
|
||||
reading when it was accepted — so a restart estimates where playback should be
|
||||
rather than restarting a slide. Catch-up is arithmetic: whole rotations are
|
||||
skipped in one step between eligibility boundaries, so a display offline for a
|
||||
month resolves its position in milliseconds instead of stepping through every
|
||||
missed slide. Mixed durations, wraparound, `[start, end)` schedule boundaries,
|
||||
and asset expirations are all evaluated locally from the cached manifest. A null
|
||||
expiration stays permanent no matter how long the display has been away.
|
||||
|
||||
Transport state is preserved exactly. Mode off stays black while the hidden
|
||||
cursor keeps advancing; a paused display holds its slide and remainder and never
|
||||
advances merely because time passed or the browser restarted; static holds
|
||||
indefinitely and releases only when the cached metadata says the asset genuinely
|
||||
expired. If the server had queued an activation for a playlist the display does
|
||||
not hold, the display holds its last good frame at that boundary rather than
|
||||
continuing deeper into a playlist that is being replaced.
|
||||
|
||||
The distinction between "the playlist is genuinely empty" and "the playlist
|
||||
should play but the local bytes are unusable" is preserved: the first is
|
||||
correctly black, the second keeps the last good frame and reports an error.
|
||||
|
||||
This evaluator is a small client-side implementation, deliberately separate from
|
||||
the verified Phase 3B engine. Contract tests run both over identical scenarios —
|
||||
mixed durations, wraparound, schedule starts and ends, expirations, long
|
||||
outages, and empty playlists — and assert they resolve the same slide, start,
|
||||
and deadline.
|
||||
|
||||
### Cold start
|
||||
|
||||
After one successful online visit the display can start with the entire stack
|
||||
unavailable. The Service Worker serves the application shell, IndexedDB
|
||||
provides the client ID and last-known-good state, Cache Storage provides the
|
||||
slide bytes, and local playback resumes from the persisted anchor. A reconnect
|
||||
attempt continues in the background.
|
||||
|
||||
A display that has never successfully cached anything has nothing trustworthy
|
||||
to show, so it stays black. It never invents placeholder content.
|
||||
|
||||
Because a remote kiosk reaches the origin through a reverse proxy, an outage
|
||||
behind that proxy arrives as a 502 rather than a refused connection. The worker
|
||||
treats a gateway error as an outage, and bounds how long it waits for the
|
||||
network when a usable shell is already cached, so a kiosk restarting during an
|
||||
outage reaches its slides quickly instead of waiting out a proxy timeout.
|
||||
|
||||
### Reconnect
|
||||
|
||||
On reconnect the authoritative snapshot wins. Required assets are staged and
|
||||
decoded first and the current frame is kept until the new one is ready, so
|
||||
resynchronization never exposes black and never replays the slides that were
|
||||
missed. If the local calculation already agrees with the server, the timing
|
||||
anchor is simply re-anchored with no visual churn. A playlist changed during the
|
||||
outage is staged and adopted without mixing old and new entries, and the
|
||||
superseded playlist's bytes are pruned once the new manifest is promoted.
|
||||
|
||||
The server's own 45-second heartbeat threshold is unchanged. A display can be
|
||||
showing announcements perfectly while disconnected and still be reported as
|
||||
`display.offline`, which is the useful distinction: the audience sees slides,
|
||||
and the operator still learns that the display lost contact.
|
||||
|
||||
### Display errors as incidents
|
||||
|
||||
Validated display error codes become Home Assistant incidents through the
|
||||
existing Phase 4 machinery, keyed per display and condition so a persisting
|
||||
error is reported once and recovers once:
|
||||
|
||||
| Code | Severity |
|
||||
| --- | --- |
|
||||
| `slide_decode_failed` | warning |
|
||||
| `asset_fetch_failed` | warning |
|
||||
| `cache_write_failed` | warning |
|
||||
| `manifest_invalid` | critical |
|
||||
| `offline_capability_unavailable` | warning |
|
||||
|
||||
Arbitrary browser text remains telemetry and never becomes a notification. If
|
||||
an error occurs while the display is completely disconnected the server cannot
|
||||
learn of it at that instant; the display persists its current error and reports
|
||||
it on reconnect. A transient offline error that fully resolves before
|
||||
reconnection therefore produces no historical notification. This is a documented
|
||||
limitation rather than a browser-side durable event queue.
|
||||
|
||||
## Storage and health
|
||||
|
||||
The API uses a small path-style S3 adapter configured by `S3_ENDPOINT`,
|
||||
|
||||
@@ -0,0 +1,291 @@
|
||||
import { expect, test } from "@playwright/test";
|
||||
import { ensureFixtures } from "./fixtures.js";
|
||||
import { activatePlaylist, resetPlayback } from "./helpers.js";
|
||||
import {
|
||||
BASE_URL,
|
||||
cachedMedia,
|
||||
diagnostics,
|
||||
launchKiosk,
|
||||
opaqueLayerCount,
|
||||
waitForCachedAssets,
|
||||
waitForSlide,
|
||||
} from "./offline-helpers.js";
|
||||
import { CONTAINERS, start, stop, waitUntilHealthy, waitUntilUnreachable } from "./stack.js";
|
||||
|
||||
test.describe.configure({ mode: "serial" });
|
||||
test.use({ baseURL: BASE_URL });
|
||||
|
||||
async function restoreStack() {
|
||||
await start(CONTAINERS.storage, CONTAINERS.api, CONTAINERS.web);
|
||||
await waitUntilHealthy(BASE_URL);
|
||||
}
|
||||
|
||||
test.afterAll(async () => {
|
||||
await restoreStack();
|
||||
});
|
||||
|
||||
test("the app shell is served from the worker when the origin returns a gateway error", async ({
|
||||
request,
|
||||
}) => {
|
||||
const library = await ensureFixtures(request);
|
||||
await resetPlayback(request);
|
||||
await activatePlaylist(request, "Gateway", [
|
||||
{ assetId: library["wide-a"].id, durationOverrideMs: 5_000 },
|
||||
]);
|
||||
|
||||
const { context, page } = await launchKiosk();
|
||||
try {
|
||||
await page.goto(`${BASE_URL}/display`);
|
||||
await waitForSlide(page);
|
||||
await expect
|
||||
.poll(async () =>
|
||||
page.evaluate(async () =>
|
||||
Boolean((await navigator.serviceWorker.getRegistration("/"))?.active),
|
||||
),
|
||||
)
|
||||
.toBe(true);
|
||||
await context.close();
|
||||
|
||||
// A remote kiosk reaches the origin through a reverse proxy, so losing the
|
||||
// application behind it produces 5xx rather than a refused connection.
|
||||
await stop(CONTAINERS.web);
|
||||
await waitUntilUnreachable(BASE_URL);
|
||||
|
||||
const restarted = await launchKiosk();
|
||||
try {
|
||||
const response = await restarted.page.goto(`${BASE_URL}/display`, {
|
||||
waitUntil: "domcontentloaded",
|
||||
});
|
||||
expect(
|
||||
response?.status(),
|
||||
"a proxied 5xx outage must still yield the cached shell",
|
||||
).toBe(200);
|
||||
await waitForSlide(restarted.page);
|
||||
expect((await diagnostics(restarted.page)).offlineActive).toBe(true);
|
||||
} finally {
|
||||
await restarted.context.close();
|
||||
}
|
||||
} finally {
|
||||
await restoreStack();
|
||||
}
|
||||
});
|
||||
|
||||
test("a damaged cached slide is skipped offline without stalling rotation", async ({
|
||||
request,
|
||||
}) => {
|
||||
const library = await ensureFixtures(request);
|
||||
await resetPlayback(request);
|
||||
await activatePlaylist(request, "Damaged", [
|
||||
{ assetId: library["wide-a"].id, durationOverrideMs: 3_000 },
|
||||
{ assetId: library["wide-d"].id, durationOverrideMs: 3_000 },
|
||||
{ assetId: library["wide-c"].id, durationOverrideMs: 3_000 },
|
||||
]);
|
||||
|
||||
const { context, page } = await launchKiosk();
|
||||
try {
|
||||
await page.goto(`${BASE_URL}/display`);
|
||||
await waitForSlide(page);
|
||||
await waitForCachedAssets(page, [
|
||||
library["wide-a"].id,
|
||||
library["wide-d"].id,
|
||||
library["wide-c"].id,
|
||||
]);
|
||||
|
||||
await stop(CONTAINERS.api, CONTAINERS.storage);
|
||||
await waitUntilUnreachable(BASE_URL);
|
||||
|
||||
// Replace one cached slide's bytes with something undecodable, then reload
|
||||
// so the damage is discovered while completely offline.
|
||||
await page.evaluate(async (assetId) => {
|
||||
const cache = await caches.open("announcements-display-assets-v1");
|
||||
for (const request of await cache.keys()) {
|
||||
if (!request.url.includes(assetId)) continue;
|
||||
await cache.put(
|
||||
request,
|
||||
new Response(new Blob([new Uint8Array([0, 1, 2, 3, 4])], { type: "image/webp" }), {
|
||||
status: 200,
|
||||
}),
|
||||
);
|
||||
}
|
||||
}, library["wide-d"].id);
|
||||
await page.reload({ waitUntil: "domcontentloaded" });
|
||||
await waitForSlide(page);
|
||||
|
||||
const seen = new Set();
|
||||
const deadline = Date.now() + 20_000;
|
||||
while (Date.now() < deadline) {
|
||||
const state = await page.evaluate(() => ({
|
||||
rendered: window.__display?.diagnostics()?.renderedAssetId ?? null,
|
||||
opaque: [...document.querySelectorAll(".scene")].filter(
|
||||
(layer) => Number(getComputedStyle(layer).opacity) > 0.99,
|
||||
).length,
|
||||
}));
|
||||
expect(state.opaque, "the last good frame is never dropped").toBeGreaterThanOrEqual(1);
|
||||
if (state.rendered) seen.add(state.rendered);
|
||||
await page.waitForTimeout(200);
|
||||
}
|
||||
|
||||
expect(seen.has(library["wide-d"].id), "the damaged slide never displays").toBe(false);
|
||||
expect(seen.size, "the other cached slides keep rotating").toBeGreaterThanOrEqual(2);
|
||||
const state = await diagnostics(page);
|
||||
expect(state.unavailableEntryIds.length).toBeGreaterThanOrEqual(1);
|
||||
expect(state.mostRecentError).not.toBeNull();
|
||||
} finally {
|
||||
await context.close();
|
||||
await restoreStack();
|
||||
}
|
||||
});
|
||||
|
||||
test("a missing media cache offline fails safe instead of fabricating state", async ({
|
||||
request,
|
||||
}) => {
|
||||
const library = await ensureFixtures(request);
|
||||
await resetPlayback(request);
|
||||
await activatePlaylist(request, "Empty cache", [
|
||||
{ assetId: library["wide-a"].id, durationOverrideMs: 5_000 },
|
||||
]);
|
||||
|
||||
const { context, page } = await launchKiosk();
|
||||
try {
|
||||
await page.goto(`${BASE_URL}/display`);
|
||||
await waitForSlide(page);
|
||||
const clientId = await page.evaluate(() => window.__display.clientId);
|
||||
|
||||
await stop(CONTAINERS.api, CONTAINERS.storage, CONTAINERS.web);
|
||||
await waitUntilUnreachable(BASE_URL);
|
||||
|
||||
// Shell and IndexedDB survive; only the slide bytes are gone.
|
||||
await page.evaluate(() => caches.delete("announcements-display-assets-v1"));
|
||||
await page.reload({ waitUntil: "domcontentloaded" });
|
||||
await page.waitForTimeout(6_000);
|
||||
|
||||
// No crash, no invented content, and the identity is intact.
|
||||
expect(await page.evaluate(() => Boolean(window.__display))).toBe(true);
|
||||
expect(await page.evaluate(() => window.__display.clientId)).toBe(clientId);
|
||||
const state = await diagnostics(page);
|
||||
expect(state.renderedAssetId, "nothing is fabricated").toBeNull();
|
||||
expect(state.mostRecentError, "the condition is recorded for later reporting").not.toBeNull();
|
||||
|
||||
// Once the infrastructure returns, the cache is rebuilt with no intervention.
|
||||
await restoreStack();
|
||||
await expect
|
||||
.poll(async () => (await diagnostics(page))?.renderedAssetId, { timeout: 90_000 })
|
||||
.not.toBeNull();
|
||||
await expect.poll(async () => (await cachedMedia(page)).length).toBeGreaterThanOrEqual(1);
|
||||
} finally {
|
||||
await context.close();
|
||||
await restoreStack();
|
||||
}
|
||||
});
|
||||
|
||||
test("corrupt persisted state offline fails safe and recovers when the server returns", async ({
|
||||
request,
|
||||
}) => {
|
||||
const library = await ensureFixtures(request);
|
||||
await resetPlayback(request);
|
||||
await activatePlaylist(request, "Corrupt", [
|
||||
{ assetId: library["wide-a"].id, durationOverrideMs: 5_000 },
|
||||
]);
|
||||
|
||||
const { context, page } = await launchKiosk();
|
||||
try {
|
||||
await page.goto(`${BASE_URL}/display`);
|
||||
await waitForSlide(page);
|
||||
const clientId = await page.evaluate(() => window.__display.clientId);
|
||||
|
||||
await page.evaluate(async () => {
|
||||
const database = await new Promise((resolve) => {
|
||||
const open = indexedDB.open("announcements-display", 1);
|
||||
open.onsuccess = () => resolve(open.result);
|
||||
});
|
||||
await new Promise((resolve) => {
|
||||
const transaction = database.transaction("state", "readwrite");
|
||||
transaction
|
||||
.objectStore("state")
|
||||
.put({ schemaVersion: 99, manifest: "not-an-object" }, "lastKnownGood");
|
||||
transaction.oncomplete = resolve;
|
||||
});
|
||||
database.close();
|
||||
});
|
||||
|
||||
await stop(CONTAINERS.api, CONTAINERS.storage);
|
||||
await waitUntilUnreachable(BASE_URL);
|
||||
await page.reload({ waitUntil: "domcontentloaded" });
|
||||
await page.waitForTimeout(5_000);
|
||||
|
||||
// An unreadable record is discarded rather than trusted; the display stays
|
||||
// alive and keeps its identity.
|
||||
expect(await page.evaluate(() => Boolean(window.__display))).toBe(true);
|
||||
expect(await page.evaluate(() => window.__display.clientId)).toBe(clientId);
|
||||
|
||||
await restoreStack();
|
||||
await expect
|
||||
.poll(async () => (await diagnostics(page))?.renderedAssetId, { timeout: 90_000 })
|
||||
.toBe(library["wide-a"].id);
|
||||
} finally {
|
||||
await context.close();
|
||||
await restoreStack();
|
||||
}
|
||||
});
|
||||
|
||||
test("a worker update keeps the separate slide media cache intact", async ({ request }) => {
|
||||
const library = await ensureFixtures(request);
|
||||
await resetPlayback(request);
|
||||
await activatePlaylist(request, "Worker update", [
|
||||
{ assetId: library["wide-a"].id, durationOverrideMs: 60_000 },
|
||||
]);
|
||||
|
||||
const { context, page } = await launchKiosk();
|
||||
try {
|
||||
await page.goto(`${BASE_URL}/display`);
|
||||
await waitForSlide(page);
|
||||
await expect
|
||||
.poll(async () =>
|
||||
page.evaluate(async () =>
|
||||
Boolean((await navigator.serviceWorker.getRegistration("/"))?.active),
|
||||
),
|
||||
)
|
||||
.toBe(true);
|
||||
const mediaBefore = await cachedMedia(page);
|
||||
expect(mediaBefore.length).toBeGreaterThanOrEqual(1);
|
||||
|
||||
// Simulate a stale shell generation from an older application version.
|
||||
await page.evaluate(async () => {
|
||||
const stale = await caches.open("announcements-display-shell-v0");
|
||||
await stale.put("/display", new Response("<html>old</html>", { status: 200 }));
|
||||
});
|
||||
// Cleanup runs when a worker activates, so force a real install/activate
|
||||
// cycle rather than calling update() on an unchanged worker file.
|
||||
await page.evaluate(async () => {
|
||||
const registration = await navigator.serviceWorker.getRegistration("/");
|
||||
await registration.unregister();
|
||||
});
|
||||
await page.reload({ waitUntil: "domcontentloaded" });
|
||||
await waitForSlide(page);
|
||||
await expect
|
||||
.poll(
|
||||
async () =>
|
||||
page.evaluate(async () =>
|
||||
Boolean((await navigator.serviceWorker.getRegistration("/"))?.active),
|
||||
),
|
||||
{ timeout: 45_000 },
|
||||
)
|
||||
.toBe(true);
|
||||
|
||||
const cacheNames = await page.evaluate(() => caches.keys());
|
||||
expect(
|
||||
cacheNames.includes("announcements-display-shell-v0"),
|
||||
"superseded shell generations are cleaned up",
|
||||
).toBe(false);
|
||||
expect(cacheNames).toContain("announcements-display-assets-v1");
|
||||
const mediaAfter = await cachedMedia(page);
|
||||
expect(
|
||||
mediaAfter,
|
||||
"a worker update must never delete the explicit slide media cache",
|
||||
).toEqual(expect.arrayContaining(mediaBefore));
|
||||
expect(await opaqueLayerCount(page)).toBeGreaterThanOrEqual(1);
|
||||
} finally {
|
||||
await context.close();
|
||||
await restoreStack();
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,363 @@
|
||||
import { expect, test } from "@playwright/test";
|
||||
import { ensureFixtures } from "./fixtures.js";
|
||||
import { activatePlaylist, api, resetPlayback } from "./helpers.js";
|
||||
import {
|
||||
BASE_URL,
|
||||
cachedMedia,
|
||||
diagnostics,
|
||||
launchKiosk,
|
||||
opaqueLayerCount,
|
||||
waitForCachedAssets,
|
||||
waitForSlide,
|
||||
} from "./offline-helpers.js";
|
||||
import { CONTAINERS, start, stop, waitUntilHealthy, waitUntilUnreachable } from "./stack.js";
|
||||
|
||||
// These scenarios remove real infrastructure, so they must not interleave.
|
||||
test.describe.configure({ mode: "serial" });
|
||||
test.use({ baseURL: BASE_URL });
|
||||
|
||||
async function restoreStack() {
|
||||
await start(CONTAINERS.storage, CONTAINERS.api, CONTAINERS.web);
|
||||
await waitUntilHealthy(BASE_URL);
|
||||
}
|
||||
|
||||
test.afterAll(async () => {
|
||||
await restoreStack();
|
||||
});
|
||||
|
||||
test("cached rotation continues through a complete server outage", async ({ request }) => {
|
||||
const library = await ensureFixtures(request);
|
||||
await resetPlayback(request);
|
||||
await activatePlaylist(request, "Outage", [
|
||||
{ assetId: library["wide-a"].id, durationOverrideMs: 3_000 },
|
||||
{ assetId: library["wide-b"].id, durationOverrideMs: 9_000 },
|
||||
{ assetId: library["wide-c"].id, durationOverrideMs: 4_000 },
|
||||
]);
|
||||
|
||||
const { context, page } = await launchKiosk();
|
||||
try {
|
||||
await page.goto(`${BASE_URL}/display`);
|
||||
await waitForSlide(page);
|
||||
// Every slide of this playlist must be cached before the outage begins.
|
||||
await waitForCachedAssets(page, [
|
||||
library["wide-a"].id,
|
||||
library["wide-b"].id,
|
||||
library["wide-c"].id,
|
||||
]);
|
||||
|
||||
await stop(CONTAINERS.api, CONTAINERS.storage);
|
||||
await waitUntilUnreachable(BASE_URL);
|
||||
|
||||
// Sample continuously across multiple full cycles with nothing serving.
|
||||
const seen = new Set();
|
||||
const starts = new Map();
|
||||
const deadline = Date.now() + 40_000;
|
||||
while (Date.now() < deadline) {
|
||||
const state = await page.evaluate(() => ({
|
||||
rendered: window.__display?.diagnostics()?.renderedAssetId ?? null,
|
||||
offline: window.__display?.diagnostics()?.offlineActive ?? false,
|
||||
opaque: [...document.querySelectorAll(".scene")].filter(
|
||||
(layer) => Number(getComputedStyle(layer).opacity) > 0.99,
|
||||
).length,
|
||||
gate: document.querySelector(".output-gate").dataset.active,
|
||||
}));
|
||||
// The original black-frame bug must remain impossible offline too.
|
||||
expect(state.opaque, "a good frame is always on screen offline").toBeGreaterThanOrEqual(1);
|
||||
expect(state.gate).toBe("false");
|
||||
if (state.rendered) {
|
||||
seen.add(state.rendered);
|
||||
if (!starts.has(state.rendered)) starts.set(state.rendered, Date.now());
|
||||
}
|
||||
await page.waitForTimeout(150);
|
||||
}
|
||||
|
||||
expect(await diagnostics(page).then((d) => d.offlineActive)).toBe(true);
|
||||
expect(seen.size, "mixed-duration rotation kept cycling with no server").toBe(3);
|
||||
expect(await page.evaluate(() => window.__loadCount ?? 0)).toBe(0);
|
||||
|
||||
await restoreStack();
|
||||
// The server is authoritative again the moment it speaks.
|
||||
await expect
|
||||
.poll(async () => (await diagnostics(page))?.offlineActive, { timeout: 60_000 })
|
||||
.toBe(false);
|
||||
const playback = await api(request, "/api/playback");
|
||||
await expect
|
||||
.poll(async () => (await diagnostics(page))?.acceptedRevision, { timeout: 30_000 })
|
||||
.toBeGreaterThanOrEqual(playback.playbackRevision);
|
||||
} finally {
|
||||
await context.close();
|
||||
await restoreStack();
|
||||
}
|
||||
});
|
||||
|
||||
test("a cold Chromium restart with the whole stack down restores signage", async ({
|
||||
request,
|
||||
}) => {
|
||||
const library = await ensureFixtures(request);
|
||||
await resetPlayback(request);
|
||||
await activatePlaylist(request, "Cold start", [
|
||||
{ assetId: library["wide-a"].id, durationOverrideMs: 4_000 },
|
||||
{ assetId: library["wide-d"].id, durationOverrideMs: 6_000 },
|
||||
]);
|
||||
|
||||
// A. Warm the persistent profile while everything is healthy.
|
||||
let clientId;
|
||||
let cachedBefore;
|
||||
{
|
||||
const { context, page } = await launchKiosk();
|
||||
await page.goto(`${BASE_URL}/display`);
|
||||
await waitForSlide(page);
|
||||
await expect
|
||||
.poll(async () =>
|
||||
page.evaluate(async () =>
|
||||
Boolean((await navigator.serviceWorker.getRegistration("/"))?.active),
|
||||
),
|
||||
)
|
||||
.toBe(true);
|
||||
await expect.poll(async () => (await cachedMedia(page)).length).toBeGreaterThanOrEqual(2);
|
||||
clientId = await page.evaluate(() => window.__display.clientId);
|
||||
cachedBefore = await cachedMedia(page);
|
||||
await context.close();
|
||||
}
|
||||
|
||||
// B. Remove the entire origin: no web server, no API, no object storage.
|
||||
await stop(CONTAINERS.web, CONTAINERS.api, CONTAINERS.storage);
|
||||
await waitUntilUnreachable(BASE_URL);
|
||||
|
||||
try {
|
||||
// C. Relaunch the kiosk against the same URL with the same profile.
|
||||
const { context, page } = await launchKiosk();
|
||||
try {
|
||||
const response = await page.goto(`${BASE_URL}/display`, { waitUntil: "domcontentloaded" });
|
||||
expect(
|
||||
response?.status(),
|
||||
"the Service Worker served the app shell with the origin down",
|
||||
).toBe(200);
|
||||
|
||||
await waitForSlide(page);
|
||||
const offline = await diagnostics(page);
|
||||
expect(offline.offlineActive, "the display took local authority").toBe(true);
|
||||
expect(
|
||||
await page.evaluate(() => window.__display.clientId),
|
||||
"the persistent display identity survived the restart",
|
||||
).toBe(clientId);
|
||||
expect(cachedBefore).toContain(
|
||||
(await cachedMedia(page)).find((path) => path.includes(library["wide-a"].id)),
|
||||
);
|
||||
expect(await opaqueLayerCount(page)).toBeGreaterThanOrEqual(1);
|
||||
|
||||
// D. It must keep advancing, not merely show one restored frame.
|
||||
const first = offline.renderedAssetId;
|
||||
await expect
|
||||
.poll(async () => (await diagnostics(page))?.renderedAssetId, { timeout: 30_000 })
|
||||
.not.toBe(first);
|
||||
expect(await opaqueLayerCount(page)).toBeGreaterThanOrEqual(1);
|
||||
|
||||
// E. Restore the stack and confirm it surrenders authority cleanly.
|
||||
await restoreStack();
|
||||
await expect
|
||||
.poll(async () => (await diagnostics(page))?.offlineActive, { timeout: 90_000 })
|
||||
.toBe(false);
|
||||
const playback = await api(request, "/api/playback");
|
||||
await expect
|
||||
.poll(async () => (await diagnostics(page))?.renderedEntryId, { timeout: 45_000 })
|
||||
.toBe(playback.currentEntryId);
|
||||
|
||||
const displays = await api(request, "/api/displays");
|
||||
expect(
|
||||
displays.displays.filter((display) => display.clientId === clientId).length,
|
||||
"one physical display stays one record across a cold restart",
|
||||
).toBe(1);
|
||||
} finally {
|
||||
await context.close();
|
||||
}
|
||||
} finally {
|
||||
await restoreStack();
|
||||
}
|
||||
});
|
||||
|
||||
test("an object storage outage never disturbs already cached slides", async ({ request }) => {
|
||||
const library = await ensureFixtures(request);
|
||||
await resetPlayback(request);
|
||||
await activatePlaylist(request, "Storage outage", [
|
||||
{ assetId: library["wide-a"].id, durationOverrideMs: 3_000 },
|
||||
{ assetId: library["wide-b"].id, durationOverrideMs: 3_000 },
|
||||
]);
|
||||
|
||||
const { context, page } = await launchKiosk();
|
||||
try {
|
||||
await page.goto(`${BASE_URL}/display`);
|
||||
await waitForSlide(page);
|
||||
await waitForCachedAssets(page, [library["wide-a"].id, library["wide-b"].id]);
|
||||
|
||||
// The API stays up; only the bytes become unreachable.
|
||||
await stop(CONTAINERS.storage);
|
||||
|
||||
const seen = new Set();
|
||||
const deadline = Date.now() + 14_000;
|
||||
while (Date.now() < deadline) {
|
||||
const state = await page.evaluate(() => ({
|
||||
rendered: window.__display?.diagnostics()?.renderedAssetId ?? null,
|
||||
opaque: [...document.querySelectorAll(".scene")].filter(
|
||||
(layer) => Number(getComputedStyle(layer).opacity) > 0.99,
|
||||
).length,
|
||||
}));
|
||||
expect(state.opaque).toBeGreaterThanOrEqual(1);
|
||||
if (state.rendered) seen.add(state.rendered);
|
||||
await page.waitForTimeout(200);
|
||||
}
|
||||
expect(seen.size, "cached slides keep rotating without object storage").toBe(2);
|
||||
|
||||
// A brand new asset cannot be fetched; the last good frame must survive.
|
||||
const held = await diagnostics(page);
|
||||
await activatePlaylist(request, "Uncached", [
|
||||
{ assetId: library.standalone.id, durationOverrideMs: 60_000 },
|
||||
]);
|
||||
await page.waitForTimeout(6_000);
|
||||
expect(await opaqueLayerCount(page)).toBeGreaterThanOrEqual(1);
|
||||
const during = await diagnostics(page);
|
||||
expect(during.renderedAssetId, "the last good frame is retained").toBe(held.renderedAssetId);
|
||||
// Nothing in the newly active playlist can be rendered locally, which is a
|
||||
// more severe condition than one failed asset and is reported as such.
|
||||
expect(during.mostRecentError?.kind).toBe("manifest_invalid");
|
||||
|
||||
// That condition reaches the server as a critical incident, exactly once.
|
||||
const clientId = await page.evaluate(() => window.__display.clientId);
|
||||
await page.evaluate(() => window.__display.heartbeat());
|
||||
await expect
|
||||
.poll(
|
||||
async () => {
|
||||
const displays = await api(request, "/api/displays");
|
||||
return (
|
||||
displays.displays.find((display) => display.clientId === clientId)?.mostRecentError
|
||||
?.kind ?? null
|
||||
);
|
||||
},
|
||||
{ timeout: 30_000 },
|
||||
)
|
||||
.toBe("manifest_invalid");
|
||||
|
||||
// Restoring storage recovers without a page reload.
|
||||
await start(CONTAINERS.storage);
|
||||
await waitUntilHealthy(BASE_URL);
|
||||
await expect
|
||||
.poll(async () => (await diagnostics(page))?.renderedAssetId, { timeout: 90_000 })
|
||||
.toBe(library.standalone.id);
|
||||
expect(await page.evaluate(() => window.__loadCount ?? 0)).toBe(0);
|
||||
} finally {
|
||||
await context.close();
|
||||
await restoreStack();
|
||||
}
|
||||
});
|
||||
|
||||
test("an API outage keeps local playback running and resynchronizes on return", async ({
|
||||
request,
|
||||
}) => {
|
||||
const library = await ensureFixtures(request);
|
||||
await resetPlayback(request);
|
||||
await activatePlaylist(request, "API outage", [
|
||||
{ assetId: library["wide-a"].id, durationOverrideMs: 3_000 },
|
||||
{ assetId: library["wide-b"].id, durationOverrideMs: 3_000 },
|
||||
{ assetId: library["wide-c"].id, durationOverrideMs: 3_000 },
|
||||
]);
|
||||
|
||||
const { context, page } = await launchKiosk();
|
||||
try {
|
||||
await page.goto(`${BASE_URL}/display`);
|
||||
await waitForSlide(page);
|
||||
await waitForCachedAssets(page, [
|
||||
library["wide-a"].id,
|
||||
library["wide-b"].id,
|
||||
library["wide-c"].id,
|
||||
]);
|
||||
|
||||
await stop(CONTAINERS.api);
|
||||
await expect
|
||||
.poll(async () => (await diagnostics(page))?.offlineActive, { timeout: 45_000 })
|
||||
.toBe(true);
|
||||
|
||||
const seen = new Set();
|
||||
const deadline = Date.now() + 14_000;
|
||||
while (Date.now() < deadline) {
|
||||
const state = await diagnostics(page);
|
||||
if (state?.renderedAssetId) seen.add(state.renderedAssetId);
|
||||
expect(await opaqueLayerCount(page)).toBeGreaterThanOrEqual(1);
|
||||
await page.waitForTimeout(250);
|
||||
}
|
||||
expect(seen.size).toBeGreaterThanOrEqual(2);
|
||||
|
||||
// While the display was blind, the operator changed authoritative state.
|
||||
await start(CONTAINERS.api);
|
||||
await waitUntilHealthy(BASE_URL);
|
||||
await api(request, "/api/playback/mode", { method: "PUT", data: { mode: "off" } });
|
||||
|
||||
// Server wins immediately on reconnect, including the visual gate.
|
||||
await expect(page.getByTestId("output-gate")).toHaveAttribute("data-active", "true", {
|
||||
timeout: 60_000,
|
||||
});
|
||||
expect((await diagnostics(page)).offlineActive).toBe(false);
|
||||
|
||||
await api(request, "/api/playback/mode", { method: "PUT", data: { mode: "on" } });
|
||||
await expect(page.getByTestId("output-gate")).toHaveAttribute("data-active", "false", {
|
||||
timeout: 30_000,
|
||||
});
|
||||
const playback = await api(request, "/api/playback");
|
||||
await expect
|
||||
.poll(async () => (await diagnostics(page))?.renderedEntryId, { timeout: 30_000 })
|
||||
.toBe(playback.currentEntryId);
|
||||
expect(await page.evaluate(() => window.__loadCount ?? 0)).toBe(0);
|
||||
} finally {
|
||||
await context.close();
|
||||
await restoreStack();
|
||||
}
|
||||
});
|
||||
|
||||
test("a playlist changed during the outage is adopted cleanly on reconnect", async ({
|
||||
request,
|
||||
}) => {
|
||||
const library = await ensureFixtures(request);
|
||||
await resetPlayback(request);
|
||||
await activatePlaylist(request, "Before outage", [
|
||||
{ assetId: library["wide-a"].id, durationOverrideMs: 5_000 },
|
||||
{ assetId: library["wide-b"].id, durationOverrideMs: 5_000 },
|
||||
]);
|
||||
|
||||
const { context, page } = await launchKiosk();
|
||||
try {
|
||||
await page.goto(`${BASE_URL}/display`);
|
||||
await waitForSlide(page);
|
||||
const before = await diagnostics(page);
|
||||
|
||||
await stop(CONTAINERS.api);
|
||||
await expect
|
||||
.poll(async () => (await diagnostics(page))?.offlineActive, { timeout: 45_000 })
|
||||
.toBe(true);
|
||||
await page.waitForTimeout(6_000);
|
||||
|
||||
await start(CONTAINERS.api);
|
||||
await waitUntilHealthy(BASE_URL);
|
||||
// A different playlist entirely, activated while the display was offline.
|
||||
await activatePlaylist(request, "After outage", [
|
||||
{ assetId: library.square.id, durationOverrideMs: 60_000 },
|
||||
]);
|
||||
|
||||
await expect
|
||||
.poll(async () => (await diagnostics(page))?.renderedAssetId, { timeout: 90_000 })
|
||||
.toBe(library.square.id);
|
||||
const after = await diagnostics(page);
|
||||
expect(after.offlineActive).toBe(false);
|
||||
expect(after.renderedAssetId).not.toBe(before.renderedAssetId);
|
||||
expect(await opaqueLayerCount(page)).toBeGreaterThanOrEqual(1);
|
||||
|
||||
// The superseded playlist's bytes are pruned once the new one is promoted.
|
||||
await expect
|
||||
.poll(async () => (await cachedMedia(page)).some((path) => path.includes(library.square.id)))
|
||||
.toBe(true);
|
||||
const cached = await cachedMedia(page);
|
||||
expect(cached.length, "repeated playlist changes must not grow the cache forever")
|
||||
.toBeLessThanOrEqual(6);
|
||||
} finally {
|
||||
await context.close();
|
||||
await restoreStack();
|
||||
}
|
||||
});
|
||||
@@ -138,10 +138,13 @@ test("the persistent client identity and last known good state survive a refresh
|
||||
database.close();
|
||||
return value;
|
||||
});
|
||||
expect(persisted.schemaVersion).toBe(1);
|
||||
// The record schema is versioned independently of the database version.
|
||||
expect(persisted.schemaVersion).toBe(2);
|
||||
expect(persisted.playbackRevision).toBeGreaterThan(0);
|
||||
expect(persisted.manifest.entries.length).toBeGreaterThan(0);
|
||||
expect(persisted.timing.serverTime).toBeGreaterThan(0);
|
||||
// The clock anchor a cold offline start depends on.
|
||||
expect(persisted.timing.recordedAt).toBeGreaterThan(0);
|
||||
expect(persisted.mode).toBe("on");
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
import { chromium, expect } from "@playwright/test";
|
||||
|
||||
export const PROFILE_DIR = process.env.DISPLAY_PROFILE_DIR || "/profiles/kiosk";
|
||||
export const BASE_URL = process.env.DISPLAY_BASE_URL || "http://127.0.0.1:5199";
|
||||
|
||||
// A real kiosk keeps a persistent Chromium profile: the Service Worker,
|
||||
// Cache Storage, and IndexedDB that make offline startup possible all live
|
||||
// there. Incognito or an ephemeral user-data-dir would defeat the whole phase.
|
||||
export async function launchKiosk() {
|
||||
const context = await chromium.launchPersistentContext(PROFILE_DIR, {
|
||||
viewport: { width: 1280, height: 720 },
|
||||
serviceWorkers: "allow",
|
||||
});
|
||||
const page = context.pages()[0] || (await context.newPage());
|
||||
return { context, page };
|
||||
}
|
||||
|
||||
export function diagnostics(page) {
|
||||
return page.evaluate(() => window.__display?.diagnostics() ?? null);
|
||||
}
|
||||
|
||||
export async function waitForSlide(page, timeout = 60_000) {
|
||||
await expect
|
||||
.poll(async () => (await diagnostics(page))?.renderedAssetId ?? null, { timeout })
|
||||
.not.toBeNull();
|
||||
}
|
||||
|
||||
export function opaqueLayerCount(page) {
|
||||
return page.evaluate(
|
||||
() =>
|
||||
[...document.querySelectorAll(".scene")].filter(
|
||||
(layer) => Number(getComputedStyle(layer).opacity) > 0.99,
|
||||
).length,
|
||||
);
|
||||
}
|
||||
|
||||
// Waiting for "some" cached assets is not enough: a scenario that removes
|
||||
// storage must first know that the specific slides it needs are present.
|
||||
export async function waitForCachedAssets(page, assetIds, timeout = 60_000) {
|
||||
const { expect } = await import("@playwright/test");
|
||||
await expect
|
||||
.poll(
|
||||
async () => {
|
||||
const cached = await cachedMedia(page);
|
||||
return assetIds.every((id) => cached.some((path) => path.includes(id)));
|
||||
},
|
||||
{ timeout },
|
||||
)
|
||||
.toBe(true);
|
||||
}
|
||||
|
||||
export async function cachedMedia(page) {
|
||||
return page.evaluate(async () => {
|
||||
if (!(await caches.has("announcements-display-assets-v1"))) return [];
|
||||
const cache = await caches.open("announcements-display-assets-v1");
|
||||
return (await cache.keys()).map((request) => new URL(request.url).pathname);
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
import http from "node:http";
|
||||
|
||||
// The offline suite must genuinely remove infrastructure, not simulate it.
|
||||
// Talking to the Docker Engine socket directly avoids installing a CLI into
|
||||
// the browser image and keeps every scenario inside one Playwright run.
|
||||
const SOCKET = "/var/run/docker.sock";
|
||||
|
||||
function engine(method, path) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const request = http.request({ socketPath: SOCKET, method, path }, (response) => {
|
||||
let body = "";
|
||||
response.on("data", (chunk) => (body += chunk));
|
||||
response.on("end", () => resolve({ status: response.statusCode, body }));
|
||||
});
|
||||
request.on("error", reject);
|
||||
request.end();
|
||||
});
|
||||
}
|
||||
|
||||
const PROJECT = process.env.VERIFY_PROJECT || "announcements-p5b-verify";
|
||||
export const CONTAINERS = {
|
||||
web: `${PROJECT}-web-1`,
|
||||
api: `${PROJECT}-api-1`,
|
||||
storage: `${PROJECT}-seaweedfs-1`,
|
||||
};
|
||||
|
||||
export async function stop(...names) {
|
||||
for (const name of names) await engine("POST", `/containers/${name}/stop?t=2`);
|
||||
}
|
||||
|
||||
export async function start(...names) {
|
||||
for (const name of names) await engine("POST", `/containers/${name}/start`);
|
||||
}
|
||||
|
||||
export async function isRunning(name) {
|
||||
const { status, body } = await engine("GET", `/containers/${name}/json`);
|
||||
if (status !== 200) return false;
|
||||
return Boolean(JSON.parse(body).State?.Running);
|
||||
}
|
||||
|
||||
export async function waitUntilHealthy(baseUrl, timeoutMs = 120_000) {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
try {
|
||||
const response = await fetch(`${baseUrl}/api/playback`, {
|
||||
signal: AbortSignal.timeout(2_000),
|
||||
});
|
||||
if (response.ok) return true;
|
||||
} catch {
|
||||
// Still coming up.
|
||||
}
|
||||
await new Promise((resolve) => setTimeout(resolve, 500));
|
||||
}
|
||||
throw new Error("stack did not become reachable in time");
|
||||
}
|
||||
|
||||
// With the API stopped but Nginx still running, the proxy answers 502. That is
|
||||
// a response, not a reachable API, so a gateway error counts as unreachable
|
||||
// just like a refused connection does.
|
||||
export async function waitUntilUnreachable(baseUrl, timeoutMs = 60_000) {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
try {
|
||||
const response = await fetch(`${baseUrl}/api/playback`, {
|
||||
signal: AbortSignal.timeout(1_000),
|
||||
});
|
||||
if (!response.ok) return true;
|
||||
} catch {
|
||||
return true;
|
||||
}
|
||||
await new Promise((resolve) => setTimeout(resolve, 250));
|
||||
}
|
||||
throw new Error("stack stayed reachable");
|
||||
}
|
||||
@@ -7,6 +7,11 @@
|
||||
*/
|
||||
const SHELL_CACHE = "announcements-display-shell-v1";
|
||||
const SHELL_FALLBACK = "/display";
|
||||
// A kiosk must reach its slides quickly. When a usable shell is already
|
||||
// cached, a slow or dead origin is never allowed to delay startup: a reverse
|
||||
// proxy whose backend is down can otherwise hold a navigation open for its
|
||||
// full connect timeout, leaving the screen blank the whole time.
|
||||
const SHELL_NETWORK_TIMEOUT_MS = 3_000;
|
||||
|
||||
// Paths the worker must never handle. Media is the asset manager's; the API and
|
||||
// socket are live server state and must never be answered from a cache.
|
||||
@@ -51,17 +56,33 @@ self.addEventListener("activate", (event) => {
|
||||
|
||||
async function networkFirst(request) {
|
||||
const cache = await caches.open(SHELL_CACHE);
|
||||
try {
|
||||
const cached =
|
||||
(await cache.match(SHELL_FALLBACK)) || (await cache.match("/index.html"));
|
||||
|
||||
// Without a cached shell there is nothing to fall back to, so wait normally.
|
||||
if (!cached) {
|
||||
const response = await fetch(request);
|
||||
if (response && response.ok) {
|
||||
// Keep the shell fallback fresh so an online kiosk always updates.
|
||||
await cache.put(SHELL_FALLBACK, response.clone());
|
||||
}
|
||||
if (response && response.ok) await cache.put(SHELL_FALLBACK, response.clone());
|
||||
return response;
|
||||
} catch (error) {
|
||||
const cached = (await cache.match(SHELL_FALLBACK)) || (await cache.match("/index.html"));
|
||||
if (cached) return cached;
|
||||
throw error;
|
||||
}
|
||||
|
||||
const abort = new AbortController();
|
||||
const timer = setTimeout(() => abort.abort(), SHELL_NETWORK_TIMEOUT_MS);
|
||||
try {
|
||||
const response = await fetch(request, { signal: abort.signal });
|
||||
clearTimeout(timer);
|
||||
if (response && response.ok) {
|
||||
// Keep the shell fresh so an online kiosk always updates.
|
||||
await cache.put(SHELL_FALLBACK, response.clone());
|
||||
return response;
|
||||
}
|
||||
// A production kiosk reaches this origin through a reverse proxy, so an
|
||||
// outage behind it arrives as 502/503/504 rather than a refused
|
||||
// connection. That is still an outage: serve the cached shell.
|
||||
return cached;
|
||||
} catch {
|
||||
clearTimeout(timer);
|
||||
return cached;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@ import { useEffect, useRef } from "react";
|
||||
import { io } from "socket.io-client";
|
||||
import { createAssetManager } from "./assetCache.js";
|
||||
import { createDisplayController } from "./displayController.js";
|
||||
import { DISPLAY_ERROR_KINDS, createDisplayError } from "./errors.js";
|
||||
import {
|
||||
createDisplayPersistence,
|
||||
createDisplayStore,
|
||||
@@ -80,6 +81,25 @@ export default function DisplayApp() {
|
||||
heartbeat: () => controller.sendHeartbeat(),
|
||||
};
|
||||
}
|
||||
|
||||
// Offline capability needs a secure context. Rather than silently
|
||||
// degrading, the display reports it so it is diagnosable from
|
||||
// /api/displays and Home Assistant instead of on screen.
|
||||
if (!window.isSecureContext || !("serviceWorker" in navigator)) {
|
||||
controller.reportCapabilityLimit(
|
||||
createDisplayError(DISPLAY_ERROR_KINDS.OFFLINE_CAPABILITY_UNAVAILABLE, {
|
||||
message: window.isSecureContext
|
||||
? "This browser provides no Service Worker; offline startup is unavailable"
|
||||
: "This origin is not a secure context; offline caching is unavailable",
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
// Restore signage from the last known good state before any server
|
||||
// contact, so a cold start with the infrastructure down still shows
|
||||
// announcements.
|
||||
await controller.bootstrapFromPersistence();
|
||||
if (cancelled) return;
|
||||
controller.start();
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,263 @@
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import {
|
||||
nextOfflineBoundary,
|
||||
resolveOfflineState,
|
||||
} from "../offlineTimeline.js";
|
||||
|
||||
const T0 = 2_000_000_000_000;
|
||||
|
||||
function entry(id, position, overrides = {}) {
|
||||
return {
|
||||
id,
|
||||
position,
|
||||
assetId: `asset-${id}`,
|
||||
effectiveDurationMs: 5_000,
|
||||
scheduleStartAt: null,
|
||||
scheduleEndAt: null,
|
||||
expirationAt: null,
|
||||
playable: true,
|
||||
asset: { id: `asset-${id}`, media: { url: `/media/${id}.webp`, width: 1920, height: 1080 } },
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
const MANIFEST = {
|
||||
entries: [
|
||||
entry("a", 0, { effectiveDurationMs: 3_000 }),
|
||||
entry("b", 1, { effectiveDurationMs: 9_000 }),
|
||||
entry("c", 2, { effectiveDurationMs: 4_000 }),
|
||||
],
|
||||
};
|
||||
|
||||
function anchor(overrides = {}) {
|
||||
return {
|
||||
entryId: "a",
|
||||
startedAt: T0,
|
||||
deadlineAt: T0 + 3_000,
|
||||
transportState: "playing",
|
||||
mode: "on",
|
||||
pausedRemainingMs: null,
|
||||
staticAssetId: null,
|
||||
staticTarget: null,
|
||||
pendingPlaylistId: null,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
test("mixed durations advance on absolute deadlines without drift", () => {
|
||||
const cycle = 3_000 + 9_000 + 4_000;
|
||||
const cases = [
|
||||
[0, "a", 0],
|
||||
[2_999, "a", 0],
|
||||
[3_000, "b", 3_000],
|
||||
[11_999, "b", 3_000],
|
||||
[12_000, "c", 12_000],
|
||||
[15_999, "c", 12_000],
|
||||
[cycle, "a", cycle],
|
||||
];
|
||||
for (const [offset, expectedId, expectedStart] of cases) {
|
||||
const resolved = resolveOfflineState({ manifest: MANIFEST, anchor: anchor(), at: T0 + offset });
|
||||
assert.equal(resolved.kind, "normal", `+${offset}`);
|
||||
assert.equal(resolved.entryId, expectedId, `+${offset} entry`);
|
||||
assert.equal(resolved.startedAt, T0 + expectedStart, `+${offset} start`);
|
||||
}
|
||||
});
|
||||
|
||||
test("many cycles offline stay exactly on the original phase", () => {
|
||||
const cycle = 16_000;
|
||||
for (const cycles of [10, 1_000, 100_000]) {
|
||||
const resolved = resolveOfflineState({
|
||||
manifest: MANIFEST,
|
||||
anchor: anchor(),
|
||||
at: T0 + cycles * cycle + 12_500,
|
||||
});
|
||||
assert.equal(resolved.entryId, "c");
|
||||
assert.equal(
|
||||
resolved.startedAt,
|
||||
T0 + cycles * cycle + 12_000,
|
||||
"no accumulated drift after many cycles",
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test("a schedule start offline brings an entry back into rotation", () => {
|
||||
const manifest = {
|
||||
entries: [
|
||||
entry("always", 0, { effectiveDurationMs: 1_000 }),
|
||||
entry("joins", 1, { effectiveDurationMs: 1_000, scheduleStartAt: T0 + 20_000 }),
|
||||
],
|
||||
};
|
||||
const before = resolveOfflineState({
|
||||
manifest,
|
||||
anchor: anchor({ entryId: "always", deadlineAt: T0 + 1_000 }),
|
||||
at: T0 + 19_500,
|
||||
});
|
||||
assert.equal(before.entryId, "always", "the future entry is not used early");
|
||||
|
||||
const after = resolveOfflineState({
|
||||
manifest,
|
||||
anchor: anchor({ entryId: "always", deadlineAt: T0 + 1_000 }),
|
||||
at: T0 + 20_500,
|
||||
});
|
||||
assert.equal(after.entryId, "joins", "[start, end) is inclusive at start");
|
||||
});
|
||||
|
||||
test("a schedule end and an expiration offline both retire an entry", () => {
|
||||
const ending = {
|
||||
entries: [
|
||||
entry("ends", 0, { effectiveDurationMs: 5_000, scheduleEndAt: T0 + 12_500 }),
|
||||
entry("stays", 1, { effectiveDurationMs: 5_000 }),
|
||||
],
|
||||
};
|
||||
const afterEnd = resolveOfflineState({
|
||||
manifest: ending,
|
||||
anchor: anchor({ entryId: "ends", deadlineAt: T0 + 5_000 }),
|
||||
at: T0 + 40_000,
|
||||
});
|
||||
assert.equal(afterEnd.entryId, "stays");
|
||||
|
||||
const expiring = {
|
||||
entries: [
|
||||
entry("expires", 0, { effectiveDurationMs: 6_000, expirationAt: T0 + 15_000 }),
|
||||
entry("permanent", 1, { effectiveDurationMs: 4_000 }),
|
||||
],
|
||||
};
|
||||
const afterExpiry = resolveOfflineState({
|
||||
manifest: expiring,
|
||||
anchor: anchor({ entryId: "expires", deadlineAt: T0 + 6_000 }),
|
||||
at: T0 + 60_000,
|
||||
});
|
||||
assert.equal(afterExpiry.entryId, "permanent");
|
||||
});
|
||||
|
||||
test("a cached playlist with nothing eligible is legitimately black", () => {
|
||||
const manifest = { entries: [entry("only", 0, { scheduleEndAt: T0 + 8_000 })] };
|
||||
const resolved = resolveOfflineState({
|
||||
manifest,
|
||||
anchor: anchor({ entryId: "only", deadlineAt: T0 + 5_000 }),
|
||||
at: T0 + 30_000,
|
||||
});
|
||||
assert.equal(resolved.kind, "black");
|
||||
assert.equal(resolved.reason, "no_playable_content");
|
||||
});
|
||||
|
||||
test("mode off keeps advancing the hidden transport underneath", () => {
|
||||
const resolved = resolveOfflineState({
|
||||
manifest: MANIFEST,
|
||||
anchor: anchor({ mode: "off" }),
|
||||
at: T0 + 12_500,
|
||||
});
|
||||
// Mode is a visual gate applied by the renderer; the cursor still moves.
|
||||
assert.equal(resolved.kind, "normal");
|
||||
assert.equal(resolved.entryId, "c");
|
||||
});
|
||||
|
||||
test("a paused display holds its slide and its remainder offline", () => {
|
||||
const resolved = resolveOfflineState({
|
||||
manifest: MANIFEST,
|
||||
anchor: anchor({
|
||||
transportState: "paused",
|
||||
entryId: "b",
|
||||
pausedRemainingMs: 4_000,
|
||||
startedAt: null,
|
||||
deadlineAt: null,
|
||||
}),
|
||||
at: T0 + 6 * 3_600_000,
|
||||
});
|
||||
assert.equal(resolved.kind, "paused");
|
||||
assert.equal(resolved.entryId, "b", "pause never advances merely because time passed");
|
||||
assert.equal(resolved.remainingMs, 4_000);
|
||||
});
|
||||
|
||||
test("a paused display still reconciles a held slide that became ineligible", () => {
|
||||
const manifest = {
|
||||
entries: [
|
||||
entry("held", 0, { expirationAt: T0 + 10_000 }),
|
||||
entry("survivor", 1),
|
||||
],
|
||||
};
|
||||
const resolved = resolveOfflineState({
|
||||
manifest,
|
||||
anchor: anchor({ transportState: "paused", entryId: "held", pausedRemainingMs: 2_000 }),
|
||||
at: T0 + 20_000,
|
||||
});
|
||||
assert.equal(resolved.kind, "paused", "it stays paused");
|
||||
assert.equal(resolved.entryId, "survivor", "but not on an expired slide");
|
||||
});
|
||||
|
||||
test("static holds indefinitely offline and releases only on real expiration", () => {
|
||||
const held = resolveOfflineState({
|
||||
manifest: MANIFEST,
|
||||
anchor: anchor({
|
||||
transportState: "static",
|
||||
staticAssetId: "static-1",
|
||||
staticTarget: { assetId: "static-1", asset: { expirationAt: null } },
|
||||
pausedRemainingMs: 2_000,
|
||||
}),
|
||||
at: T0 + 30 * 24 * 3_600_000,
|
||||
});
|
||||
assert.equal(held.kind, "static");
|
||||
assert.equal(held.assetId, "static-1", "a permanent static asset holds forever");
|
||||
|
||||
const released = resolveOfflineState({
|
||||
manifest: MANIFEST,
|
||||
anchor: anchor({
|
||||
entryId: "a",
|
||||
transportState: "static",
|
||||
staticAssetId: "static-1",
|
||||
staticTarget: { assetId: "static-1", expirationAt: T0 + 10_000 },
|
||||
pausedRemainingMs: 2_000,
|
||||
}),
|
||||
at: T0 + 11_000,
|
||||
});
|
||||
// Normal playback resumes from the expiration instant with its remainder.
|
||||
assert.equal(released.kind, "normal");
|
||||
assert.equal(released.entryId, "a");
|
||||
assert.equal(released.deadlineAt, T0 + 12_000);
|
||||
});
|
||||
|
||||
test("an unknown pending activation holds the last good frame at the boundary", () => {
|
||||
const beforeBoundary = resolveOfflineState({
|
||||
manifest: MANIFEST,
|
||||
anchor: anchor({ pendingPlaylistId: "playlist-b" }),
|
||||
at: T0 + 1_000,
|
||||
});
|
||||
assert.equal(beforeBoundary.kind, "normal", "the current slide still finishes");
|
||||
|
||||
const atBoundary = resolveOfflineState({
|
||||
manifest: MANIFEST,
|
||||
anchor: anchor({ pendingPlaylistId: "playlist-b" }),
|
||||
at: T0 + 3_000,
|
||||
});
|
||||
assert.equal(atBoundary.kind, "hold");
|
||||
assert.equal(
|
||||
atBoundary.reason,
|
||||
"pending_activation_unknown",
|
||||
"never continue deeper into a playlist the server is replacing",
|
||||
);
|
||||
});
|
||||
|
||||
test("the next offline wake-up is the earliest real boundary", () => {
|
||||
const resolved = resolveOfflineState({ manifest: MANIFEST, anchor: anchor(), at: T0 + 1_000 });
|
||||
assert.equal(
|
||||
nextOfflineBoundary({ manifest: MANIFEST, resolved, at: T0 + 1_000 }),
|
||||
T0 + 3_000,
|
||||
"the current slide deadline",
|
||||
);
|
||||
|
||||
const scheduled = {
|
||||
entries: [entry("future", 0, { scheduleStartAt: T0 + 50_000 })],
|
||||
};
|
||||
const black = resolveOfflineState({
|
||||
manifest: scheduled,
|
||||
anchor: anchor({ entryId: "future", startedAt: null, deadlineAt: null }),
|
||||
at: T0,
|
||||
});
|
||||
assert.equal(black.kind, "black");
|
||||
assert.equal(
|
||||
nextOfflineBoundary({ manifest: scheduled, resolved: black, at: T0 }),
|
||||
T0 + 50_000,
|
||||
"a black display still wakes for a cached schedule start",
|
||||
);
|
||||
});
|
||||
@@ -1,5 +1,6 @@
|
||||
import { createAssetManager, DisplayAssetError } from "./assetCache.js";
|
||||
import { DISPLAY_ERROR_KINDS, createDisplayError, serializeDisplayError, shouldClearError } from "./errors.js";
|
||||
import { nextOfflineBoundary, resolveOfflineState } from "./offlineTimeline.js";
|
||||
import { resolveScenePresentation } from "./presentation.js";
|
||||
import {
|
||||
canPredictLocally,
|
||||
@@ -39,6 +40,7 @@ export function createDisplayController({
|
||||
setIntervalFn = (...args) => setInterval(...args),
|
||||
clearIntervalFn = (...args) => clearInterval(...args),
|
||||
heartbeatIntervalMs = HEARTBEAT_INTERVAL_MS,
|
||||
logger = { warn() {} },
|
||||
onDiagnostics = () => {},
|
||||
} = {}) {
|
||||
let accepted = null;
|
||||
@@ -54,6 +56,10 @@ export function createDisplayController({
|
||||
let currentError = null;
|
||||
let destroyed = false;
|
||||
let stagingToken = 0;
|
||||
let online = false;
|
||||
let offlineTimer = null;
|
||||
let offlineAnchor = null;
|
||||
let offlineActive = false;
|
||||
const layerUrls = [null, null];
|
||||
const unavailableEntryIds = new Set();
|
||||
const failedMediaUrls = new Set();
|
||||
@@ -68,6 +74,9 @@ export function createDisplayController({
|
||||
renderedAssetId: renderedTarget?.assetId ?? null,
|
||||
renderedEntryId: renderedTarget?.entryId ?? null,
|
||||
predictedEntryId,
|
||||
online,
|
||||
offlineActive,
|
||||
offlineAnchor,
|
||||
unavailableEntryIds: [...unavailableEntryIds],
|
||||
mostRecentError: currentError,
|
||||
activeObjectUrls: assets.activeObjectUrlCount(),
|
||||
@@ -85,6 +94,13 @@ export function createDisplayController({
|
||||
diagnostics();
|
||||
}
|
||||
|
||||
// A capability limit is a standing condition, not a transient failure, so it
|
||||
// is only recorded when nothing more specific is already being reported.
|
||||
function reportCapabilityLimit(displayError) {
|
||||
if (currentError) return;
|
||||
recordError(displayError);
|
||||
}
|
||||
|
||||
function clearErrorIfResolved() {
|
||||
const manifestAssetIds = (promotedManifest.entries || []).map(
|
||||
(entry) => entry.assetId,
|
||||
@@ -321,6 +337,177 @@ export function createDisplayController({
|
||||
armPrediction(at + (next.effectiveDurationMs ?? 0));
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------ offline
|
||||
|
||||
function anchorFrom(source) {
|
||||
if (!source) return null;
|
||||
return {
|
||||
entryId: source.currentEntryId ?? null,
|
||||
startedAt: source.timing?.slideStartedAt ?? source.timeline?.slideStartedAt ?? null,
|
||||
deadlineAt: source.timing?.nextSlideAt ?? source.timeline?.nextSlideAt ?? null,
|
||||
transportState: source.transportState,
|
||||
mode: source.mode,
|
||||
pausedRemainingMs:
|
||||
source.pausedRemainingMs ?? source.timeline?.pausedRemainingMs ?? null,
|
||||
staticAssetId: source.staticAssetId ?? source.staticTarget?.assetId ?? null,
|
||||
staticTarget: source.staticTarget ?? null,
|
||||
pendingPlaylistId: source.pendingPlaylistId ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
function clearOfflineTimer() {
|
||||
if (offlineTimer !== null) {
|
||||
clearTimeoutFn(offlineTimer);
|
||||
offlineTimer = null;
|
||||
}
|
||||
}
|
||||
|
||||
// A locally broken asset must not stall rotation. The timeline itself stays
|
||||
// authoritative-equivalent; only what is painted is substituted.
|
||||
function usableEntryFor(entryId, at) {
|
||||
const entries = [...(promotedManifest.entries || [])].sort(
|
||||
(left, right) => left.position - right.position,
|
||||
);
|
||||
const resolved = entries.find((entry) => entry.id === entryId);
|
||||
if (resolved && !unavailableEntryIds.has(resolved.id)) return resolved;
|
||||
const index = entries.findIndex((entry) => entry.id === entryId);
|
||||
for (let offset = 1; offset <= entries.length; offset += 1) {
|
||||
const candidate = entries[(Math.max(index, 0) + offset) % entries.length];
|
||||
if (unavailableEntryIds.has(candidate.id)) continue;
|
||||
if (
|
||||
(candidate.scheduleStartAt == null || at >= candidate.scheduleStartAt) &&
|
||||
(candidate.scheduleEndAt == null || at < candidate.scheduleEndAt) &&
|
||||
(candidate.expirationAt == null || at < candidate.expirationAt)
|
||||
) {
|
||||
return candidate;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
async function renderOffline(resolved, at) {
|
||||
if (resolved.kind === "hold") {
|
||||
// Correct-but-held: the next boundary belongs to a playlist this display
|
||||
// does not have, so the last good frame stays up.
|
||||
return;
|
||||
}
|
||||
if (offlineAnchor?.mode === "off") setGate(true);
|
||||
|
||||
if (resolved.kind === "black") {
|
||||
// The cached playlist genuinely has nothing eligible; black is correct.
|
||||
if (offlineAnchor?.mode !== "off") hideContent();
|
||||
renderedTarget = null;
|
||||
return;
|
||||
}
|
||||
if (resolved.kind === "static") {
|
||||
const target = offlineAnchor?.staticTarget;
|
||||
if (!target?.asset?.media?.url) {
|
||||
// Missing local bytes never mean the operator ended static.
|
||||
recordError(
|
||||
createDisplayError(DISPLAY_ERROR_KINDS.ASSET_FETCH_FAILED, {
|
||||
assetId: resolved.assetId,
|
||||
message: "The held static asset is not available in the local cache",
|
||||
}),
|
||||
);
|
||||
return;
|
||||
}
|
||||
generation += 1;
|
||||
const scene = buildScene(target, "static");
|
||||
if (scene) await showScene(scene, generation, 0);
|
||||
return;
|
||||
}
|
||||
|
||||
const entry = usableEntryFor(resolved.entryId, at);
|
||||
if (!entry) {
|
||||
// Authoritative content exists but nothing local can render it. The last
|
||||
// good frame is kept rather than pretending the playlist went black.
|
||||
recordError(
|
||||
createDisplayError(DISPLAY_ERROR_KINDS.MANIFEST_INVALID, {
|
||||
entryId: resolved.entryId,
|
||||
message: "No cached slide of the active playlist can be rendered offline",
|
||||
}),
|
||||
);
|
||||
return;
|
||||
}
|
||||
generation += 1;
|
||||
const scene = buildScene(entry, "normal");
|
||||
if (scene) await showScene(scene, generation, accepted?.crossfadeDurationMs ?? 0);
|
||||
}
|
||||
|
||||
async function runOfflineCycle() {
|
||||
if (destroyed || online || !offlineAnchor) return;
|
||||
const at = estimatedServerNow();
|
||||
let resolved;
|
||||
try {
|
||||
resolved = resolveOfflineState({ manifest: promotedManifest, anchor: offlineAnchor, at });
|
||||
} catch (error) {
|
||||
warnOffline(error);
|
||||
return;
|
||||
}
|
||||
await renderOffline(resolved, at);
|
||||
if (destroyed || online) return;
|
||||
|
||||
const boundary = nextOfflineBoundary({ manifest: promotedManifest, resolved, at });
|
||||
clearOfflineTimer();
|
||||
if (boundary === null) return;
|
||||
offlineTimer = setTimeoutFn(
|
||||
() => {
|
||||
offlineTimer = null;
|
||||
void runOfflineCycle();
|
||||
},
|
||||
Math.max(0, boundary - at),
|
||||
);
|
||||
diagnostics();
|
||||
}
|
||||
|
||||
function warnOffline(error) {
|
||||
try {
|
||||
logger?.warn?.(`Offline playback evaluation failed: ${error?.message}`);
|
||||
} catch {
|
||||
// Offline rendering must never depend on logging.
|
||||
}
|
||||
}
|
||||
|
||||
// The display becomes a deterministic local continuation of the last
|
||||
// authoritative state. This is resilience, never an attempt to become a
|
||||
// second authority.
|
||||
function enterOffline(source) {
|
||||
if (destroyed) return;
|
||||
const anchor = anchorFrom(source);
|
||||
if (!anchor) return;
|
||||
online = false;
|
||||
offlineActive = true;
|
||||
offlineAnchor = anchor;
|
||||
clearPrediction();
|
||||
void runOfflineCycle();
|
||||
}
|
||||
|
||||
function exitOffline() {
|
||||
offlineActive = false;
|
||||
clearOfflineTimer();
|
||||
}
|
||||
|
||||
// Cold start: restore signage from persisted state before, and without, any
|
||||
// server contact.
|
||||
async function bootstrapFromPersistence() {
|
||||
if (!persistence) return false;
|
||||
let record;
|
||||
try {
|
||||
record = await persistence.readLastKnownGood();
|
||||
} catch {
|
||||
record = null;
|
||||
}
|
||||
if (!record) return false;
|
||||
|
||||
promotedManifest = record.manifest || { entries: [] };
|
||||
currentError = record.mostRecentError ?? null;
|
||||
// Reconstruct the authoritative clock from the persisted anchor.
|
||||
serverOffset = record.timing.serverTime - record.timing.recordedAt;
|
||||
setGate(record.mode === "off");
|
||||
enterOffline(record);
|
||||
return true;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- snapshots
|
||||
|
||||
async function applySnapshot(snapshot) {
|
||||
@@ -331,8 +518,13 @@ export function createDisplayController({
|
||||
return { accepted: false, reason: decision.reason };
|
||||
}
|
||||
|
||||
// The server is authoritative again the instant it speaks. Any local
|
||||
// offline cursor is abandoned rather than reconciled or reported back.
|
||||
online = true;
|
||||
exitOffline();
|
||||
// Packet arrival time is not slide start time; re-anchor on every accepted
|
||||
// snapshot so local scheduling follows the server's clock.
|
||||
// snapshot so local scheduling follows the server's clock. A fresh anchor
|
||||
// also discards any local clock drift accumulated while disconnected.
|
||||
serverOffset = snapshot.serverTime - now();
|
||||
accepted = snapshot;
|
||||
predictedEntryId = snapshot.currentEntryId ?? null;
|
||||
@@ -418,13 +610,16 @@ export function createDisplayController({
|
||||
});
|
||||
socket.on("connect", () => {
|
||||
// A reconnect resynchronizes from the server; it never reloads the page
|
||||
// and never clears the visible frame.
|
||||
// and never clears the visible frame. The same persistent client ID is
|
||||
// reused, so the server sees one display, not a new one.
|
||||
void sendHeartbeat();
|
||||
diagnostics();
|
||||
});
|
||||
socket.on("disconnect", () => {
|
||||
// The last good scene stays exactly where it is.
|
||||
// The last good scene stays exactly where it is while local playback
|
||||
// takes over from the last authoritative state.
|
||||
clearPrediction();
|
||||
enterOffline(accepted ?? offlineAnchor);
|
||||
diagnostics();
|
||||
});
|
||||
socket.connect();
|
||||
@@ -436,6 +631,7 @@ export function createDisplayController({
|
||||
generation += 1;
|
||||
stagingToken += 1;
|
||||
clearPrediction();
|
||||
clearOfflineTimer();
|
||||
stopHeartbeat();
|
||||
socket.off("playback:state");
|
||||
socket.off("connect");
|
||||
@@ -448,7 +644,10 @@ export function createDisplayController({
|
||||
|
||||
return {
|
||||
applySnapshot,
|
||||
bootstrapFromPersistence,
|
||||
destroy,
|
||||
enterOffline,
|
||||
reportCapabilityLimit,
|
||||
diagnostics,
|
||||
estimatedServerNow,
|
||||
sendHeartbeat,
|
||||
|
||||
@@ -5,6 +5,11 @@
|
||||
export const DISPLAY_DB_NAME = "announcements-display";
|
||||
export const DISPLAY_DB_VERSION = 1;
|
||||
export const DISPLAY_STORE = "state";
|
||||
// The record schema is versioned independently of the database so adding
|
||||
// fields does not require an IndexedDB upgrade. A display carrying an older
|
||||
// record simply discards it and rebuilds from the server on its next
|
||||
// successful connection.
|
||||
export const LKG_SCHEMA_VERSION = 2;
|
||||
|
||||
export const STATE_KEYS = Object.freeze({
|
||||
CLIENT_ID: "clientId",
|
||||
@@ -70,10 +75,14 @@ export function createDisplayStore({ database }) {
|
||||
// than allowed to corrupt startup.
|
||||
export function isValidLastKnownGood(value) {
|
||||
if (!value || typeof value !== "object" || Array.isArray(value)) return false;
|
||||
if (value.schemaVersion !== DISPLAY_DB_VERSION) return false;
|
||||
if (value.schemaVersion !== LKG_SCHEMA_VERSION) return false;
|
||||
if (typeof value.playbackRevision !== "number") return false;
|
||||
if (!value.manifest || typeof value.manifest !== "object") return false;
|
||||
if (!Array.isArray(value.manifest.entries)) return false;
|
||||
if (!value.timing || typeof value.timing !== "object") return false;
|
||||
// Without a usable clock anchor the record cannot drive offline playback.
|
||||
if (typeof value.timing.serverTime !== "number") return false;
|
||||
if (typeof value.timing.recordedAt !== "number") return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
@@ -103,7 +112,7 @@ export function createDisplayPersistence({ store, createId, now = Date.now }) {
|
||||
|
||||
async function writeLastKnownGood(snapshot, extra = {}) {
|
||||
const record = {
|
||||
schemaVersion: DISPLAY_DB_VERSION,
|
||||
schemaVersion: LKG_SCHEMA_VERSION,
|
||||
savedAt: now(),
|
||||
playbackRevision: snapshot.playbackRevision,
|
||||
settingsRevision: snapshot.settingsRevision ?? null,
|
||||
@@ -113,6 +122,8 @@ export function createDisplayPersistence({ store, createId, now = Date.now }) {
|
||||
transportState: snapshot.transportState,
|
||||
currentEntryId: snapshot.currentEntryId ?? null,
|
||||
staticAssetId: snapshot.staticTarget?.assetId ?? null,
|
||||
pendingPlaylistId: snapshot.pendingPlaylistId ?? null,
|
||||
pausedRemainingMs: snapshot.timeline?.pausedRemainingMs ?? null,
|
||||
// The timing anchor is what lets a future offline start reason about
|
||||
// where playback should be, without a server to ask.
|
||||
timing: {
|
||||
@@ -126,6 +137,9 @@ export function createDisplayPersistence({ store, createId, now = Date.now }) {
|
||||
staticTarget: snapshot.staticTarget || null,
|
||||
mostRecentError: extra.mostRecentError ?? null,
|
||||
};
|
||||
// One record in one transaction: a cold start can never observe a new
|
||||
// manifest paired with an old timing anchor. The previous complete record
|
||||
// is only replaced once this one is durably written.
|
||||
await store.write(STATE_KEYS.LAST_KNOWN_GOOD, record);
|
||||
return record;
|
||||
}
|
||||
|
||||
@@ -5,6 +5,7 @@ export const DISPLAY_ERROR_KINDS = Object.freeze({
|
||||
ASSET_FETCH_FAILED: "asset_fetch_failed",
|
||||
CACHE_WRITE_FAILED: "cache_write_failed",
|
||||
MANIFEST_INVALID: "manifest_invalid",
|
||||
OFFLINE_CAPABILITY_UNAVAILABLE: "offline_capability_unavailable",
|
||||
});
|
||||
|
||||
const KNOWN_KINDS = new Set(Object.values(DISPLAY_ERROR_KINDS));
|
||||
|
||||
@@ -0,0 +1,297 @@
|
||||
import { entryPlayableAt } from "./snapshot.js";
|
||||
|
||||
// Deterministic local continuation of the last authoritative state.
|
||||
//
|
||||
// This is a small client-side evaluator, intentionally separate from the
|
||||
// verified Phase 3B server engine: the server stays the single authority while
|
||||
// connected, and this only reproduces its documented semantics well enough to
|
||||
// keep a disconnected kiosk correct. Contract tests compare the two directly.
|
||||
|
||||
export const TRANSITION_SAFETY_LIMIT = 10_000;
|
||||
|
||||
export function entryDuration(entry) {
|
||||
const value = entry?.effectiveDurationMs;
|
||||
return Number.isFinite(value) && value > 0 ? value : 0;
|
||||
}
|
||||
|
||||
function ordered(manifest) {
|
||||
return [...(manifest?.entries || [])].sort(
|
||||
(left, right) => left.position - right.position,
|
||||
);
|
||||
}
|
||||
|
||||
function boundariesOf(entry) {
|
||||
return [entry.scheduleStartAt, entry.scheduleEndAt, entry.expirationAt];
|
||||
}
|
||||
|
||||
// The first instant in (after, through] at which any entry's eligibility can
|
||||
// change. Between such instants the eligible set is constant, which is what
|
||||
// makes whole-cycle skipping safe.
|
||||
function firstEligibilityBoundary(entries, after, through) {
|
||||
let best = null;
|
||||
for (const entry of entries) {
|
||||
for (const value of boundariesOf(entry)) {
|
||||
if (typeof value !== "number") continue;
|
||||
if (value > after && value <= through && (best === null || value < best)) {
|
||||
best = value;
|
||||
}
|
||||
}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
|
||||
// A currently displayed entry can be cut short by its own schedule end or asset
|
||||
// expiration, never by another entry's boundary.
|
||||
function invalidBoundaryFor(entry, startedAt, through) {
|
||||
if (!entry) return null;
|
||||
let best = null;
|
||||
for (const value of [entry.scheduleEndAt, entry.expirationAt]) {
|
||||
if (typeof value !== "number") continue;
|
||||
if (value > startedAt && value <= through && (best === null || value < best)) {
|
||||
best = value;
|
||||
}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
|
||||
function firstPlayable(entries, at) {
|
||||
return entries.find((entry) => entryPlayableAt(entry, at)) || null;
|
||||
}
|
||||
|
||||
function adjacentPlayable(entries, currentId, at) {
|
||||
if (entries.length === 0) return null;
|
||||
const index = entries.findIndex((entry) => entry.id === currentId);
|
||||
if (index < 0) return firstPlayable(entries, at);
|
||||
for (let offset = 1; offset <= entries.length; offset += 1) {
|
||||
const candidate = entries[(index + offset) % entries.length];
|
||||
if (entryPlayableAt(candidate, at)) return candidate;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// The next instant at which a currently ineligible playlist becomes playable
|
||||
// again, used to recover from a legitimately empty stretch.
|
||||
function nextAvailability(entries, after, through) {
|
||||
const starts = entries
|
||||
.map((entry) => entry.scheduleStartAt)
|
||||
.filter((value) => typeof value === "number" && value > after && value <= through)
|
||||
.sort((left, right) => left - right);
|
||||
return starts.find((candidate) => entries.some((entry) => entryPlayableAt(entry, candidate))) ?? null;
|
||||
}
|
||||
|
||||
// Skips whole rotations with arithmetic instead of stepping one slide at a
|
||||
// time, so a kiosk offline for a week catches up in constant work per
|
||||
// eligibility boundary rather than per missed slide.
|
||||
function fastForward(entries, cursor, at) {
|
||||
const boundary = firstEligibilityBoundary(entries, cursor.startedAt, at);
|
||||
const through = boundary === null ? at : boundary - 1;
|
||||
if (cursor.deadlineAt > through) return false;
|
||||
|
||||
const eligible = entries.filter((entry) => entryPlayableAt(entry, through));
|
||||
const index = eligible.findIndex((entry) => entry.id === cursor.entryId);
|
||||
if (index < 0 || eligible.length === 0) return false;
|
||||
|
||||
const cycleDuration = eligible.reduce((total, entry) => total + entryDuration(entry), 0);
|
||||
if (cycleDuration <= 0) return false;
|
||||
|
||||
let position = (index + 1) % eligible.length;
|
||||
let startedAt = cursor.deadlineAt;
|
||||
const cycles = Math.floor((through - startedAt) / cycleDuration);
|
||||
if (cycles > 0) startedAt += cycles * cycleDuration;
|
||||
|
||||
let deadlineAt = startedAt + entryDuration(eligible[position]);
|
||||
while (deadlineAt <= through) {
|
||||
startedAt = deadlineAt;
|
||||
position = (position + 1) % eligible.length;
|
||||
deadlineAt = startedAt + entryDuration(eligible[position]);
|
||||
}
|
||||
|
||||
cursor.entryId = eligible[position].id;
|
||||
cursor.startedAt = startedAt;
|
||||
cursor.deadlineAt = deadlineAt;
|
||||
return boundary === null;
|
||||
}
|
||||
|
||||
// Walks the rotation forward from an anchor to `at`, honouring schedule and
|
||||
// expiration boundaries exactly as the server does.
|
||||
function advanceRotation(entries, anchor, at) {
|
||||
const cursor = {
|
||||
entryId: anchor.entryId,
|
||||
startedAt: anchor.startedAt,
|
||||
deadlineAt: anchor.deadlineAt,
|
||||
};
|
||||
|
||||
for (let step = 0; step < TRANSITION_SAFETY_LIMIT; step += 1) {
|
||||
let current = entries.find((entry) => entry.id === cursor.entryId);
|
||||
|
||||
if (!current) {
|
||||
const availableAt = nextAvailability(
|
||||
entries,
|
||||
Number.isFinite(cursor.startedAt) ? cursor.startedAt : at,
|
||||
at,
|
||||
);
|
||||
const startAt = availableAt ?? at;
|
||||
const replacement = firstPlayable(entries, startAt);
|
||||
if (!replacement) return { kind: "black", reason: "no_playable_content" };
|
||||
cursor.entryId = replacement.id;
|
||||
cursor.startedAt = startAt;
|
||||
cursor.deadlineAt = startAt + entryDuration(replacement);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (cursor.deadlineAt <= at && fastForward(entries, cursor, at)) {
|
||||
current = entries.find((entry) => entry.id === cursor.entryId);
|
||||
}
|
||||
if (cursor.entryId !== current?.id) continue;
|
||||
|
||||
const invalidAt = invalidBoundaryFor(current, cursor.startedAt, at);
|
||||
const playableNow = entryPlayableAt(current, at);
|
||||
if (playableNow && cursor.deadlineAt > at) {
|
||||
return {
|
||||
kind: "normal",
|
||||
entryId: cursor.entryId,
|
||||
startedAt: cursor.startedAt,
|
||||
deadlineAt: cursor.deadlineAt,
|
||||
};
|
||||
}
|
||||
|
||||
let boundary = cursor.deadlineAt <= at ? cursor.deadlineAt : at;
|
||||
if (invalidAt !== null) boundary = Math.min(boundary, invalidAt);
|
||||
|
||||
const next = adjacentPlayable(entries, cursor.entryId, boundary);
|
||||
if (!next) {
|
||||
const availableAt = nextAvailability(entries, boundary, at);
|
||||
if (availableAt === null) {
|
||||
const now = firstPlayable(entries, at);
|
||||
if (!now) return { kind: "black", reason: "no_playable_content" };
|
||||
return {
|
||||
kind: "normal",
|
||||
entryId: now.id,
|
||||
startedAt: at,
|
||||
deadlineAt: at + entryDuration(now),
|
||||
};
|
||||
}
|
||||
const replacement = firstPlayable(entries, availableAt);
|
||||
cursor.entryId = replacement.id;
|
||||
cursor.startedAt = availableAt;
|
||||
cursor.deadlineAt = availableAt + entryDuration(replacement);
|
||||
continue;
|
||||
}
|
||||
|
||||
cursor.entryId = next.id;
|
||||
cursor.startedAt = boundary;
|
||||
cursor.deadlineAt = boundary + entryDuration(next);
|
||||
}
|
||||
|
||||
throw new Error("Offline catch-up exceeded its transition safety limit");
|
||||
}
|
||||
|
||||
function staticStillValid(staticTarget, at) {
|
||||
if (!staticTarget) return false;
|
||||
const expiration = staticTarget.expirationAt ?? staticTarget.asset?.expirationAt ?? null;
|
||||
// A null expiration is permanent; local age never invents one.
|
||||
if (typeof expiration !== "number") return true;
|
||||
return at < expiration;
|
||||
}
|
||||
|
||||
function heldEntryAt(entries, anchor, at) {
|
||||
const current = entries.find((entry) => entry.id === anchor.entryId);
|
||||
if (current && entryPlayableAt(current, at)) return current;
|
||||
// A held slide can still be invalidated by schedule or expiration; the
|
||||
// replacement is held in turn rather than starting to advance.
|
||||
return adjacentPlayable(entries, anchor.entryId, at);
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves what a disconnected display should be showing at `at`, from the last
|
||||
* authoritative state it persisted. Pure and deterministic.
|
||||
*/
|
||||
export function resolveOfflineState({ manifest, anchor, at }) {
|
||||
const entries = ordered(manifest);
|
||||
|
||||
if (anchor.pendingPlaylistId) {
|
||||
// The next boundary belongs to a playlist this display does not have. Going
|
||||
// deeper into the old playlist would show known-wrong content, so the last
|
||||
// good frame is held until the server can speak again.
|
||||
if (typeof anchor.deadlineAt === "number" && at >= anchor.deadlineAt) {
|
||||
return { kind: "hold", reason: "pending_activation_unknown" };
|
||||
}
|
||||
}
|
||||
|
||||
if (anchor.staticAssetId) {
|
||||
if (staticStillValid(anchor.staticTarget, at)) {
|
||||
return { kind: "static", assetId: anchor.staticAssetId };
|
||||
}
|
||||
// A genuinely expired static asset releases the hold and normal playback
|
||||
// resumes from that exact instant.
|
||||
const expiration =
|
||||
anchor.staticTarget?.expirationAt ?? anchor.staticTarget?.asset?.expirationAt ?? at;
|
||||
const resumeAt = Math.min(at, expiration);
|
||||
const current = entries.find((entry) => entry.id === anchor.entryId);
|
||||
const full = entryDuration(current);
|
||||
const remaining = Math.min(
|
||||
typeof anchor.pausedRemainingMs === "number" ? anchor.pausedRemainingMs : full,
|
||||
full,
|
||||
);
|
||||
const deadlineAt = resumeAt + remaining;
|
||||
return advanceRotation(
|
||||
entries,
|
||||
{ entryId: anchor.entryId, startedAt: deadlineAt - full, deadlineAt },
|
||||
at,
|
||||
);
|
||||
}
|
||||
|
||||
if (anchor.transportState === "paused") {
|
||||
const held = heldEntryAt(entries, anchor, at);
|
||||
if (!held) return { kind: "black", reason: "no_playable_content" };
|
||||
return {
|
||||
kind: "paused",
|
||||
entryId: held.id,
|
||||
remainingMs:
|
||||
held.id === anchor.entryId && typeof anchor.pausedRemainingMs === "number"
|
||||
? Math.min(anchor.pausedRemainingMs, entryDuration(held))
|
||||
: entryDuration(held),
|
||||
};
|
||||
}
|
||||
|
||||
if (entries.length === 0) return { kind: "black", reason: "no_playable_content" };
|
||||
if (typeof anchor.deadlineAt !== "number" || typeof anchor.startedAt !== "number") {
|
||||
const now = firstPlayable(entries, at);
|
||||
if (!now) return { kind: "black", reason: "no_playable_content" };
|
||||
return {
|
||||
kind: "normal",
|
||||
entryId: now.id,
|
||||
startedAt: at,
|
||||
deadlineAt: at + entryDuration(now),
|
||||
};
|
||||
}
|
||||
|
||||
return advanceRotation(
|
||||
entries,
|
||||
{ entryId: anchor.entryId, startedAt: anchor.startedAt, deadlineAt: anchor.deadlineAt },
|
||||
at,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* The next instant local playback must re-evaluate: the current slide's
|
||||
* deadline, or the next cached eligibility boundary when nothing is playable.
|
||||
*/
|
||||
export function nextOfflineBoundary({ manifest, resolved, at }) {
|
||||
const entries = ordered(manifest);
|
||||
const candidates = [];
|
||||
if (resolved?.kind === "normal" && typeof resolved.deadlineAt === "number") {
|
||||
candidates.push(resolved.deadlineAt);
|
||||
}
|
||||
for (const entry of entries) {
|
||||
for (const value of boundariesOf(entry)) {
|
||||
if (typeof value === "number" && value > at) candidates.push(value);
|
||||
}
|
||||
}
|
||||
if (resolved?.kind === "static") {
|
||||
const expiration =
|
||||
resolved.expirationAt ?? resolved.staticExpirationAt ?? null;
|
||||
if (typeof expiration === "number" && expiration > at) candidates.push(expiration);
|
||||
}
|
||||
return candidates.length > 0 ? Math.min(...candidates) : null;
|
||||
}
|
||||
@@ -66,6 +66,32 @@ function parseDisplayError(value) {
|
||||
return value;
|
||||
}
|
||||
|
||||
// Only validated error codes become notifications. Arbitrary browser text
|
||||
// stays telemetry and never reaches Home Assistant.
|
||||
const DISPLAY_ERROR_SEVERITY = Object.freeze({
|
||||
slide_decode_failed: "warning",
|
||||
asset_fetch_failed: "warning",
|
||||
// A display that cannot write its cache still shows slides now, but has lost
|
||||
// its ability to survive the next outage.
|
||||
cache_write_failed: "warning",
|
||||
// Nothing renderable for an active playlist is an on-air problem.
|
||||
manifest_invalid: "critical",
|
||||
offline_capability_unavailable: "warning",
|
||||
});
|
||||
|
||||
const DISPLAY_ERROR_MESSAGES = Object.freeze({
|
||||
slide_decode_failed: "could not decode a slide",
|
||||
asset_fetch_failed: "could not fetch a slide",
|
||||
cache_write_failed: "could not store slides for offline use",
|
||||
manifest_invalid: "cannot render any slide of its active playlist",
|
||||
offline_capability_unavailable: "cannot cache slides for offline use",
|
||||
});
|
||||
|
||||
function errorConditionKey(clientId, error) {
|
||||
const scope = error.assetId ? `:${error.assetId}` : "";
|
||||
return `display:${clientId}:${error.kind}${scope}`;
|
||||
}
|
||||
|
||||
function validateHeartbeat(payload) {
|
||||
if (payload === null || typeof payload !== "object" || Array.isArray(payload)) {
|
||||
throw new TypeError("heartbeat must be an object");
|
||||
@@ -221,6 +247,47 @@ function createDisplayService({
|
||||
return result.recovered;
|
||||
}
|
||||
|
||||
// Reconciles the one display-error condition this client currently reports
|
||||
// against whatever it had open before. Phase 4 deduplication stays
|
||||
// authoritative: a persisting error is never re-notified, and a resolved one
|
||||
// recovers exactly once.
|
||||
function reconcileErrorIncidents(clientId, row, at) {
|
||||
if (!notifications) return;
|
||||
const parsed = parseDisplayError(row.last_error);
|
||||
const structured =
|
||||
parsed && typeof parsed === "object" && DISPLAY_ERROR_SEVERITY[parsed.kind]
|
||||
? parsed
|
||||
: null;
|
||||
const desired = structured ? errorConditionKey(clientId, structured) : null;
|
||||
const offlineKey = conditionKey(clientId);
|
||||
|
||||
for (const key of notifications.openConditionKeys(`display:${clientId}:`)) {
|
||||
// The display-offline condition is a separate concern and is never
|
||||
// touched here.
|
||||
if (key === offlineKey || key === desired) continue;
|
||||
notifications.observeRecovery(key, {
|
||||
event: "display.error_cleared",
|
||||
message: `Display ${clientId} no longer reports that error.`,
|
||||
context: { clientId, recoveredAt: new Date(at).toISOString() },
|
||||
});
|
||||
}
|
||||
|
||||
if (!structured) return;
|
||||
notifications.observeFailure(desired, {
|
||||
event: `display.${structured.kind}`,
|
||||
severity: DISPLAY_ERROR_SEVERITY[structured.kind],
|
||||
message: `Display ${clientId} ${DISPLAY_ERROR_MESSAGES[structured.kind]}.`,
|
||||
context: {
|
||||
clientId,
|
||||
outputId: row.output_id,
|
||||
kind: structured.kind,
|
||||
assetId: structured.assetId ?? null,
|
||||
entryId: structured.entryId ?? null,
|
||||
detail: structured.message ?? null,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function heartbeat(payload) {
|
||||
const telemetry = validateHeartbeat(payload);
|
||||
const at = now();
|
||||
@@ -237,6 +304,7 @@ function createDisplayService({
|
||||
const row = findClient.get(telemetry.clientId);
|
||||
const wasOffline = existing ? isOffline(existing, at) : false;
|
||||
const recovered = recoverOffline(row, at);
|
||||
reconcileErrorIncidents(telemetry.clientId, row, at);
|
||||
const errorChanged = (existing?.last_error ?? null) !== telemetry.lastError;
|
||||
|
||||
if (!existing) {
|
||||
@@ -339,7 +407,9 @@ module.exports = {
|
||||
DEFAULT_OFFLINE_THRESHOLD_MS,
|
||||
HEARTBEAT_PROTOCOL_VERSION,
|
||||
MAX_ERROR_LENGTH,
|
||||
DISPLAY_ERROR_SEVERITY,
|
||||
createDisplayService,
|
||||
errorConditionKey,
|
||||
parseDisplayError,
|
||||
validateHeartbeat,
|
||||
};
|
||||
|
||||
@@ -232,6 +232,15 @@ function createNotificationService({
|
||||
return Boolean(findOpen.get(conditionKey));
|
||||
}
|
||||
|
||||
// Lets a caller find the conditions it currently has open, so a changed
|
||||
// condition can be recovered without inventing a second bookkeeping store.
|
||||
function openConditionKeys(prefix) {
|
||||
return listOpenIncidents
|
||||
.all()
|
||||
.map((incident) => incident.condition_key)
|
||||
.filter((key) => (prefix ? key.startsWith(prefix) : true));
|
||||
}
|
||||
|
||||
async function deliverOne(row) {
|
||||
const at = now();
|
||||
const headers = { "content-type": "application/json" };
|
||||
@@ -341,6 +350,7 @@ function createNotificationService({
|
||||
deliverDue,
|
||||
isOpen,
|
||||
observeFailure,
|
||||
openConditionKeys,
|
||||
observeRecovery,
|
||||
start,
|
||||
status,
|
||||
|
||||
@@ -3,7 +3,9 @@ const assert = require("node:assert/strict");
|
||||
const { openDatabase } = require("../src/database");
|
||||
const { runMigrations } = require("../src/database/migrate");
|
||||
const {
|
||||
DISPLAY_ERROR_SEVERITY,
|
||||
createDisplayService,
|
||||
errorConditionKey,
|
||||
parseDisplayError,
|
||||
} = require("../src/displays/service");
|
||||
const { createNotificationService } = require("../src/notifications");
|
||||
@@ -422,3 +424,163 @@ test("a structured display error round-trips without loosening validation", (t)
|
||||
assert.equal(parseDisplayError("{not json"), "{not json");
|
||||
assert.equal(parseDisplayError('{"no":"kind"}'), '{"no":"kind"}');
|
||||
});
|
||||
|
||||
function structuredError(kind, overrides = {}) {
|
||||
return JSON.stringify({
|
||||
kind,
|
||||
assetId: null,
|
||||
entryId: null,
|
||||
message: `${kind} detail`,
|
||||
at: BASE_TIME,
|
||||
...overrides,
|
||||
});
|
||||
}
|
||||
|
||||
test("a structured display error opens exactly one incident and never repeats", (t) => {
|
||||
const harness = createHarness();
|
||||
t.after(() => harness.database.close());
|
||||
|
||||
for (let beat = 0; beat < 6; beat += 1) {
|
||||
harness.clock.set(BASE_TIME + beat * 12_000);
|
||||
harness.service.heartbeat(
|
||||
heartbeatPayload({
|
||||
mostRecentError: structuredError("slide_decode_failed", { assetId: "asset-a" }),
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
const incidents = harness.incidents();
|
||||
assert.equal(incidents.length, 1, "a persisting error is reported once, not per heartbeat");
|
||||
assert.equal(incidents[0].event, "display.slide_decode_failed");
|
||||
assert.equal(incidents[0].severity, "warning");
|
||||
assert.equal(
|
||||
incidents[0].condition_key,
|
||||
errorConditionKey("lobby-display", { kind: "slide_decode_failed", assetId: "asset-a" }),
|
||||
);
|
||||
assert.equal(JSON.parse(incidents[0].context_json).assetId, "asset-a");
|
||||
assert.equal(harness.outbox().length, 1);
|
||||
});
|
||||
|
||||
test("a resolved display error recovers exactly once", (t) => {
|
||||
const harness = createHarness();
|
||||
t.after(() => harness.database.close());
|
||||
|
||||
harness.service.heartbeat(
|
||||
heartbeatPayload({
|
||||
mostRecentError: structuredError("asset_fetch_failed", { assetId: "asset-a" }),
|
||||
}),
|
||||
);
|
||||
assert.equal(harness.outbox().length, 1);
|
||||
|
||||
for (let beat = 1; beat <= 3; beat += 1) {
|
||||
harness.clock.set(BASE_TIME + beat * 12_000);
|
||||
harness.service.heartbeat(heartbeatPayload({ mostRecentError: null }));
|
||||
}
|
||||
|
||||
const incidents = harness.incidents();
|
||||
assert.equal(incidents.length, 1);
|
||||
assert.equal(incidents[0].state, "recovered");
|
||||
assert.equal(harness.outbox().length, 2, "one failure and one recovery");
|
||||
assert.equal(harness.outbox()[1].transition, "recovery");
|
||||
});
|
||||
|
||||
test("an error moving to another asset recovers the old condition and opens the new one", (t) => {
|
||||
const harness = createHarness();
|
||||
t.after(() => harness.database.close());
|
||||
|
||||
harness.service.heartbeat(
|
||||
heartbeatPayload({
|
||||
mostRecentError: structuredError("asset_fetch_failed", { assetId: "asset-a" }),
|
||||
}),
|
||||
);
|
||||
harness.clock.set(BASE_TIME + 12_000);
|
||||
harness.service.heartbeat(
|
||||
heartbeatPayload({
|
||||
mostRecentError: structuredError("asset_fetch_failed", { assetId: "asset-b" }),
|
||||
}),
|
||||
);
|
||||
|
||||
const byKey = Object.fromEntries(
|
||||
harness.incidents().map((incident) => [incident.condition_key, incident]),
|
||||
);
|
||||
const keyA = errorConditionKey("lobby-display", {
|
||||
kind: "asset_fetch_failed",
|
||||
assetId: "asset-a",
|
||||
});
|
||||
const keyB = errorConditionKey("lobby-display", {
|
||||
kind: "asset_fetch_failed",
|
||||
assetId: "asset-b",
|
||||
});
|
||||
assert.equal(byKey[keyA].state, "recovered", "the resolved asset condition closes");
|
||||
assert.equal(byKey[keyB].state, "open", "the new asset condition opens");
|
||||
assert.equal(harness.outbox().length, 3);
|
||||
});
|
||||
|
||||
test("only validated error codes become incidents", (t) => {
|
||||
const harness = createHarness();
|
||||
t.after(() => harness.database.close());
|
||||
|
||||
// Arbitrary browser text is telemetry, never a Home Assistant notification.
|
||||
harness.service.heartbeat(
|
||||
heartbeatPayload({ mostRecentError: "TypeError: undefined is not a function" }),
|
||||
);
|
||||
assert.equal(harness.incidents().length, 0);
|
||||
assert.equal(
|
||||
harness.service.list()[0].mostRecentError,
|
||||
"TypeError: undefined is not a function",
|
||||
"the raw telemetry is still visible to operators",
|
||||
);
|
||||
|
||||
// A structured payload with an unrecognised code is equally inert.
|
||||
harness.clock.set(BASE_TIME + 12_000);
|
||||
harness.service.heartbeat(
|
||||
heartbeatPayload({ mostRecentError: structuredError("something_invented") }),
|
||||
);
|
||||
assert.equal(harness.incidents().length, 0);
|
||||
assert.equal(harness.outbox().length, 0);
|
||||
assert.equal(harness.service.list()[0].mostRecentError.kind, "something_invented");
|
||||
});
|
||||
|
||||
test("a manifest that cannot render locally is critical", (t) => {
|
||||
const harness = createHarness();
|
||||
t.after(() => harness.database.close());
|
||||
|
||||
harness.service.heartbeat(
|
||||
heartbeatPayload({ mostRecentError: structuredError("manifest_invalid") }),
|
||||
);
|
||||
|
||||
assert.equal(harness.incidents()[0].severity, "critical");
|
||||
assert.equal(DISPLAY_ERROR_SEVERITY.manifest_invalid, "critical");
|
||||
assert.equal(DISPLAY_ERROR_SEVERITY.offline_capability_unavailable, "warning");
|
||||
});
|
||||
|
||||
test("display error incidents never disturb the display offline condition", (t) => {
|
||||
const harness = createHarness();
|
||||
t.after(() => harness.database.close());
|
||||
|
||||
harness.service.heartbeat(heartbeatPayload());
|
||||
// The display goes offline, then returns while reporting a slide problem.
|
||||
harness.clock.runTo(BASE_TIME + THRESHOLD);
|
||||
assert.equal(harness.incidents().length, 1);
|
||||
assert.equal(harness.incidents()[0].event, "display.offline");
|
||||
|
||||
harness.clock.set(BASE_TIME + 60_000);
|
||||
harness.service.heartbeat(
|
||||
heartbeatPayload({
|
||||
mostRecentError: structuredError("slide_decode_failed", { assetId: "asset-a" }),
|
||||
}),
|
||||
);
|
||||
|
||||
const offline = harness
|
||||
.incidents()
|
||||
.find((incident) => incident.condition_key === "display:lobby-display");
|
||||
assert.equal(offline.state, "recovered", "returning still recovers the offline condition once");
|
||||
const decode = harness
|
||||
.incidents()
|
||||
.find((incident) => incident.event === "display.slide_decode_failed");
|
||||
assert.equal(decode.state, "open");
|
||||
assert.equal(
|
||||
harness.incidents().filter((incident) => incident.event === "display.offline").length,
|
||||
1,
|
||||
);
|
||||
});
|
||||
|
||||
@@ -0,0 +1,288 @@
|
||||
const test = require("node:test");
|
||||
const assert = require("node:assert/strict");
|
||||
const path = require("node:path");
|
||||
const { pathToFileURL } = require("node:url");
|
||||
const { openDatabase } = require("../src/database");
|
||||
const { runMigrations } = require("../src/database/migrate");
|
||||
const { createPlaybackService } = require("../src/playback/service");
|
||||
|
||||
// Client/server contract.
|
||||
//
|
||||
// The disconnected display necessarily reimplements part of the playback
|
||||
// timeline. Rather than trusting that the two agree, every scenario below runs
|
||||
// the real Phase 3B engine and the browser evaluator over identical data and
|
||||
// asserts they land on the same slide, start, and deadline. Phase 3B is not
|
||||
// refactored to share code; it stays the verified authority.
|
||||
const CLIENT_MODULE = pathToFileURL(
|
||||
path.join(__dirname, "..", "..", "client", "src", "display", "offlineTimeline.js"),
|
||||
).href;
|
||||
|
||||
const BASE_TIME = 2_000_000_000_000;
|
||||
|
||||
function createHarness({ defaultDurationMs = 1_000 } = {}) {
|
||||
const database = openDatabase({ path: ":memory:" });
|
||||
runMigrations(database);
|
||||
database
|
||||
.prepare("UPDATE app_settings SET default_duration_ms = ?, updated_at = ? WHERE id = 1")
|
||||
.run(defaultDurationMs, BASE_TIME);
|
||||
|
||||
let current = BASE_TIME;
|
||||
const clock = {
|
||||
now: () => current,
|
||||
set(value) {
|
||||
current = value;
|
||||
},
|
||||
};
|
||||
let assetSequence = 0;
|
||||
let service = null;
|
||||
|
||||
function seedAsset({ expirationAt = null } = {}) {
|
||||
const ordinal = ++assetSequence;
|
||||
const id = `asset-${ordinal}`;
|
||||
const sha = ordinal.toString(16).padStart(64, "0");
|
||||
database
|
||||
.prepare(
|
||||
`INSERT INTO assets (
|
||||
id, source_sha256, display_name, original_filename, status, expiration_at,
|
||||
display_object_key, display_sha256, display_content_type, display_byte_size,
|
||||
display_width, display_height, created_at, updated_at
|
||||
) VALUES (?, ?, ?, ?, 'ready', ?, ?, ?, 'image/webp', 10, 1920, 1080, ?, ?)`,
|
||||
)
|
||||
.run(id, sha, id, `${id}.png`, expirationAt, `private/${id}.webp`, sha, BASE_TIME, BASE_TIME);
|
||||
return id;
|
||||
}
|
||||
|
||||
function seedPlaylist(entries) {
|
||||
database
|
||||
.prepare(
|
||||
"INSERT INTO playlists (id, name, revision, created_at, updated_at) VALUES ('p', 'P', 1, ?, ?)",
|
||||
)
|
||||
.run(BASE_TIME, BASE_TIME);
|
||||
entries.forEach((options, position) => {
|
||||
const assetId = seedAsset({ expirationAt: options.assetExpirationAt ?? null });
|
||||
database
|
||||
.prepare(
|
||||
`INSERT INTO playlist_entries (
|
||||
id, playlist_id, asset_id, enabled, duration_override_ms,
|
||||
schedule_start_at, schedule_end_at, position, created_at, updated_at
|
||||
) VALUES (?, 'p', ?, 1, ?, ?, ?, ?, ?, ?)`,
|
||||
)
|
||||
.run(
|
||||
options.id,
|
||||
assetId,
|
||||
options.durationMs ?? null,
|
||||
options.scheduleStartAt ?? null,
|
||||
options.scheduleEndAt ?? null,
|
||||
position,
|
||||
BASE_TIME,
|
||||
BASE_TIME,
|
||||
);
|
||||
});
|
||||
return "p";
|
||||
}
|
||||
|
||||
function startService() {
|
||||
service = createPlaybackService({
|
||||
database,
|
||||
now: clock.now,
|
||||
setTimeoutFn: () => null,
|
||||
clearTimeoutFn: () => {},
|
||||
logger: { warn() {} },
|
||||
});
|
||||
service.initialize();
|
||||
return service;
|
||||
}
|
||||
|
||||
function playback() {
|
||||
return database.prepare("SELECT * FROM playback_state WHERE output_id = 'global'").get();
|
||||
}
|
||||
|
||||
return {
|
||||
clock,
|
||||
close: () => database.close(),
|
||||
database,
|
||||
playback,
|
||||
seedPlaylist,
|
||||
startService,
|
||||
get service() {
|
||||
return service;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
// Runs one disconnected scenario through both implementations.
|
||||
async function compare(t, { entries, defaultDurationMs, offlineAt, label }) {
|
||||
const { resolveOfflineState } = await import(CLIENT_MODULE);
|
||||
const harness = createHarness({ defaultDurationMs });
|
||||
t.after(() => harness.close());
|
||||
harness.seedPlaylist(entries);
|
||||
|
||||
const service = harness.startService();
|
||||
service.activatePlaylist("p");
|
||||
|
||||
// What the display would have persisted as its last authoritative state.
|
||||
const snapshot = service.getSnapshot();
|
||||
const anchor = {
|
||||
entryId: snapshot.currentEntryId,
|
||||
startedAt: snapshot.timeline.slideStartedAt,
|
||||
deadlineAt: snapshot.timeline.nextSlideAt,
|
||||
transportState: snapshot.transportState,
|
||||
pausedRemainingMs: snapshot.timeline.pausedRemainingMs,
|
||||
staticAssetId: snapshot.staticTarget?.assetId ?? null,
|
||||
staticTarget: snapshot.staticTarget,
|
||||
pendingPlaylistId: snapshot.pendingPlaylistId ?? null,
|
||||
};
|
||||
const manifest = snapshot.manifest;
|
||||
|
||||
// The server catches up the same outage on restart.
|
||||
harness.clock.set(offlineAt);
|
||||
harness.startService();
|
||||
const server = harness.playback();
|
||||
|
||||
const client = resolveOfflineState({ manifest, anchor, at: offlineAt });
|
||||
|
||||
if (server.current_entry_id === null) {
|
||||
assert.equal(client.kind, "black", `${label}: both must report no playable content`);
|
||||
return { client, server };
|
||||
}
|
||||
|
||||
assert.equal(client.kind, "normal", `${label}: client must resolve a normal slide`);
|
||||
assert.equal(client.entryId, server.current_entry_id, `${label}: same slide`);
|
||||
assert.equal(client.startedAt, server.slide_started_at, `${label}: same start instant`);
|
||||
assert.equal(client.deadlineAt, server.next_slide_at, `${label}: same deadline`);
|
||||
return { client, server };
|
||||
}
|
||||
|
||||
test("offline catch-up matches the server across mixed durations and wraparound", async (t) => {
|
||||
const entries = [
|
||||
{ id: "a", durationMs: 3_000 },
|
||||
{ id: "b", durationMs: 9_000 },
|
||||
{ id: "c", durationMs: 4_000 },
|
||||
];
|
||||
// One cycle is 16s; each instant lands in a different part of the rotation.
|
||||
for (const offset of [1_000, 3_000, 5_000, 12_000, 16_000, 17_000, 31_999, 32_000, 100_000]) {
|
||||
await compare(t, {
|
||||
entries,
|
||||
offlineAt: BASE_TIME + offset,
|
||||
label: `mixed durations +${offset}ms`,
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
test("offline catch-up matches the server over hours and days offline", async (t) => {
|
||||
const entries = [
|
||||
{ id: "a", durationMs: 3_000 },
|
||||
{ id: "b", durationMs: 9_000 },
|
||||
{ id: "c", durationMs: 4_000 },
|
||||
];
|
||||
const started = process.hrtime.bigint();
|
||||
for (const offset of [
|
||||
3_600_000, // one hour
|
||||
86_400_000, // one day
|
||||
604_800_000, // one week
|
||||
2_592_000_000, // thirty days
|
||||
]) {
|
||||
await compare(t, {
|
||||
entries,
|
||||
offlineAt: BASE_TIME + offset + 1_234,
|
||||
label: `long outage +${offset}ms`,
|
||||
});
|
||||
}
|
||||
const elapsedMs = Number(process.hrtime.bigint() - started) / 1e6;
|
||||
// Thirty days of 16-second cycles is ~160k slides; arithmetic catch-up must
|
||||
// not step through them one at a time.
|
||||
assert.ok(
|
||||
elapsedMs < 4_000,
|
||||
`catch-up must be arithmetic, not iterative (took ${elapsedMs.toFixed(0)}ms)`,
|
||||
);
|
||||
});
|
||||
|
||||
test("offline catch-up matches the server across a schedule start", async (t) => {
|
||||
const entries = [
|
||||
{ id: "always", durationMs: 1_000 },
|
||||
{ id: "joins", durationMs: 1_000, scheduleStartAt: BASE_TIME + 20_000 },
|
||||
];
|
||||
for (const offset of [19_999, 20_000, 20_001, 25_000, 60_000]) {
|
||||
await compare(t, {
|
||||
entries,
|
||||
offlineAt: BASE_TIME + offset,
|
||||
label: `schedule start +${offset}ms`,
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
test("offline catch-up matches the server across a schedule end", async (t) => {
|
||||
const entries = [
|
||||
{ id: "ends", durationMs: 5_000, scheduleEndAt: BASE_TIME + 12_500 },
|
||||
{ id: "stays", durationMs: 5_000 },
|
||||
];
|
||||
for (const offset of [12_499, 12_500, 12_501, 30_000, 120_000]) {
|
||||
await compare(t, {
|
||||
entries,
|
||||
offlineAt: BASE_TIME + offset,
|
||||
label: `schedule end +${offset}ms`,
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
test("offline catch-up matches the server across an asset expiration", async (t) => {
|
||||
const entries = [
|
||||
{ id: "expires", durationMs: 6_000, assetExpirationAt: BASE_TIME + 15_000 },
|
||||
{ id: "permanent", durationMs: 4_000 },
|
||||
];
|
||||
for (const offset of [14_999, 15_000, 15_001, 40_000, 300_000]) {
|
||||
await compare(t, {
|
||||
entries,
|
||||
offlineAt: BASE_TIME + offset,
|
||||
label: `expiration +${offset}ms`,
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
test("offline evaluation agrees with the server when nothing is playable", async (t) => {
|
||||
const entries = [{ id: "only", durationMs: 5_000, scheduleEndAt: BASE_TIME + 8_000 }];
|
||||
const { client } = await compare(t, {
|
||||
entries,
|
||||
offlineAt: BASE_TIME + 30_000,
|
||||
label: "no playable content",
|
||||
});
|
||||
assert.equal(client.kind, "black");
|
||||
assert.equal(client.reason, "no_playable_content");
|
||||
});
|
||||
|
||||
test("a permanent asset never expires locally no matter how long the display is offline", async (t) => {
|
||||
const { resolveOfflineState } = await import(CLIENT_MODULE);
|
||||
const manifest = {
|
||||
entries: [
|
||||
{
|
||||
id: "permanent",
|
||||
position: 0,
|
||||
assetId: "asset-permanent",
|
||||
effectiveDurationMs: 5_000,
|
||||
scheduleStartAt: null,
|
||||
scheduleEndAt: null,
|
||||
// A null expiration is permanent and must stay permanent.
|
||||
expirationAt: null,
|
||||
},
|
||||
],
|
||||
};
|
||||
const anchor = {
|
||||
entryId: "permanent",
|
||||
startedAt: BASE_TIME,
|
||||
deadlineAt: BASE_TIME + 5_000,
|
||||
transportState: "playing",
|
||||
};
|
||||
|
||||
const afterAYear = resolveOfflineState({
|
||||
manifest,
|
||||
anchor,
|
||||
at: BASE_TIME + 365 * 24 * 3_600_000,
|
||||
});
|
||||
assert.equal(afterAYear.kind, "normal");
|
||||
assert.equal(afterAYear.entryId, "permanent");
|
||||
assert.equal(
|
||||
(afterAYear.deadlineAt - afterAYear.startedAt) % 5_000,
|
||||
0,
|
||||
"the slide keeps its configured duration on the original phase",
|
||||
);
|
||||
});
|
||||
Reference in New Issue
Block a user