diff --git a/README.md b/README.md index 5fdacf6..3c5b8d9 100644 --- a/README.md +++ b/README.md @@ -369,18 +369,74 @@ direct navigation and refresh through the existing Nginx SPA fallback. The display stays a Chromium kiosk pointed at a web application: v2 makes that application reliable rather than replacing it with a native client. -Chromium should open `/display` in kiosk mode against an origin the browser -treats as secure — `http://localhost`, a private hostname over HTTPS, or the -equivalent. Cache Storage, Service Workers, and therefore offline behaviour are -unavailable on an insecure origin, and the display degrades to a network-only -renderer without them. +### Deployment topology + +The kiosk runs on a different computer from the Announcements host, so the +localhost secure-context exception is not available in production. A remote +display **must** use HTTPS: + +``` +Remote Chromium kiosk + | + HTTPS + | + v +Announcements web / reverse proxy + | + +-- React /display + +-- /api + +-- /socket.io + +-- /media +``` + +The reverse proxy terminates TLS; the API and web containers may keep speaking +plain HTTP behind it. The kiosk browser must trust the certificate — a public +certificate or an internal CA installed on the kiosk machine both work. + +Secure-context rules decide what the display can do: + +| Origin | Status | +| --- | --- | +| `https://display.example.org/display` | Production. Full offline capability. | +| `http://localhost:5175/display` | Development and automated testing only. Secure context, so offline works. | +| `http://10.0.0.5:5175/display` | Degraded, online only. **Not a valid kiosk configuration.** | + +Plain LAN HTTP has no Service Worker, no Cache Storage, and no offline startup. +The display still shows slides while the server is reachable, but it cannot +survive an outage. It detects this at startup and reports +`offline_capability_unavailable` through its heartbeat so the limitation is +visible in `/api/displays` and Home Assistant rather than on screen. + +Do not work around this with `--ignore-certificate-errors`, +`--disable-web-security`, or an insecure-origin-as-secure flag. They disable +the protections the offline design relies on and do not represent a supportable +deployment. + +### Kiosk browser requirements + +The kiosk must use a **persistent** Chromium profile. The Service Worker, Cache +Storage, and IndexedDB that make offline operation possible all live in that +profile, and an incognito window, a guest profile, or a throwaway +`--user-data-dir` discards them on every launch. + +```sh +chromium \ + --kiosk \ + --user-data-dir=/var/lib/announcements-kiosk \ + https://display.example.org/display +``` The old v1 kiosk flags that disabled the browser cache are no longer required -and should be removed at cutover. They existed because v1 reused mutable slide +and must be removed at cutover. They existed because v1 reused mutable slide URLs; v2 media URLs are content-addressed, so a changed image always arrives under a new URL and stale bytes are impossible. Those flags now actively interfere with the Service Worker and Cache Storage the display depends on. +Clearing the browser's site data deletes the cached slides, the last-known-good +state, and the display's identity. Nothing is lost permanently: on its next +connection the display registers as a new client, re-caches the active playlist, +and resumes. No routine cache management is expected of an operator. + ### Transition guarantee The renderer owns two permanent full-screen scene layers. Each layer is a @@ -483,6 +539,95 @@ was required; `/api/displays` parses them back into an object. An error clears when the failing asset renders successfully or leaves the manifest — never merely because time passed. +### Offline resilience + +While connected the server is authoritative. While disconnected the display +becomes a deterministic local continuation of the last authoritative state it +persisted. The moment the server speaks again it is authoritative once more: +the local cursor is abandoned, never reconciled and never reported back. +Offline playback is resilience, not distributed consensus. + +Local continuation uses the persisted manifest and timing anchor. Elapsed time +is derived from the anchor — the authoritative `serverTime` and the local clock +reading when it was accepted — so a restart estimates where playback should be +rather than restarting a slide. Catch-up is arithmetic: whole rotations are +skipped in one step between eligibility boundaries, so a display offline for a +month resolves its position in milliseconds instead of stepping through every +missed slide. Mixed durations, wraparound, `[start, end)` schedule boundaries, +and asset expirations are all evaluated locally from the cached manifest. A null +expiration stays permanent no matter how long the display has been away. + +Transport state is preserved exactly. Mode off stays black while the hidden +cursor keeps advancing; a paused display holds its slide and remainder and never +advances merely because time passed or the browser restarted; static holds +indefinitely and releases only when the cached metadata says the asset genuinely +expired. If the server had queued an activation for a playlist the display does +not hold, the display holds its last good frame at that boundary rather than +continuing deeper into a playlist that is being replaced. + +The distinction between "the playlist is genuinely empty" and "the playlist +should play but the local bytes are unusable" is preserved: the first is +correctly black, the second keeps the last good frame and reports an error. + +This evaluator is a small client-side implementation, deliberately separate from +the verified Phase 3B engine. Contract tests run both over identical scenarios — +mixed durations, wraparound, schedule starts and ends, expirations, long +outages, and empty playlists — and assert they resolve the same slide, start, +and deadline. + +### Cold start + +After one successful online visit the display can start with the entire stack +unavailable. The Service Worker serves the application shell, IndexedDB +provides the client ID and last-known-good state, Cache Storage provides the +slide bytes, and local playback resumes from the persisted anchor. A reconnect +attempt continues in the background. + +A display that has never successfully cached anything has nothing trustworthy +to show, so it stays black. It never invents placeholder content. + +Because a remote kiosk reaches the origin through a reverse proxy, an outage +behind that proxy arrives as a 502 rather than a refused connection. The worker +treats a gateway error as an outage, and bounds how long it waits for the +network when a usable shell is already cached, so a kiosk restarting during an +outage reaches its slides quickly instead of waiting out a proxy timeout. + +### Reconnect + +On reconnect the authoritative snapshot wins. Required assets are staged and +decoded first and the current frame is kept until the new one is ready, so +resynchronization never exposes black and never replays the slides that were +missed. If the local calculation already agrees with the server, the timing +anchor is simply re-anchored with no visual churn. A playlist changed during the +outage is staged and adopted without mixing old and new entries, and the +superseded playlist's bytes are pruned once the new manifest is promoted. + +The server's own 45-second heartbeat threshold is unchanged. A display can be +showing announcements perfectly while disconnected and still be reported as +`display.offline`, which is the useful distinction: the audience sees slides, +and the operator still learns that the display lost contact. + +### Display errors as incidents + +Validated display error codes become Home Assistant incidents through the +existing Phase 4 machinery, keyed per display and condition so a persisting +error is reported once and recovers once: + +| Code | Severity | +| --- | --- | +| `slide_decode_failed` | warning | +| `asset_fetch_failed` | warning | +| `cache_write_failed` | warning | +| `manifest_invalid` | critical | +| `offline_capability_unavailable` | warning | + +Arbitrary browser text remains telemetry and never becomes a notification. If +an error occurs while the display is completely disconnected the server cannot +learn of it at that instant; the display persists its current error and reports +it on reconnect. A transient offline error that fully resolves before +reconnection therefore produces no historical notification. This is a documented +limitation rather than a browser-side durable event queue. + ## Storage and health The API uses a small path-style S3 adapter configured by `S3_ENDPOINT`, diff --git a/client/e2e/display-offline-degraded.spec.js b/client/e2e/display-offline-degraded.spec.js new file mode 100644 index 0000000..5ea40d1 --- /dev/null +++ b/client/e2e/display-offline-degraded.spec.js @@ -0,0 +1,291 @@ +import { expect, test } from "@playwright/test"; +import { ensureFixtures } from "./fixtures.js"; +import { activatePlaylist, resetPlayback } from "./helpers.js"; +import { + BASE_URL, + cachedMedia, + diagnostics, + launchKiosk, + opaqueLayerCount, + waitForCachedAssets, + waitForSlide, +} from "./offline-helpers.js"; +import { CONTAINERS, start, stop, waitUntilHealthy, waitUntilUnreachable } from "./stack.js"; + +test.describe.configure({ mode: "serial" }); +test.use({ baseURL: BASE_URL }); + +async function restoreStack() { + await start(CONTAINERS.storage, CONTAINERS.api, CONTAINERS.web); + await waitUntilHealthy(BASE_URL); +} + +test.afterAll(async () => { + await restoreStack(); +}); + +test("the app shell is served from the worker when the origin returns a gateway error", async ({ + request, +}) => { + const library = await ensureFixtures(request); + await resetPlayback(request); + await activatePlaylist(request, "Gateway", [ + { assetId: library["wide-a"].id, durationOverrideMs: 5_000 }, + ]); + + const { context, page } = await launchKiosk(); + try { + await page.goto(`${BASE_URL}/display`); + await waitForSlide(page); + await expect + .poll(async () => + page.evaluate(async () => + Boolean((await navigator.serviceWorker.getRegistration("/"))?.active), + ), + ) + .toBe(true); + await context.close(); + + // A remote kiosk reaches the origin through a reverse proxy, so losing the + // application behind it produces 5xx rather than a refused connection. + await stop(CONTAINERS.web); + await waitUntilUnreachable(BASE_URL); + + const restarted = await launchKiosk(); + try { + const response = await restarted.page.goto(`${BASE_URL}/display`, { + waitUntil: "domcontentloaded", + }); + expect( + response?.status(), + "a proxied 5xx outage must still yield the cached shell", + ).toBe(200); + await waitForSlide(restarted.page); + expect((await diagnostics(restarted.page)).offlineActive).toBe(true); + } finally { + await restarted.context.close(); + } + } finally { + await restoreStack(); + } +}); + +test("a damaged cached slide is skipped offline without stalling rotation", async ({ + request, +}) => { + const library = await ensureFixtures(request); + await resetPlayback(request); + await activatePlaylist(request, "Damaged", [ + { assetId: library["wide-a"].id, durationOverrideMs: 3_000 }, + { assetId: library["wide-d"].id, durationOverrideMs: 3_000 }, + { assetId: library["wide-c"].id, durationOverrideMs: 3_000 }, + ]); + + const { context, page } = await launchKiosk(); + try { + await page.goto(`${BASE_URL}/display`); + await waitForSlide(page); + await waitForCachedAssets(page, [ + library["wide-a"].id, + library["wide-d"].id, + library["wide-c"].id, + ]); + + await stop(CONTAINERS.api, CONTAINERS.storage); + await waitUntilUnreachable(BASE_URL); + + // Replace one cached slide's bytes with something undecodable, then reload + // so the damage is discovered while completely offline. + await page.evaluate(async (assetId) => { + const cache = await caches.open("announcements-display-assets-v1"); + for (const request of await cache.keys()) { + if (!request.url.includes(assetId)) continue; + await cache.put( + request, + new Response(new Blob([new Uint8Array([0, 1, 2, 3, 4])], { type: "image/webp" }), { + status: 200, + }), + ); + } + }, library["wide-d"].id); + await page.reload({ waitUntil: "domcontentloaded" }); + await waitForSlide(page); + + const seen = new Set(); + const deadline = Date.now() + 20_000; + while (Date.now() < deadline) { + const state = await page.evaluate(() => ({ + rendered: window.__display?.diagnostics()?.renderedAssetId ?? null, + opaque: [...document.querySelectorAll(".scene")].filter( + (layer) => Number(getComputedStyle(layer).opacity) > 0.99, + ).length, + })); + expect(state.opaque, "the last good frame is never dropped").toBeGreaterThanOrEqual(1); + if (state.rendered) seen.add(state.rendered); + await page.waitForTimeout(200); + } + + expect(seen.has(library["wide-d"].id), "the damaged slide never displays").toBe(false); + expect(seen.size, "the other cached slides keep rotating").toBeGreaterThanOrEqual(2); + const state = await diagnostics(page); + expect(state.unavailableEntryIds.length).toBeGreaterThanOrEqual(1); + expect(state.mostRecentError).not.toBeNull(); + } finally { + await context.close(); + await restoreStack(); + } +}); + +test("a missing media cache offline fails safe instead of fabricating state", async ({ + request, +}) => { + const library = await ensureFixtures(request); + await resetPlayback(request); + await activatePlaylist(request, "Empty cache", [ + { assetId: library["wide-a"].id, durationOverrideMs: 5_000 }, + ]); + + const { context, page } = await launchKiosk(); + try { + await page.goto(`${BASE_URL}/display`); + await waitForSlide(page); + const clientId = await page.evaluate(() => window.__display.clientId); + + await stop(CONTAINERS.api, CONTAINERS.storage, CONTAINERS.web); + await waitUntilUnreachable(BASE_URL); + + // Shell and IndexedDB survive; only the slide bytes are gone. + await page.evaluate(() => caches.delete("announcements-display-assets-v1")); + await page.reload({ waitUntil: "domcontentloaded" }); + await page.waitForTimeout(6_000); + + // No crash, no invented content, and the identity is intact. + expect(await page.evaluate(() => Boolean(window.__display))).toBe(true); + expect(await page.evaluate(() => window.__display.clientId)).toBe(clientId); + const state = await diagnostics(page); + expect(state.renderedAssetId, "nothing is fabricated").toBeNull(); + expect(state.mostRecentError, "the condition is recorded for later reporting").not.toBeNull(); + + // Once the infrastructure returns, the cache is rebuilt with no intervention. + await restoreStack(); + await expect + .poll(async () => (await diagnostics(page))?.renderedAssetId, { timeout: 90_000 }) + .not.toBeNull(); + await expect.poll(async () => (await cachedMedia(page)).length).toBeGreaterThanOrEqual(1); + } finally { + await context.close(); + await restoreStack(); + } +}); + +test("corrupt persisted state offline fails safe and recovers when the server returns", async ({ + request, +}) => { + const library = await ensureFixtures(request); + await resetPlayback(request); + await activatePlaylist(request, "Corrupt", [ + { assetId: library["wide-a"].id, durationOverrideMs: 5_000 }, + ]); + + const { context, page } = await launchKiosk(); + try { + await page.goto(`${BASE_URL}/display`); + await waitForSlide(page); + const clientId = await page.evaluate(() => window.__display.clientId); + + await page.evaluate(async () => { + const database = await new Promise((resolve) => { + const open = indexedDB.open("announcements-display", 1); + open.onsuccess = () => resolve(open.result); + }); + await new Promise((resolve) => { + const transaction = database.transaction("state", "readwrite"); + transaction + .objectStore("state") + .put({ schemaVersion: 99, manifest: "not-an-object" }, "lastKnownGood"); + transaction.oncomplete = resolve; + }); + database.close(); + }); + + await stop(CONTAINERS.api, CONTAINERS.storage); + await waitUntilUnreachable(BASE_URL); + await page.reload({ waitUntil: "domcontentloaded" }); + await page.waitForTimeout(5_000); + + // An unreadable record is discarded rather than trusted; the display stays + // alive and keeps its identity. + expect(await page.evaluate(() => Boolean(window.__display))).toBe(true); + expect(await page.evaluate(() => window.__display.clientId)).toBe(clientId); + + await restoreStack(); + await expect + .poll(async () => (await diagnostics(page))?.renderedAssetId, { timeout: 90_000 }) + .toBe(library["wide-a"].id); + } finally { + await context.close(); + await restoreStack(); + } +}); + +test("a worker update keeps the separate slide media cache intact", async ({ request }) => { + const library = await ensureFixtures(request); + await resetPlayback(request); + await activatePlaylist(request, "Worker update", [ + { assetId: library["wide-a"].id, durationOverrideMs: 60_000 }, + ]); + + const { context, page } = await launchKiosk(); + try { + await page.goto(`${BASE_URL}/display`); + await waitForSlide(page); + await expect + .poll(async () => + page.evaluate(async () => + Boolean((await navigator.serviceWorker.getRegistration("/"))?.active), + ), + ) + .toBe(true); + const mediaBefore = await cachedMedia(page); + expect(mediaBefore.length).toBeGreaterThanOrEqual(1); + + // Simulate a stale shell generation from an older application version. + await page.evaluate(async () => { + const stale = await caches.open("announcements-display-shell-v0"); + await stale.put("/display", new Response("old", { status: 200 })); + }); + // Cleanup runs when a worker activates, so force a real install/activate + // cycle rather than calling update() on an unchanged worker file. + await page.evaluate(async () => { + const registration = await navigator.serviceWorker.getRegistration("/"); + await registration.unregister(); + }); + await page.reload({ waitUntil: "domcontentloaded" }); + await waitForSlide(page); + await expect + .poll( + async () => + page.evaluate(async () => + Boolean((await navigator.serviceWorker.getRegistration("/"))?.active), + ), + { timeout: 45_000 }, + ) + .toBe(true); + + const cacheNames = await page.evaluate(() => caches.keys()); + expect( + cacheNames.includes("announcements-display-shell-v0"), + "superseded shell generations are cleaned up", + ).toBe(false); + expect(cacheNames).toContain("announcements-display-assets-v1"); + const mediaAfter = await cachedMedia(page); + expect( + mediaAfter, + "a worker update must never delete the explicit slide media cache", + ).toEqual(expect.arrayContaining(mediaBefore)); + expect(await opaqueLayerCount(page)).toBeGreaterThanOrEqual(1); + } finally { + await context.close(); + await restoreStack(); + } +}); diff --git a/client/e2e/display-offline.spec.js b/client/e2e/display-offline.spec.js new file mode 100644 index 0000000..fafb653 --- /dev/null +++ b/client/e2e/display-offline.spec.js @@ -0,0 +1,363 @@ +import { expect, test } from "@playwright/test"; +import { ensureFixtures } from "./fixtures.js"; +import { activatePlaylist, api, resetPlayback } from "./helpers.js"; +import { + BASE_URL, + cachedMedia, + diagnostics, + launchKiosk, + opaqueLayerCount, + waitForCachedAssets, + waitForSlide, +} from "./offline-helpers.js"; +import { CONTAINERS, start, stop, waitUntilHealthy, waitUntilUnreachable } from "./stack.js"; + +// These scenarios remove real infrastructure, so they must not interleave. +test.describe.configure({ mode: "serial" }); +test.use({ baseURL: BASE_URL }); + +async function restoreStack() { + await start(CONTAINERS.storage, CONTAINERS.api, CONTAINERS.web); + await waitUntilHealthy(BASE_URL); +} + +test.afterAll(async () => { + await restoreStack(); +}); + +test("cached rotation continues through a complete server outage", async ({ request }) => { + const library = await ensureFixtures(request); + await resetPlayback(request); + await activatePlaylist(request, "Outage", [ + { assetId: library["wide-a"].id, durationOverrideMs: 3_000 }, + { assetId: library["wide-b"].id, durationOverrideMs: 9_000 }, + { assetId: library["wide-c"].id, durationOverrideMs: 4_000 }, + ]); + + const { context, page } = await launchKiosk(); + try { + await page.goto(`${BASE_URL}/display`); + await waitForSlide(page); + // Every slide of this playlist must be cached before the outage begins. + await waitForCachedAssets(page, [ + library["wide-a"].id, + library["wide-b"].id, + library["wide-c"].id, + ]); + + await stop(CONTAINERS.api, CONTAINERS.storage); + await waitUntilUnreachable(BASE_URL); + + // Sample continuously across multiple full cycles with nothing serving. + const seen = new Set(); + const starts = new Map(); + const deadline = Date.now() + 40_000; + while (Date.now() < deadline) { + const state = await page.evaluate(() => ({ + rendered: window.__display?.diagnostics()?.renderedAssetId ?? null, + offline: window.__display?.diagnostics()?.offlineActive ?? false, + opaque: [...document.querySelectorAll(".scene")].filter( + (layer) => Number(getComputedStyle(layer).opacity) > 0.99, + ).length, + gate: document.querySelector(".output-gate").dataset.active, + })); + // The original black-frame bug must remain impossible offline too. + expect(state.opaque, "a good frame is always on screen offline").toBeGreaterThanOrEqual(1); + expect(state.gate).toBe("false"); + if (state.rendered) { + seen.add(state.rendered); + if (!starts.has(state.rendered)) starts.set(state.rendered, Date.now()); + } + await page.waitForTimeout(150); + } + + expect(await diagnostics(page).then((d) => d.offlineActive)).toBe(true); + expect(seen.size, "mixed-duration rotation kept cycling with no server").toBe(3); + expect(await page.evaluate(() => window.__loadCount ?? 0)).toBe(0); + + await restoreStack(); + // The server is authoritative again the moment it speaks. + await expect + .poll(async () => (await diagnostics(page))?.offlineActive, { timeout: 60_000 }) + .toBe(false); + const playback = await api(request, "/api/playback"); + await expect + .poll(async () => (await diagnostics(page))?.acceptedRevision, { timeout: 30_000 }) + .toBeGreaterThanOrEqual(playback.playbackRevision); + } finally { + await context.close(); + await restoreStack(); + } +}); + +test("a cold Chromium restart with the whole stack down restores signage", async ({ + request, +}) => { + const library = await ensureFixtures(request); + await resetPlayback(request); + await activatePlaylist(request, "Cold start", [ + { assetId: library["wide-a"].id, durationOverrideMs: 4_000 }, + { assetId: library["wide-d"].id, durationOverrideMs: 6_000 }, + ]); + + // A. Warm the persistent profile while everything is healthy. + let clientId; + let cachedBefore; + { + const { context, page } = await launchKiosk(); + await page.goto(`${BASE_URL}/display`); + await waitForSlide(page); + await expect + .poll(async () => + page.evaluate(async () => + Boolean((await navigator.serviceWorker.getRegistration("/"))?.active), + ), + ) + .toBe(true); + await expect.poll(async () => (await cachedMedia(page)).length).toBeGreaterThanOrEqual(2); + clientId = await page.evaluate(() => window.__display.clientId); + cachedBefore = await cachedMedia(page); + await context.close(); + } + + // B. Remove the entire origin: no web server, no API, no object storage. + await stop(CONTAINERS.web, CONTAINERS.api, CONTAINERS.storage); + await waitUntilUnreachable(BASE_URL); + + try { + // C. Relaunch the kiosk against the same URL with the same profile. + const { context, page } = await launchKiosk(); + try { + const response = await page.goto(`${BASE_URL}/display`, { waitUntil: "domcontentloaded" }); + expect( + response?.status(), + "the Service Worker served the app shell with the origin down", + ).toBe(200); + + await waitForSlide(page); + const offline = await diagnostics(page); + expect(offline.offlineActive, "the display took local authority").toBe(true); + expect( + await page.evaluate(() => window.__display.clientId), + "the persistent display identity survived the restart", + ).toBe(clientId); + expect(cachedBefore).toContain( + (await cachedMedia(page)).find((path) => path.includes(library["wide-a"].id)), + ); + expect(await opaqueLayerCount(page)).toBeGreaterThanOrEqual(1); + + // D. It must keep advancing, not merely show one restored frame. + const first = offline.renderedAssetId; + await expect + .poll(async () => (await diagnostics(page))?.renderedAssetId, { timeout: 30_000 }) + .not.toBe(first); + expect(await opaqueLayerCount(page)).toBeGreaterThanOrEqual(1); + + // E. Restore the stack and confirm it surrenders authority cleanly. + await restoreStack(); + await expect + .poll(async () => (await diagnostics(page))?.offlineActive, { timeout: 90_000 }) + .toBe(false); + const playback = await api(request, "/api/playback"); + await expect + .poll(async () => (await diagnostics(page))?.renderedEntryId, { timeout: 45_000 }) + .toBe(playback.currentEntryId); + + const displays = await api(request, "/api/displays"); + expect( + displays.displays.filter((display) => display.clientId === clientId).length, + "one physical display stays one record across a cold restart", + ).toBe(1); + } finally { + await context.close(); + } + } finally { + await restoreStack(); + } +}); + +test("an object storage outage never disturbs already cached slides", async ({ request }) => { + const library = await ensureFixtures(request); + await resetPlayback(request); + await activatePlaylist(request, "Storage outage", [ + { assetId: library["wide-a"].id, durationOverrideMs: 3_000 }, + { assetId: library["wide-b"].id, durationOverrideMs: 3_000 }, + ]); + + const { context, page } = await launchKiosk(); + try { + await page.goto(`${BASE_URL}/display`); + await waitForSlide(page); + await waitForCachedAssets(page, [library["wide-a"].id, library["wide-b"].id]); + + // The API stays up; only the bytes become unreachable. + await stop(CONTAINERS.storage); + + const seen = new Set(); + const deadline = Date.now() + 14_000; + while (Date.now() < deadline) { + const state = await page.evaluate(() => ({ + rendered: window.__display?.diagnostics()?.renderedAssetId ?? null, + opaque: [...document.querySelectorAll(".scene")].filter( + (layer) => Number(getComputedStyle(layer).opacity) > 0.99, + ).length, + })); + expect(state.opaque).toBeGreaterThanOrEqual(1); + if (state.rendered) seen.add(state.rendered); + await page.waitForTimeout(200); + } + expect(seen.size, "cached slides keep rotating without object storage").toBe(2); + + // A brand new asset cannot be fetched; the last good frame must survive. + const held = await diagnostics(page); + await activatePlaylist(request, "Uncached", [ + { assetId: library.standalone.id, durationOverrideMs: 60_000 }, + ]); + await page.waitForTimeout(6_000); + expect(await opaqueLayerCount(page)).toBeGreaterThanOrEqual(1); + const during = await diagnostics(page); + expect(during.renderedAssetId, "the last good frame is retained").toBe(held.renderedAssetId); + // Nothing in the newly active playlist can be rendered locally, which is a + // more severe condition than one failed asset and is reported as such. + expect(during.mostRecentError?.kind).toBe("manifest_invalid"); + + // That condition reaches the server as a critical incident, exactly once. + const clientId = await page.evaluate(() => window.__display.clientId); + await page.evaluate(() => window.__display.heartbeat()); + await expect + .poll( + async () => { + const displays = await api(request, "/api/displays"); + return ( + displays.displays.find((display) => display.clientId === clientId)?.mostRecentError + ?.kind ?? null + ); + }, + { timeout: 30_000 }, + ) + .toBe("manifest_invalid"); + + // Restoring storage recovers without a page reload. + await start(CONTAINERS.storage); + await waitUntilHealthy(BASE_URL); + await expect + .poll(async () => (await diagnostics(page))?.renderedAssetId, { timeout: 90_000 }) + .toBe(library.standalone.id); + expect(await page.evaluate(() => window.__loadCount ?? 0)).toBe(0); + } finally { + await context.close(); + await restoreStack(); + } +}); + +test("an API outage keeps local playback running and resynchronizes on return", async ({ + request, +}) => { + const library = await ensureFixtures(request); + await resetPlayback(request); + await activatePlaylist(request, "API outage", [ + { assetId: library["wide-a"].id, durationOverrideMs: 3_000 }, + { assetId: library["wide-b"].id, durationOverrideMs: 3_000 }, + { assetId: library["wide-c"].id, durationOverrideMs: 3_000 }, + ]); + + const { context, page } = await launchKiosk(); + try { + await page.goto(`${BASE_URL}/display`); + await waitForSlide(page); + await waitForCachedAssets(page, [ + library["wide-a"].id, + library["wide-b"].id, + library["wide-c"].id, + ]); + + await stop(CONTAINERS.api); + await expect + .poll(async () => (await diagnostics(page))?.offlineActive, { timeout: 45_000 }) + .toBe(true); + + const seen = new Set(); + const deadline = Date.now() + 14_000; + while (Date.now() < deadline) { + const state = await diagnostics(page); + if (state?.renderedAssetId) seen.add(state.renderedAssetId); + expect(await opaqueLayerCount(page)).toBeGreaterThanOrEqual(1); + await page.waitForTimeout(250); + } + expect(seen.size).toBeGreaterThanOrEqual(2); + + // While the display was blind, the operator changed authoritative state. + await start(CONTAINERS.api); + await waitUntilHealthy(BASE_URL); + await api(request, "/api/playback/mode", { method: "PUT", data: { mode: "off" } }); + + // Server wins immediately on reconnect, including the visual gate. + await expect(page.getByTestId("output-gate")).toHaveAttribute("data-active", "true", { + timeout: 60_000, + }); + expect((await diagnostics(page)).offlineActive).toBe(false); + + await api(request, "/api/playback/mode", { method: "PUT", data: { mode: "on" } }); + await expect(page.getByTestId("output-gate")).toHaveAttribute("data-active", "false", { + timeout: 30_000, + }); + const playback = await api(request, "/api/playback"); + await expect + .poll(async () => (await diagnostics(page))?.renderedEntryId, { timeout: 30_000 }) + .toBe(playback.currentEntryId); + expect(await page.evaluate(() => window.__loadCount ?? 0)).toBe(0); + } finally { + await context.close(); + await restoreStack(); + } +}); + +test("a playlist changed during the outage is adopted cleanly on reconnect", async ({ + request, +}) => { + const library = await ensureFixtures(request); + await resetPlayback(request); + await activatePlaylist(request, "Before outage", [ + { assetId: library["wide-a"].id, durationOverrideMs: 5_000 }, + { assetId: library["wide-b"].id, durationOverrideMs: 5_000 }, + ]); + + const { context, page } = await launchKiosk(); + try { + await page.goto(`${BASE_URL}/display`); + await waitForSlide(page); + const before = await diagnostics(page); + + await stop(CONTAINERS.api); + await expect + .poll(async () => (await diagnostics(page))?.offlineActive, { timeout: 45_000 }) + .toBe(true); + await page.waitForTimeout(6_000); + + await start(CONTAINERS.api); + await waitUntilHealthy(BASE_URL); + // A different playlist entirely, activated while the display was offline. + await activatePlaylist(request, "After outage", [ + { assetId: library.square.id, durationOverrideMs: 60_000 }, + ]); + + await expect + .poll(async () => (await diagnostics(page))?.renderedAssetId, { timeout: 90_000 }) + .toBe(library.square.id); + const after = await diagnostics(page); + expect(after.offlineActive).toBe(false); + expect(after.renderedAssetId).not.toBe(before.renderedAssetId); + expect(await opaqueLayerCount(page)).toBeGreaterThanOrEqual(1); + + // The superseded playlist's bytes are pruned once the new one is promoted. + await expect + .poll(async () => (await cachedMedia(page)).some((path) => path.includes(library.square.id))) + .toBe(true); + const cached = await cachedMedia(page); + expect(cached.length, "repeated playlist changes must not grow the cache forever") + .toBeLessThanOrEqual(6); + } finally { + await context.close(); + await restoreStack(); + } +}); diff --git a/client/e2e/display-persistence.spec.js b/client/e2e/display-persistence.spec.js index 9f9d0ea..b3a7413 100644 --- a/client/e2e/display-persistence.spec.js +++ b/client/e2e/display-persistence.spec.js @@ -138,10 +138,13 @@ test("the persistent client identity and last known good state survive a refresh database.close(); return value; }); - expect(persisted.schemaVersion).toBe(1); + // The record schema is versioned independently of the database version. + expect(persisted.schemaVersion).toBe(2); expect(persisted.playbackRevision).toBeGreaterThan(0); expect(persisted.manifest.entries.length).toBeGreaterThan(0); expect(persisted.timing.serverTime).toBeGreaterThan(0); + // The clock anchor a cold offline start depends on. + expect(persisted.timing.recordedAt).toBeGreaterThan(0); expect(persisted.mode).toBe("on"); }); diff --git a/client/e2e/offline-helpers.js b/client/e2e/offline-helpers.js new file mode 100644 index 0000000..862f8a4 --- /dev/null +++ b/client/e2e/offline-helpers.js @@ -0,0 +1,58 @@ +import { chromium, expect } from "@playwright/test"; + +export const PROFILE_DIR = process.env.DISPLAY_PROFILE_DIR || "/profiles/kiosk"; +export const BASE_URL = process.env.DISPLAY_BASE_URL || "http://127.0.0.1:5199"; + +// A real kiosk keeps a persistent Chromium profile: the Service Worker, +// Cache Storage, and IndexedDB that make offline startup possible all live +// there. Incognito or an ephemeral user-data-dir would defeat the whole phase. +export async function launchKiosk() { + const context = await chromium.launchPersistentContext(PROFILE_DIR, { + viewport: { width: 1280, height: 720 }, + serviceWorkers: "allow", + }); + const page = context.pages()[0] || (await context.newPage()); + return { context, page }; +} + +export function diagnostics(page) { + return page.evaluate(() => window.__display?.diagnostics() ?? null); +} + +export async function waitForSlide(page, timeout = 60_000) { + await expect + .poll(async () => (await diagnostics(page))?.renderedAssetId ?? null, { timeout }) + .not.toBeNull(); +} + +export function opaqueLayerCount(page) { + return page.evaluate( + () => + [...document.querySelectorAll(".scene")].filter( + (layer) => Number(getComputedStyle(layer).opacity) > 0.99, + ).length, + ); +} + +// Waiting for "some" cached assets is not enough: a scenario that removes +// storage must first know that the specific slides it needs are present. +export async function waitForCachedAssets(page, assetIds, timeout = 60_000) { + const { expect } = await import("@playwright/test"); + await expect + .poll( + async () => { + const cached = await cachedMedia(page); + return assetIds.every((id) => cached.some((path) => path.includes(id))); + }, + { timeout }, + ) + .toBe(true); +} + +export async function cachedMedia(page) { + return page.evaluate(async () => { + if (!(await caches.has("announcements-display-assets-v1"))) return []; + const cache = await caches.open("announcements-display-assets-v1"); + return (await cache.keys()).map((request) => new URL(request.url).pathname); + }); +} diff --git a/client/e2e/stack.js b/client/e2e/stack.js new file mode 100644 index 0000000..6f0a3ad --- /dev/null +++ b/client/e2e/stack.js @@ -0,0 +1,74 @@ +import http from "node:http"; + +// The offline suite must genuinely remove infrastructure, not simulate it. +// Talking to the Docker Engine socket directly avoids installing a CLI into +// the browser image and keeps every scenario inside one Playwright run. +const SOCKET = "/var/run/docker.sock"; + +function engine(method, path) { + return new Promise((resolve, reject) => { + const request = http.request({ socketPath: SOCKET, method, path }, (response) => { + let body = ""; + response.on("data", (chunk) => (body += chunk)); + response.on("end", () => resolve({ status: response.statusCode, body })); + }); + request.on("error", reject); + request.end(); + }); +} + +const PROJECT = process.env.VERIFY_PROJECT || "announcements-p5b-verify"; +export const CONTAINERS = { + web: `${PROJECT}-web-1`, + api: `${PROJECT}-api-1`, + storage: `${PROJECT}-seaweedfs-1`, +}; + +export async function stop(...names) { + for (const name of names) await engine("POST", `/containers/${name}/stop?t=2`); +} + +export async function start(...names) { + for (const name of names) await engine("POST", `/containers/${name}/start`); +} + +export async function isRunning(name) { + const { status, body } = await engine("GET", `/containers/${name}/json`); + if (status !== 200) return false; + return Boolean(JSON.parse(body).State?.Running); +} + +export async function waitUntilHealthy(baseUrl, timeoutMs = 120_000) { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + try { + const response = await fetch(`${baseUrl}/api/playback`, { + signal: AbortSignal.timeout(2_000), + }); + if (response.ok) return true; + } catch { + // Still coming up. + } + await new Promise((resolve) => setTimeout(resolve, 500)); + } + throw new Error("stack did not become reachable in time"); +} + +// With the API stopped but Nginx still running, the proxy answers 502. That is +// a response, not a reachable API, so a gateway error counts as unreachable +// just like a refused connection does. +export async function waitUntilUnreachable(baseUrl, timeoutMs = 60_000) { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + try { + const response = await fetch(`${baseUrl}/api/playback`, { + signal: AbortSignal.timeout(1_000), + }); + if (!response.ok) return true; + } catch { + return true; + } + await new Promise((resolve) => setTimeout(resolve, 250)); + } + throw new Error("stack stayed reachable"); +} diff --git a/client/public/display-sw.js b/client/public/display-sw.js index 477637a..5fe8cc0 100644 --- a/client/public/display-sw.js +++ b/client/public/display-sw.js @@ -7,6 +7,11 @@ */ const SHELL_CACHE = "announcements-display-shell-v1"; const SHELL_FALLBACK = "/display"; +// A kiosk must reach its slides quickly. When a usable shell is already +// cached, a slow or dead origin is never allowed to delay startup: a reverse +// proxy whose backend is down can otherwise hold a navigation open for its +// full connect timeout, leaving the screen blank the whole time. +const SHELL_NETWORK_TIMEOUT_MS = 3_000; // Paths the worker must never handle. Media is the asset manager's; the API and // socket are live server state and must never be answered from a cache. @@ -51,17 +56,33 @@ self.addEventListener("activate", (event) => { async function networkFirst(request) { const cache = await caches.open(SHELL_CACHE); - try { + const cached = + (await cache.match(SHELL_FALLBACK)) || (await cache.match("/index.html")); + + // Without a cached shell there is nothing to fall back to, so wait normally. + if (!cached) { const response = await fetch(request); - if (response && response.ok) { - // Keep the shell fallback fresh so an online kiosk always updates. - await cache.put(SHELL_FALLBACK, response.clone()); - } + if (response && response.ok) await cache.put(SHELL_FALLBACK, response.clone()); return response; - } catch (error) { - const cached = (await cache.match(SHELL_FALLBACK)) || (await cache.match("/index.html")); - if (cached) return cached; - throw error; + } + + const abort = new AbortController(); + const timer = setTimeout(() => abort.abort(), SHELL_NETWORK_TIMEOUT_MS); + try { + const response = await fetch(request, { signal: abort.signal }); + clearTimeout(timer); + if (response && response.ok) { + // Keep the shell fresh so an online kiosk always updates. + await cache.put(SHELL_FALLBACK, response.clone()); + return response; + } + // A production kiosk reaches this origin through a reverse proxy, so an + // outage behind it arrives as 502/503/504 rather than a refused + // connection. That is still an outage: serve the cached shell. + return cached; + } catch { + clearTimeout(timer); + return cached; } } diff --git a/client/src/display/DisplayApp.jsx b/client/src/display/DisplayApp.jsx index 5ccab84..471b555 100644 --- a/client/src/display/DisplayApp.jsx +++ b/client/src/display/DisplayApp.jsx @@ -2,6 +2,7 @@ import { useEffect, useRef } from "react"; import { io } from "socket.io-client"; import { createAssetManager } from "./assetCache.js"; import { createDisplayController } from "./displayController.js"; +import { DISPLAY_ERROR_KINDS, createDisplayError } from "./errors.js"; import { createDisplayPersistence, createDisplayStore, @@ -80,6 +81,25 @@ export default function DisplayApp() { heartbeat: () => controller.sendHeartbeat(), }; } + + // Offline capability needs a secure context. Rather than silently + // degrading, the display reports it so it is diagnosable from + // /api/displays and Home Assistant instead of on screen. + if (!window.isSecureContext || !("serviceWorker" in navigator)) { + controller.reportCapabilityLimit( + createDisplayError(DISPLAY_ERROR_KINDS.OFFLINE_CAPABILITY_UNAVAILABLE, { + message: window.isSecureContext + ? "This browser provides no Service Worker; offline startup is unavailable" + : "This origin is not a secure context; offline caching is unavailable", + }), + ); + } + + // Restore signage from the last known good state before any server + // contact, so a cold start with the infrastructure down still shows + // announcements. + await controller.bootstrapFromPersistence(); + if (cancelled) return; controller.start(); } diff --git a/client/src/display/__tests__/offline-timeline.test.js b/client/src/display/__tests__/offline-timeline.test.js new file mode 100644 index 0000000..4b9e2f5 --- /dev/null +++ b/client/src/display/__tests__/offline-timeline.test.js @@ -0,0 +1,263 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import { + nextOfflineBoundary, + resolveOfflineState, +} from "../offlineTimeline.js"; + +const T0 = 2_000_000_000_000; + +function entry(id, position, overrides = {}) { + return { + id, + position, + assetId: `asset-${id}`, + effectiveDurationMs: 5_000, + scheduleStartAt: null, + scheduleEndAt: null, + expirationAt: null, + playable: true, + asset: { id: `asset-${id}`, media: { url: `/media/${id}.webp`, width: 1920, height: 1080 } }, + ...overrides, + }; +} + +const MANIFEST = { + entries: [ + entry("a", 0, { effectiveDurationMs: 3_000 }), + entry("b", 1, { effectiveDurationMs: 9_000 }), + entry("c", 2, { effectiveDurationMs: 4_000 }), + ], +}; + +function anchor(overrides = {}) { + return { + entryId: "a", + startedAt: T0, + deadlineAt: T0 + 3_000, + transportState: "playing", + mode: "on", + pausedRemainingMs: null, + staticAssetId: null, + staticTarget: null, + pendingPlaylistId: null, + ...overrides, + }; +} + +test("mixed durations advance on absolute deadlines without drift", () => { + const cycle = 3_000 + 9_000 + 4_000; + const cases = [ + [0, "a", 0], + [2_999, "a", 0], + [3_000, "b", 3_000], + [11_999, "b", 3_000], + [12_000, "c", 12_000], + [15_999, "c", 12_000], + [cycle, "a", cycle], + ]; + for (const [offset, expectedId, expectedStart] of cases) { + const resolved = resolveOfflineState({ manifest: MANIFEST, anchor: anchor(), at: T0 + offset }); + assert.equal(resolved.kind, "normal", `+${offset}`); + assert.equal(resolved.entryId, expectedId, `+${offset} entry`); + assert.equal(resolved.startedAt, T0 + expectedStart, `+${offset} start`); + } +}); + +test("many cycles offline stay exactly on the original phase", () => { + const cycle = 16_000; + for (const cycles of [10, 1_000, 100_000]) { + const resolved = resolveOfflineState({ + manifest: MANIFEST, + anchor: anchor(), + at: T0 + cycles * cycle + 12_500, + }); + assert.equal(resolved.entryId, "c"); + assert.equal( + resolved.startedAt, + T0 + cycles * cycle + 12_000, + "no accumulated drift after many cycles", + ); + } +}); + +test("a schedule start offline brings an entry back into rotation", () => { + const manifest = { + entries: [ + entry("always", 0, { effectiveDurationMs: 1_000 }), + entry("joins", 1, { effectiveDurationMs: 1_000, scheduleStartAt: T0 + 20_000 }), + ], + }; + const before = resolveOfflineState({ + manifest, + anchor: anchor({ entryId: "always", deadlineAt: T0 + 1_000 }), + at: T0 + 19_500, + }); + assert.equal(before.entryId, "always", "the future entry is not used early"); + + const after = resolveOfflineState({ + manifest, + anchor: anchor({ entryId: "always", deadlineAt: T0 + 1_000 }), + at: T0 + 20_500, + }); + assert.equal(after.entryId, "joins", "[start, end) is inclusive at start"); +}); + +test("a schedule end and an expiration offline both retire an entry", () => { + const ending = { + entries: [ + entry("ends", 0, { effectiveDurationMs: 5_000, scheduleEndAt: T0 + 12_500 }), + entry("stays", 1, { effectiveDurationMs: 5_000 }), + ], + }; + const afterEnd = resolveOfflineState({ + manifest: ending, + anchor: anchor({ entryId: "ends", deadlineAt: T0 + 5_000 }), + at: T0 + 40_000, + }); + assert.equal(afterEnd.entryId, "stays"); + + const expiring = { + entries: [ + entry("expires", 0, { effectiveDurationMs: 6_000, expirationAt: T0 + 15_000 }), + entry("permanent", 1, { effectiveDurationMs: 4_000 }), + ], + }; + const afterExpiry = resolveOfflineState({ + manifest: expiring, + anchor: anchor({ entryId: "expires", deadlineAt: T0 + 6_000 }), + at: T0 + 60_000, + }); + assert.equal(afterExpiry.entryId, "permanent"); +}); + +test("a cached playlist with nothing eligible is legitimately black", () => { + const manifest = { entries: [entry("only", 0, { scheduleEndAt: T0 + 8_000 })] }; + const resolved = resolveOfflineState({ + manifest, + anchor: anchor({ entryId: "only", deadlineAt: T0 + 5_000 }), + at: T0 + 30_000, + }); + assert.equal(resolved.kind, "black"); + assert.equal(resolved.reason, "no_playable_content"); +}); + +test("mode off keeps advancing the hidden transport underneath", () => { + const resolved = resolveOfflineState({ + manifest: MANIFEST, + anchor: anchor({ mode: "off" }), + at: T0 + 12_500, + }); + // Mode is a visual gate applied by the renderer; the cursor still moves. + assert.equal(resolved.kind, "normal"); + assert.equal(resolved.entryId, "c"); +}); + +test("a paused display holds its slide and its remainder offline", () => { + const resolved = resolveOfflineState({ + manifest: MANIFEST, + anchor: anchor({ + transportState: "paused", + entryId: "b", + pausedRemainingMs: 4_000, + startedAt: null, + deadlineAt: null, + }), + at: T0 + 6 * 3_600_000, + }); + assert.equal(resolved.kind, "paused"); + assert.equal(resolved.entryId, "b", "pause never advances merely because time passed"); + assert.equal(resolved.remainingMs, 4_000); +}); + +test("a paused display still reconciles a held slide that became ineligible", () => { + const manifest = { + entries: [ + entry("held", 0, { expirationAt: T0 + 10_000 }), + entry("survivor", 1), + ], + }; + const resolved = resolveOfflineState({ + manifest, + anchor: anchor({ transportState: "paused", entryId: "held", pausedRemainingMs: 2_000 }), + at: T0 + 20_000, + }); + assert.equal(resolved.kind, "paused", "it stays paused"); + assert.equal(resolved.entryId, "survivor", "but not on an expired slide"); +}); + +test("static holds indefinitely offline and releases only on real expiration", () => { + const held = resolveOfflineState({ + manifest: MANIFEST, + anchor: anchor({ + transportState: "static", + staticAssetId: "static-1", + staticTarget: { assetId: "static-1", asset: { expirationAt: null } }, + pausedRemainingMs: 2_000, + }), + at: T0 + 30 * 24 * 3_600_000, + }); + assert.equal(held.kind, "static"); + assert.equal(held.assetId, "static-1", "a permanent static asset holds forever"); + + const released = resolveOfflineState({ + manifest: MANIFEST, + anchor: anchor({ + entryId: "a", + transportState: "static", + staticAssetId: "static-1", + staticTarget: { assetId: "static-1", expirationAt: T0 + 10_000 }, + pausedRemainingMs: 2_000, + }), + at: T0 + 11_000, + }); + // Normal playback resumes from the expiration instant with its remainder. + assert.equal(released.kind, "normal"); + assert.equal(released.entryId, "a"); + assert.equal(released.deadlineAt, T0 + 12_000); +}); + +test("an unknown pending activation holds the last good frame at the boundary", () => { + const beforeBoundary = resolveOfflineState({ + manifest: MANIFEST, + anchor: anchor({ pendingPlaylistId: "playlist-b" }), + at: T0 + 1_000, + }); + assert.equal(beforeBoundary.kind, "normal", "the current slide still finishes"); + + const atBoundary = resolveOfflineState({ + manifest: MANIFEST, + anchor: anchor({ pendingPlaylistId: "playlist-b" }), + at: T0 + 3_000, + }); + assert.equal(atBoundary.kind, "hold"); + assert.equal( + atBoundary.reason, + "pending_activation_unknown", + "never continue deeper into a playlist the server is replacing", + ); +}); + +test("the next offline wake-up is the earliest real boundary", () => { + const resolved = resolveOfflineState({ manifest: MANIFEST, anchor: anchor(), at: T0 + 1_000 }); + assert.equal( + nextOfflineBoundary({ manifest: MANIFEST, resolved, at: T0 + 1_000 }), + T0 + 3_000, + "the current slide deadline", + ); + + const scheduled = { + entries: [entry("future", 0, { scheduleStartAt: T0 + 50_000 })], + }; + const black = resolveOfflineState({ + manifest: scheduled, + anchor: anchor({ entryId: "future", startedAt: null, deadlineAt: null }), + at: T0, + }); + assert.equal(black.kind, "black"); + assert.equal( + nextOfflineBoundary({ manifest: scheduled, resolved: black, at: T0 }), + T0 + 50_000, + "a black display still wakes for a cached schedule start", + ); +}); diff --git a/client/src/display/displayController.js b/client/src/display/displayController.js index 38a674f..24847bd 100644 --- a/client/src/display/displayController.js +++ b/client/src/display/displayController.js @@ -1,5 +1,6 @@ import { createAssetManager, DisplayAssetError } from "./assetCache.js"; import { DISPLAY_ERROR_KINDS, createDisplayError, serializeDisplayError, shouldClearError } from "./errors.js"; +import { nextOfflineBoundary, resolveOfflineState } from "./offlineTimeline.js"; import { resolveScenePresentation } from "./presentation.js"; import { canPredictLocally, @@ -39,6 +40,7 @@ export function createDisplayController({ setIntervalFn = (...args) => setInterval(...args), clearIntervalFn = (...args) => clearInterval(...args), heartbeatIntervalMs = HEARTBEAT_INTERVAL_MS, + logger = { warn() {} }, onDiagnostics = () => {}, } = {}) { let accepted = null; @@ -54,6 +56,10 @@ export function createDisplayController({ let currentError = null; let destroyed = false; let stagingToken = 0; + let online = false; + let offlineTimer = null; + let offlineAnchor = null; + let offlineActive = false; const layerUrls = [null, null]; const unavailableEntryIds = new Set(); const failedMediaUrls = new Set(); @@ -68,6 +74,9 @@ export function createDisplayController({ renderedAssetId: renderedTarget?.assetId ?? null, renderedEntryId: renderedTarget?.entryId ?? null, predictedEntryId, + online, + offlineActive, + offlineAnchor, unavailableEntryIds: [...unavailableEntryIds], mostRecentError: currentError, activeObjectUrls: assets.activeObjectUrlCount(), @@ -85,6 +94,13 @@ export function createDisplayController({ diagnostics(); } + // A capability limit is a standing condition, not a transient failure, so it + // is only recorded when nothing more specific is already being reported. + function reportCapabilityLimit(displayError) { + if (currentError) return; + recordError(displayError); + } + function clearErrorIfResolved() { const manifestAssetIds = (promotedManifest.entries || []).map( (entry) => entry.assetId, @@ -321,6 +337,177 @@ export function createDisplayController({ armPrediction(at + (next.effectiveDurationMs ?? 0)); } + // ------------------------------------------------------------------ offline + + function anchorFrom(source) { + if (!source) return null; + return { + entryId: source.currentEntryId ?? null, + startedAt: source.timing?.slideStartedAt ?? source.timeline?.slideStartedAt ?? null, + deadlineAt: source.timing?.nextSlideAt ?? source.timeline?.nextSlideAt ?? null, + transportState: source.transportState, + mode: source.mode, + pausedRemainingMs: + source.pausedRemainingMs ?? source.timeline?.pausedRemainingMs ?? null, + staticAssetId: source.staticAssetId ?? source.staticTarget?.assetId ?? null, + staticTarget: source.staticTarget ?? null, + pendingPlaylistId: source.pendingPlaylistId ?? null, + }; + } + + function clearOfflineTimer() { + if (offlineTimer !== null) { + clearTimeoutFn(offlineTimer); + offlineTimer = null; + } + } + + // A locally broken asset must not stall rotation. The timeline itself stays + // authoritative-equivalent; only what is painted is substituted. + function usableEntryFor(entryId, at) { + const entries = [...(promotedManifest.entries || [])].sort( + (left, right) => left.position - right.position, + ); + const resolved = entries.find((entry) => entry.id === entryId); + if (resolved && !unavailableEntryIds.has(resolved.id)) return resolved; + const index = entries.findIndex((entry) => entry.id === entryId); + for (let offset = 1; offset <= entries.length; offset += 1) { + const candidate = entries[(Math.max(index, 0) + offset) % entries.length]; + if (unavailableEntryIds.has(candidate.id)) continue; + if ( + (candidate.scheduleStartAt == null || at >= candidate.scheduleStartAt) && + (candidate.scheduleEndAt == null || at < candidate.scheduleEndAt) && + (candidate.expirationAt == null || at < candidate.expirationAt) + ) { + return candidate; + } + } + return null; + } + + async function renderOffline(resolved, at) { + if (resolved.kind === "hold") { + // Correct-but-held: the next boundary belongs to a playlist this display + // does not have, so the last good frame stays up. + return; + } + if (offlineAnchor?.mode === "off") setGate(true); + + if (resolved.kind === "black") { + // The cached playlist genuinely has nothing eligible; black is correct. + if (offlineAnchor?.mode !== "off") hideContent(); + renderedTarget = null; + return; + } + if (resolved.kind === "static") { + const target = offlineAnchor?.staticTarget; + if (!target?.asset?.media?.url) { + // Missing local bytes never mean the operator ended static. + recordError( + createDisplayError(DISPLAY_ERROR_KINDS.ASSET_FETCH_FAILED, { + assetId: resolved.assetId, + message: "The held static asset is not available in the local cache", + }), + ); + return; + } + generation += 1; + const scene = buildScene(target, "static"); + if (scene) await showScene(scene, generation, 0); + return; + } + + const entry = usableEntryFor(resolved.entryId, at); + if (!entry) { + // Authoritative content exists but nothing local can render it. The last + // good frame is kept rather than pretending the playlist went black. + recordError( + createDisplayError(DISPLAY_ERROR_KINDS.MANIFEST_INVALID, { + entryId: resolved.entryId, + message: "No cached slide of the active playlist can be rendered offline", + }), + ); + return; + } + generation += 1; + const scene = buildScene(entry, "normal"); + if (scene) await showScene(scene, generation, accepted?.crossfadeDurationMs ?? 0); + } + + async function runOfflineCycle() { + if (destroyed || online || !offlineAnchor) return; + const at = estimatedServerNow(); + let resolved; + try { + resolved = resolveOfflineState({ manifest: promotedManifest, anchor: offlineAnchor, at }); + } catch (error) { + warnOffline(error); + return; + } + await renderOffline(resolved, at); + if (destroyed || online) return; + + const boundary = nextOfflineBoundary({ manifest: promotedManifest, resolved, at }); + clearOfflineTimer(); + if (boundary === null) return; + offlineTimer = setTimeoutFn( + () => { + offlineTimer = null; + void runOfflineCycle(); + }, + Math.max(0, boundary - at), + ); + diagnostics(); + } + + function warnOffline(error) { + try { + logger?.warn?.(`Offline playback evaluation failed: ${error?.message}`); + } catch { + // Offline rendering must never depend on logging. + } + } + + // The display becomes a deterministic local continuation of the last + // authoritative state. This is resilience, never an attempt to become a + // second authority. + function enterOffline(source) { + if (destroyed) return; + const anchor = anchorFrom(source); + if (!anchor) return; + online = false; + offlineActive = true; + offlineAnchor = anchor; + clearPrediction(); + void runOfflineCycle(); + } + + function exitOffline() { + offlineActive = false; + clearOfflineTimer(); + } + + // Cold start: restore signage from persisted state before, and without, any + // server contact. + async function bootstrapFromPersistence() { + if (!persistence) return false; + let record; + try { + record = await persistence.readLastKnownGood(); + } catch { + record = null; + } + if (!record) return false; + + promotedManifest = record.manifest || { entries: [] }; + currentError = record.mostRecentError ?? null; + // Reconstruct the authoritative clock from the persisted anchor. + serverOffset = record.timing.serverTime - record.timing.recordedAt; + setGate(record.mode === "off"); + enterOffline(record); + return true; + } + // ---------------------------------------------------------------- snapshots async function applySnapshot(snapshot) { @@ -331,8 +518,13 @@ export function createDisplayController({ return { accepted: false, reason: decision.reason }; } + // The server is authoritative again the instant it speaks. Any local + // offline cursor is abandoned rather than reconciled or reported back. + online = true; + exitOffline(); // Packet arrival time is not slide start time; re-anchor on every accepted - // snapshot so local scheduling follows the server's clock. + // snapshot so local scheduling follows the server's clock. A fresh anchor + // also discards any local clock drift accumulated while disconnected. serverOffset = snapshot.serverTime - now(); accepted = snapshot; predictedEntryId = snapshot.currentEntryId ?? null; @@ -418,13 +610,16 @@ export function createDisplayController({ }); socket.on("connect", () => { // A reconnect resynchronizes from the server; it never reloads the page - // and never clears the visible frame. + // and never clears the visible frame. The same persistent client ID is + // reused, so the server sees one display, not a new one. void sendHeartbeat(); diagnostics(); }); socket.on("disconnect", () => { - // The last good scene stays exactly where it is. + // The last good scene stays exactly where it is while local playback + // takes over from the last authoritative state. clearPrediction(); + enterOffline(accepted ?? offlineAnchor); diagnostics(); }); socket.connect(); @@ -436,6 +631,7 @@ export function createDisplayController({ generation += 1; stagingToken += 1; clearPrediction(); + clearOfflineTimer(); stopHeartbeat(); socket.off("playback:state"); socket.off("connect"); @@ -448,7 +644,10 @@ export function createDisplayController({ return { applySnapshot, + bootstrapFromPersistence, destroy, + enterOffline, + reportCapabilityLimit, diagnostics, estimatedServerNow, sendHeartbeat, diff --git a/client/src/display/displayDb.js b/client/src/display/displayDb.js index d426ed3..f425881 100644 --- a/client/src/display/displayDb.js +++ b/client/src/display/displayDb.js @@ -5,6 +5,11 @@ export const DISPLAY_DB_NAME = "announcements-display"; export const DISPLAY_DB_VERSION = 1; export const DISPLAY_STORE = "state"; +// The record schema is versioned independently of the database so adding +// fields does not require an IndexedDB upgrade. A display carrying an older +// record simply discards it and rebuilds from the server on its next +// successful connection. +export const LKG_SCHEMA_VERSION = 2; export const STATE_KEYS = Object.freeze({ CLIENT_ID: "clientId", @@ -70,10 +75,14 @@ export function createDisplayStore({ database }) { // than allowed to corrupt startup. export function isValidLastKnownGood(value) { if (!value || typeof value !== "object" || Array.isArray(value)) return false; - if (value.schemaVersion !== DISPLAY_DB_VERSION) return false; + if (value.schemaVersion !== LKG_SCHEMA_VERSION) return false; if (typeof value.playbackRevision !== "number") return false; if (!value.manifest || typeof value.manifest !== "object") return false; if (!Array.isArray(value.manifest.entries)) return false; + if (!value.timing || typeof value.timing !== "object") return false; + // Without a usable clock anchor the record cannot drive offline playback. + if (typeof value.timing.serverTime !== "number") return false; + if (typeof value.timing.recordedAt !== "number") return false; return true; } @@ -103,7 +112,7 @@ export function createDisplayPersistence({ store, createId, now = Date.now }) { async function writeLastKnownGood(snapshot, extra = {}) { const record = { - schemaVersion: DISPLAY_DB_VERSION, + schemaVersion: LKG_SCHEMA_VERSION, savedAt: now(), playbackRevision: snapshot.playbackRevision, settingsRevision: snapshot.settingsRevision ?? null, @@ -113,6 +122,8 @@ export function createDisplayPersistence({ store, createId, now = Date.now }) { transportState: snapshot.transportState, currentEntryId: snapshot.currentEntryId ?? null, staticAssetId: snapshot.staticTarget?.assetId ?? null, + pendingPlaylistId: snapshot.pendingPlaylistId ?? null, + pausedRemainingMs: snapshot.timeline?.pausedRemainingMs ?? null, // The timing anchor is what lets a future offline start reason about // where playback should be, without a server to ask. timing: { @@ -126,6 +137,9 @@ export function createDisplayPersistence({ store, createId, now = Date.now }) { staticTarget: snapshot.staticTarget || null, mostRecentError: extra.mostRecentError ?? null, }; + // One record in one transaction: a cold start can never observe a new + // manifest paired with an old timing anchor. The previous complete record + // is only replaced once this one is durably written. await store.write(STATE_KEYS.LAST_KNOWN_GOOD, record); return record; } diff --git a/client/src/display/errors.js b/client/src/display/errors.js index 965af32..f5d2b75 100644 --- a/client/src/display/errors.js +++ b/client/src/display/errors.js @@ -5,6 +5,7 @@ export const DISPLAY_ERROR_KINDS = Object.freeze({ ASSET_FETCH_FAILED: "asset_fetch_failed", CACHE_WRITE_FAILED: "cache_write_failed", MANIFEST_INVALID: "manifest_invalid", + OFFLINE_CAPABILITY_UNAVAILABLE: "offline_capability_unavailable", }); const KNOWN_KINDS = new Set(Object.values(DISPLAY_ERROR_KINDS)); diff --git a/client/src/display/offlineTimeline.js b/client/src/display/offlineTimeline.js new file mode 100644 index 0000000..555b52d --- /dev/null +++ b/client/src/display/offlineTimeline.js @@ -0,0 +1,297 @@ +import { entryPlayableAt } from "./snapshot.js"; + +// Deterministic local continuation of the last authoritative state. +// +// This is a small client-side evaluator, intentionally separate from the +// verified Phase 3B server engine: the server stays the single authority while +// connected, and this only reproduces its documented semantics well enough to +// keep a disconnected kiosk correct. Contract tests compare the two directly. + +export const TRANSITION_SAFETY_LIMIT = 10_000; + +export function entryDuration(entry) { + const value = entry?.effectiveDurationMs; + return Number.isFinite(value) && value > 0 ? value : 0; +} + +function ordered(manifest) { + return [...(manifest?.entries || [])].sort( + (left, right) => left.position - right.position, + ); +} + +function boundariesOf(entry) { + return [entry.scheduleStartAt, entry.scheduleEndAt, entry.expirationAt]; +} + +// The first instant in (after, through] at which any entry's eligibility can +// change. Between such instants the eligible set is constant, which is what +// makes whole-cycle skipping safe. +function firstEligibilityBoundary(entries, after, through) { + let best = null; + for (const entry of entries) { + for (const value of boundariesOf(entry)) { + if (typeof value !== "number") continue; + if (value > after && value <= through && (best === null || value < best)) { + best = value; + } + } + } + return best; +} + +// A currently displayed entry can be cut short by its own schedule end or asset +// expiration, never by another entry's boundary. +function invalidBoundaryFor(entry, startedAt, through) { + if (!entry) return null; + let best = null; + for (const value of [entry.scheduleEndAt, entry.expirationAt]) { + if (typeof value !== "number") continue; + if (value > startedAt && value <= through && (best === null || value < best)) { + best = value; + } + } + return best; +} + +function firstPlayable(entries, at) { + return entries.find((entry) => entryPlayableAt(entry, at)) || null; +} + +function adjacentPlayable(entries, currentId, at) { + if (entries.length === 0) return null; + const index = entries.findIndex((entry) => entry.id === currentId); + if (index < 0) return firstPlayable(entries, at); + for (let offset = 1; offset <= entries.length; offset += 1) { + const candidate = entries[(index + offset) % entries.length]; + if (entryPlayableAt(candidate, at)) return candidate; + } + return null; +} + +// The next instant at which a currently ineligible playlist becomes playable +// again, used to recover from a legitimately empty stretch. +function nextAvailability(entries, after, through) { + const starts = entries + .map((entry) => entry.scheduleStartAt) + .filter((value) => typeof value === "number" && value > after && value <= through) + .sort((left, right) => left - right); + return starts.find((candidate) => entries.some((entry) => entryPlayableAt(entry, candidate))) ?? null; +} + +// Skips whole rotations with arithmetic instead of stepping one slide at a +// time, so a kiosk offline for a week catches up in constant work per +// eligibility boundary rather than per missed slide. +function fastForward(entries, cursor, at) { + const boundary = firstEligibilityBoundary(entries, cursor.startedAt, at); + const through = boundary === null ? at : boundary - 1; + if (cursor.deadlineAt > through) return false; + + const eligible = entries.filter((entry) => entryPlayableAt(entry, through)); + const index = eligible.findIndex((entry) => entry.id === cursor.entryId); + if (index < 0 || eligible.length === 0) return false; + + const cycleDuration = eligible.reduce((total, entry) => total + entryDuration(entry), 0); + if (cycleDuration <= 0) return false; + + let position = (index + 1) % eligible.length; + let startedAt = cursor.deadlineAt; + const cycles = Math.floor((through - startedAt) / cycleDuration); + if (cycles > 0) startedAt += cycles * cycleDuration; + + let deadlineAt = startedAt + entryDuration(eligible[position]); + while (deadlineAt <= through) { + startedAt = deadlineAt; + position = (position + 1) % eligible.length; + deadlineAt = startedAt + entryDuration(eligible[position]); + } + + cursor.entryId = eligible[position].id; + cursor.startedAt = startedAt; + cursor.deadlineAt = deadlineAt; + return boundary === null; +} + +// Walks the rotation forward from an anchor to `at`, honouring schedule and +// expiration boundaries exactly as the server does. +function advanceRotation(entries, anchor, at) { + const cursor = { + entryId: anchor.entryId, + startedAt: anchor.startedAt, + deadlineAt: anchor.deadlineAt, + }; + + for (let step = 0; step < TRANSITION_SAFETY_LIMIT; step += 1) { + let current = entries.find((entry) => entry.id === cursor.entryId); + + if (!current) { + const availableAt = nextAvailability( + entries, + Number.isFinite(cursor.startedAt) ? cursor.startedAt : at, + at, + ); + const startAt = availableAt ?? at; + const replacement = firstPlayable(entries, startAt); + if (!replacement) return { kind: "black", reason: "no_playable_content" }; + cursor.entryId = replacement.id; + cursor.startedAt = startAt; + cursor.deadlineAt = startAt + entryDuration(replacement); + continue; + } + + if (cursor.deadlineAt <= at && fastForward(entries, cursor, at)) { + current = entries.find((entry) => entry.id === cursor.entryId); + } + if (cursor.entryId !== current?.id) continue; + + const invalidAt = invalidBoundaryFor(current, cursor.startedAt, at); + const playableNow = entryPlayableAt(current, at); + if (playableNow && cursor.deadlineAt > at) { + return { + kind: "normal", + entryId: cursor.entryId, + startedAt: cursor.startedAt, + deadlineAt: cursor.deadlineAt, + }; + } + + let boundary = cursor.deadlineAt <= at ? cursor.deadlineAt : at; + if (invalidAt !== null) boundary = Math.min(boundary, invalidAt); + + const next = adjacentPlayable(entries, cursor.entryId, boundary); + if (!next) { + const availableAt = nextAvailability(entries, boundary, at); + if (availableAt === null) { + const now = firstPlayable(entries, at); + if (!now) return { kind: "black", reason: "no_playable_content" }; + return { + kind: "normal", + entryId: now.id, + startedAt: at, + deadlineAt: at + entryDuration(now), + }; + } + const replacement = firstPlayable(entries, availableAt); + cursor.entryId = replacement.id; + cursor.startedAt = availableAt; + cursor.deadlineAt = availableAt + entryDuration(replacement); + continue; + } + + cursor.entryId = next.id; + cursor.startedAt = boundary; + cursor.deadlineAt = boundary + entryDuration(next); + } + + throw new Error("Offline catch-up exceeded its transition safety limit"); +} + +function staticStillValid(staticTarget, at) { + if (!staticTarget) return false; + const expiration = staticTarget.expirationAt ?? staticTarget.asset?.expirationAt ?? null; + // A null expiration is permanent; local age never invents one. + if (typeof expiration !== "number") return true; + return at < expiration; +} + +function heldEntryAt(entries, anchor, at) { + const current = entries.find((entry) => entry.id === anchor.entryId); + if (current && entryPlayableAt(current, at)) return current; + // A held slide can still be invalidated by schedule or expiration; the + // replacement is held in turn rather than starting to advance. + return adjacentPlayable(entries, anchor.entryId, at); +} + +/** + * Resolves what a disconnected display should be showing at `at`, from the last + * authoritative state it persisted. Pure and deterministic. + */ +export function resolveOfflineState({ manifest, anchor, at }) { + const entries = ordered(manifest); + + if (anchor.pendingPlaylistId) { + // The next boundary belongs to a playlist this display does not have. Going + // deeper into the old playlist would show known-wrong content, so the last + // good frame is held until the server can speak again. + if (typeof anchor.deadlineAt === "number" && at >= anchor.deadlineAt) { + return { kind: "hold", reason: "pending_activation_unknown" }; + } + } + + if (anchor.staticAssetId) { + if (staticStillValid(anchor.staticTarget, at)) { + return { kind: "static", assetId: anchor.staticAssetId }; + } + // A genuinely expired static asset releases the hold and normal playback + // resumes from that exact instant. + const expiration = + anchor.staticTarget?.expirationAt ?? anchor.staticTarget?.asset?.expirationAt ?? at; + const resumeAt = Math.min(at, expiration); + const current = entries.find((entry) => entry.id === anchor.entryId); + const full = entryDuration(current); + const remaining = Math.min( + typeof anchor.pausedRemainingMs === "number" ? anchor.pausedRemainingMs : full, + full, + ); + const deadlineAt = resumeAt + remaining; + return advanceRotation( + entries, + { entryId: anchor.entryId, startedAt: deadlineAt - full, deadlineAt }, + at, + ); + } + + if (anchor.transportState === "paused") { + const held = heldEntryAt(entries, anchor, at); + if (!held) return { kind: "black", reason: "no_playable_content" }; + return { + kind: "paused", + entryId: held.id, + remainingMs: + held.id === anchor.entryId && typeof anchor.pausedRemainingMs === "number" + ? Math.min(anchor.pausedRemainingMs, entryDuration(held)) + : entryDuration(held), + }; + } + + if (entries.length === 0) return { kind: "black", reason: "no_playable_content" }; + if (typeof anchor.deadlineAt !== "number" || typeof anchor.startedAt !== "number") { + const now = firstPlayable(entries, at); + if (!now) return { kind: "black", reason: "no_playable_content" }; + return { + kind: "normal", + entryId: now.id, + startedAt: at, + deadlineAt: at + entryDuration(now), + }; + } + + return advanceRotation( + entries, + { entryId: anchor.entryId, startedAt: anchor.startedAt, deadlineAt: anchor.deadlineAt }, + at, + ); +} + +/** + * The next instant local playback must re-evaluate: the current slide's + * deadline, or the next cached eligibility boundary when nothing is playable. + */ +export function nextOfflineBoundary({ manifest, resolved, at }) { + const entries = ordered(manifest); + const candidates = []; + if (resolved?.kind === "normal" && typeof resolved.deadlineAt === "number") { + candidates.push(resolved.deadlineAt); + } + for (const entry of entries) { + for (const value of boundariesOf(entry)) { + if (typeof value === "number" && value > at) candidates.push(value); + } + } + if (resolved?.kind === "static") { + const expiration = + resolved.expirationAt ?? resolved.staticExpirationAt ?? null; + if (typeof expiration === "number" && expiration > at) candidates.push(expiration); + } + return candidates.length > 0 ? Math.min(...candidates) : null; +} diff --git a/server/src/displays/service.js b/server/src/displays/service.js index 2bac2f5..925445a 100644 --- a/server/src/displays/service.js +++ b/server/src/displays/service.js @@ -66,6 +66,32 @@ function parseDisplayError(value) { return value; } +// Only validated error codes become notifications. Arbitrary browser text +// stays telemetry and never reaches Home Assistant. +const DISPLAY_ERROR_SEVERITY = Object.freeze({ + slide_decode_failed: "warning", + asset_fetch_failed: "warning", + // A display that cannot write its cache still shows slides now, but has lost + // its ability to survive the next outage. + cache_write_failed: "warning", + // Nothing renderable for an active playlist is an on-air problem. + manifest_invalid: "critical", + offline_capability_unavailable: "warning", +}); + +const DISPLAY_ERROR_MESSAGES = Object.freeze({ + slide_decode_failed: "could not decode a slide", + asset_fetch_failed: "could not fetch a slide", + cache_write_failed: "could not store slides for offline use", + manifest_invalid: "cannot render any slide of its active playlist", + offline_capability_unavailable: "cannot cache slides for offline use", +}); + +function errorConditionKey(clientId, error) { + const scope = error.assetId ? `:${error.assetId}` : ""; + return `display:${clientId}:${error.kind}${scope}`; +} + function validateHeartbeat(payload) { if (payload === null || typeof payload !== "object" || Array.isArray(payload)) { throw new TypeError("heartbeat must be an object"); @@ -221,6 +247,47 @@ function createDisplayService({ return result.recovered; } + // Reconciles the one display-error condition this client currently reports + // against whatever it had open before. Phase 4 deduplication stays + // authoritative: a persisting error is never re-notified, and a resolved one + // recovers exactly once. + function reconcileErrorIncidents(clientId, row, at) { + if (!notifications) return; + const parsed = parseDisplayError(row.last_error); + const structured = + parsed && typeof parsed === "object" && DISPLAY_ERROR_SEVERITY[parsed.kind] + ? parsed + : null; + const desired = structured ? errorConditionKey(clientId, structured) : null; + const offlineKey = conditionKey(clientId); + + for (const key of notifications.openConditionKeys(`display:${clientId}:`)) { + // The display-offline condition is a separate concern and is never + // touched here. + if (key === offlineKey || key === desired) continue; + notifications.observeRecovery(key, { + event: "display.error_cleared", + message: `Display ${clientId} no longer reports that error.`, + context: { clientId, recoveredAt: new Date(at).toISOString() }, + }); + } + + if (!structured) return; + notifications.observeFailure(desired, { + event: `display.${structured.kind}`, + severity: DISPLAY_ERROR_SEVERITY[structured.kind], + message: `Display ${clientId} ${DISPLAY_ERROR_MESSAGES[structured.kind]}.`, + context: { + clientId, + outputId: row.output_id, + kind: structured.kind, + assetId: structured.assetId ?? null, + entryId: structured.entryId ?? null, + detail: structured.message ?? null, + }, + }); + } + function heartbeat(payload) { const telemetry = validateHeartbeat(payload); const at = now(); @@ -237,6 +304,7 @@ function createDisplayService({ const row = findClient.get(telemetry.clientId); const wasOffline = existing ? isOffline(existing, at) : false; const recovered = recoverOffline(row, at); + reconcileErrorIncidents(telemetry.clientId, row, at); const errorChanged = (existing?.last_error ?? null) !== telemetry.lastError; if (!existing) { @@ -339,7 +407,9 @@ module.exports = { DEFAULT_OFFLINE_THRESHOLD_MS, HEARTBEAT_PROTOCOL_VERSION, MAX_ERROR_LENGTH, + DISPLAY_ERROR_SEVERITY, createDisplayService, + errorConditionKey, parseDisplayError, validateHeartbeat, }; diff --git a/server/src/notifications/index.js b/server/src/notifications/index.js index 6331a61..eb6ff8a 100644 --- a/server/src/notifications/index.js +++ b/server/src/notifications/index.js @@ -232,6 +232,15 @@ function createNotificationService({ return Boolean(findOpen.get(conditionKey)); } + // Lets a caller find the conditions it currently has open, so a changed + // condition can be recovered without inventing a second bookkeeping store. + function openConditionKeys(prefix) { + return listOpenIncidents + .all() + .map((incident) => incident.condition_key) + .filter((key) => (prefix ? key.startsWith(prefix) : true)); + } + async function deliverOne(row) { const at = now(); const headers = { "content-type": "application/json" }; @@ -341,6 +350,7 @@ function createNotificationService({ deliverDue, isOpen, observeFailure, + openConditionKeys, observeRecovery, start, status, diff --git a/server/test/displays.test.js b/server/test/displays.test.js index 6c3931d..45a59af 100644 --- a/server/test/displays.test.js +++ b/server/test/displays.test.js @@ -3,7 +3,9 @@ const assert = require("node:assert/strict"); const { openDatabase } = require("../src/database"); const { runMigrations } = require("../src/database/migrate"); const { + DISPLAY_ERROR_SEVERITY, createDisplayService, + errorConditionKey, parseDisplayError, } = require("../src/displays/service"); const { createNotificationService } = require("../src/notifications"); @@ -422,3 +424,163 @@ test("a structured display error round-trips without loosening validation", (t) assert.equal(parseDisplayError("{not json"), "{not json"); assert.equal(parseDisplayError('{"no":"kind"}'), '{"no":"kind"}'); }); + +function structuredError(kind, overrides = {}) { + return JSON.stringify({ + kind, + assetId: null, + entryId: null, + message: `${kind} detail`, + at: BASE_TIME, + ...overrides, + }); +} + +test("a structured display error opens exactly one incident and never repeats", (t) => { + const harness = createHarness(); + t.after(() => harness.database.close()); + + for (let beat = 0; beat < 6; beat += 1) { + harness.clock.set(BASE_TIME + beat * 12_000); + harness.service.heartbeat( + heartbeatPayload({ + mostRecentError: structuredError("slide_decode_failed", { assetId: "asset-a" }), + }), + ); + } + + const incidents = harness.incidents(); + assert.equal(incidents.length, 1, "a persisting error is reported once, not per heartbeat"); + assert.equal(incidents[0].event, "display.slide_decode_failed"); + assert.equal(incidents[0].severity, "warning"); + assert.equal( + incidents[0].condition_key, + errorConditionKey("lobby-display", { kind: "slide_decode_failed", assetId: "asset-a" }), + ); + assert.equal(JSON.parse(incidents[0].context_json).assetId, "asset-a"); + assert.equal(harness.outbox().length, 1); +}); + +test("a resolved display error recovers exactly once", (t) => { + const harness = createHarness(); + t.after(() => harness.database.close()); + + harness.service.heartbeat( + heartbeatPayload({ + mostRecentError: structuredError("asset_fetch_failed", { assetId: "asset-a" }), + }), + ); + assert.equal(harness.outbox().length, 1); + + for (let beat = 1; beat <= 3; beat += 1) { + harness.clock.set(BASE_TIME + beat * 12_000); + harness.service.heartbeat(heartbeatPayload({ mostRecentError: null })); + } + + const incidents = harness.incidents(); + assert.equal(incidents.length, 1); + assert.equal(incidents[0].state, "recovered"); + assert.equal(harness.outbox().length, 2, "one failure and one recovery"); + assert.equal(harness.outbox()[1].transition, "recovery"); +}); + +test("an error moving to another asset recovers the old condition and opens the new one", (t) => { + const harness = createHarness(); + t.after(() => harness.database.close()); + + harness.service.heartbeat( + heartbeatPayload({ + mostRecentError: structuredError("asset_fetch_failed", { assetId: "asset-a" }), + }), + ); + harness.clock.set(BASE_TIME + 12_000); + harness.service.heartbeat( + heartbeatPayload({ + mostRecentError: structuredError("asset_fetch_failed", { assetId: "asset-b" }), + }), + ); + + const byKey = Object.fromEntries( + harness.incidents().map((incident) => [incident.condition_key, incident]), + ); + const keyA = errorConditionKey("lobby-display", { + kind: "asset_fetch_failed", + assetId: "asset-a", + }); + const keyB = errorConditionKey("lobby-display", { + kind: "asset_fetch_failed", + assetId: "asset-b", + }); + assert.equal(byKey[keyA].state, "recovered", "the resolved asset condition closes"); + assert.equal(byKey[keyB].state, "open", "the new asset condition opens"); + assert.equal(harness.outbox().length, 3); +}); + +test("only validated error codes become incidents", (t) => { + const harness = createHarness(); + t.after(() => harness.database.close()); + + // Arbitrary browser text is telemetry, never a Home Assistant notification. + harness.service.heartbeat( + heartbeatPayload({ mostRecentError: "TypeError: undefined is not a function" }), + ); + assert.equal(harness.incidents().length, 0); + assert.equal( + harness.service.list()[0].mostRecentError, + "TypeError: undefined is not a function", + "the raw telemetry is still visible to operators", + ); + + // A structured payload with an unrecognised code is equally inert. + harness.clock.set(BASE_TIME + 12_000); + harness.service.heartbeat( + heartbeatPayload({ mostRecentError: structuredError("something_invented") }), + ); + assert.equal(harness.incidents().length, 0); + assert.equal(harness.outbox().length, 0); + assert.equal(harness.service.list()[0].mostRecentError.kind, "something_invented"); +}); + +test("a manifest that cannot render locally is critical", (t) => { + const harness = createHarness(); + t.after(() => harness.database.close()); + + harness.service.heartbeat( + heartbeatPayload({ mostRecentError: structuredError("manifest_invalid") }), + ); + + assert.equal(harness.incidents()[0].severity, "critical"); + assert.equal(DISPLAY_ERROR_SEVERITY.manifest_invalid, "critical"); + assert.equal(DISPLAY_ERROR_SEVERITY.offline_capability_unavailable, "warning"); +}); + +test("display error incidents never disturb the display offline condition", (t) => { + const harness = createHarness(); + t.after(() => harness.database.close()); + + harness.service.heartbeat(heartbeatPayload()); + // The display goes offline, then returns while reporting a slide problem. + harness.clock.runTo(BASE_TIME + THRESHOLD); + assert.equal(harness.incidents().length, 1); + assert.equal(harness.incidents()[0].event, "display.offline"); + + harness.clock.set(BASE_TIME + 60_000); + harness.service.heartbeat( + heartbeatPayload({ + mostRecentError: structuredError("slide_decode_failed", { assetId: "asset-a" }), + }), + ); + + const offline = harness + .incidents() + .find((incident) => incident.condition_key === "display:lobby-display"); + assert.equal(offline.state, "recovered", "returning still recovers the offline condition once"); + const decode = harness + .incidents() + .find((incident) => incident.event === "display.slide_decode_failed"); + assert.equal(decode.state, "open"); + assert.equal( + harness.incidents().filter((incident) => incident.event === "display.offline").length, + 1, + ); +}); diff --git a/server/test/offline-contract.test.js b/server/test/offline-contract.test.js new file mode 100644 index 0000000..2330dce --- /dev/null +++ b/server/test/offline-contract.test.js @@ -0,0 +1,288 @@ +const test = require("node:test"); +const assert = require("node:assert/strict"); +const path = require("node:path"); +const { pathToFileURL } = require("node:url"); +const { openDatabase } = require("../src/database"); +const { runMigrations } = require("../src/database/migrate"); +const { createPlaybackService } = require("../src/playback/service"); + +// Client/server contract. +// +// The disconnected display necessarily reimplements part of the playback +// timeline. Rather than trusting that the two agree, every scenario below runs +// the real Phase 3B engine and the browser evaluator over identical data and +// asserts they land on the same slide, start, and deadline. Phase 3B is not +// refactored to share code; it stays the verified authority. +const CLIENT_MODULE = pathToFileURL( + path.join(__dirname, "..", "..", "client", "src", "display", "offlineTimeline.js"), +).href; + +const BASE_TIME = 2_000_000_000_000; + +function createHarness({ defaultDurationMs = 1_000 } = {}) { + const database = openDatabase({ path: ":memory:" }); + runMigrations(database); + database + .prepare("UPDATE app_settings SET default_duration_ms = ?, updated_at = ? WHERE id = 1") + .run(defaultDurationMs, BASE_TIME); + + let current = BASE_TIME; + const clock = { + now: () => current, + set(value) { + current = value; + }, + }; + let assetSequence = 0; + let service = null; + + function seedAsset({ expirationAt = null } = {}) { + const ordinal = ++assetSequence; + const id = `asset-${ordinal}`; + const sha = ordinal.toString(16).padStart(64, "0"); + database + .prepare( + `INSERT INTO assets ( + id, source_sha256, display_name, original_filename, status, expiration_at, + display_object_key, display_sha256, display_content_type, display_byte_size, + display_width, display_height, created_at, updated_at + ) VALUES (?, ?, ?, ?, 'ready', ?, ?, ?, 'image/webp', 10, 1920, 1080, ?, ?)`, + ) + .run(id, sha, id, `${id}.png`, expirationAt, `private/${id}.webp`, sha, BASE_TIME, BASE_TIME); + return id; + } + + function seedPlaylist(entries) { + database + .prepare( + "INSERT INTO playlists (id, name, revision, created_at, updated_at) VALUES ('p', 'P', 1, ?, ?)", + ) + .run(BASE_TIME, BASE_TIME); + entries.forEach((options, position) => { + const assetId = seedAsset({ expirationAt: options.assetExpirationAt ?? null }); + database + .prepare( + `INSERT INTO playlist_entries ( + id, playlist_id, asset_id, enabled, duration_override_ms, + schedule_start_at, schedule_end_at, position, created_at, updated_at + ) VALUES (?, 'p', ?, 1, ?, ?, ?, ?, ?, ?)`, + ) + .run( + options.id, + assetId, + options.durationMs ?? null, + options.scheduleStartAt ?? null, + options.scheduleEndAt ?? null, + position, + BASE_TIME, + BASE_TIME, + ); + }); + return "p"; + } + + function startService() { + service = createPlaybackService({ + database, + now: clock.now, + setTimeoutFn: () => null, + clearTimeoutFn: () => {}, + logger: { warn() {} }, + }); + service.initialize(); + return service; + } + + function playback() { + return database.prepare("SELECT * FROM playback_state WHERE output_id = 'global'").get(); + } + + return { + clock, + close: () => database.close(), + database, + playback, + seedPlaylist, + startService, + get service() { + return service; + }, + }; +} + +// Runs one disconnected scenario through both implementations. +async function compare(t, { entries, defaultDurationMs, offlineAt, label }) { + const { resolveOfflineState } = await import(CLIENT_MODULE); + const harness = createHarness({ defaultDurationMs }); + t.after(() => harness.close()); + harness.seedPlaylist(entries); + + const service = harness.startService(); + service.activatePlaylist("p"); + + // What the display would have persisted as its last authoritative state. + const snapshot = service.getSnapshot(); + const anchor = { + entryId: snapshot.currentEntryId, + startedAt: snapshot.timeline.slideStartedAt, + deadlineAt: snapshot.timeline.nextSlideAt, + transportState: snapshot.transportState, + pausedRemainingMs: snapshot.timeline.pausedRemainingMs, + staticAssetId: snapshot.staticTarget?.assetId ?? null, + staticTarget: snapshot.staticTarget, + pendingPlaylistId: snapshot.pendingPlaylistId ?? null, + }; + const manifest = snapshot.manifest; + + // The server catches up the same outage on restart. + harness.clock.set(offlineAt); + harness.startService(); + const server = harness.playback(); + + const client = resolveOfflineState({ manifest, anchor, at: offlineAt }); + + if (server.current_entry_id === null) { + assert.equal(client.kind, "black", `${label}: both must report no playable content`); + return { client, server }; + } + + assert.equal(client.kind, "normal", `${label}: client must resolve a normal slide`); + assert.equal(client.entryId, server.current_entry_id, `${label}: same slide`); + assert.equal(client.startedAt, server.slide_started_at, `${label}: same start instant`); + assert.equal(client.deadlineAt, server.next_slide_at, `${label}: same deadline`); + return { client, server }; +} + +test("offline catch-up matches the server across mixed durations and wraparound", async (t) => { + const entries = [ + { id: "a", durationMs: 3_000 }, + { id: "b", durationMs: 9_000 }, + { id: "c", durationMs: 4_000 }, + ]; + // One cycle is 16s; each instant lands in a different part of the rotation. + for (const offset of [1_000, 3_000, 5_000, 12_000, 16_000, 17_000, 31_999, 32_000, 100_000]) { + await compare(t, { + entries, + offlineAt: BASE_TIME + offset, + label: `mixed durations +${offset}ms`, + }); + } +}); + +test("offline catch-up matches the server over hours and days offline", async (t) => { + const entries = [ + { id: "a", durationMs: 3_000 }, + { id: "b", durationMs: 9_000 }, + { id: "c", durationMs: 4_000 }, + ]; + const started = process.hrtime.bigint(); + for (const offset of [ + 3_600_000, // one hour + 86_400_000, // one day + 604_800_000, // one week + 2_592_000_000, // thirty days + ]) { + await compare(t, { + entries, + offlineAt: BASE_TIME + offset + 1_234, + label: `long outage +${offset}ms`, + }); + } + const elapsedMs = Number(process.hrtime.bigint() - started) / 1e6; + // Thirty days of 16-second cycles is ~160k slides; arithmetic catch-up must + // not step through them one at a time. + assert.ok( + elapsedMs < 4_000, + `catch-up must be arithmetic, not iterative (took ${elapsedMs.toFixed(0)}ms)`, + ); +}); + +test("offline catch-up matches the server across a schedule start", async (t) => { + const entries = [ + { id: "always", durationMs: 1_000 }, + { id: "joins", durationMs: 1_000, scheduleStartAt: BASE_TIME + 20_000 }, + ]; + for (const offset of [19_999, 20_000, 20_001, 25_000, 60_000]) { + await compare(t, { + entries, + offlineAt: BASE_TIME + offset, + label: `schedule start +${offset}ms`, + }); + } +}); + +test("offline catch-up matches the server across a schedule end", async (t) => { + const entries = [ + { id: "ends", durationMs: 5_000, scheduleEndAt: BASE_TIME + 12_500 }, + { id: "stays", durationMs: 5_000 }, + ]; + for (const offset of [12_499, 12_500, 12_501, 30_000, 120_000]) { + await compare(t, { + entries, + offlineAt: BASE_TIME + offset, + label: `schedule end +${offset}ms`, + }); + } +}); + +test("offline catch-up matches the server across an asset expiration", async (t) => { + const entries = [ + { id: "expires", durationMs: 6_000, assetExpirationAt: BASE_TIME + 15_000 }, + { id: "permanent", durationMs: 4_000 }, + ]; + for (const offset of [14_999, 15_000, 15_001, 40_000, 300_000]) { + await compare(t, { + entries, + offlineAt: BASE_TIME + offset, + label: `expiration +${offset}ms`, + }); + } +}); + +test("offline evaluation agrees with the server when nothing is playable", async (t) => { + const entries = [{ id: "only", durationMs: 5_000, scheduleEndAt: BASE_TIME + 8_000 }]; + const { client } = await compare(t, { + entries, + offlineAt: BASE_TIME + 30_000, + label: "no playable content", + }); + assert.equal(client.kind, "black"); + assert.equal(client.reason, "no_playable_content"); +}); + +test("a permanent asset never expires locally no matter how long the display is offline", async (t) => { + const { resolveOfflineState } = await import(CLIENT_MODULE); + const manifest = { + entries: [ + { + id: "permanent", + position: 0, + assetId: "asset-permanent", + effectiveDurationMs: 5_000, + scheduleStartAt: null, + scheduleEndAt: null, + // A null expiration is permanent and must stay permanent. + expirationAt: null, + }, + ], + }; + const anchor = { + entryId: "permanent", + startedAt: BASE_TIME, + deadlineAt: BASE_TIME + 5_000, + transportState: "playing", + }; + + const afterAYear = resolveOfflineState({ + manifest, + anchor, + at: BASE_TIME + 365 * 24 * 3_600_000, + }); + assert.equal(afterAYear.kind, "normal"); + assert.equal(afterAYear.entryId, "permanent"); + assert.equal( + (afterAYear.deadlineAt - afterAYear.startedAt) % 5_000, + 0, + "the slide keeps its configured duration on the original phase", + ); +});