Implement Phase 4 integrations and display telemetry

Add MQTT playback control, durable Home Assistant incident notifications,
and display heartbeat telemetry on top of the verified Phase 3B playback
authority. No playback logic is duplicated: every MQTT command and every
observed condition goes through the existing playback service, and all
integration work happens after the authoritative SQLite commit.

MQTT (server/src/mqtt) subscribes the exact required command topics under
a configurable prefix that defaults to announcements. Durable commands use
QoS 1; next and previous stay at QoS 0 because they are not idempotent and
carry no command ID. Retained authoritative state is published only on the
separate availability, mode/state, and playback/state topics, never onto a
command topic, and only after a mutation commits. A registered last will
reports an ungraceful disappearance, a clean shutdown retracts it, and a
broker outage degrades status while playback, HTTP, and the display
protocol continue untouched.

Notifications (server/src/notifications) use the Phase 1 incident and
outbox tables. A failing condition opens one incident and queues exactly
one notification; continued failure only updates its observation time;
recovery queues exactly one recovery; a later failure is a new episode.
Delivery retries on a documented capped schedule with a stable notification
ID, survives restart, and never blocks playback. With no webhook URL
configured, incidents are still recorded but no delivery work is queued.
Conditions come from storage health transitions, committed playback state,
display heartbeat staleness, and newly failed uploads.

Displays (server/src/displays) ingest validated, bounded display:heartbeat
telemetry keyed by persistent client ID rather than socket ID. Telemetry is
read-only and never touches playback or playlist revisions. Offline is
decided by heartbeat staleness, not by a closed socket, and a startup grace
period keeps an API restart from manufacturing offline alerts. Adds
GET /api/displays and meaningful-change display:status events.

Health and status now report redacted storage, MQTT, notification,
playback, and display state. External integrations degrade status and never
affect readiness, so no integration outage can cause a restart loop. No
MQTT password, S3 credential, webhook URL, or webhook secret is exposed.

Uses the existing Phase 1 schema unchanged; no migration was required.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Media
2026-08-16 22:52:59 -05:00
co-authored by Claude Opus 5
parent 993986c25e
commit ae70bf1dd8
22 changed files with 3691 additions and 24 deletions
+188
View File
@@ -181,6 +181,186 @@ exit static safely, return to normal rotation, and only then delete the asset.
Phase 3B does not depend on the refusal, so the change belongs to the later
phase that revisits asset lifecycle.
## MQTT integration
The API connects to `MQTT_URL` when it is configured and stays disabled when it
is not. Commands are dispatched straight into the authoritative playback
service, so an MQTT command and its REST equivalent are the same operation with
the same semantics; no playback logic is duplicated in the integration.
With the default `MQTT_TOPIC_PREFIX` of `announcements` the command topics are:
| Topic | Payload | QoS |
| ------------------------- | ---------------------- | --- |
| `announcements/mode` | `on` or `off` | 1 |
| `announcements/pause` | ignored | 1 |
| `announcements/resume` | ignored | 1 |
| `announcements/reload` | ignored | 1 |
| `announcements/static` | LibraryAsset UUID | 1 |
| `announcements/next` | ignored | 0 |
| `announcements/previous` | ignored | 0 |
Mode payloads are trimmed and case-insensitive. `off` is only the visual black
gate; it never pauses, restarts, or resets the playlist, cursor, static hold, or
timing. Static accepts only a stable LibraryAsset UUID: URLs, filenames, and S3
object keys are rejected. The asset must exist, be `ready`, and be unexpired,
but it does not need to appear in any playlist. A rejected command is logged as
a structured warning, changes nothing, and publishes no state.
`next` and `previous` stay at QoS 0 on purpose. They are not idempotent and
carry no command ID, so an at-least-once redelivery would silently skip a
slide. Commands are never retained, and the server never publishes onto a
command topic.
The server publishes retained state on separate topics:
| Topic | Payload |
| --------------------------------- | ---------------------------------- |
| `announcements/availability` | `online` / `offline` |
| `announcements/mode/state` | `on` / `off` |
| `announcements/playback/state` | small versioned JSON snapshot |
Availability describes this API's MQTT integration, not a display. A registered
last will and testament lets the broker publish `offline` when the API
disappears without a clean disconnect; a planned shutdown retracts it directly.
The compact playback topic carries mode, transport, active playlist, current
entry and asset, static asset, and revision. It never carries the Socket.IO
manifest or any private storage detail.
State is published only after a playback mutation has committed, and MQTT
publication happens after the SQLite commit. A publish failure degrades the
integration and is never allowed to roll playback back. A broker outage leaves
the API, HTTP, and playback untouched: the client reconnects on its own, then
resubscribes and republishes availability and authoritative state.
## Home Assistant notifications
Announcements sends outgoing generic webhook POSTs to `HA_WEBHOOK_URL`. Home
Assistant is only the receiver; Announcements never calls Companion directly and
needs no Home Assistant authentication beyond the configured webhook. The URL
and `HA_WEBHOOK_SECRET` are configuration, never API or status output.
Notifications are durable and deduplicated through `notification_incidents` and
`notification_outbox`. A condition that starts failing opens one incident and
queues exactly one failure notification. While it keeps failing only its
observation time is updated, so a thirty-minute outage still sends one message.
Recovery closes the incident and queues exactly one recovery notification. If
the same condition fails again later that is a new episode with its own
failure/recovery pair. Open incidents and undelivered notifications survive an
API restart.
Reported conditions are `storage.unavailable` / `storage.recovered`, driven by
the Phase 2 storage health transition rather than individual request failures;
`display.offline` / `display.recovered` per display client; and
`playback.no_playable_slides` / `playback.playable` from committed playback
state. A static override does not suppress the playback condition, because
holding one asset does not repair an empty playlist. A failed upload opens one
`asset.processing_failed` warning per asset; an individual failed upload has no
automatic recovery, so the operator resolves it by deleting or re-uploading.
The payload is versioned:
```json
{
"version": 1,
"notificationId": "stable-delivery-id",
"incidentId": "stable-incident-id",
"event": "storage.unavailable",
"severity": "critical",
"state": "failure",
"message": "Announcement storage is unavailable.",
"timestamp": "2026-08-16T12:00:00.000Z",
"context": { "conditionKey": "storage", "episode": 1 }
}
```
Severities are `info`, `warning`, `critical`, and `recovery`. Home Assistant can
treat them all as ordinary notifications; the field exists so later Ekklesia or
HA handling can discriminate.
Delivery is asynchronous and bounded. A 2xx marks the notification delivered.
Any other response and any transport failure leaves it pending and retries after
30 seconds, 1 minute, 5 minutes, 15 minutes, and 30 minutes, then every 30
minutes. The notification ID is stable across retries so a receiver can
deduplicate, retries resume after a restart, and a failing webhook never blocks
playback, MQTT, storage, or heartbeats. When no webhook URL is configured,
incidents are still recorded but no delivery work is queued, and status reports
the integration as disabled.
An example Home Assistant receiver:
```yaml
automation:
- alias: Announcements Notifications
triggers:
- trigger: webhook
webhook_id: announcements
allowed_methods:
- POST
local_only: true
actions:
- action: notify.mobile_app_YOUR_DEVICE
data:
title: "Announcements"
message: "{{ trigger.json.message }}"
```
Substitute any notify target; Companion is not required. Announcements cannot
send a webhook while the Announcements API itself is completely down. That is a
limitation of self-reporting, not a bug: if server-down alerting is wanted, an
external monitor such as Home Assistant should poll `/health/live`.
## Display telemetry
The future `/display` client reports read-only telemetry over the Socket.IO
event `display:heartbeat`, expected every 10-15 seconds:
```json
{
"protocolVersion": 1,
"clientId": "persistent-display-id",
"outputId": "global",
"activePlaylistId": "playlist-id",
"playlistRevision": 12,
"playbackRevision": 42,
"currentEntryId": "entry-id",
"currentAssetId": "asset-id",
"cachedAssetCount": 10,
"playableAssetCount": 10,
"mostRecentError": null
}
```
Every field is validated and bounded, `mostRecentError` is truncated, and
malformed telemetry is ignored without creating a display. Heartbeats can never
mutate playback and never increment a playback or playlist revision. Identity is
the persistent `clientId` stored in `display_clients`, never a Socket.IO
connection ID, so a browser reconnect remains the same display.
A display is online while heartbeats are arriving and offline once
`DISPLAY_OFFLINE_THRESHOLD_MS` (45 seconds, about three missed heartbeats) has
passed since the last one. A closed socket alone never marks a display offline,
because reconnects and brief network hiccups are normal. Crossing the threshold
opens one `display.offline` incident that does not repeat while the display
stays away, and the next valid heartbeat recovers it exactly once.
Display history is persisted, so a restart does not erase monitoring. The API
cannot know what happened to a display while the API itself was down, so it is
deliberately conservative: nothing is reported offline until a full threshold has
elapsed since startup. A healthy display that reconnects during that window is
never reported, a display that is still missing afterwards is reported once, and
an incident that was already open before the restart is not duplicated.
`GET /api/displays` returns each known display with a status derived from
heartbeat freshness rather than mere database existence, and exposes no socket
internals. `display:status` is emitted only for meaningful changes — first seen,
offline, recovered, or a changed error — so steady telemetry creates no realtime
churn. `mostRecentError` is telemetry only; arbitrary browser strings never
become Home Assistant notifications. Structured display-side errors such as
decode failures arrive with the Phase 5 renderer, which will report them through
this same field and event.
## Storage and health
The API uses a small path-style S3 adapter configured by `S3_ENDPOINT`,
@@ -193,3 +373,11 @@ Storage outages do not stop the API or fail process readiness. `/health/ready`
reports storage as degraded while remaining HTTP 200 when SQLite and the API
runtime are healthy. `/api/health` and `/api/status` expose cached, redacted
storage state and recover after a successful probe or storage operation.
`/health/live` reports only that the process is running. Readiness stays limited
to SQLite, migrations, and playback initialization. Every external integration
degrades the reported status instead of failing readiness, so a broker, webhook
receiver, or display problem can never put the API into a restart loop.
`/api/health` and `/api/status` report redacted storage, MQTT, notification,
playback, and display state. No MQTT password, S3 credential, webhook URL, or
webhook secret appears in any response.