Zayin_Admin/Video-Player-Controller
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
8db46df69e257f60ef0db12a070a3d5fa12a7f38
Video-Player-Controller/README.md
T
Media c2544c44c7 fix: close Audio Router self-echo PUBACK/subscription-delivery race 
publish_router_mode registered its self-echo correlation marker before
publishing but removed it in a finally block the instant the QoS 1 publish()
call observed its PUBACK. A broker may legally deliver a Controller's own
subscription copy of that same publish after the PUBACK, since MQTT does not
order the two; when that happened, the Controller mistook its own late echo
for live operator intent and relinquished automatic restoration ownership.

Keep the marker claimable for a short, bounded mqtt.router_echo_grace_seconds
window (default 2s) after the PUBACK instead of clearing it immediately.
Genuine external/operator commands are unaffected: once the grace window
elapses, a matching payload is still correctly treated as operator intent
and ownership is relinquished as before.

Verified with deterministic fake-clock unit tests plus a real Mosquitto
broker integration test (tests/test_router_echo_mosquitto_integration.py,
skipped automatically when Docker is unavailable).
2026-08-07 17:13:54 -05:00

983 lines
47 KiB
Markdown
Raw Blame History

# Video Player Controller v2
Video Player Controller downloads media into ten stable slots, publishes the
existing Home Assistant and Companion state topics, and coordinates playback
between the authoritative Video Player Node and the Audio Router.
Controller v2 is deliberately conservative: publishing a command is not proof
that playback changed. Only fresh Player Node status or telemetry can establish
playing, paused, or stopped state. A command accepted by the MQTT broker remains
`pending` until that evidence arrives.
The Controller does not run or supervise `mpv`. The Player Node owns the MPV
lifecycle and sends audio only to `pipewire/ytplayer_bus`; the Controller owns
the request/confirmation/restoration workflow for Audio Router mode.
## Architecture and safety model
```text
Browser / HA / Companion
|
| download requests and Controller commands
v
Flask -> bounded event queue -> single Controller coordinator
| | |
| | +-> SQLite state/audit
| +------------> bounded yt-dlp workers
+---------------------> MQTT
|
+-----------------------------+------------------+
| |
v v
Video Player Node Audio Router
(authoritative state) (physical audio mode)
|
+-> mpv -> pipewire/ytplayer_bus
```
MQTT callbacks, web requests, timers, and worker notifications converge on a
single coordinator thread for orchestration and publication. Download workers
perform their external work independently and commit only generation-fenced
SQLite transitions. SQLite transactions are the final persistence boundary.
This provides the following invariants:
- Slots `1` through `10` are permanent identities. A hole stays a hole; deleting
slot 3 never renames slot 4 or its directory.
- A download slot is reserved in an immediate SQLite transaction before a
worker is submitted. A partial unique index prevents two live jobs in one
slot.
- Every reuse increments the slot generation. A late worker or deletion token
cannot commit against a newer occupant.
- Active, pending, uncertain, or operator-review playback guards block slot
replacement and deletion.
- A reserved or running download blocks deletion and expiry.
- Filesystem deletion is limited to the canonical `slotN` directory. The
database first atomically fences the generation as `deleting`; filesystem
work then runs asynchronously. A directory is atomically moved under
`.controller-trash` before recursive removal, and interrupted deletion
recovery is deferred until Player safety is established.
- MQTT can be unavailable at startup. The web/coordinator process remains up,
reports degraded health, and retries with bounded exponential backoff.
The canonical managed media layout is:
```text
/mnt/Media/YTPlayer/
slot1/playlist.m3u
slot2/playlist.m3u
...
slot10/playlist.m3u
```
The base path is configurable, but both Controller and Player Node must see the
same absolute paths.
## Player-authoritative operation model
Controller commands are serialized. A second transition is rejected while a
play, stop, pause, or resume operation is waiting for evidence.
For a play request, the Controller:
1. Validates the stable slot or explicit local path and protects a managed
slot before any external command is sent.
2. Acquires the Audio Router mode using the bounded policy described below.
3. Publishes the Player command at QoS 1 with `retain=false` and waits up to
`mqtt.publish_ack_timeout_seconds` for the broker PUBACK.
4. Reports the request as pending. It does **not** publish `playing` merely
because the command was sent.
5. Confirms, rejects, or marks the operation uncertain from matching Player
status/telemetry. The normal Player-evidence deadline is 10 seconds.
A PUBACK proves only broker receipt, not Player execution. If the PUBACK is
uncertain, the Controller replaces its MQTT transport so an unacknowledged QoS
1 command cannot be replayed later, and it preserves uncertain playback state
for reconciliation.
Player evidence is interpreted using the current Player Node contract:
- `playing` status or telemetry with a path establishes active playback.
- Telemetry's boolean `paused` field is authoritative for pause state. Player
`paused` and `toggled` status only show that control was issued; Controller
waits for telemetry before confirming the resulting pause value. If pause,
resume, or toggle confirmation times out, both phase and pause value project
as `unknown` until fresh evidence arrives.
- `busy` means the new play did not replace the Player's existing process. When
a slot/path is present, or an earlier identity is still known, that existing
playback remains authoritative. A real pathless `busy` with no prior identity
becomes unknown; Controller keeps the Router lease and global cleanup fence
until identity/terminal evidence is reasserted or an administrator reconciles
the uncertainty.
- Terminal `idle` requires one of `stop`, `finished`, `mpv_error`, `sigterm`,
`sigint`, or `shutdown`, or cleared telemetry. An ambiguous idle does not
erase active ownership.
- A generic Player `error` is not proof that MPV stopped. Known managed or
external playback becomes reviewable uncertainty; an identity-less error
keeps the global cleanup fence closed. Only error reasons `startup_error` and
`worker_start_error` prove that playback never became active and reduce to
idle even if cleared telemetry was lost.
- On reconnect, `idle` with `message:"player online"` and a path means the
Player is still active. The same announcement without a path means idle.
- The retained flat `youtube/player/state` topic is never playback or cleanup
evidence. Its value is historical across outages, and Player reconnect
deliberately emits flat `idle` even while MPV is active. Controller keeps the
subscription for Companion compatibility but does not use it to unlock
downloads, expiry, deletion recovery, or playback guards.
Use `youtube/controller/cmd` for operational control. Publishing directly to
`youtube/player/cmd` bypasses Controller Audio Router acquisition and slot
guard creation. Controller will eventually reduce resulting Player evidence,
but there is an unavoidable cross-service race before that evidence arrives.
## MQTT contracts
Defaults remain compatible with the existing Player Node, Home Assistant, and
Companion integrations. Topic names can be configured except where another
component treats one as fixed; coordinate any change across all consumers.
### Topic matrix
| Direction | Default topic | Payload | QoS used by Controller | Retained | Contract status |
|---|---|---|---:|---:|---|
| Controller -> Player | `youtube/player/cmd` | JSON command | publish 1 | no | Preserved; required |
| Player -> Controller | `youtube/player/status` | JSON status | subscribe 1 | no | Player-authoritative |
| Player -> Controller | `youtube/player/state` | plain flat state | subscribe 0 | yes, by Player | Preserved Companion contract only; never cleanup evidence |
| Player -> Controller | `youtube/player/telemetry` | JSON telemetry | subscribe 0 | no | Player-authoritative |
| HA/Companion -> Controller | `youtube/controller/cmd` | JSON command | subscribe 1 | must be no | Preserved; retained commands are rejected |
| Controller -> consumers | `youtube/ingest/progress` | JSON progress | publish 0 | no | Preserved |
| Controller -> consumers | `youtube/media/playback/<slot>/title` | plain string | publish 0 | yes | Preserved |
| Controller -> consumers | `youtube/media/playback/<slot>/state` | plain string | publish 0 | yes | Preserved |
| Controller -> consumers | `youtube/media/playback/<slot>/count` | decimal string | publish 0 | yes | Preserved |
| Controller -> consumers | `youtube/media/playback/<slot>/items/<position>` | plain string | publish 0 | yes | Preserved; empty payload clears stale items |
| Controller/operator -> Router | `audio/player/mode/set` | plain mode string | publish and subscribe 1 | Controller publishes no | Router contract |
| Router -> Controller | `audio/player/mode/state` | plain mode string | subscribe 1 | Router-owned | Router confirmation |
| Controller -> consumers | `youtube/controller/v2/events` | JSON event | publish 1 | no | Optional, additive v2 contract |
| Controller -> Home Assistant | `homeassistant/sensor/yt_slot_<slot>_<field>/config` | discovery JSON | publish 0 | yes | Preserved IDs |
QoS in the incoming rows is the Controller's requested subscription QoS; the
publisher and broker determine the delivered QoS. Player commands and Router
mode commands are always emitted with QoS 1 and `retain=false`.
The Controller uses Paho manual acknowledgement for incoming QoS 1 messages.
It acknowledges after successful coordinator handling and deduplicates repeated
packets by MQTT connection generation, topic, and packet ID. Retained Controller
commands are rejected so a reconnect cannot replay an old control operation.
Detailed Player status and telemetry are valid evidence only when
`retain=false`. If either arrives retained, Controller treats it as historical,
does not reduce it, does not unlock cleanup, and does not clear guards. It logs
and audits `retained_player_evidence_rejected`, publishing that warning when the
optional v2 event topic is enabled; a retained QoS 1 message is deliberately
acknowledged after rejection so the same invalid evidence is not redelivered
indefinitely.
### Controller command payloads
Publish all examples at QoS 1 without retain:
```bash
mosquitto_pub -h 10.40.0.2 -t youtube/controller/cmd -q 1 \
-m '{"action":"play","slot":1,"label":"Opening"}'
```
Supported JSON objects are:
```json
{"action":"play","slot":1,"label":"Opening"}
```
```json
{"action":"play","path":"/mnt/Media/YTPlayer/special/welcome.mp4","label":"Welcome"}
```
```json
{"action":"play","playlist":"/mnt/Media/YTPlayer/special/playlist.m3u","title":"Special"}
```
```json
{"action":"stop"}
```
```json
{"action":"pause"}
```
```json
{"action":"resume"}
```
```json
{"action":"toggle"}
```
```json
{"action":"delete_slot","slot":4,"url":"optional stale-command guard"}
```
`play` requires a ready managed slot or an existing local `path`/`playlist`.
An explicit path inside a managed slot derives that stable slot identity; an
explicit conflicting slot is rejected. `toggle` is translated to `pause` or
`resume` from the latest telemetry and is rejected when pause state is unknown.
When the global cleanup gate lacks fresh Player evidence, manual deletion also
requires the exact boolean override:
```json
{"action":"delete_slot","slot":4,"confirm_player_unknown":true}
```
That override only bypasses the global evidence gate. It does not bypass an
active download or a per-slot active/uncertain playback guard. It is separate
from administrative reconciliation, authorizes only that manual delete, and
does not open automatic expiry/recovery or consume a cleanup token.
An accepted manual delete is atomically fenced in SQLite and returns
`{"status":"deleting","slot":4}`, not `deleted`. Filesystem removal happens
on a worker afterward. MQTT commands have no synchronous response topic, so an
MQTT consumer observes the retained slot state (and optional v2 warnings) rather
than depending on that in-process return value.
### Preserved slot and Home Assistant state
For every stable slot, Controller publishes retained `title`, `state`, and
`count` values plus retained item titles. Projected state values include
`empty`, `queued`, `downloading`, `ready`, `error`, `deleting`, `playing`,
`paused`, `unknown`, and `operator_review`. A pending play remains `ready`
until Player evidence says otherwise.
The thirty discovery entities keep the established identifiers:
- Discovery topic: `homeassistant/sensor/yt_slot_<N>_<title|state|count>/config`
- Unique ID: `yt_slot_<N>_<title|state|count>`
- Device identifier: `ytplayer-controller`
- Device name/manufacturer/model: `YouTube Controller` / `River of Glory` /
`YT Controller`
Changing `homeassistant.discovery_prefix` changes only the discovery prefix.
Set `homeassistant.enabled: false` to suppress discovery; legacy slot state
publication continues.
### Ingest progress
`youtube/ingest/progress` is a non-retained JSON stream. Payloads always contain
`slot` and `state`, with applicable fields such as `title`, `count`, `message`,
`url`, `submitter`, and `note`. A `queued` event also contains `reused_slot`.
Common states are `queued`, `downloading`, `ready`, and `error`.
### Optional v2 events
`topics.controller_result` defaults to `youtube/controller/v2/events`. Set it
to YAML `null` to disable publication. This topic is deliberately additive:
legacy MQTT, Home Assistant, and Companion consumers never need it.
Events are QoS 1, non-retained JSON:
```json
{
"version": 2,
"event_id": "6ed8f0e49a8b4d598e5285ca14ec89ef",
"type": "audio_router_warning",
"severity": "warning",
"message": "Audio Router did not confirm 'video' within 3 seconds",
"ts": 1786060800.0,
"slot": 1,
"operation_id": "4e32c8d628454426bff69c879e709f33",
"continued": true
}
```
`slot`, `operation_id`, and `continued` appear only when relevant. Each event
is also written to `audit_events` and logged, so the non-retained stream is not
the durable record. Event publication itself depends on MQTT availability.
## Audio Router orchestration
The normal playback target is the plain mode `video`:
- Command: `audio/player/mode/set`
- Confirmed state: `audio/player/mode/state`
Before play, Controller snapshots the latest confirmed Router state and creates
one persisted lease for the playback session. Router acquisition and restoration
both use the finite `router.confirmation_timeout_seconds`; neither can hang a
play operation indefinitely.
### Unknown-state and failure policy
The four Router settings are independent:
| Setting | Default | Effect |
|---|---|---|
| `router.confirmation_timeout_seconds` | `3` | Maximum wait for acquisition or restoration confirmation |
| `router.when_state_unknown` | `attempt` | `attempt`, `continue`, or `reject` when no current state exists |
| `router.continue_on_confirmation_failure` | `true` | Continue playback after publish failure, disconnect, operator conflict, or confirmation timeout |
| `router.fallback_restore_mode` | `null` | Optional restoration target when the prior mode was unknown |
`when_state_unknown` has these meanings:
- `attempt`: request `video`, wait for the bounded confirmation, then apply
`continue_on_confirmation_failure` if it cannot be confirmed.
- `continue`: request `video` and start playback immediately without claiming
restoration ownership. This still records and publishes a warning.
- `reject`: reject this play request without sending a Player command. The
caller may retry after a current Router state becomes available.
The safe service default is `attempt` plus
`continue_on_confirmation_failure: true`: it tries to set and confirm `video`
for three seconds, but unavailable Router state cannot hang or permanently
disable playback. The warning is persisted, logged, and published on the
optional v2 event topic when MQTT permits.
`fallback_restore_mode` is used only when the previous mode was unknown and
acquisition subsequently becomes confirmed. A play that continues immediately
or after failed confirmation has no safe restoration ownership, so it does not
apply the fallback later.
### Safe restoration and manual override
At authoritative playback termination, Controller restores the prior mode only
when all of these remain true:
- Controller actually changed the mode and received confirmation of `video`.
- The same lease is still held.
- The latest Router state is still the Controller-owned target.
- MQTT continuity was not lost.
- No live, non-self mode command and no conflicting state change indicated an
operator override.
Controller subscribes to both Router mode set and state topics. Its own
mode-set publication is correlated by exact payload and stays recognized as
a self-echo both while the publish is in flight and, because a broker may
legally deliver a subscription echo after the PUBACK that acknowledged the
same publish, for a short `mqtt.router_echo_grace_seconds` window afterward
(default 2 seconds). Every other live mode-set publication is treated
conservatively as operator intent, even if it repeats `video`. A retained set
message is ignored as stale intent. Controller does not publish Router state;
it accepts a received live or retained state message as the Router's current
mode. Once ownership is lost, Controller does not take it back merely because
state later returns to `video`.
Restoration is requested once and confirmed within the same bounded timeout.
Failure produces a warning and no automatic retry. The Router's plain string
protocol has no publisher identity or compare-and-set token, so an exactly
simultaneous same-value command cannot be distinguished perfectly; conservative
ownership loss avoids overwriting a known manual choice.
## Downloads, retention, and stable slots
The web form reserves the first empty stable slot, or reuses the slot already
associated with the same URL when it is safe. Reservation and the `reserved`
download job commit before a worker starts. `downloads.max_workers` bounds
concurrent yt-dlp work; it does not change the ten-slot capacity.
Each successful download builds `slotN/playlist.m3u` atomically and records item
metadata in the same slot generation. Downloads run yt-dlp with an explicit Node
JavaScript runtime and ffmpeg support. On restart, database jobs left in
`reserved` or `running` become `interrupted`, and their slots become `error`
instead of pretending success. Whenever a failed or restart-interrupted job
would otherwise leave an error slot without expiry, Controller assigns
`now + storage.retention_seconds`. This fallback does not replace an existing
expiry and prevents repeated failed downloads from permanently exhausting all
ten slots.
Ready and error media can expire after `storage.retention_seconds`, checked at
`storage.expiry_interval_seconds`. Cleanup skips any slot with a live job or
playback guard. Cleanup and interrupted-deletion recovery are globally deferred
until safe detailed Player status/telemetry or an explicit audited
reconciliation makes cleanup safe. The retained flat Player state never unlocks
this fence.
Manual and expiry deletion use the same asynchronous sequence:
1. A transaction rechecks media state, URL guard, live jobs, and playback
guards, increments the generation, and commits state `deleting`.
2. A bounded worker rechecks playback guards before touching the filesystem,
atomically moves the canonical directory to generation-specific trash, and
checks again before recursive removal. If a late guard appears after the
move, the worker restores the directory and leaves deletion pending.
3. Only successful filesystem completion changes the database row to `empty`.
A failure remains durably `deleting`, emits `slot_deletion_pending`, and is
retried without blocking the coordinator. Error retries are accelerated to
at most 30 seconds; otherwise pending deletions are reconsidered during each
cleanup/expiry cycle and whenever safe Player evidence or reconciliation
releases the fence.
Only one worker handles a given slot generation at a time. A late guard or a
stale generation therefore defers/fences completion instead of deleting a newer
occupant. A `deleting` state can be visible for more than one interval during a
filesystem failure or active playback review; that is deliberate, recoverable
state rather than a completed delete.
Expiry never compacts slots. An expired slot becomes empty; every other slot
keeps its number and canonical directory.
## Restart uncertainty and administrative reconciliation
On startup, a last-known managed active slot or pending play is converted to a
`restart_uncertain` guard. An active path outside managed slots receives its
own persisted reconciliation ID. These records remain deletion-protected until
fresh Player evidence confirms active playback or a terminal state clears it.
Separately, every Controller start and every MQTT connect/disconnect closes the
global cleanup fence until safe detailed Player evidence arrives. These are two
different reconciliation scopes: durable playback ownership reviews protect a
known slot/path, while a connection-scoped cleanup token handles the narrower
case where no active identity or guard is known but Player evidence is absent.
### Durable playback ownership reviews
`storage.uncertain_review_after_seconds` defaults to 900 seconds. Reaching that
deadline changes `awaiting_evidence` to `operator_review`; it **never** assumes
the Player stopped and never removes protection. Set the value to `0` for no
automatic review deadline. Review events are re-announced after each broker
connection.
Find durable review tokens in `GET /healthz` under
`reconciliation_reviews`. When the optional event topic is enabled, the same
token appears as `operation_id` on `playback_reconciliation_required`.
To clear a managed-slot review after independently verifying that playback is
not active:
```json
{
"action": "reconcile_playback",
"decision": "clear",
"confirm": true,
"slot": 1,
"review_id": "copy-the-current-review-id",
"actor": "operator-name",
"reason": "Player host was replaced and old playback cannot still exist"
}
```
For uncertain external playback, omit `slot`:
```json
{
"action": "reconcile_playback",
"decision": "clear",
"confirm": true,
"review_id": "copy-the-current-review-id",
"actor": "operator-name",
"reason": "Retired Player host verified offline"
}
```
The exact current review ID, `decision:"clear"`, and boolean `confirm:true`
are required. A stale ID is rejected, as is reconciliation while a Player
operation is pending. Clearing a review does not send stop to the Player; it
states that the operator has established safety and can permit later cleanup.
An incorrect clear can therefore allow deletion of media still in use.
### Connection-scoped cleanup reconciliation
Detailed evidence unlocks the global cleanup fence only when it proves either
terminal idle or an online active identity. Examples are cleared telemetry,
terminal/reconnect-idle status without a path, matching active telemetry/status,
or a busy/control/reconnect status backed by a known slot/path. Retained flat
state, pathless `busy`, identity-less generic `error`, malformed messages, and
mere MQTT connectivity do not qualify.
When cleanup is fenced but there is no known active identity and no live slot
guard, `/healthz` exposes the current token:
```json
{
"cleanup_reconciliation": {
"review_id": "copy-the-current-connection-token",
"kind": "player_evidence",
"state": "awaiting_evidence",
"clearable": true
}
}
```
The value is `null` once detailed evidence/override has opened cleanup, and is
also withheld when a known identity or guard requires the durable playback
review workflow instead. `clearable:false` means a Router-gated play or Player
operation is still pending.
After independently verifying that no Player can be using managed media,
publish this exact administrative command:
```json
{
"action": "reconcile_playback",
"scope": "cleanup",
"decision": "clear",
"confirm": true,
"review_id": "copy-the-current-connection-token",
"actor": "operator-name",
"reason": "Player appliance was independently verified offline"
}
```
The accepted in-process result is:
```json
{
"status": "cleared",
"review_id": "copy-the-current-connection-token",
"scope": "cleanup"
}
```
This token is intentionally not durable: it rotates on Controller startup and
every MQTT connection or continuity loss. Fetch `/healthz` again immediately
before publishing; an older token is rejected. The clear is connection-scoped,
recorded as `player_evidence_admin_override` in the audit database, and announced
as the optional `playback_reconciled` event. Later MQTT continuity loss closes
the fence and creates a new token.
For a pathless `busy`, accepting this cleanup reconciliation also abandons the
held Router lease instead of guessing that playback ended and restoring an old
mode. The operator then owns any subsequent Router change. Fresh reasserted
playback followed by terminal evidence uses the normal safe restoration path.
Cleanup scope accepts no `slot` and cannot clear or bypass a known active path,
slot guard, active download, or pending Player/Router transition. Use the
durable slot/external review payloads above for known playback uncertainty. This
override opens cleanup scheduling; each slot deletion still performs its own
transactional job/guard checks and late-guard checks.
`actor` and `reason` are durable audit fields, but `actor` is a caller-supplied
label, not authentication or authorization. Enforce administrative access with
broker credentials and topic ACLs.
## SQLite persistence
Controller v2 uses one embedded SQLite database at
`<storage.data_base>/<storage.database_name>`, by default
`/controller-data/controller.sqlite3`. It introduces no external database
service.
Schema version 2 contains:
| Table | Purpose and important constraints |
|---|---|
| `schema_migrations` | Applied internal schema versions; v1 databases upgrade transactionally to v2 |
| `data_migrations` | One-time import provenance, source hash, backup path, and warnings |
| `slots` | Exactly one row for each stable slot 1-10, generation, media state, identity, timestamps, expiry, and error |
| `slot_items` | Ordered item title/path metadata keyed by slot and position |
| `download_jobs` | Reservation/worker lifecycle keyed by job ID and slot generation; one live job per slot |
| `slot_guards` | Active, pending, restart-uncertain, and legacy-uncertain playback protection; one live guard per slot |
| `player_observation` | Singleton last status/telemetry reduction, generation, active identity, pause, and reconciliation state/token |
| `audio_router_observation` | Singleton last mode, freshness, timestamp, and warning |
| `audio_leases` | Acquisition, ownership, restoration, deadline, and failure history for Router sessions |
| `audit_events` | Durable migration, safety, reconciliation, and warning history |
Runtime connections enable foreign keys, a 5-second busy timeout, WAL journaling,
and `synchronous=FULL`. Writes use `BEGIN IMMEDIATE` transactions behind a
re-entrant process lock. Opening and migrating the store also takes the
cross-process `/controller-data/.controller-store.lock` file lock. Production
must run one application worker; these database protections are not permission
to start duplicate coordinators or MQTT clients.
For a consistent manual backup, stop Controller cleanly before copying the
entire data directory. If an unclean live copy is unavoidable, preserve
`controller.sqlite3`, `controller.sqlite3-wal`, and `controller.sqlite3-shm` as
a set.
## One-time `playlists.yaml` migration
Migration runs automatically only when both conditions are true:
1. The final SQLite database does not exist.
2. `<storage.data_base>/<storage.legacy_yaml_name>` exists, normally
`/controller-data/playlists.yaml`.
The migration is non-destructive and idempotent:
1. It takes the process lock and rechecks database existence.
2. It reads at most 8 MiB and rejects a source whose inode, size, or mtime
changes during the read.
3. It computes the full SHA-256 and creates a content-addressed, fsynced backup
named `playlists.yaml.pre-v2.<full-sha256>.bak`. An existing backup is reused
only if its hash matches.
4. It validates UTF-8, unique YAML mapping keys, slot keys in 1-10, and agreement
between each key and embedded `slot` identity.
5. It creates a separate staging SQLite file, initializes all ten slots, imports
the YAML in one transaction, records migration provenance, and checks schema,
foreign keys, integrity, and the exact 1-10 slot set.
6. It fsyncs and atomically replaces the final database path only after all
validation succeeds.
The original YAML is never renamed, edited, or removed. The backup is created
before YAML parsing, so even malformed YAML can leave a verified recovery copy.
A parsing, import, or validation failure rolls back the transaction, removes the
staging database, and occurs before final database installation. The final step
is an atomic replace followed by a directory fsync; if that last fsync itself
fails, a validated final database may already be present and must be inspected
rather than overwritten or blindly reimported. Fix or restore the YAML and retry
only when the final database is absent, or return to the old deployment; do not
discard the original or hash-named backup.
Once `controller.sqlite3` exists, startup never imports YAML again and later
YAML edits are ignored. That database-presence rule plus the `data_migrations`
record makes the migration idempotent. An existing schema-v1 Controller SQLite
database is upgraded transactionally rather than reimporting YAML.
Legacy state mapping is explicit:
| Legacy state | v2 media state | Extra treatment |
|---|---|---|
| `ready`, `idle`, `finished` | `ready` | Preserves slot number, metadata, and expiry |
| `playing`, `paused`, `busy` | `ready` | Creates a `legacy_uncertain` operator-review guard |
| `queued`, `downloading` | `error` | Records an interrupted download job and warning |
| `error` | `error` | Preserves or supplies the error message |
| any unknown value | `error` | Records a warning rather than guessing |
Missing URLs and invalid counts/timestamps are recorded as errors or migration
warnings. Parseable legacy UTC timestamps are converted to Unix timestamps.
Any migrated error/interrupted slot with no usable legacy expiry receives the
same migration-time `now + storage.retention_seconds` fallback, recorded as a
migration warning, so it can later expire when cleanup is safe. An existing
legacy expiry is preserved.
Migration does not move, rename, compact, or delete media. Legacy YAML does not
contain complete per-item paths, so v2 does not invent them; existing playlist
files and retained item topics remain untouched until authoritative v2 metadata
is available.
## Configuration
`config.yaml` is loaded relative to the application, not the invoking shell's
working directory. The production image bind-mounts it read-only at
`/app/config.yaml`. Configuration is validated before service startup and has no
hot reload; restart the one-worker service after changes.
### Environment overrides
Only the following environment variables override YAML:
| Environment variable | YAML setting | Application default |
|---|---|---|
| `MQTT_HOST` | `mqtt.host` | `10.40.0.2` |
| `MQTT_PORT` | `mqtt.port` | `1883` |
| `MQTT_USERNAME` | `mqtt.username` | unset |
| `MQTT_PASSWORD` | `mqtt.password` | unset |
| `MQTT_CLIENT_ID` | `mqtt.client_id` | `yt-controller` |
| `YTPLAYER_MEDIA_BASE` | `storage.media_base` | `/mnt/Media/YTPlayer` |
| `YTPLAYER_DATA_BASE` | `storage.data_base` | `/controller-data` |
| `YTPLAYER_RETENTION_SECONDS` | `storage.retention_seconds` | `172800` |
| `YTPLAYER_COOKIE_PATH` | `downloads.cookie_path` | `/run/secrets/yt-dlp-cookies.txt` |
| `PORT` | `web.port` | `7531` |
MQTT username and password must either both be set or both be unset. Media,
data, and cookie paths must be absolute. `storage.slot_count` is validated as
exactly 10 and cannot be used to compact or resize the stable namespace.
Compose-only deployment variables are documented in `.env.example`:
| Variable | Purpose |
|---|---|
| `FLASK_SECRET_KEY` | Required long random Flask signing secret; no production fallback |
| `CONTROLLER_DATA_DIR` | Host directory mounted at `/controller-data` |
| `YTPLAYER_MEDIA_DIR` | Host media directory mounted at `/mnt/Media/YTPlayer` |
| `CONTROLLER_SECRETS_DIR` | Host secrets directory mounted read-only at `/run/secrets` |
| `CONTROLLER_WEB_PORT` | Host-side published HTTP port |
| `CONTROLLER_UID`, `CONTROLLER_GID` | Non-root image user/group matching writable host paths |
### YAML settings
The checked-in `config.yaml` is the canonical example. Important defaults are:
- MQTT: 30-second keepalive, 1-to-30-second reconnect backoff, 512-entry event
queue, and 2-second QoS 1 PUBACK timeout.
- Storage: 48-hour retention, 60-second expiry scan, and 15-minute uncertainty
review deadline.
- Downloads: two workers, 2-hour subprocess timeout, maximum height 1080, and
optional cookies.
- Router: required mode `video`, 3-second confirmation timeout, unknown-state
policy `attempt`, continue after confirmation failure, and no fallback restore
mode.
- Home Assistant: discovery enabled under `homeassistant`.
- Web: bind `0.0.0.0:7531`.
All remaining MQTT topic defaults are listed in the topic matrix. Numeric values
must be finite and within their validated ranges.
### Cookies
Place an optional Netscape-format cookie file at:
```text
<CONTROLLER_SECRETS_DIR>/yt-dlp-cookies.txt
```
Compose exposes it as `/run/secrets/yt-dlp-cookies.txt`. For each yt-dlp command,
Controller makes a mode-0600 temporary copy under `/tmp`, passes that copy to
yt-dlp, and removes it afterward. The source is never copied into the image.
With `downloads.cookie_required: false` (the default), a missing cookie file is
logged and download continues without cookies. Set it to `true` when silent
cookie omission is unacceptable. Keep the host file out of Git, restrict its
permissions, and rotate it as a credential.
## HTTP and health
The HTTP surface is intentionally small:
- `GET /` renders the playlist submission form.
- `POST /submit` accepts an HTTP(S) URL plus optional submitter and note.
- `GET /healthz` returns service and reconciliation state as JSON.
Health includes `status`, `started`, `coordinator_alive`, `mqtt_connected`,
`router_state`, `player`, durable `reconciliation_reviews`, connection-scoped
`cleanup_reconciliation`, and `degraded_reasons`. Typical degraded reasons are
`mqtt_disconnected`, `awaiting_player_evidence`, and
`router_state_unavailable`.
The endpoint returns HTTP 200 whenever the service is started and its coordinator
thread is alive, including degraded MQTT or Router conditions. It returns 503
when the service is stopped, the coordinator is dead, or health evaluation
fails. This means a broker outage does not make Docker restart an otherwise
healthy Controller. A configuration, storage, or migration failure prevents
startup and therefore cannot produce a healthy endpoint.
## Docker deployment
The production image and Compose file intentionally differ from Controller v1:
- Gunicorn runs one `gthread` worker with four HTTP threads and `preload=false`.
One process must own the singleton coordinator, MQTT client, executor, and
SQLite connection. Do not increase `workers` or enable development hot reload.
- The container runs as configurable non-root UID/GID with all Linux
capabilities dropped and `no-new-privileges` enabled.
- The root filesystem is read-only. Only the data/media bind mounts and bounded
`/tmp` tmpfs are writable.
- The persistent Controller data mount is `/controller-data`; media is mounted
at the same `/mnt/Media/YTPlayer` path used by Player Node.
- Cookies are supplied through the read-only `/run/secrets` bind mount instead
of the old `/cookies_ro.txt` path.
- The image includes ffmpeg and a Node 22 runtime required by the pinned yt-dlp
YouTube JavaScript support.
- The build currently uses `node:22.23.1-bookworm-slim` as a JavaScript-runtime
stage and `python:3.12-slim-bookworm` as the final base. Runtime Python
requirements are exact pins, including Flask 3.1.3, Gunicorn 26.0.0,
paho-mqtt 2.1.0, PyYAML 6.0.3, and yt-dlp 2026.7.4.
- The HTTP health check, 45-second stop grace, bounded JSON logs, init process,
and `unless-stopped` restart policy are explicit.
- `tabernet` remains an external Docker network and must already exist.
Create a deployment environment file and secret:
```bash
cp .env.example .env
python3 -c 'import secrets; print(secrets.token_urlsafe(48))'
```
Put the generated value in `FLASK_SECRET_KEY`, set every host path, and ensure
`CONTROLLER_UID`/`CONTROLLER_GID` can write the data and media directories.
Then validate and start:
```bash
docker network inspect tabernet
docker compose config --quiet
docker compose build --pull
docker compose up -d
docker compose ps
docker compose logs --tail=200 controller
curl -fsS http://127.0.0.1:7531/healthz
```
## Upgrade from Controller v1
Use a maintenance window. Replace the example paths below with the paths from
the existing deployment; the historical Compose deployment commonly used
`/home/media/Scripts/Video-Player` as `/controller-data`.
1. Record the exact old Git commit or image ID, Compose file, configuration,
environment, UID/GID, and MQTT settings. Keep the old image locally.
2. Stop new submissions and let downloads finish. Prefer to stop playback and
verify authoritative Player terminal/cleared telemetry before the cutover.
If Player must remain active, leave it running; v2 will start conservatively
and protect last-known activity until fresh evidence arrives.
3. Stop only the old Controller. Do not modify, upgrade, or replace Player Node
as part of this migration.
4. While Controller is stopped, take a matched, read-only rollback snapshot of
the old data directory and `/mnt/Media/YTPlayer`. Preserve permissions and
timestamps. Record a SHA-256 of `playlists.yaml`.
5. Create a dedicated v2 data directory. Copy, do not move, the legacy
`playlists.yaml` into its root before first v2 startup. The v2 directory must
not already contain `controller.sqlite3`, or automatic YAML import will not
run.
6. Create the secrets directory and, if needed, install the old cookies in it as
`yt-dlp-cookies.txt`. Do not bake cookies into the image.
7. Copy `.env.example` to `.env`; set a new long `FLASK_SECRET_KEY`, the dedicated
`CONTROLLER_DATA_DIR`, the shared `YTPLAYER_MEDIA_DIR`, secrets directory,
MQTT values, web port, and matching UID/GID. Review `config.yaml`, especially
Router policy and uncertainty deadline.
8. Validate and build the v2 deployment, then start it once. Watch logs for the
migration report or any startup failure.
9. Verify that the original YAML still exists, its full-hash `.pre-v2.*.bak`
exists, and `controller.sqlite3` exists. Confirm the source and backup hashes
match.
10. Check `/healthz`. Resolve or deliberately reconcile any review token; do not
equate a 200/degraded response with current Player or Router evidence.
11. Verify retained slot `title/state/count` values and all thirty HA discovery
IDs. Perform a controlled play/stop through `youtube/controller/cmd`, confirm
Player evidence drives state, and verify Router restoration/manual override
behavior.
A concrete copy/start sequence, using the common legacy path, is:
```bash
sudo install -d -o 1000 -g 1000 -m 0750 /srv/video-player-controller/data
sudo install -m 0640 -o 1000 -g 1000 \
/home/media/Scripts/Video-Player/playlists.yaml \
/srv/video-player-controller/data/playlists.yaml
sudo install -d -o root -g 1000 -m 0750 /srv/video-player-controller/secrets
cp .env.example .env
# Edit .env and config.yaml before continuing.
docker compose config --quiet
docker compose build --pull
docker compose up -d
docker compose logs --tail=200 controller
curl -fsS http://127.0.0.1:7531/healthz
sha256sum \
/srv/video-player-controller/data/playlists.yaml \
/srv/video-player-controller/data/playlists.yaml.pre-v2.*.bak
ls -l /srv/video-player-controller/data/controller.sqlite3
```
If first startup reports migration failure, stop the restart loop, retain the
original YAML and generated hash backup, inspect/fix a copy of the source, and
retry only while no final SQLite database exists. Returning to the recorded v1
image with the untouched v1 data/media snapshot remains safe because migration
does not modify them.
## Rollback
Code rollback and state rollback are different operations. Controller v2 does
not write changes back to `playlists.yaml`; v1 does not understand
`controller.sqlite3`. After any v2 download, delete, expiry, or slot reuse, the
pre-migration YAML and current media tree may no longer agree.
1. Stop Controller v2 cleanly. Leave Player Node unchanged.
2. Archive the complete v2 data directory, logs, `.env`, `config.yaml`, image ID,
and current media tree. Do not delete the SQLite database, original YAML, or
hash backup.
3. Restore the matched pre-upgrade v1 `playlists.yaml` **and media snapshot as a
pair** into the paths used by the old deployment. Do not point v1 at the live
v2 SQLite file or mix old YAML with media already mutated by v2.
4. Restore the recorded v1 image/commit, Compose file, cookie mount, environment,
and path layout.
5. Start only v1 and verify MQTT connectivity, retained stable-topic consumers,
submissions, and a controlled Player operation.
6. Keep the archived v2 database/media pair. A later roll-forward must restore
those together before starting v2 again.
If v2 has made no media or inventory mutation, the untouched original YAML and
pre-upgrade media may still be a valid rollback pair. Verify that fact rather
than assuming it. Controller v1 can compact and rename slots during prune or
delete, so starting it against mismatched/current v2 media can immediately undo
the stable-slot guarantee.
For a migration-only failure before final SQLite installation, simply keep v2
stopped and restart the recorded v1 deployment against the untouched original
data and media; no v2 database state needs conversion.
## Development and tests
Application imports are side-effect free. Constructing the Flask app does not
open SQLite, start MQTT, create directories, or launch threads. `wsgi.py` owns
the production lifecycle; `python app.py` explicitly owns the development
lifecycle.
Create an isolated environment and run the complete regression suite:
```bash
python3 -m venv .venv
.venv/bin/python -m pip install --upgrade pip
.venv/bin/python -m pip install --requirement requirements.txt
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -m unittest discover -s tests -v
```
The suite covers configuration and import side effects, atomic reservation and
stale generations, asynchronous deletion/retry/late-guard safety, migration
recovery/idempotence, MQTT startup/reconnect/PUBACK/manual acknowledgement,
Player generation, pathless busy/error handling and pause-timeout reduction,
durable playback plus connection-scoped cleanup reconciliation, Audio Router
timeouts and ownership loss, service integration, HA compatibility, HTTP
health, and Docker/runtime assets. It uses fakes and temporary directories; a
real-broker/container smoke test is still required for a deployment.
Useful static and container checks are:
```bash
docker compose config --quiet
docker compose build
docker compose up -d
curl -fsS http://127.0.0.1:7531/healthz
docker compose logs --tail=200 controller
```
The authoritative Player Node has its own isolated regression suite. Validate
the sibling without installing dependencies globally or modifying its source:
```bash
cd ../Video-Player-Node
python3 -m venv .venv
.venv/bin/python -m pip install --upgrade pip
.venv/bin/python -m pip install --requirement requirements.txt
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -m unittest discover -s tests -v
```
## Security and operational limitations
- The web form is unauthenticated and listens on `0.0.0.0` by default. Restrict
the published port with host firewall/network ACLs or place it behind an
authenticated reverse proxy. Submitted URLs must use `http`/`https`, carry no
embedded userinfo, and resolve (by exact, normalized hostname match) to an
official YouTube host: `youtube.com`, `www.youtube.com`, `m.youtube.com`,
`music.youtube.com`, or `youtu.be`. URL, submitter, note, and total request
size are also server-side bounded. yt-dlp performs the outbound request.
- MQTT username/password support is available, but application-level TLS is not
configured here. Keep this connection on a trusted/protected network or add a
separately managed secure tunnel/proxy, and grant the Controller only the
required publish/subscribe topic ACLs. In particular, restrict
`youtube/controller/cmd` and Router mode commands.
- Administrative reconciliation's `actor` is an audit label, not an identity
proof. Authorization comes from who can publish to the command topic.
- Direct commands to `youtube/player/cmd` bypass Controller orchestration and
can race its Router and deletion guards. Operational publishers should use
`youtube/controller/cmd` exclusively.
- Player command PUBACK is transport acknowledgement, not an application-level
correlated acknowledgement. Controller infers completion from authoritative
status/telemetry and serializes transitions; the Player protocol still has no
request ID or generation echoed in evidence.
- Audio Router's plain set/state topics have no publisher identity or atomic
compare-and-set. Manual override detection is deliberately conservative but
cannot eliminate a perfectly simultaneous same-value race.
- Optional v2 events are non-retained and can be missed while a consumer or MQTT
is offline. SQLite audit rows and durable playback reconciliation records are
the historical sources; the cleanup-only `/healthz` token is deliberately
connection-scoped and must be refetched. Existing consumers must not depend
on the event stream.
- Production is intentionally one Gunicorn worker with no preload or hot reload.
More workers would duplicate the coordinator, MQTT client, and downloader.
- Shutdown actively terminates tracked yt-dlp process groups (TERM, then KILL
after a short bound) rather than waiting for the configured 2-hour command
timeout. The current job fails instead of resuming. A container kill before
that bookkeeping completes is recovered as `interrupted`/`error`; queued
deletion work remains durably `deleting` for the evidence-gated retry path.
- The container is read-only and non-root, but the data and media mounts must be
writable. Controller can intentionally delete canonical expired or
administratively deleted slot directories; maintain independent backups.
- Python dependencies and yt-dlp are exactly pinned, but Docker base images are
tag-pinned rather than digest-pinned and Debian apt packages are not version
pinned. Builds are therefore not bit-for-bit reproducible. Pin verified image
digests and use a controlled package mirror if that is required.
- YouTube changes regularly. Update the explicit `yt-dlp[default,pin]` version
in `requirements.txt`, rebuild the image, and run both regression and download
smoke tests on a deliberate maintenance cadence. Runtime self-update is not
enabled. The copied Node binary must match the target image architecture and
remain supported by yt-dlp.
- SQLite and its `fcntl` process lock are designed for a local persistent
filesystem. Do not place `/controller-data` on storage whose locking or atomic
rename semantics are unreliable.
Reference in New Issue View Git Blame Copy Permalink