# Video Player Controller v2 Video Player Controller downloads media into ten stable slots, publishes the existing Home Assistant and Companion state topics, and coordinates playback between the authoritative Video Player Node and the Audio Router. Controller v2 is deliberately conservative: publishing a command is not proof that playback changed. Only fresh Player Node status or telemetry can establish playing, paused, or stopped state. A command accepted by the MQTT broker remains `pending` until that evidence arrives. The Controller does not run or supervise `mpv`. The Player Node owns the MPV lifecycle and sends audio only to `pipewire/ytplayer_bus`; the Controller owns the request/confirmation/restoration workflow for Audio Router mode. ## Architecture and safety model ```text Browser / HA / Companion | | download requests and Controller commands v Flask -> bounded event queue -> single Controller coordinator | | | | | +-> SQLite state/audit | +------------> Player-slot workers | +------------> utility-export workers +---------------------> MQTT | +-----------------------------+------------------+ | | v v Video Player Node Audio Router (authoritative state) (physical audio mode) | +-> mpv -> pipewire/ytplayer_bus ``` MQTT callbacks, web requests, timers, and worker notifications converge on a single coordinator thread for orchestration and publication. Download workers perform their external work independently and commit only generation-fenced SQLite transitions. SQLite transactions are the final persistence boundary. This provides the following invariants: - Slots `1` through `10` are permanent identities. A hole stays a hole; deleting slot 3 never renames slot 4 or its directory. - A download slot is reserved in an immediate SQLite transaction before a worker is submitted. A partial unique index prevents two live jobs in one slot. - Every reuse increments the slot generation. A late worker or deletion token cannot commit against a newer occupant. - Active, pending, uncertain, or operator-review playback guards block slot replacement and deletion. - A reserved or running download blocks deletion and expiry. - Filesystem deletion is limited to the canonical `slotN` directory. The database first atomically fences the generation as `deleting`; filesystem work then runs asynchronously. A directory is atomically moved under `.controller-trash` before recursive removal, and interrupted deletion recovery is deferred until Player safety is established. - MQTT can be unavailable at startup. The web/coordinator process remains up, reports degraded health, and retries with bounded exponential backoff. The canonical managed media layout is: ```text /mnt/Media/YTPlayer/ slot1/playlist.m3u slot2/playlist.m3u ... slot10/playlist.m3u ``` The base path is configurable, but both Controller and Player Node must see the same absolute paths. ## Player-authoritative operation model Controller commands are serialized. A second transition is rejected while a play, stop, pause, or resume operation is waiting for evidence. For a play request, the Controller: 1. Validates the stable slot or explicit local path and protects a managed slot before any external command is sent. 2. Acquires the Audio Router mode using the bounded policy described below. 3. Publishes the Player command at QoS 1 with `retain=false` and waits up to `mqtt.publish_ack_timeout_seconds` for the broker PUBACK. 4. Reports the request as pending. It does **not** publish `playing` merely because the command was sent. 5. Confirms, rejects, or marks the operation uncertain from matching Player status/telemetry. The normal Player-evidence deadline is 10 seconds. A PUBACK proves only broker receipt, not Player execution. If the PUBACK is uncertain, the Controller replaces its MQTT transport so an unacknowledged QoS 1 command cannot be replayed later, and it preserves uncertain playback state for reconciliation. Player evidence is interpreted using the current Player Node contract: - `playing` status or telemetry with a path establishes active playback. - Telemetry's boolean `paused` field is authoritative for pause state. Player `paused` and `toggled` status only show that control was issued; Controller waits for telemetry before confirming the resulting pause value. If pause, resume, or toggle confirmation times out, both phase and pause value project as `unknown` until fresh evidence arrives. - `busy` means the new play did not replace the Player's existing process. When a slot/path is present, or an earlier identity is still known, that existing playback remains authoritative. A real pathless `busy` with no prior identity becomes unknown; Controller keeps the Router lease and global cleanup fence until identity/terminal evidence is reasserted or an administrator reconciles the uncertainty. - Terminal `idle` requires one of `stop`, `finished`, `mpv_error`, `sigterm`, `sigint`, or `shutdown`, or cleared telemetry. An ambiguous idle does not erase active ownership. - A generic Player `error` is not proof that MPV stopped. Known managed or external playback becomes reviewable uncertainty; an identity-less error keeps the global cleanup fence closed. Only error reasons `startup_error` and `worker_start_error` prove that playback never became active and reduce to idle even if cleared telemetry was lost. - On reconnect, `idle` with `message:"player online"` and a path means the Player is still active. The same announcement without a path means idle. - The retained flat `youtube/player/state` topic is never playback or cleanup evidence. Its value is historical across outages, and Player reconnect deliberately emits flat `idle` even while MPV is active. Controller keeps the subscription for Companion compatibility but does not use it to unlock downloads, expiry, deletion recovery, or playback guards. Use `youtube/controller/cmd` for operational control. Publishing directly to `youtube/player/cmd` bypasses Controller Audio Router acquisition and slot guard creation. Controller will eventually reduce resulting Player evidence, but there is an unavoidable cross-service race before that evidence arrives. ## MQTT contracts Defaults remain compatible with the existing Player Node, Home Assistant, and Companion integrations. Topic names can be configured except where another component treats one as fixed; coordinate any change across all consumers. ### Topic matrix | Direction | Default topic | Payload | QoS used by Controller | Retained | Contract status | |---|---|---|---:|---:|---| | Controller -> Player | `youtube/player/cmd` | JSON command | publish 1 | no | Preserved; required | | Player -> Controller | `youtube/player/status` | JSON status | subscribe 1 | no | Player-authoritative | | Player -> Controller | `youtube/player/state` | plain flat state | subscribe 0 | yes, by Player | Preserved Companion contract only; never cleanup evidence | | Player -> Controller | `youtube/player/telemetry` | JSON telemetry | subscribe 0 | no | Player-authoritative | | HA/Companion -> Controller | `youtube/controller/cmd` | JSON command | subscribe 1 | must be no | Preserved; retained commands are rejected | | Controller -> consumers | `youtube/ingest/progress` | JSON progress | publish 0 | no | Preserved | | Controller -> consumers | `youtube/media/playback//title` | plain string | publish 0 | yes | Preserved | | Controller -> consumers | `youtube/media/playback//state` | plain string | publish 0 | yes | Preserved | | Controller -> consumers | `youtube/media/playback//count` | decimal string | publish 0 | yes | Preserved | | Controller -> consumers | `youtube/media/playback//items/` | plain string | publish 0 | yes | Preserved; empty payload clears stale items | | Controller/operator -> Router | `audio/player/mode/set` | plain mode string | publish and subscribe 1 | Controller publishes no | Router contract | | Router -> Controller | `audio/player/mode/state` | plain mode string | subscribe 1 | Router-owned | Router confirmation | | Controller -> consumers | `youtube/controller/v2/events` | JSON event | publish 1 | no | Optional, additive v2 contract | | Controller -> Home Assistant | `homeassistant/sensor/yt_slot__/config` | discovery JSON | publish 0 | yes | Preserved IDs | QoS in the incoming rows is the Controller's requested subscription QoS; the publisher and broker determine the delivered QoS. Player commands and Router mode commands are always emitted with QoS 1 and `retain=false`. The Controller uses Paho manual acknowledgement for incoming QoS 1 messages. It acknowledges after successful coordinator handling and deduplicates repeated packets by MQTT connection generation, topic, and packet ID. Retained Controller commands are rejected so a reconnect cannot replay an old control operation. Detailed Player status and telemetry are valid evidence only when `retain=false`. If either arrives retained, Controller treats it as historical, does not reduce it, does not unlock cleanup, and does not clear guards. It logs and audits `retained_player_evidence_rejected`, publishing that warning when the optional v2 event topic is enabled; a retained QoS 1 message is deliberately acknowledged after rejection so the same invalid evidence is not redelivered indefinitely. ### Controller command payloads Publish all examples at QoS 1 without retain: ```bash mosquitto_pub -h 10.40.0.2 -t youtube/controller/cmd -q 1 \ -m '{"action":"play","slot":1,"label":"Opening"}' ``` Supported JSON objects are: ```json {"action":"play","slot":1,"label":"Opening"} ``` ```json {"action":"play","path":"/mnt/Media/YTPlayer/special/welcome.mp4","label":"Welcome"} ``` ```json {"action":"play","playlist":"/mnt/Media/YTPlayer/special/playlist.m3u","title":"Special"} ``` ```json {"action":"stop"} ``` ```json {"action":"pause"} ``` ```json {"action":"resume"} ``` ```json {"action":"toggle"} ``` ```json {"action":"delete_slot","slot":4,"url":"optional stale-command guard"} ``` `play` requires a ready managed slot or an existing local `path`/`playlist`. An explicit path inside a managed slot derives that stable slot identity; an explicit conflicting slot is rejected. `toggle` is translated to `pause` or `resume` from the latest telemetry and is rejected when pause state is unknown. When the global cleanup gate lacks fresh Player evidence, manual deletion also requires the exact boolean override: ```json {"action":"delete_slot","slot":4,"confirm_player_unknown":true} ``` That override only bypasses the global evidence gate. It does not bypass an active download or a per-slot active/uncertain playback guard. It is separate from administrative reconciliation, authorizes only that manual delete, and does not open automatic expiry/recovery or consume a cleanup token. An accepted manual delete is atomically fenced in SQLite and returns `{"status":"deleting","slot":4}`, not `deleted`. Filesystem removal happens on a worker afterward. MQTT commands have no synchronous response topic, so an MQTT consumer observes the retained slot state (and optional v2 warnings) rather than depending on that in-process return value. ### Preserved slot and Home Assistant state For every stable slot, Controller publishes retained `title`, `state`, and `count` values plus retained item titles. Projected state values include `empty`, `queued`, `downloading`, `ready`, `error`, `deleting`, `playing`, `paused`, `unknown`, and `operator_review`. A pending play remains `ready` until Player evidence says otherwise. The thirty discovery entities keep the established identifiers: - Discovery topic: `homeassistant/sensor/yt_slot__/config` - Unique ID: `yt_slot__` - Device identifier: `ytplayer-controller` - Device name/manufacturer/model: `YouTube Controller` / `River of Glory` / `YT Controller` Changing `homeassistant.discovery_prefix` changes only the discovery prefix. Set `homeassistant.enabled: false` to suppress discovery; legacy slot state publication continues. ### Ingest progress `youtube/ingest/progress` is a non-retained JSON stream. Payloads always contain `slot` and `state`, with applicable fields such as `title`, `count`, `message`, `url`, `submitter`, and `note`. A `queued` event also contains `reused_slot`. Common states are `queued`, `downloading`, `ready`, and `error`. ### Optional v2 events `topics.controller_result` defaults to `youtube/controller/v2/events`. Set it to YAML `null` to disable publication. This topic is deliberately additive: legacy MQTT, Home Assistant, and Companion consumers never need it. Events are QoS 1, non-retained JSON: ```json { "version": 2, "event_id": "6ed8f0e49a8b4d598e5285ca14ec89ef", "type": "audio_router_warning", "severity": "warning", "message": "Audio Router did not confirm 'video' within 3 seconds", "ts": 1786060800.0, "slot": 1, "operation_id": "4e32c8d628454426bff69c879e709f33", "continued": true } ``` `slot`, `operation_id`, and `continued` appear only when relevant. Each event is also written to `audit_events` and logged, so the non-retained stream is not the durable record. Event publication itself depends on MQTT availability. ## Audio Router orchestration The normal playback target is the plain mode `video`: - Command: `audio/player/mode/set` - Confirmed state: `audio/player/mode/state` Before play, Controller snapshots the latest confirmed Router state and creates one persisted lease for the playback session. Router acquisition and restoration both use the finite `router.confirmation_timeout_seconds`; neither can hang a play operation indefinitely. ### Unknown-state and failure policy The four Router settings are independent: | Setting | Default | Effect | |---|---|---| | `router.confirmation_timeout_seconds` | `3` | Maximum wait for acquisition or restoration confirmation | | `router.when_state_unknown` | `attempt` | `attempt`, `continue`, or `reject` when no current state exists | | `router.continue_on_confirmation_failure` | `true` | Continue playback after publish failure, disconnect, operator conflict, or confirmation timeout | | `router.fallback_restore_mode` | `null` | Optional restoration target when the prior mode was unknown | `when_state_unknown` has these meanings: - `attempt`: request `video`, wait for the bounded confirmation, then apply `continue_on_confirmation_failure` if it cannot be confirmed. - `continue`: request `video` and start playback immediately without claiming restoration ownership. This still records and publishes a warning. - `reject`: reject this play request without sending a Player command. The caller may retry after a current Router state becomes available. The safe service default is `attempt` plus `continue_on_confirmation_failure: true`: it tries to set and confirm `video` for three seconds, but unavailable Router state cannot hang or permanently disable playback. The warning is persisted, logged, and published on the optional v2 event topic when MQTT permits. `fallback_restore_mode` is used only when the previous mode was unknown and acquisition subsequently becomes confirmed. A play that continues immediately or after failed confirmation has no safe restoration ownership, so it does not apply the fallback later. ### Safe restoration and manual override At authoritative playback termination, Controller restores the prior mode only when all of these remain true: - Controller actually changed the mode and received confirmation of `video`. - The same lease is still held. - The latest Router state is still the Controller-owned target. - MQTT continuity was not lost. - No live, non-self mode command and no conflicting state change indicated an operator override. Controller subscribes to both Router mode set and state topics. Its own mode-set publication is correlated by exact payload and stays recognized as a self-echo both while the publish is in flight and, because a broker may legally deliver a subscription echo after the PUBACK that acknowledged the same publish, for a short `mqtt.router_echo_grace_seconds` window afterward (default 2 seconds). Every other live mode-set publication is treated conservatively as operator intent, even if it repeats `video`. A retained set message is ignored as stale intent. Controller does not publish Router state; it accepts a received live or retained state message as the Router's current mode. Once ownership is lost, Controller does not take it back merely because state later returns to `video`. Restoration is requested once and confirmed within the same bounded timeout. Failure produces a warning and no automatic retry. The Router's plain string protocol has no publisher identity or compare-and-set token, so an exactly simultaneous same-value command cannot be distinguished perfectly; conservative ownership loss avoids overwriting a known manual choice. ## Downloads, retention, and stable slots The web form reserves the first empty stable slot, or reuses the slot already associated with the same URL when it is safe. Reservation and the `reserved` download job commit before a worker starts. `downloads.max_workers` bounds concurrent yt-dlp work; it does not change the ten-slot capacity. Each successful download builds `slotN/playlist.m3u` atomically and records item metadata in the same slot generation. Downloads run yt-dlp with an explicit Node JavaScript runtime and ffmpeg support. On restart, database jobs left in `reserved` or `running` become `interrupted`, and their slots become `error` instead of pretending success. Whenever a failed or restart-interrupted job would otherwise leave an error slot without expiry, Controller assigns `now + storage.retention_seconds`. This fallback does not replace an existing expiry and prevents repeated failed downloads from permanently exhausting all ten slots. Ready and error media can expire after `storage.retention_seconds`, checked at `storage.expiry_interval_seconds`. Cleanup skips any slot with a live job or playback guard. Cleanup and interrupted-deletion recovery are globally deferred until safe detailed Player status/telemetry or an explicit audited reconciliation makes cleanup safe. The retained flat Player state never unlocks this fence. Manual and expiry deletion use the same asynchronous sequence: 1. A transaction rechecks media state, URL guard, live jobs, and playback guards, increments the generation, and commits state `deleting`. 2. A bounded worker rechecks playback guards before touching the filesystem, atomically moves the canonical directory to generation-specific trash, and checks again before recursive removal. If a late guard appears after the move, the worker restores the directory and leaves deletion pending. 3. Only successful filesystem completion changes the database row to `empty`. A failure remains durably `deleting`, emits `slot_deletion_pending`, and is retried without blocking the coordinator. Error retries are accelerated to at most 30 seconds; otherwise pending deletions are reconsidered during each cleanup/expiry cycle and whenever safe Player evidence or reconciliation releases the fence. Only one worker handles a given slot generation at a time. A late guard or a stale generation therefore defers/fences completion instead of deleting a newer occupant. A `deleting` state can be visible for more than one interval during a filesystem failure or active playback review; that is deliberate, recoverable state rather than a completed delete. Expiry never compacts slots. An expired slot becomes empty; every other slot keeps its number and canonical directory. ## Utility exports The submission page has four explicit modes. `Player Slot` is selected by default, and an older client that omits `download_mode` still takes that path. `MP4 Export`, `M4A Audio`, and `MP3 Audio` use a separate durable job queue and dedicated worker pool. They never reserve or mutate a Player slot, publish ingest/HA/MQTT state, acquire the Audio Router, create Player metadata, or enter Player playback, expiry, deletion, or reconciliation gates. Player Slot remains quality-first and does not force MP4 or H.264. Its strict yt-dlp selector is `bv*[height<=H]+ba/b[height<=H]`: separate 1080p video and audio are eligible, a lower-resolution source is not upscaled, and no unfiltered fallback may exceed `downloads.max_height`. Utility modes behave as follows: - MP4 selects the best source at or below `downloads.export_max_height`, with native H.264/AAC preferred only among otherwise equal-quality choices. It probes every result with ffprobe. An existing MP4/H.264/AAC/yuv420p result is moved without remuxing or transcoding; otherwise ffmpeg copies each compatible stream and converts only an incompatible video stream to H.264 or audio stream to AAC. The output height must equal the selected source height, so export never upscales, and the final file is verified as MP4/H.264/AAC. - M4A selects best available audio, prefers native AAC among equal-quality choices, extracts audio only, and verifies an AAC stream in an M4A/MP4 container. Native AAC is losslessly repackaged by yt-dlp; another source codec is encoded only when required. - MP3 selects best available audio, extracts audio only, encodes with yt-dlp's highest-quality VBR setting when required, and verifies an MP3-only result. Playlist exports preserve item order and use bounded, human-readable Unicode filenames with unsafe path/control characters removed and case-insensitive collision suffixes. Each job publishes atomically into its own directory: ```text /mnt/Media/YTPlayer/exports/ video//*.mp4 audio//*.m4a audio//*.mp3 ``` Work is staged under `exports/.work/`. A fsynced manifest makes the publication boundary recoverable: after restart, Controller verifies and finishes a fully prepared or already-published job, while a known partial job is marked interrupted and its private staging directory is removed. It does not resume a partial yt-dlp process. Unknown directories are left untouched. The web page lists live jobs before recent completed/failed/interrupted jobs and shows their output paths or error. It intentionally provides no file-serving, cancellation, or deletion endpoint. Utility output is deliberately outside slot retention. Controller never expires it; operators should monitor capacity and remove an entire completed `video/` or `audio/` directory according to local policy. Do not write untrusted files or symlinks into the media/export tree. ## Restart uncertainty and administrative reconciliation On startup, a last-known managed active slot or pending play is converted to a `restart_uncertain` guard. An active path outside managed slots receives its own persisted reconciliation ID. These records remain deletion-protected until fresh Player evidence confirms active playback or a terminal state clears it. Separately, every Controller start and every MQTT connect/disconnect closes the global cleanup fence until safe detailed Player evidence arrives. These are two different reconciliation scopes: durable playback ownership reviews protect a known slot/path, while a connection-scoped cleanup token handles the narrower case where no active identity or guard is known but Player evidence is absent. ### Durable playback ownership reviews `storage.uncertain_review_after_seconds` defaults to 900 seconds. Reaching that deadline changes `awaiting_evidence` to `operator_review`; it **never** assumes the Player stopped and never removes protection. Set the value to `0` for no automatic review deadline. Review events are re-announced after each broker connection. Find durable review tokens in `GET /healthz` under `reconciliation_reviews`. When the optional event topic is enabled, the same token appears as `operation_id` on `playback_reconciliation_required`. To clear a managed-slot review after independently verifying that playback is not active: ```json { "action": "reconcile_playback", "decision": "clear", "confirm": true, "slot": 1, "review_id": "copy-the-current-review-id", "actor": "operator-name", "reason": "Player host was replaced and old playback cannot still exist" } ``` For uncertain external playback, omit `slot`: ```json { "action": "reconcile_playback", "decision": "clear", "confirm": true, "review_id": "copy-the-current-review-id", "actor": "operator-name", "reason": "Retired Player host verified offline" } ``` The exact current review ID, `decision:"clear"`, and boolean `confirm:true` are required. A stale ID is rejected, as is reconciliation while a Player operation is pending. Clearing a review does not send stop to the Player; it states that the operator has established safety and can permit later cleanup. An incorrect clear can therefore allow deletion of media still in use. ### Connection-scoped cleanup reconciliation Detailed evidence unlocks the global cleanup fence only when it proves either terminal idle or an online active identity. Examples are cleared telemetry, terminal/reconnect-idle status without a path, matching active telemetry/status, or a busy/control/reconnect status backed by a known slot/path. Retained flat state, pathless `busy`, identity-less generic `error`, malformed messages, and mere MQTT connectivity do not qualify. When cleanup is fenced but there is no known active identity and no live slot guard, `/healthz` exposes the current token: ```json { "cleanup_reconciliation": { "review_id": "copy-the-current-connection-token", "kind": "player_evidence", "state": "awaiting_evidence", "clearable": true } } ``` The value is `null` once detailed evidence/override has opened cleanup, and is also withheld when a known identity or guard requires the durable playback review workflow instead. `clearable:false` means a Router-gated play or Player operation is still pending. After independently verifying that no Player can be using managed media, publish this exact administrative command: ```json { "action": "reconcile_playback", "scope": "cleanup", "decision": "clear", "confirm": true, "review_id": "copy-the-current-connection-token", "actor": "operator-name", "reason": "Player appliance was independently verified offline" } ``` The accepted in-process result is: ```json { "status": "cleared", "review_id": "copy-the-current-connection-token", "scope": "cleanup" } ``` This token is intentionally not durable: it rotates on Controller startup and every MQTT connection or continuity loss. Fetch `/healthz` again immediately before publishing; an older token is rejected. The clear is connection-scoped, recorded as `player_evidence_admin_override` in the audit database, and announced as the optional `playback_reconciled` event. Later MQTT continuity loss closes the fence and creates a new token. For a pathless `busy`, accepting this cleanup reconciliation also abandons the held Router lease instead of guessing that playback ended and restoring an old mode. The operator then owns any subsequent Router change. Fresh reasserted playback followed by terminal evidence uses the normal safe restoration path. Cleanup scope accepts no `slot` and cannot clear or bypass a known active path, slot guard, active download, or pending Player/Router transition. Use the durable slot/external review payloads above for known playback uncertainty. This override opens cleanup scheduling; each slot deletion still performs its own transactional job/guard checks and late-guard checks. `actor` and `reason` are durable audit fields, but `actor` is a caller-supplied label, not authentication or authorization. Enforce administrative access with broker credentials and topic ACLs. ## SQLite persistence Controller v2 uses one embedded SQLite database at `/`, by default `/controller-data/controller.sqlite3`. It introduces no external database service. Schema version 3 contains: | Table | Purpose and important constraints | |---|---| | `schema_migrations` | Applied internal schema versions; v1/v2 databases upgrade transactionally to v3 | | `data_migrations` | One-time import provenance, source hash, backup path, and warnings | | `slots` | Exactly one row for each stable slot 1-10, generation, media state, identity, timestamps, expiry, and error | | `slot_items` | Ordered item title/path metadata keyed by slot and position | | `download_jobs` | Reservation/worker lifecycle keyed by job ID and slot generation; one live job per slot | | `export_jobs` | Slot-free utility job mode/status, request metadata, output paths, and error; no slot foreign key | | `slot_guards` | Active, pending, restart-uncertain, and legacy-uncertain playback protection; one live guard per slot | | `player_observation` | Singleton last status/telemetry reduction, generation, active identity, pause, and reconciliation state/token | | `audio_router_observation` | Singleton last mode, freshness, timestamp, and warning | | `audio_leases` | Acquisition, ownership, restoration, deadline, and failure history for Router sessions | | `audit_events` | Durable migration, safety, reconciliation, and warning history | Runtime connections enable foreign keys, a 5-second busy timeout, WAL journaling, and `synchronous=FULL`. Writes use `BEGIN IMMEDIATE` transactions behind a re-entrant process lock. Opening and migrating the store also takes the cross-process `/controller-data/.controller-store.lock` file lock. Production must run one application worker; these database protections are not permission to start duplicate coordinators or MQTT clients. For a consistent manual backup, stop Controller cleanly before copying the entire data directory. If an unclean live copy is unavoidable, preserve `controller.sqlite3`, `controller.sqlite3-wal`, and `controller.sqlite3-shm` as a set. ## One-time `playlists.yaml` migration Migration runs automatically only when both conditions are true: 1. The final SQLite database does not exist. 2. `/` exists, normally `/controller-data/playlists.yaml`. The migration is non-destructive and idempotent: 1. It takes the process lock and rechecks database existence. 2. It reads at most 8 MiB and rejects a source whose inode, size, or mtime changes during the read. 3. It computes the full SHA-256 and creates a content-addressed, fsynced backup named `playlists.yaml.pre-v2..bak`. An existing backup is reused only if its hash matches. 4. It validates UTF-8, unique YAML mapping keys, slot keys in 1-10, and agreement between each key and embedded `slot` identity. 5. It creates a separate staging SQLite file, initializes all ten slots, imports the YAML in one transaction, records migration provenance, and checks schema, foreign keys, integrity, and the exact 1-10 slot set. 6. It fsyncs and atomically replaces the final database path only after all validation succeeds. The original YAML is never renamed, edited, or removed. The backup is created before YAML parsing, so even malformed YAML can leave a verified recovery copy. A parsing, import, or validation failure rolls back the transaction, removes the staging database, and occurs before final database installation. The final step is an atomic replace followed by a directory fsync; if that last fsync itself fails, a validated final database may already be present and must be inspected rather than overwritten or blindly reimported. Fix or restore the YAML and retry only when the final database is absent, or return to the old deployment; do not discard the original or hash-named backup. Once `controller.sqlite3` exists, startup never imports YAML again and later YAML edits are ignored. That database-presence rule plus the `data_migrations` record makes the migration idempotent. Existing schema-v1 or schema-v2 Controller SQLite databases are upgraded transactionally to schema v3 rather than reimporting YAML. The v2-to-v3 upgrade adds only the slot-free `export_jobs` table, its recent-jobs index, and migration record. Legacy state mapping is explicit: | Legacy state | v2 media state | Extra treatment | |---|---|---| | `ready`, `idle`, `finished` | `ready` | Preserves slot number, metadata, and expiry | | `playing`, `paused`, `busy` | `ready` | Creates a `legacy_uncertain` operator-review guard | | `queued`, `downloading` | `error` | Records an interrupted download job and warning | | `error` | `error` | Preserves or supplies the error message | | any unknown value | `error` | Records a warning rather than guessing | Missing URLs and invalid counts/timestamps are recorded as errors or migration warnings. Parseable legacy UTC timestamps are converted to Unix timestamps. Any migrated error/interrupted slot with no usable legacy expiry receives the same migration-time `now + storage.retention_seconds` fallback, recorded as a migration warning, so it can later expire when cleanup is safe. An existing legacy expiry is preserved. Migration does not move, rename, compact, or delete media. Legacy YAML does not contain complete per-item paths, so v2 does not invent them; existing playlist files and retained item topics remain untouched until authoritative v2 metadata is available. ## Configuration `config.yaml` is loaded relative to the application, not the invoking shell's working directory. The production image bind-mounts it read-only at `/app/config.yaml`. Configuration is validated before service startup and has no hot reload; restart the one-worker service after changes. ### Environment overrides Only the following environment variables override YAML: | Environment variable | YAML setting | Application default | |---|---|---| | `MQTT_HOST` | `mqtt.host` | `10.40.0.2` | | `MQTT_PORT` | `mqtt.port` | `1883` | | `MQTT_USERNAME` | `mqtt.username` | unset | | `MQTT_PASSWORD` | `mqtt.password` | unset | | `MQTT_CLIENT_ID` | `mqtt.client_id` | `yt-controller` | | `YTPLAYER_MEDIA_BASE` | `storage.media_base` | `/mnt/Media/YTPlayer` | | `YTPLAYER_DATA_BASE` | `storage.data_base` | `/controller-data` | | `YTPLAYER_RETENTION_SECONDS` | `storage.retention_seconds` | `172800` | | `YTPLAYER_COOKIE_PATH` | `downloads.cookie_path` | `/run/secrets/yt-dlp-cookies.txt` | | `PORT` | `web.port` | `7531` | MQTT username and password must either both be set or both be unset. Media, data, and cookie paths must be absolute. `storage.slot_count` is validated as exactly 10 and cannot be used to compact or resize the stable namespace. Compose-only deployment variables are documented in `.env.example`: | Variable | Purpose | |---|---| | `FLASK_SECRET_KEY` | Required long random Flask signing secret; no production fallback | | `CONTROLLER_DATA_DIR` | Host directory mounted at `/controller-data` | | `YTPLAYER_MEDIA_DIR` | Host media directory mounted at `/mnt/Media/YTPlayer` | | `CONTROLLER_SECRETS_DIR` | Host secrets directory mounted read-only at `/run/secrets` | | `CONTROLLER_WEB_PORT` | Host-side published HTTP port | | `CONTROLLER_UID`, `CONTROLLER_GID` | Non-root image user/group matching writable host paths | ### YAML settings The checked-in `config.yaml` is the canonical example. Important defaults are: - MQTT: 30-second keepalive, 1-to-30-second reconnect backoff, 512-entry event queue, and 2-second QoS 1 PUBACK timeout. - Storage: 48-hour retention, 60-second expiry scan, and 15-minute uncertainty review deadline. - Downloads: two Player-slot workers, one dedicated utility-export worker, 2-hour subprocess timeout, strict Player/export maximum height 1080, at most 20 queued/running exports, output below `exports`, and optional cookies. - Router: required mode `video`, 3-second confirmation timeout, unknown-state policy `attempt`, continue after confirmation failure, and no fallback restore mode. - Home Assistant: discovery enabled under `homeassistant`. - Web: bind `0.0.0.0:7531`. All remaining MQTT topic defaults are listed in the topic matrix. Numeric values must be finite and within their validated ranges. ### Cookies Place an optional Netscape-format cookie file at: ```text /yt-dlp-cookies.txt ``` Compose exposes it as `/run/secrets/yt-dlp-cookies.txt`. For each yt-dlp command, Controller makes a mode-0600 temporary copy under `/tmp`, passes that copy to yt-dlp, and removes it afterward. The source is never copied into the image. With `downloads.cookie_required: false` (the default), a missing cookie file is logged and download continues without cookies. Set it to `true` when silent cookie omission is unacceptable. Keep the host file out of Git, restrict its permissions, and rotate it as a credential. ## HTTP and health The HTTP surface is intentionally small: - `GET /` renders the submission form and recent utility-export status. - `POST /submit` accepts an HTTP(S) URL, optional submitter and note, and `download_mode` equal to `player_slot`, `mp4`, `m4a`, or `mp3`. Omitting the field remains equivalent to `player_slot`. - `GET /healthz` returns service and reconciliation state as JSON. Health includes `status`, `started`, `coordinator_alive`, `mqtt_connected`, `router_state`, `player`, durable `reconciliation_reviews`, connection-scoped `cleanup_reconciliation`, and `degraded_reasons`. Typical degraded reasons are `mqtt_disconnected`, `awaiting_player_evidence`, and `router_state_unavailable`. The endpoint returns HTTP 200 whenever the service is started and its coordinator thread is alive, including degraded MQTT or Router conditions. It returns 503 when the service is stopped, the coordinator is dead, or health evaluation fails. This means a broker outage does not make Docker restart an otherwise healthy Controller. A configuration, storage, or migration failure prevents startup and therefore cannot produce a healthy endpoint. ## Docker deployment The production image and Compose file intentionally differ from Controller v1: - Gunicorn runs one `gthread` worker with four HTTP threads and `preload=false`. One process must own the singleton coordinator, MQTT client, executor, and SQLite connection. Do not increase `workers` or enable development hot reload. - The container runs as configurable non-root UID/GID with all Linux capabilities dropped and `no-new-privileges` enabled. - The root filesystem is read-only. Only the data/media bind mounts and bounded `/tmp` tmpfs are writable. - The persistent Controller data mount is `/controller-data`; media is mounted at the same `/mnt/Media/YTPlayer` path used by Player Node. - Cookies are supplied through the read-only `/run/secrets` bind mount instead of the old `/cookies_ro.txt` path. - The image includes ffmpeg and a Node 22 runtime required by the pinned yt-dlp YouTube JavaScript support. - Utility exports use the existing writable media bind mount and ffmpeg/ffprobe installation; they add no service, port, volume, or external database. - The build currently uses `node:22.23.1-bookworm-slim` as a JavaScript-runtime stage and `python:3.12-slim-bookworm` as the final base. Runtime Python requirements are exact pins, including Flask 3.1.3, Gunicorn 26.0.0, paho-mqtt 2.1.0, PyYAML 6.0.3, and yt-dlp 2026.7.4. - The HTTP health check, 45-second stop grace, bounded JSON logs, init process, and `unless-stopped` restart policy are explicit. - `tabernet` remains an external Docker network and must already exist. Create a deployment environment file and secret: ```bash cp .env.example .env python3 -c 'import secrets; print(secrets.token_urlsafe(48))' ``` Put the generated value in `FLASK_SECRET_KEY`, set every host path, and ensure `CONTROLLER_UID`/`CONTROLLER_GID` can write the data and media directories. Then validate and start: ```bash docker network inspect tabernet docker compose config --quiet docker compose build --pull docker compose up -d docker compose ps docker compose logs --tail=200 controller curl -fsS http://127.0.0.1:7531/healthz ``` ## Upgrade from Controller v1 Use a maintenance window. Replace the example paths below with the paths from the existing deployment; the historical Compose deployment commonly used `/home/media/Scripts/Video-Player` as `/controller-data`. 1. Record the exact old Git commit or image ID, Compose file, configuration, environment, UID/GID, and MQTT settings. Keep the old image locally. 2. Stop new submissions and let downloads finish. Prefer to stop playback and verify authoritative Player terminal/cleared telemetry before the cutover. If Player must remain active, leave it running; v2 will start conservatively and protect last-known activity until fresh evidence arrives. 3. Stop only the old Controller. Do not modify, upgrade, or replace Player Node as part of this migration. 4. While Controller is stopped, take a matched, read-only rollback snapshot of the old data directory and `/mnt/Media/YTPlayer`. Preserve permissions and timestamps. Record a SHA-256 of `playlists.yaml`. 5. Create a dedicated v2 data directory. Copy, do not move, the legacy `playlists.yaml` into its root before first v2 startup. The v2 directory must not already contain `controller.sqlite3`, or automatic YAML import will not run. 6. Create the secrets directory and, if needed, install the old cookies in it as `yt-dlp-cookies.txt`. Do not bake cookies into the image. 7. Copy `.env.example` to `.env`; set a new long `FLASK_SECRET_KEY`, the dedicated `CONTROLLER_DATA_DIR`, the shared `YTPLAYER_MEDIA_DIR`, secrets directory, MQTT values, web port, and matching UID/GID. Review `config.yaml`, especially Router policy and uncertainty deadline. 8. Validate and build the v2 deployment, then start it once. Watch logs for the migration report or any startup failure. 9. Verify that the original YAML still exists, its full-hash `.pre-v2.*.bak` exists, and `controller.sqlite3` exists. Confirm the source and backup hashes match. 10. Check `/healthz`. Resolve or deliberately reconcile any review token; do not equate a 200/degraded response with current Player or Router evidence. 11. Verify retained slot `title/state/count` values and all thirty HA discovery IDs. Perform a controlled play/stop through `youtube/controller/cmd`, confirm Player evidence drives state, and verify Router restoration/manual override behavior. A concrete copy/start sequence, using the common legacy path, is: ```bash sudo install -d -o 1000 -g 1000 -m 0750 /srv/video-player-controller/data sudo install -m 0640 -o 1000 -g 1000 \ /home/media/Scripts/Video-Player/playlists.yaml \ /srv/video-player-controller/data/playlists.yaml sudo install -d -o root -g 1000 -m 0750 /srv/video-player-controller/secrets cp .env.example .env # Edit .env and config.yaml before continuing. docker compose config --quiet docker compose build --pull docker compose up -d docker compose logs --tail=200 controller curl -fsS http://127.0.0.1:7531/healthz sha256sum \ /srv/video-player-controller/data/playlists.yaml \ /srv/video-player-controller/data/playlists.yaml.pre-v2.*.bak ls -l /srv/video-player-controller/data/controller.sqlite3 ``` If first startup reports migration failure, stop the restart loop, retain the original YAML and generated hash backup, inspect/fix a copy of the source, and retry only while no final SQLite database exists. Returning to the recorded v1 image with the untouched v1 data/media snapshot remains safe because migration does not modify them. ### Upgrade from an earlier Controller v2 build Stop Controller and back up the complete data directory before installing this build. On first start, schema v1 or v2 is upgraded in one transaction to schema v3; existing slots, guards, observations, leases, jobs, and audit rows are not rewritten. Confirm `exports/.work`, `exports/video`, and `exports/audio` can be created by the configured container UID/GID, then submit one controlled export and inspect the path shown on the web page. The legacy `playlists.yaml` import does not rerun when the SQLite database already exists. ## Rollback Code rollback and state rollback are different operations. Controller v2 does not write changes back to `playlists.yaml`; v1 does not understand `controller.sqlite3`. Earlier Controller v2 builds also do not understand schema v3. After any v2 download, delete, expiry, or slot reuse, the pre-migration YAML and current media tree may no longer agree. 1. Stop Controller v2 cleanly. Leave Player Node unchanged. 2. Archive the complete v2 data directory, logs, `.env`, `config.yaml`, image ID, and current media tree. Do not delete the SQLite database, original YAML, or hash backup. 3. Restore the matched pre-upgrade v1 `playlists.yaml` **and media snapshot as a pair** into the paths used by the old deployment. Do not point v1 at the live v2 SQLite file or mix old YAML with media already mutated by v2. 4. Restore the recorded v1 image/commit, Compose file, cookie mount, environment, and path layout. 5. Start only v1 and verify MQTT connectivity, retained stable-topic consumers, submissions, and a controlled Player operation. 6. Keep the archived v2 database/media pair. A later roll-forward must restore those together before starting v2 again. If v2 has made no media or inventory mutation, the untouched original YAML and pre-upgrade media may still be a valid rollback pair. Verify that fact rather than assuming it. Controller v1 can compact and rename slots during prune or delete, so starting it against mismatched/current v2 media can immediately undo the stable-slot guarantee. For a migration-only failure before final SQLite installation, simply keep v2 stopped and restart the recorded v1 deployment against the untouched original data and media; no v2 database state needs conversion. To roll back only this utility-export extension to an earlier v2 build, stop the service and restore the complete schema-v2 data-directory backup made before upgrade; do not point the older binary at the schema-v3 database. Utility files already published below `exports` are independent of slots and may be archived or left in place, but the older build will not list or manage them. Rolling forward again requires restoring a matched newer database backup or allowing a fresh schema-v2 backup to upgrade once more. ## Development and tests Application imports are side-effect free. Constructing the Flask app does not open SQLite, start MQTT, create directories, or launch threads. `wsgi.py` owns the production lifecycle; `python app.py` explicitly owns the development lifecycle. Create an isolated environment and run the complete regression suite: ```bash python3 -m venv .venv .venv/bin/python -m pip install --upgrade pip .venv/bin/python -m pip install --requirement requirements.txt PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -m unittest discover -s tests -v ``` The suite covers configuration and import side effects, atomic reservation and stale generations, asynchronous deletion/retry/late-guard safety, migration recovery/idempotence, MQTT startup/reconnect/PUBACK/manual acknowledgement, Player generation, pathless busy/error handling and pause-timeout reduction, durable playback plus connection-scoped cleanup reconciliation, Audio Router timeouts and ownership loss, service integration, HA compatibility, HTTP health, and Docker/runtime assets. It uses fakes and temporary directories; a real-broker/container smoke test is still required for a deployment. Utility coverage additionally exercises actual pinned yt-dlp selection semantics, Player/export isolation, queue bounds, codec command paths, safe names, failure/restart publication recovery, and schema-v2-to-v3 migration. Useful static and container checks are: ```bash docker compose config --quiet docker compose build docker compose up -d curl -fsS http://127.0.0.1:7531/healthz docker compose logs --tail=200 controller ``` The authoritative Player Node has its own isolated regression suite. Validate the sibling without installing dependencies globally or modifying its source: ```bash cd ../Video-Player-Node python3 -m venv .venv .venv/bin/python -m pip install --upgrade pip .venv/bin/python -m pip install --requirement requirements.txt PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -m unittest discover -s tests -v ``` ## Security and operational limitations - The web form is unauthenticated and listens on `0.0.0.0` by default. Restrict the published port with host firewall/network ACLs or place it behind an authenticated reverse proxy. Submitted URLs must use `http`/`https`, carry no embedded userinfo, and resolve (by exact, normalized hostname match) to an official YouTube host: `youtube.com`, `www.youtube.com`, `m.youtube.com`, `music.youtube.com`, or `youtu.be`. URL, submitter, note, and total request size are also server-side bounded. yt-dlp performs the outbound request. - MQTT username/password support is available, but application-level TLS is not configured here. Keep this connection on a trusted/protected network or add a separately managed secure tunnel/proxy, and grant the Controller only the required publish/subscribe topic ACLs. In particular, restrict `youtube/controller/cmd` and Router mode commands. - Administrative reconciliation's `actor` is an audit label, not an identity proof. Authorization comes from who can publish to the command topic. - Direct commands to `youtube/player/cmd` bypass Controller orchestration and can race its Router and deletion guards. Operational publishers should use `youtube/controller/cmd` exclusively. - Player command PUBACK is transport acknowledgement, not an application-level correlated acknowledgement. Controller infers completion from authoritative status/telemetry and serializes transitions; the Player protocol still has no request ID or generation echoed in evidence. - Audio Router's plain set/state topics have no publisher identity or atomic compare-and-set. Manual override detection is deliberately conservative but cannot eliminate a perfectly simultaneous same-value race. - Optional v2 events are non-retained and can be missed while a consumer or MQTT is offline. SQLite audit rows and durable playback reconciliation records are the historical sources; the cleanup-only `/healthz` token is deliberately connection-scoped and must be refetched. Existing consumers must not depend on the event stream. - Production is intentionally one Gunicorn worker with no preload or hot reload. More workers would duplicate the coordinator, MQTT client, and downloader. - Shutdown actively terminates tracked yt-dlp process groups (TERM, then KILL after a short bound) rather than waiting for the configured 2-hour command timeout. The current job fails instead of resuming. A container kill before that bookkeeping completes is recovered as `interrupted`/`error`; queued deletion work remains durably `deleting` for the evidence-gated retry path. - Utility exports are not subject to slot retention and have no automatic disk quota or cleanup policy. Monitor the media filesystem and remove completed job directories administratively. Interrupted partial jobs are not resumed. - Export-path checks reject configured slot/trash overlap and encountered symlinks, but the writable media tree is not a hostile multi-user sandbox. Only trusted host accounts should be able to modify it while Controller runs. - The container is read-only and non-root, but the data and media mounts must be writable. Controller can intentionally delete canonical expired or administratively deleted slot directories; maintain independent backups. - Python dependencies and yt-dlp are exactly pinned, but Docker base images are tag-pinned rather than digest-pinned and Debian apt packages are not version pinned. Builds are therefore not bit-for-bit reproducible. Pin verified image digests and use a controlled package mirror if that is required. - YouTube changes regularly. Update the explicit `yt-dlp[default,pin]` version in `requirements.txt`, rebuild the image, and run both regression and download smoke tests on a deliberate maintenance cadence. Runtime self-update is not enabled. The copied Node binary must match the target image architecture and remain supported by yt-dlp. - SQLite and its `fcntl` process lock are designed for a local persistent filesystem. Do not place `/controller-data` on storage whose locking or atomic rename semantics are unreliable.