From d339e10c476c54b98166a1d6379465444f64a959 Mon Sep 17 00:00:00 2001 From: Media Date: Wed, 12 Aug 2026 18:50:48 -0500 Subject: [PATCH] docs: document utility export workflow --- README.md | 114 ++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 103 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index ecb02ff..9f9c7b6 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,8 @@ Browser / HA / Companion Flask -> bounded event queue -> single Controller coordinator | | | | | +-> SQLite state/audit - | +------------> bounded yt-dlp workers + | +------------> Player-slot workers + | +------------> utility-export workers +---------------------> MQTT | +-----------------------------+------------------+ @@ -411,6 +412,61 @@ state rather than a completed delete. Expiry never compacts slots. An expired slot becomes empty; every other slot keeps its number and canonical directory. +## Utility exports + +The submission page has four explicit modes. `Player Slot` is selected by +default, and an older client that omits `download_mode` still takes that path. +`MP4 Export`, `M4A Audio`, and `MP3 Audio` use a separate durable job queue and +dedicated worker pool. They never reserve or mutate a Player slot, publish +ingest/HA/MQTT state, acquire the Audio Router, create Player metadata, or enter +Player playback, expiry, deletion, or reconciliation gates. + +Player Slot remains quality-first and does not force MP4 or H.264. Its strict +yt-dlp selector is `bv*[height<=H]+ba/b[height<=H]`: separate 1080p video and +audio are eligible, a lower-resolution source is not upscaled, and no +unfiltered fallback may exceed `downloads.max_height`. + +Utility modes behave as follows: + +- MP4 selects the best source at or below `downloads.export_max_height`, with + native H.264/AAC preferred only among otherwise equal-quality choices. It + probes every result with ffprobe. An existing MP4/H.264/AAC/yuv420p result is + moved without remuxing or transcoding; otherwise ffmpeg copies each compatible + stream and converts only an incompatible video stream to H.264 or audio stream + to AAC. The output height must equal the selected source height, so export + never upscales, and the final file is verified as MP4/H.264/AAC. +- M4A selects best available audio, prefers native AAC among equal-quality + choices, extracts audio only, and verifies an AAC stream in an M4A/MP4 + container. Native AAC is losslessly repackaged by yt-dlp; another source codec + is encoded only when required. +- MP3 selects best available audio, extracts audio only, encodes with yt-dlp's + highest-quality VBR setting when required, and verifies an MP3-only result. + +Playlist exports preserve item order and use bounded, human-readable Unicode +filenames with unsafe path/control characters removed and case-insensitive +collision suffixes. Each job publishes atomically into its own directory: + +```text +/mnt/Media/YTPlayer/exports/ + video//*.mp4 + audio//*.m4a + audio//*.mp3 +``` + +Work is staged under `exports/.work/`. A fsynced manifest makes the +publication boundary recoverable: after restart, Controller verifies and +finishes a fully prepared or already-published job, while a known partial job +is marked interrupted and its private staging directory is removed. It does +not resume a partial yt-dlp process. Unknown directories are left untouched. +The web page lists live jobs before recent completed/failed/interrupted jobs and +shows their output paths or error. It intentionally provides no file-serving, +cancellation, or deletion endpoint. + +Utility output is deliberately outside slot retention. Controller never +expires it; operators should monitor capacity and remove an entire completed +`video/` or `audio/` directory according to local policy. Do not +write untrusted files or symlinks into the media/export tree. + ## Restart uncertainty and administrative reconciliation On startup, a last-known managed active slot or pending play is converted to a @@ -552,15 +608,16 @@ Controller v2 uses one embedded SQLite database at `/controller-data/controller.sqlite3`. It introduces no external database service. -Schema version 2 contains: +Schema version 3 contains: | Table | Purpose and important constraints | |---|---| -| `schema_migrations` | Applied internal schema versions; v1 databases upgrade transactionally to v2 | +| `schema_migrations` | Applied internal schema versions; v1/v2 databases upgrade transactionally to v3 | | `data_migrations` | One-time import provenance, source hash, backup path, and warnings | | `slots` | Exactly one row for each stable slot 1-10, generation, media state, identity, timestamps, expiry, and error | | `slot_items` | Ordered item title/path metadata keyed by slot and position | | `download_jobs` | Reservation/worker lifecycle keyed by job ID and slot generation; one live job per slot | +| `export_jobs` | Slot-free utility job mode/status, request metadata, output paths, and error; no slot foreign key | | `slot_guards` | Active, pending, restart-uncertain, and legacy-uncertain playback protection; one live guard per slot | | `player_observation` | Singleton last status/telemetry reduction, generation, active identity, pause, and reconciliation state/token | | `audio_router_observation` | Singleton last mode, freshness, timestamp, and warning | @@ -615,8 +672,10 @@ discard the original or hash-named backup. Once `controller.sqlite3` exists, startup never imports YAML again and later YAML edits are ignored. That database-presence rule plus the `data_migrations` -record makes the migration idempotent. An existing schema-v1 Controller SQLite -database is upgraded transactionally rather than reimporting YAML. +record makes the migration idempotent. Existing schema-v1 or schema-v2 +Controller SQLite databases are upgraded transactionally to schema v3 rather +than reimporting YAML. The v2-to-v3 upgrade adds only the slot-free +`export_jobs` table, its recent-jobs index, and migration record. Legacy state mapping is explicit: @@ -686,8 +745,9 @@ The checked-in `config.yaml` is the canonical example. Important defaults are: queue, and 2-second QoS 1 PUBACK timeout. - Storage: 48-hour retention, 60-second expiry scan, and 15-minute uncertainty review deadline. -- Downloads: two workers, 2-hour subprocess timeout, maximum height 1080, and - optional cookies. +- Downloads: two Player-slot workers, one dedicated utility-export worker, + 2-hour subprocess timeout, strict Player/export maximum height 1080, at most + 20 queued/running exports, output below `exports`, and optional cookies. - Router: required mode `video`, 3-second confirmation timeout, unknown-state policy `attempt`, continue after confirmation failure, and no fallback restore mode. @@ -718,8 +778,10 @@ permissions, and rotate it as a credential. The HTTP surface is intentionally small: -- `GET /` renders the playlist submission form. -- `POST /submit` accepts an HTTP(S) URL plus optional submitter and note. +- `GET /` renders the submission form and recent utility-export status. +- `POST /submit` accepts an HTTP(S) URL, optional submitter and note, and + `download_mode` equal to `player_slot`, `mp4`, `m4a`, or `mp3`. Omitting the + field remains equivalent to `player_slot`. - `GET /healthz` returns service and reconciliation state as JSON. Health includes `status`, `started`, `coordinator_alive`, `mqtt_connected`, @@ -752,6 +814,8 @@ The production image and Compose file intentionally differ from Controller v1: of the old `/cookies_ro.txt` path. - The image includes ffmpeg and a Node 22 runtime required by the pinned yt-dlp YouTube JavaScript support. +- Utility exports use the existing writable media bind mount and ffmpeg/ffprobe + installation; they add no service, port, volume, or external database. - The build currently uses `node:22.23.1-bookworm-slim` as a JavaScript-runtime stage and `python:3.12-slim-bookworm` as the final base. Runtime Python requirements are exact pins, including Flask 3.1.3, Gunicorn 26.0.0, @@ -850,11 +914,22 @@ retry only while no final SQLite database exists. Returning to the recorded v1 image with the untouched v1 data/media snapshot remains safe because migration does not modify them. +### Upgrade from an earlier Controller v2 build + +Stop Controller and back up the complete data directory before installing this +build. On first start, schema v1 or v2 is upgraded in one transaction to schema +v3; existing slots, guards, observations, leases, jobs, and audit rows are not +rewritten. Confirm `exports/.work`, `exports/video`, and `exports/audio` can be +created by the configured container UID/GID, then submit one controlled export +and inspect the path shown on the web page. The legacy `playlists.yaml` import +does not rerun when the SQLite database already exists. + ## Rollback Code rollback and state rollback are different operations. Controller v2 does not write changes back to `playlists.yaml`; v1 does not understand -`controller.sqlite3`. After any v2 download, delete, expiry, or slot reuse, the +`controller.sqlite3`. Earlier Controller v2 builds also do not understand +schema v3. After any v2 download, delete, expiry, or slot reuse, the pre-migration YAML and current media tree may no longer agree. 1. Stop Controller v2 cleanly. Leave Player Node unchanged. @@ -881,6 +956,14 @@ For a migration-only failure before final SQLite installation, simply keep v2 stopped and restart the recorded v1 deployment against the untouched original data and media; no v2 database state needs conversion. +To roll back only this utility-export extension to an earlier v2 build, stop the +service and restore the complete schema-v2 data-directory backup made before +upgrade; do not point the older binary at the schema-v3 database. Utility files +already published below `exports` are independent of slots and may be archived +or left in place, but the older build will not list or manage them. Rolling +forward again requires restoring a matched newer database backup or allowing a +fresh schema-v2 backup to upgrade once more. + ## Development and tests Application imports are side-effect free. Constructing the Flask app does not @@ -904,7 +987,10 @@ Player generation, pathless busy/error handling and pause-timeout reduction, durable playback plus connection-scoped cleanup reconciliation, Audio Router timeouts and ownership loss, service integration, HA compatibility, HTTP health, and Docker/runtime assets. It uses fakes and temporary directories; a -real-broker/container smoke test is still required for a deployment. +real-broker/container smoke test is still required for a deployment. Utility +coverage additionally exercises actual pinned yt-dlp selection semantics, +Player/export isolation, queue bounds, codec command paths, safe names, +failure/restart publication recovery, and schema-v2-to-v3 migration. Useful static and container checks are: @@ -965,6 +1051,12 @@ PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -m unittest discover -s tests -v timeout. The current job fails instead of resuming. A container kill before that bookkeeping completes is recovered as `interrupted`/`error`; queued deletion work remains durably `deleting` for the evidence-gated retry path. +- Utility exports are not subject to slot retention and have no automatic disk + quota or cleanup policy. Monitor the media filesystem and remove completed + job directories administratively. Interrupted partial jobs are not resumed. +- Export-path checks reject configured slot/trash overlap and encountered + symlinks, but the writable media tree is not a hostile multi-user sandbox. + Only trusted host accounts should be able to modify it while Controller runs. - The container is read-only and non-root, but the data and media mounts must be writable. Controller can intentionally delete canonical expired or administratively deleted slot directories; maintain independent backups.