docs: document utility export workflow
This commit is contained in:
Media
@@ -23,7 +23,8 @@ Browser / HA / Companion
|
||||
Flask -> bounded event queue -> single Controller coordinator
|
||||
| | |
|
||||
| | +-> SQLite state/audit
|
||||
| +------------> bounded yt-dlp workers
|
||||
| +------------> Player-slot workers
|
||||
| +------------> utility-export workers
|
||||
+---------------------> MQTT
|
||||
|
|
||||
+-----------------------------+------------------+
|
||||
@@ -411,6 +412,61 @@ state rather than a completed delete.
|
||||
Expiry never compacts slots. An expired slot becomes empty; every other slot
|
||||
keeps its number and canonical directory.
|
||||
|
||||
## Utility exports
|
||||
|
||||
The submission page has four explicit modes. `Player Slot` is selected by
|
||||
default, and an older client that omits `download_mode` still takes that path.
|
||||
`MP4 Export`, `M4A Audio`, and `MP3 Audio` use a separate durable job queue and
|
||||
dedicated worker pool. They never reserve or mutate a Player slot, publish
|
||||
ingest/HA/MQTT state, acquire the Audio Router, create Player metadata, or enter
|
||||
Player playback, expiry, deletion, or reconciliation gates.
|
||||
|
||||
Player Slot remains quality-first and does not force MP4 or H.264. Its strict
|
||||
yt-dlp selector is `bv*[height<=H]+ba/b[height<=H]`: separate 1080p video and
|
||||
audio are eligible, a lower-resolution source is not upscaled, and no
|
||||
unfiltered fallback may exceed `downloads.max_height`.
|
||||
|
||||
Utility modes behave as follows:
|
||||
|
||||
- MP4 selects the best source at or below `downloads.export_max_height`, with
|
||||
native H.264/AAC preferred only among otherwise equal-quality choices. It
|
||||
probes every result with ffprobe. An existing MP4/H.264/AAC/yuv420p result is
|
||||
moved without remuxing or transcoding; otherwise ffmpeg copies each compatible
|
||||
stream and converts only an incompatible video stream to H.264 or audio stream
|
||||
to AAC. The output height must equal the selected source height, so export
|
||||
never upscales, and the final file is verified as MP4/H.264/AAC.
|
||||
- M4A selects best available audio, prefers native AAC among equal-quality
|
||||
choices, extracts audio only, and verifies an AAC stream in an M4A/MP4
|
||||
container. Native AAC is losslessly repackaged by yt-dlp; another source codec
|
||||
is encoded only when required.
|
||||
- MP3 selects best available audio, extracts audio only, encodes with yt-dlp's
|
||||
highest-quality VBR setting when required, and verifies an MP3-only result.
|
||||
|
||||
Playlist exports preserve item order and use bounded, human-readable Unicode
|
||||
filenames with unsafe path/control characters removed and case-insensitive
|
||||
collision suffixes. Each job publishes atomically into its own directory:
|
||||
|
||||
```text
|
||||
/mnt/Media/YTPlayer/exports/
|
||||
video/<job-id>/*.mp4
|
||||
audio/<job-id>/*.m4a
|
||||
audio/<job-id>/*.mp3
|
||||
```
|
||||
|
||||
Work is staged under `exports/.work/<job-id>`. A fsynced manifest makes the
|
||||
publication boundary recoverable: after restart, Controller verifies and
|
||||
finishes a fully prepared or already-published job, while a known partial job
|
||||
is marked interrupted and its private staging directory is removed. It does
|
||||
not resume a partial yt-dlp process. Unknown directories are left untouched.
|
||||
The web page lists live jobs before recent completed/failed/interrupted jobs and
|
||||
shows their output paths or error. It intentionally provides no file-serving,
|
||||
cancellation, or deletion endpoint.
|
||||
|
||||
Utility output is deliberately outside slot retention. Controller never
|
||||
expires it; operators should monitor capacity and remove an entire completed
|
||||
`video/<job-id>` or `audio/<job-id>` directory according to local policy. Do not
|
||||
write untrusted files or symlinks into the media/export tree.
|
||||
|
||||
## Restart uncertainty and administrative reconciliation
|
||||
|
||||
On startup, a last-known managed active slot or pending play is converted to a
|
||||
@@ -552,15 +608,16 @@ Controller v2 uses one embedded SQLite database at
|
||||
`/controller-data/controller.sqlite3`. It introduces no external database
|
||||
service.
|
||||
|
||||
Schema version 2 contains:
|
||||
Schema version 3 contains:
|
||||
|
||||
| Table | Purpose and important constraints |
|
||||
|---|---|
|
||||
| `schema_migrations` | Applied internal schema versions; v1 databases upgrade transactionally to v2 |
|
||||
| `schema_migrations` | Applied internal schema versions; v1/v2 databases upgrade transactionally to v3 |
|
||||
| `data_migrations` | One-time import provenance, source hash, backup path, and warnings |
|
||||
| `slots` | Exactly one row for each stable slot 1-10, generation, media state, identity, timestamps, expiry, and error |
|
||||
| `slot_items` | Ordered item title/path metadata keyed by slot and position |
|
||||
| `download_jobs` | Reservation/worker lifecycle keyed by job ID and slot generation; one live job per slot |
|
||||
| `export_jobs` | Slot-free utility job mode/status, request metadata, output paths, and error; no slot foreign key |
|
||||
| `slot_guards` | Active, pending, restart-uncertain, and legacy-uncertain playback protection; one live guard per slot |
|
||||
| `player_observation` | Singleton last status/telemetry reduction, generation, active identity, pause, and reconciliation state/token |
|
||||
| `audio_router_observation` | Singleton last mode, freshness, timestamp, and warning |
|
||||
@@ -615,8 +672,10 @@ discard the original or hash-named backup.
|
||||
|
||||
Once `controller.sqlite3` exists, startup never imports YAML again and later
|
||||
YAML edits are ignored. That database-presence rule plus the `data_migrations`
|
||||
record makes the migration idempotent. An existing schema-v1 Controller SQLite
|
||||
database is upgraded transactionally rather than reimporting YAML.
|
||||
record makes the migration idempotent. Existing schema-v1 or schema-v2
|
||||
Controller SQLite databases are upgraded transactionally to schema v3 rather
|
||||
than reimporting YAML. The v2-to-v3 upgrade adds only the slot-free
|
||||
`export_jobs` table, its recent-jobs index, and migration record.
|
||||
|
||||
Legacy state mapping is explicit:
|
||||
|
||||
@@ -686,8 +745,9 @@ The checked-in `config.yaml` is the canonical example. Important defaults are:
|
||||
queue, and 2-second QoS 1 PUBACK timeout.
|
||||
- Storage: 48-hour retention, 60-second expiry scan, and 15-minute uncertainty
|
||||
review deadline.
|
||||
- Downloads: two workers, 2-hour subprocess timeout, maximum height 1080, and
|
||||
optional cookies.
|
||||
- Downloads: two Player-slot workers, one dedicated utility-export worker,
|
||||
2-hour subprocess timeout, strict Player/export maximum height 1080, at most
|
||||
20 queued/running exports, output below `exports`, and optional cookies.
|
||||
- Router: required mode `video`, 3-second confirmation timeout, unknown-state
|
||||
policy `attempt`, continue after confirmation failure, and no fallback restore
|
||||
mode.
|
||||
@@ -718,8 +778,10 @@ permissions, and rotate it as a credential.
|
||||
|
||||
The HTTP surface is intentionally small:
|
||||
|
||||
- `GET /` renders the playlist submission form.
|
||||
- `POST /submit` accepts an HTTP(S) URL plus optional submitter and note.
|
||||
- `GET /` renders the submission form and recent utility-export status.
|
||||
- `POST /submit` accepts an HTTP(S) URL, optional submitter and note, and
|
||||
`download_mode` equal to `player_slot`, `mp4`, `m4a`, or `mp3`. Omitting the
|
||||
field remains equivalent to `player_slot`.
|
||||
- `GET /healthz` returns service and reconciliation state as JSON.
|
||||
|
||||
Health includes `status`, `started`, `coordinator_alive`, `mqtt_connected`,
|
||||
@@ -752,6 +814,8 @@ The production image and Compose file intentionally differ from Controller v1:
|
||||
of the old `/cookies_ro.txt` path.
|
||||
- The image includes ffmpeg and a Node 22 runtime required by the pinned yt-dlp
|
||||
YouTube JavaScript support.
|
||||
- Utility exports use the existing writable media bind mount and ffmpeg/ffprobe
|
||||
installation; they add no service, port, volume, or external database.
|
||||
- The build currently uses `node:22.23.1-bookworm-slim` as a JavaScript-runtime
|
||||
stage and `python:3.12-slim-bookworm` as the final base. Runtime Python
|
||||
requirements are exact pins, including Flask 3.1.3, Gunicorn 26.0.0,
|
||||
@@ -850,11 +914,22 @@ retry only while no final SQLite database exists. Returning to the recorded v1
|
||||
image with the untouched v1 data/media snapshot remains safe because migration
|
||||
does not modify them.
|
||||
|
||||
### Upgrade from an earlier Controller v2 build
|
||||
|
||||
Stop Controller and back up the complete data directory before installing this
|
||||
build. On first start, schema v1 or v2 is upgraded in one transaction to schema
|
||||
v3; existing slots, guards, observations, leases, jobs, and audit rows are not
|
||||
rewritten. Confirm `exports/.work`, `exports/video`, and `exports/audio` can be
|
||||
created by the configured container UID/GID, then submit one controlled export
|
||||
and inspect the path shown on the web page. The legacy `playlists.yaml` import
|
||||
does not rerun when the SQLite database already exists.
|
||||
|
||||
## Rollback
|
||||
|
||||
Code rollback and state rollback are different operations. Controller v2 does
|
||||
not write changes back to `playlists.yaml`; v1 does not understand
|
||||
`controller.sqlite3`. After any v2 download, delete, expiry, or slot reuse, the
|
||||
`controller.sqlite3`. Earlier Controller v2 builds also do not understand
|
||||
schema v3. After any v2 download, delete, expiry, or slot reuse, the
|
||||
pre-migration YAML and current media tree may no longer agree.
|
||||
|
||||
1. Stop Controller v2 cleanly. Leave Player Node unchanged.
|
||||
@@ -881,6 +956,14 @@ For a migration-only failure before final SQLite installation, simply keep v2
|
||||
stopped and restart the recorded v1 deployment against the untouched original
|
||||
data and media; no v2 database state needs conversion.
|
||||
|
||||
To roll back only this utility-export extension to an earlier v2 build, stop the
|
||||
service and restore the complete schema-v2 data-directory backup made before
|
||||
upgrade; do not point the older binary at the schema-v3 database. Utility files
|
||||
already published below `exports` are independent of slots and may be archived
|
||||
or left in place, but the older build will not list or manage them. Rolling
|
||||
forward again requires restoring a matched newer database backup or allowing a
|
||||
fresh schema-v2 backup to upgrade once more.
|
||||
|
||||
## Development and tests
|
||||
|
||||
Application imports are side-effect free. Constructing the Flask app does not
|
||||
@@ -904,7 +987,10 @@ Player generation, pathless busy/error handling and pause-timeout reduction,
|
||||
durable playback plus connection-scoped cleanup reconciliation, Audio Router
|
||||
timeouts and ownership loss, service integration, HA compatibility, HTTP
|
||||
health, and Docker/runtime assets. It uses fakes and temporary directories; a
|
||||
real-broker/container smoke test is still required for a deployment.
|
||||
real-broker/container smoke test is still required for a deployment. Utility
|
||||
coverage additionally exercises actual pinned yt-dlp selection semantics,
|
||||
Player/export isolation, queue bounds, codec command paths, safe names,
|
||||
failure/restart publication recovery, and schema-v2-to-v3 migration.
|
||||
|
||||
Useful static and container checks are:
|
||||
|
||||
@@ -965,6 +1051,12 @@ PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -m unittest discover -s tests -v
|
||||
timeout. The current job fails instead of resuming. A container kill before
|
||||
that bookkeeping completes is recovered as `interrupted`/`error`; queued
|
||||
deletion work remains durably `deleting` for the evidence-gated retry path.
|
||||
- Utility exports are not subject to slot retention and have no automatic disk
|
||||
quota or cleanup policy. Monitor the media filesystem and remove completed
|
||||
job directories administratively. Interrupted partial jobs are not resumed.
|
||||
- Export-path checks reject configured slot/trash overlap and encountered
|
||||
symlinks, but the writable media tree is not a hostile multi-user sandbox.
|
||||
Only trusted host accounts should be able to modify it while Controller runs.
|
||||
- The container is read-only and non-root, but the data and media mounts must be
|
||||
writable. Controller can intentionally delete canonical expired or
|
||||
administratively deleted slot directories; maintain independent backups.
|
||||
|
||||
Reference in New Issue
Block a user