Zayin_Admin/Video-Output
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 1: V4L2/MJPEG camera path to fullscreen SDL3 output

Browse Source 
A small live-video-to-display appliance for the church stream machine,
replacing MPV/GStreamer for two fixed jobs. This lands job 1 end to end and
scaffolds job 2.

Pipeline: V4L2 mmap capture -> TurboJPEG decode to YUV planes -> interleave to
packed YUY2 -> SDL3 streaming texture -> borderless fullscreen on a chosen
Wayland output. The GPU does the colour conversion, so no CPU matrix multiply
and half the upload bandwidth of RGBA.

Latency is managed by dropping, never queueing: the driver queue is drained to
its newest buffer each wakeup, and a one-deep slot between the input thread and
the renderer discards any frame the renderer has not taken.

Dependency tree is two crates (sdl3-sys, libc). V4L2 and TurboJPEG are
hand-written FFI; the V4L2 struct layouts were taken from the target kernel's
headers and are pinned by compile-time assertions.

Three hardware behaviours found while validating against the real ATEM:

* /dev/video1 is a metadata node with no formats, so MJPEG on /dev/video0 is
  the only path on this hardware.
* The ATEM pads frames with a few bytes before the EOI marker. libjpeg warns,
  and TurboJPEG returns -1 for a warning as well as a failure, so the first
  cut rejected ~97% of frames. Warnings are now distinguished from failures
  via tj3GetErrorCode, with a structural SOI/EOI check to still catch
  genuinely truncated frames.
* On KDE Wayland, SDL reports monitor model names, not connector names.

Also fixes sticky libjpeg error state, where one corrupt frame would latch an
error and fail every valid frame after it.

87 tests. Verified on the real ATEM: sustained 30.0 fps, 0 undecodable frames,
0 frames dropped before display, ~33% of one core.
This commit is contained in:
Claude
2026-08-29 15:55:37 -05:00
commit 4b7ab4f10a
27 changed files with 5232 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+353
View File
@@ -0,0 +1,353 @@
# video-output
A small live-video-to-display appliance for Fedora. It receives live video and
puts it fullscreen on one chosen monitor. That is all it does.
Two jobs:
| Command | Path |
| --- | --- |
| `video-output camera` | ATEM Mini Pro over USB → V4L2 → MJPEG decode → GPU texture → fullscreen HDMI |
| `video-output ndi` | ProPresenter → NDI → GPU texture → fullscreen HDMI → ATEM chroma key |
No MPV, no GStreamer, no OBS, no browser, no MQTT, no audio, no recording, no
playback controls. One process that a launcher starts and kills.
**Status:** the camera path is complete and verified on real hardware. The NDI
backend is stubbed and reports that it is not implemented yet.
---
## Why Rust, SDL3, libjpeg-turbo
* **Rust** — the toolchain was already present, `cargo test` gives the test
suite with no extra machinery, and a long-running unattended process that
parses frames off a USB bus benefits from bounds checking. The whole
dependency tree is **two crates**: `sdl3-sys` and `libc`. V4L2, TurboJPEG and
(later) NDI are hand-written FFI, so there is no binding crate to rot.
* **SDL3** — the one portable way to get a borderless fullscreen GPU window on
a *chosen* Wayland output without writing raw `xdg-shell` code. Used through
`sdl3-sys` (raw FFI) rather than a safe wrapper, for direct control over
display selection and YUV texture formats.
* **libjpeg-turbo (TurboJPEG API)** — SIMD MJPEG decode, and its
`tj3DecompressToYUVPlanes8` hands back raw YUV planes so the CPU never does a
colour conversion.
### The pixel path
The ATEM emits **baseline 4:2:2 JPEG**. Rather than decode to RGBA (4 bytes per
pixel, CPU colour conversion), the planes are interleaved into packed **YUY2**
(2 bytes per pixel) and uploaded to an SDL YUY2 texture, letting the GPU sampler
convert. Half the upload bandwidth and no CPU matrix multiply.
```
MJPEG → TurboJPEG → Y/U/V planes → interleave to YUY2 → SDL texture → GPU
```
4:2:0 sources skip even the interleave (planes go straight in as I420).
Anything else (4:4:4, 4:4:0, 4:1:1, grayscale) falls back to packed BGRX.
### Latency
Two places drop stale data, both on purpose:
1. **Driver queue** — each wakeup drains every queued V4L2 buffer and decodes
only the newest, returning the rest unread.
2. **Frame slot** — a one-deep mailbox between the input thread and the
renderer. Publishing a second frame throws the first away.
A backlog can never form. If decoding falls behind, frames are dropped, not
queued. Latency stays bounded by one capture interval plus one present
interval.
---
## Fedora dependencies
```bash
sudo dnf install -y SDL3-devel turbojpeg-devel gcc
```
| Package | Why |
| --- | --- |
| `SDL3-devel` | headers and the `libSDL3.so` link symlink (the `SDL3` runtime package alone is not enough to build) |
| `turbojpeg-devel` | `libturbojpeg.so` for the MJPEG decoder |
| `gcc` | the linker driver cargo invokes |
Rust, if not present:
```bash
sudo dnf install -y cargo rust # or: https://rustup.rs
```
You must be in the `video` group to open capture devices:
```bash
groups | grep -q video || sudo usermod -aG video "$USER" # then log out and back in
```
### NDI (phase 2, not yet wired up)
Download the **NDI SDK for Linux** from ndi.video, then either install its
`libndi.so.6` into `/usr/local/lib` and run `sudo ldconfig`, or point
`LD_LIBRARY_PATH` at it. The SDK is not redistributable, so it is not vendored
here. The receiver will load it at runtime via `dlopen`, so the camera path
keeps working on machines that do not have it.
---
## Build
```bash
cargo build --release
sudo install -m755 target/release/video-output /usr/local/bin/
```
The release binary is about 560 KB.
---
## Discovering devices and displays
### Capture devices
```bash
video-output devices
```
```
/dev/video0
card : ATEM Mini Pro: Blackmagic Desig
driver : uvcvideo
bus : usb-0000:67:00.3-1
usable : yes, video capture + streaming
[0] MJPG (Motion-JPEG) [needs JPEG decode]
1920x1080 @ 29.97 fps
/dev/video1
card : ATEM Mini Pro: Blackmagic Desig
(not a video capture node -- nothing to stream from here)
```
This is the same information as `v4l2-ctl --list-formats-ext`, without needing
`v4l-utils` installed.
> **On the ATEM Mini Pro specifically:** `/dev/video1` is a *metadata* node, not
> a second video node. It exposes no formats at all, so there is no uncompressed
> (YUYV/NV12) alternative on this hardware — MJPEG is the only option. The code
> is structured so an uncompressed device would only need a new branch in
> `pixel_format_for_fourcc`, not a restructure.
### Displays
```bash
video-output displays
```
```
Video driver: wayland
2 display(s):
[0] Sceptre Tech Inc Sceptre F27 1920x1080+0+1080 @ 110.00 Hz (primary)
[1] HXA BMD HDMI 1920x1080+0+0 @ 29.97 Hz
```
> **Important, and probably not what you expect:** on KDE Plasma / Wayland, SDL
> reports the **monitor's model name**, not the connector name. You will see
> `HXA BMD HDMI`, not `HDMI-A-2`. Always run `video-output displays` and use a
> name from that list.
`--display` accepts, in order of precedence:
1. an exact name (case-insensitive) — `--display "HXA BMD HDMI"`
2. a unique prefix — `--display Sceptre`
3. a unique substring — `--display BMD`
4. an index — `--display 1` or `--display #1`
5. `primary`, the default
An ambiguous fragment is an **error**, never a guess: picking the wrong screen
mid-service is the failure this is designed to avoid. Names are strongly
preferred over indices, which shift when an output is hotplugged.
---
## Running it
Simplest possible form:
```bash
video-output camera --device /dev/video0 --display BMD
video-output ndi --source "ProPresenter Lower Thirds" --display HDMI
```
Full options: `video-output --help`.
### Profiles
So the launcher only needs a stable word:
```bash
sudo mkdir -p /etc/video-output
sudo cp examples/multiview.toml /etc/video-output/
video-output multiview
```
A profile is a flat `key = value` file using the same names as the long
options, plus `mode = "camera"` or `mode = "ndi"`. Searched in order:
```
./<name>.toml
$XDG_CONFIG_HOME/video-output/<name>.toml
~/.config/video-output/<name>.toml
/etc/video-output/<name>.toml
```
Flags given after a profile name override it:
```bash
video-output multiview --display Sceptre
```
Unknown keys are rejected at startup rather than ignored — a typo should fail
before the service, not during it.
### From a launcher
It is an ordinary foreground process:
* start it, and it opens fullscreen on the chosen output
* `SIGTERM` (or `SIGINT`/`SIGHUP`) makes it exit cleanly
* it never exits on its own because of a missing or broken source
```bash
video-output multiview &
LAUNCHER_PID=$!
# ...
kill -TERM "$LAUNCHER_PID"
```
Nothing needs to be restarted because the camera was unplugged or ProPresenter
stopped sending; it reconnects on its own. An optional systemd template is in
[`packaging/[email protected]`](packaging/[email protected]).
---
## Display behaviour
Borderless, fullscreen, no title bar, no controls, no OSD, cursor hidden,
screensaver disabled. Fullscreen is *fullscreen-desktop*: it adopts the
output's current resolution rather than forcing a mode change, so the HDMI link
is never renegotiated on a live output.
Aspect ratio is preserved by SDL's letterbox logical presentation, so a 16:9
source fills a 16:9 output exactly and is never stretched.
Wayland gives a client no way to *place* a window, only to request fullscreen on
a particular `wl_output`. The window is therefore created hidden, positioned on
the target display, set fullscreen, and only then shown — and afterwards the
program re-reads which display it actually landed on. If the compositor
overrode the request you get a warning naming both displays, rather than a
silently wrong monitor.
---
## Failure behaviour
| Situation | What happens |
| --- | --- |
| Device missing at startup | one clear error, then quiet retries with backoff (250 ms → 5 s); the window stays up and black |
| Device busy | says so explicitly, keeps retrying |
| Permission denied | tells you to join the `video` group |
| Device unplugged while running | `signal lost` logged once, black output, reopens automatically when it returns |
| Malformed frame | dropped, counted, stream continues |
| Fatal decoder error | the JPEG handle is rebuilt so one bad frame cannot poison the ones after it |
| Backend panics | caught, logged, backend restarted; the display never goes down with it |
| No valid frame | black |
Logging is edge-triggered — a camera unplugged for twenty minutes produces one
`signal lost` line, not one per retry — plus one throughput line per minute
(`--stats <seconds>`, `0` disables). Set `VIDEO_OUTPUT_LOG=debug` or
`--log debug` for per-frame diagnostics. Under systemd, journald severity
prefixes are emitted automatically so `journalctl -p warning` filters correctly.
---
## Testing
```bash
cargo test
```
87 tests covering CLI and profile parsing, invalid configuration, display
selection (including the ambiguity rules), the frame slot's drop-stale
guarantee, backoff and signal-transition state, the V4L2 struct ABI against the
kernel headers, and real JPEG decoding of all four subsampling paths — plus
truncation and bit-flip fuzzing to prove a corrupt frame cannot panic or wedge
the decoder.
Regenerate the JPEG fixtures with `python3 tests/data/generate.py` (needs
Pillow). They are solid green on purpose: green is what the ATEM keys on, so a
wrong YUV matrix or range shows up in a test rather than on stage.
### Manual validation on Fedora
Hardware paths that cannot be unit tested:
```bash
# 1. Devices and displays are discovered
video-output devices
video-output displays
# 2. Windowed, so it does not take over a screen; expect ~29.97 fps and
# "0 undecodable"
video-output camera --device /dev/video0 --display <name> --windowed --stats 3
# 3. Fullscreen on the intended output. Confirm the "Opened display:" line
# names the monitor you meant, with no placement warning after it
video-output camera --device /dev/video0 --display <name> --stats 5
# 4. Recovery: unplug the ATEM's USB cable while it runs.
# Expect "signal lost", black output, then "signal restored" on replug,
# with no restart and no repeated log lines
# 5. Clean shutdown
kill -TERM <pid> # exits promptly, no error
```
Verified on this machine: Fedora 44, KDE Plasma on Wayland, ATEM Mini Pro on
USB, 1920x1080 MJPEG @ 29.97 — sustained 30.0 fps decoded, 0 undecodable
frames, 0 stale buffers, 0 frames dropped before display, ~33% of one CPU core
and ~98 MB RSS.
---
## Troubleshooting
**`cannot find -lSDL3` or `cannot find -lturbojpeg` when building**
`SDL3-devel` / `turbojpeg-devel` are not installed. The runtime packages alone
do not provide the link symlinks.
**`cannot open /dev/video0: Permission denied`**
Join the `video` group and log out and back in.
**`device is busy`**
Something else is capturing — often another copy of this program, or OBS.
**`no display matches "..."`**
Run `video-output displays`. Remember the names are monitor models, not
connector names, on KDE Wayland.
**`display "HDMI" is ambiguous`**
Two outputs match that fragment. Use a longer one, or the exact name.
**Warning that the compositor placed the window on the wrong display**
KWin overrode the fullscreen output request. Check for a KWin window rule
matching app id `video-output`.
**Low frame rate, many "stale buffers skipped"**
You are running a debug build. Use `cargo build --release`; the YUY2 interleave
is several times slower without optimisation.
**Log line says "decoded despite a libjpeg warning"**
Normal for the ATEM. It pads frames with a few bytes before the JPEG
end-of-image marker, which makes libjpeg warn on an otherwise perfect image.
These frames are complete and are kept. Truncated frames are caught separately
by a structural check and dropped.