Classroom_Desktop/docs/releasing.md

Releasing Ekklesia Classroom

How a change becomes a package, how that package reaches a classroom, and who decides each step. Written so it can be followed months from now by somebody who was not there when it was set up.

The short version: Gitea is the record, updates.riverofglorymn.org is the delivery, and a person makes every decision that reaches a room.

 pull request ──► CI (checks, package, container gate)            automatic, no secrets
      │
   merge to master or release/X.Y
      │
 Release 1 - build ──► Gitea pre-release vX.Y.Z: .deb, SHA256SUMS,     a person runs it
      │                 build-info.json, gate report, notes
 Release 2 - Preview ──► the same file on canary; read back; recorded  a person runs it
      │
 physical checks on real hardware (docs/testing.md)                   a person does them
      │
 Release 3 - Stable ──► the same file on stable; read back; recorded   a person runs it

Nothing is rebuilt after step 1. The file the gate passed is the file the laptop installs and the file the desktops install, byte for byte, and its SHA-256 is written down at every step.

The tracks

Track	What it is	Where classrooms get it	Who is on it
Stable	Approved for production: passed the gate, ran on Preview, and passed the physical checks on real hardware	https://updates.riverofglorymn.org/stable/	The three classroom desktops
Preview	A finished release candidate, frozen, waiting for its physical checks	https://updates.riverofglorymn.org/canary/ — Preview is the canary channel	The prayer-room laptop
Nightly	Development builds of the next version, for lab machines	not yet — see Nightly	Nobody yet; never a classroom

"Preview" is a new name for an old channel. The URL, the manifest and the laptop's CHANNEL=canary are unchanged: renaming a production path would strand the one machine on it for no gain. The scripts still say canary.

A Gitea release is a pre-release while its version is only on Preview and becomes a full release when it is promoted to Stable.

Versions

As in tools/release-lib.sh: 0.2.6 is a feature release, 0.2.6.1 its first maintenance release, 0.2.7 the next feature release. Three or four numbers, no leading zeros, and ordering is always dpkg --compare-versions.

A version is used once. Once any file of that name has been on the update server it is never rebuilt or replaced — 0.2.6 was, and no room that had the first one could ever be offered the second. The release tag is v + the version (v0.2.6.6) and is never moved.

Branches

Branch	What it holds	Who merges
master	The integration branch: the next version's development (0.3.0 now). Every change arrives by pull request with CI green	Anyone with write access, after review
release/X.Y	A maintenance line (release/0.2): fixes only, each one a pull request, each release built from here. Created when a release has to be maintained while master moves on — which is now	Same
short-lived branches	One change each (fix/…, feat/…, docs/…), deleted after merge	—
experiment/*	Work that may never ship. Not built as releases	—

Why this and not more: the repository has always been one line of work on master, and what it lacked was only a place for 0.2.x to wait while 0.3.0 starts. One maintenance branch per line, created when it is needed, does that. There is no develop branch, because master already is one, and no branch per channel, because a channel is a manifest pointing at a file, not a line of code.

Releases are tags, not branches. v0.2.6.6 names the exact commit the package was built from. Stable and Preview are not branches at all.

Fixes flow forward. A fix made on release/0.2 is merged into master afterwards (git merge release/0.2, as a pull request), never cherry-picked backwards from master. A merge keeps both lines' histories honest and makes it impossible to forget a fix in the newer line. If master has moved too far for a clean merge, the fix is re-done on master in its own pull request that says so.

A frozen candidate. 0.2.6.6 is tagged v0.2.6.6 on release/0.2. While it waits for its physical checks, 0.3.0 work lands on master and cannot touch it. If the checks fail, the fix goes to release/0.2, becomes 0.2.6.7, and is merged forward.

Protecting the branches (Gitea settings)

For master and release/*: no force push, no deletion, and merging only from a pull request whose CI / checks and CI / package and container gate statuses are green. See Gitea administration.

Pull requests and commits: CI

.gitea/workflows/ci.yml, on every push to any branch and every pull request, on the ekklesia-ci runner, with no secrets:

1. checks — tools/check.sh --base <where the change started>:
   • every script's syntax by its #! line, every Python module, every JSON policy and wallpaper metadata file;
   • the change's hygiene: git diff --check, no .deb, dist/, repo/ or packaging/root/ committed, and VERSION a valid release version;
   • the teacher's manual, rebuilt and checked against itself;
   • every tests/test-*.sh: configuration, migration, desktop layout and wallpaper convergence, printing (no paper: the queue, the options, the picture pipeline), Wonder Ink workspace, Presentations, audio, network and Wi-Fi adoption, the updater and its schedule, release versions and the release tooling, deployment support, and the Gitea release record. The log of each is kept as the check-logs artifact, with a summary.md table.
2. package and container gate — builds the package the commit's VERSION describes, proves it says that version, and installs it on a clean Debian 13 container with tools/release-gate.sh --ci: fresh, and as an upgrade from whatever canary and stable serve. Between releases master's VERSION equals what canary already serves, so an upgrade that is not possible is recorded as SKIP, with the reason, not failed. The package contents and the gate report are kept; the package itself is not, because it is not a release and must never be mistaken for one.

A test added to tests/ runs in both jobs, and in every release build, with nothing else to edit: tools/build-release.sh runs the same tools/check.sh before it builds.

What CI does not do: the GUI lab (tools/lab/), which needs the nested Plasma session on the development VM, and anything on real hardware. Those are run by hand before a release, and their results written into the release commit and the Gitea release.

Building a release

1. On master (a feature release) or release/X.Y (maintenance), make the release commit: VERSION set to the new version and the manual rebuilt, nothing else:

   echo 0.2.6.7 > VERSION && tools/build-manual
   git commit -m "release: 0.2.6.7, for the canary machine only" VERSION docs/manual/build
   

   and merge it by pull request like anything else.

2. Run the GUI lab against a local build (tools/lab/dirty-upgrade, from the release stable rooms are on and from canary's), and note the result.

3. In Gitea: Actions → Release 1 - build → Run workflow, on that branch, with the version (and 0.2.2 or whatever older release rooms are still on, to rehearse that upgrade too).

tools/release-ci.sh build then:

• refuses unless the branch is master or release/*, the checkout is clean, and VERSION already says the version;
• refuses a version that has a Gitea release or tag, whose package is on the update server, or that is not newer than what canary and stable serve;
• builds with SOURCE_DATE_EPOCH set to the commit's time, so the manual's revision date is the commit's and the build changes nothing in the tree — and fails if it did change anything, because then the package would not be the commit;
• writes SHA256SUMS and build-info.json (version, package, sha256, full commit, branch, time, the run that built it, and the version dpkg reads from the file);
• runs the full gate, tools/release-gate.sh, on that file;
• creates the Gitea pre-release vX.Y.Z tagged at the commit, with the package, SHA256SUMS, build-info.json, the gate report, the check summary, and notes listing the changes since the previous tag.

The sha256 it prints is the release's name from then on.

Publishing to Preview

Actions → Release 2 - publish to Preview (canary) → Run workflow, with the version and its sha256 from the Gitea release. Two jobs:

• gate, on ekklesia-ci with no secret: downloads the release's .deb, proves it is that sha256, that the release's SHA256SUMS says so, and that dpkg reads the version from it; runs the full gate again, against what canary and stable serve now; attaches the report to the release under a new name.

• publish, on ekklesia-release with the update server's key: downloads the same file and that report and runs update_canary.sh VERSION PACKAGE with the report. update_canary.sh:

  • refuses a version that has ever been published, or is not newer than everything that has (asked of the server itself, over ssh);
  • accepts the gate report only if it names this file's sha256, passed, and upgraded from exactly the versions canary and stable serve now — a report made before either moved is refused, and the gate has to be run again;
  • uploads the package with rsync --ignore-existing and proves the server's copy is the same bytes, writes the manifest after the package, swaps canary/latest.deb by rename after the manifest, and reads all three back over HTTPS the way a classroom does.

  tools/release-ci.sh preview then checks that exactly one file was added to the server's packages and none changed, and records on the Gitea release that the file went to Preview, with its sha256 and the read-back.

The laptop takes it at its next update check (daily at 03:00, caught up after a night off) and reboots once nobody is using it.

Physical approval

The gate and the lab cannot stand in for a real television, a real lid, a real DVD drive, real speakers or a real reboot. docs/testing.md lists what a person checks for each release on real hardware — for 0.2.6.6, the section Before 0.2.6.6 goes to stable.

Only a person does these, and only a person records them. Nothing in CI claims a physical check passed.

Promoting to Stable

Actions → Release 3 - promote to Stable → Run workflow, with the version, its sha256, the approval (who, when, which checks, on which machine), and the confirmation PROMOTE X.Y.Z TO STABLE. On ekklesia-release:

• it refuses unless the Gitea release records that sha256 going to Preview: only what Preview ran can be promoted;
• update_stable.sh VERSION SHA256 refuses any other file, refuses a version not newer than stable, refuses unless the server already holds exactly that package, then points stable/manifest.json and stable/latest.deb at it and reads them back over HTTPS. No build, no upload, no new file;
• it proves no package on the server changed, records the promotion and the approval on the Gitea release, and marks it no longer a pre-release.

The desktops take it at their next 03:00 check.

Merging code never promotes anything. There is no path from a push to Stable that does not pass through a person running this workflow.

Without a runner: the same steps by hand

Every workflow step is a script, so a release never depends on a runner being up. From a checkout with the update server in ~/.ssh/config as classroom-control (as the development VM has always had) and a Gitea token with repository write access:

export GITEA_URL=https://git.riverofglorymn.org
export GITEA_REPO=River_of_Glory_Church/Ekklesia_Classroom
export GITEA_TOKEN=…                                   # never committed

tools/release-ci.sh build   0.2.6.7 0.2.2
tools/release-ci.sh gate    0.2.6.7 <sha256>
tools/release-ci.sh preview 0.2.6.7 <sha256> <gate-report name>
tools/release-ci.sh stable  0.2.6.7 <sha256> "physical checks passed on …, by …, on …"

And, as before and still true, the underlying tools alone:

./update_canary.sh 0.2.6.7 dist/ekklesia-classroom_0.2.6.7_amd64.deb   # gate + publish that file
./update_stable.sh 0.2.6.7 <sha256>                                     # promote that file

update_canary.sh VERSION with no package still builds one first, for compatibility; handing it the file you tested is the better habit.

Immutability: three locks

1. The update server. release_check_new_version refuses a version whose file name has ever been used or that is not newer than every version ever published; rsync --ignore-existing will not overwrite a package even if that check were wrong; every upload is read back by sha256.
2. The Gitea release. Gitea itself lets anyone with write access delete an asset and upload another of the same name, and Gitea 1.27 has no setting to stop it. tools/gitea-release is the only thing the workflows use to write releases, and it uploads an asset only under a name the release does not have; the same bytes again are left alone; different bytes fail loudly and change nothing. A tag is never moved. The record under "## Record" is only appended to. A person with write access can still delete things in the web UI — don't; nothing automated will put them back.
3. The checksum everywhere. The sha256 is in the Gitea release, its SHA256SUMS, build-info.json, the gate reports, the update server's manifest and the release commit's follow-up. Preview and Stable both require it as an input and refuse a file that does not match.

Gitea Releases and updates.riverofglorymn.org

	Gitea release vX.Y.Z	updates.riverofglorymn.org
Purpose	The permanent record: why, from what, how it was checked, what happened	Delivery to classrooms
Read by	People	Each classroom's updater, daily
Holds	The .deb, SHA256SUMS, build info, gate reports, notes, the record of Preview and Stable	packages/*.deb, and per channel manifest.json and latest.deb
Decides what a room installs	No	Yes: the channel's manifest

The two always hold the same bytes. The update server stays the only thing a classroom talks to: a Gitea outage, a Gitea upgrade or a Gitea migration never touches a room.

Rollback and recovery

A classroom never installs an older version. The updater only takes a version newer than its own (it has to: otherwise two channels could ping-pong a room), and update_stable.sh refuses to move stable backwards, because a room would ignore it anyway. So:

• A bad release on Preview only (the laptop): do not promote it. Fix forward: the fix, or a revert, becomes the next version, through the same gate and Preview. The bad file stays on the server — it is a version that was published, and deleting it would only hide what the laptop is running.
• A bad release on Stable: the same, faster. Build the next version from a revert of the bad change (on release/X.Y), gate it, Preview it, and promote it as soon as the specific failure is shown fixed on real hardware. Stable points at a new version whose content is the last good one.
• One room that must go back now, before a fix exists: by hand on that machine, sudo apt-get install --allow-downgrades ./ekklesia-classroom_<good>_amd64.deb with the file from its Gitea release, then reboot. It stays there until a newer version is published to its channel.
• A publish interrupted half way: the manifest is written after the package and latest.deb after the manifest, so a room either sees the old release or the whole new one. Re-running the same workflow is safe: the version checks refuse to publish twice, so check the Gitea release's record and release_verify_channel before re-running anything.
• A lost Gitea: the update server still has every package, and the tags are in every clone. Release - import an existing release rebuilds the record from the server's files.

Secrets, variables and credentials

None of these are in the repository, and none are given to CI.

Name	Kind	Holds	Used by
CLASSROOM_UPDATES_SSH_KEY	repository secret	The private key the release runner uses to publish to the update server	Release 2 (publish job), Release 3
CLASSROOM_UPDATES_ADDRESS	repository variable	The update server's address from the runner (10.40.0.2)	Release 2, 3
CLASSROOM_UPDATES_USER	repository variable	The account that key logs in as	Release 2, 3
CLASSROOM_UPDATES_KNOWN_HOSTS	repository variable	The server's ssh host key line(s), so the job refuses any other host	Release 2, 3
RELEASE_API_TOKEN	repository secret, optional	A token that may write releases. Only needed if the automatic Actions token ever cannot; measured on Gitea 1.27.2, it can	Release 1, 2, 3, import
automatic Actions token	provided by Gitea per job	Creates and records releases	Release 1, 2, 3, import

Variables are not secret: a pull request's jobs can read them (measured). Put nothing in a variable that would matter if it were printed.

Least privilege for the publish key. Today the development VM publishes as [email protected], and media is in the sudo and docker groups on that host — which also runs Gitea, Home Assistant and Vaultwarden. That key must not be the one put in Gitea. Before the release workflows are used:

1. On 10.40.0.2, create an account used for nothing else (say classroom-publish), in no group but one that may write /srv/ekklesia/classroom-updates — not sudo, not docker.
2. Generate a new key pair for it on a trusted machine; put the public half in its authorized_keys with restrict,from="<release runner address>"; put the private half in CLASSROOM_UPDATES_SSH_KEY; delete the local copy.
3. Set the three variables above.

The scripts need ls, test, sha256sum, cat, ln, mv and rsync there, and nothing else.

Runners

Label	Runs	Needs	Must not have
ekklesia-ci	CI on every push and pull request; Release 1; the gate job of Release 2; import	Debian 13, Docker (the container gate), git, python3, dpkg-deb, curl, rsync, nodejs (for actions/checkout), the fonts and tools the tests use — in practice, a VM set up like the development VM	Any credential for the update server or a classroom; a route to classrooms it does not need
ekklesia-release	Release 2's publish job, Release 3	ssh, rsync, curl, python3, dpkg-deb, git, nodejs; network to 10.40.0.2	Docker; any key on its disk; any other label
classroom-hw-* (future)	Hardware checks, by hand only	A real classroom computer in the lab	Anything to do with production rooms

How the separation holds, and where it doesn't — measured on a local Gitea 1.27.2 with act_runner 0.6.1:

• A pull request from a fork does not run at all until a maintainer approves it. Approved, it gets no secrets. It does get the variables and a token, and it runs on whichever runner its workflow names — including ekklesia-release, if it asks for that label.
• A branch pushed to this repository runs with the repository's secrets. Anyone with write access can publish, by writing a workflow that uses the key. Give write access accordingly.

The release scripts never use a runner's own ssh keys: under Actions the key comes only from the secret, is written to the job's own directory with its own known_hosts and an ssh configuration used with ssh -F (which ignores ~/.ssh/config), and is deleted when the job ends. A job without the secret stops. Never put a runner on a machine that holds the publish key by hand — the development VM does — in case that rule is ever broken.

So: the key lives only in Gitea's secret store, never on a runner's disk, where an approved fork job could read it. The ekklesia-release runner runs each job in a fresh container (ekklesia-release:docker://node:22-trixie, without the Docker socket), so no job can leave anything behind for the next. Never approve a fork's run that changes .gitea/workflows/ without reading it.

ekklesia-ci needs Docker for the container gate, and Docker is root on its host. Treat that host as something any approved pull request can take over: a VM with nothing on it worth taking, and no key to anything else.

Registering a runner (on the runner's host, as the account it runs as):

act_runner generate-config > config.yaml
#   edit runner.labels in config.yaml: [ "ekklesia-ci:host" ]
#   (measured: labels in the config file OVERRIDE --labels at registration;
#    a generated config lists ubuntu-latest etc. and the runner then
#    advertises those, and every job here waits for ever)
act_runner register --no-interactive -c config.yaml \
    --instance https://git.riverofglorymn.org --token <from Gitea> --name ekklesia-ci-1
act_runner daemon -c config.yaml        # as a systemd service

The registration token is under the repository's Settings → Actions → Runners. Register at repository level, not instance level, so these runners take no other repository's jobs.

Artifacts: actions/upload-artifact@v3 works on Gitea 1.27.2; v4 fails ("not supported on GHES", measured). The workflows use v3.

Nightly (designed, not yet active)

For installable 0.3.0 development builds, separate from a frozen Preview.

• Source: master, a commit whose CI passed.
• When: once a night if master changed since the last nightly, and on demand. Not every commit: a lab machine that reboots into a new build every hour tests nothing.
• Version: <next>~nightly.<YYYYMMDD>.<run>, for example 0.3.0~nightly.20261014.3. The ~ makes it sort before 0.3.0 and after every 0.2.x, and each night after the last — so a lab machine moves to the real 0.3.0 when it ships. The release version rules (tools/release-lib.sh, and the same pattern in the updater) refuse ~ today; they must accept it for the nightly channel only.
• Server: /nightly/manifest.json, /nightly/latest.deb, and packages in /nightly/packages/ — not /packages/. The rule that a release must be newer than everything ever published reads /packages/; a nightly there would make 0.2.6.7 "older" than a 0.3.0 nightly and block every maintenance release.
• Retention: the last 14 nightlies on the server; older ones deleted from /nightly/packages/ only. No Gitea release per nightly; the build is a workflow artifact, kept as long as Gitea keeps artifacts.
• Opting in: a lab machine's device.conf says CHANNEL=nightly, and its first nightly is installed by hand.
• Never a classroom: the nightly updater refuses CHANNEL=nightly unless /etc/ekklesia-classroom/lab-machine exists, and a nightly package refuses to install without it. A nightly machine never reads the stable or canary manifest, and it cannot drift onto them: its version is newer than any 0.2.x, so pointed at stable it would simply sit until stable passes it. Taking it back to 0.2.x is a reinstall with --allow-downgrades, by hand.
• Credential: its own key, allowed to write /srv/ekklesia/classroom-updates/nightly and nothing else (CLASSROOM_NIGHTLY_SSH_KEY).

What activating it needs, which is why it is not active: the updater and release-lib changes above ship inside the package, so they are 0.3.0 work; plus the server directory, the key, and a nightly.yml workflow. That is the next infrastructure task, after the backlog audit decides what 0.3.0 is.

The hardware lab (designed, not yet built)

Spare classroom computers, each a runner labelled classroom-hw-<kind> (classroom-hw-desktop, classroom-hw-laptop), registered to this repository, in a lab network with no path to production rooms, on CHANNEL=nightly or on whatever a job installs.

• Triggered only by hand (workflow_dispatch) with a version and sha256 — never by a pull request or a push.

• Installs the exact file from its Gitea release through apt, proves dpkg reports that version, then runs checks from tools/hw/ (to be written): HDMI connect/disconnect (the display count and the desktop's wallpaper before and after), HDMI audio (the sink appears, is chosen, plays), the DVD drive (seen, a disc read), the printer (queue state; a real page only with an explicit input), Wi-Fi recovery (down, up, reconnected with nobody typing), sleep/wake (rtcwake, and what is standing after), and reboot/update (a job that reboots the machine and a second job that reads what came back).

• Reports one line per check into hardware-report-<machine>-<date>.txt:

  AUTO    PASS  hdmi-hotplug      2 -> 1 -> 2 outputs; desktop 1 kept its wallpaper
  AUTO    FAIL  dvd-read          no disc in /dev/sr0
  MANUAL  -     picture-visible   not checked by a machine
  

  attached to the Gitea release, with a record line summarising it. AUTO lines come only from a script that measured something. MANUAL lines are only ever - from a machine; a person's result is written by the person, in the Stable workflow's approval, and nowhere else. A machine never ticks a physical check.

• Does not block CI or releases. Until it exists, physical checks are done by a person from docs/testing.md, exactly as now.

Gitea administration still to do

Things only a Gitea administrator can do, none of which this repository can do for itself:

• Say which host the existing "Ekklesia" runner is on, and its labels. It is not on 10.40.0.2, and the Gitea API does not show runners to anyone but an administrator. Relabel it ekklesia-ci:host (in its config file, see above) if it is fit for that role, or register a new one.
• Set up an ekklesia-release runner as above: container mode, no Docker socket, repository level.
• Create the classroom-publish account and key on 10.40.0.2, and the secret and three variables above.
• Protect master and release/*: no force push, no deletion, required status checks CI / checks and CI / package and container gate.
• Under the repository's Actions settings, keep approval required for runs from forks (the default, measured).
• Run Release - import an existing release for 0.2.6.6 (version, sha256 and commit are in the release commit and in docs/testing.md) so its Gitea release exists before its promotion.

What was measured

On a throwaway Gitea 1.27.2 with act_runner 0.6.1 (the versions production runs), on the development VM, 2026-09-29:

Question	Answer
Do all five workflows parse, and plan only on their own events?	Yes: CI on push and pull_request; the release workflows only on a manual run
Does the CI checks job pass on a real runner?	Yes: all 31 checks, and their logs upload as an artifact
Does CI's package job build, and run the gate in its CI mode?	Yes: built, the tree unchanged, fresh install and both upgrades passed, the report marked mode ci
Does Release 1 - build work end to end?	Yes, at a throwaway version 9.9.9 against what canary (0.2.6.6) and stable (0.2.6.4) really serve: built from the commit with the tree unchanged, full gate passed, pre-release v9.9.9 tagged at exactly that commit with the package, SHA256SUMS, build-info.json, gate report and check summary
Is a second build of the same version refused?	Yes: "v9.9.9 already has a Gitea release; a version is built once"
Does Preview's gate job work?	Yes: it downloaded the release's file, proved it against the sha256 given and SHA256SUMS, passed the full gate, attached a uniquely named report, recorded it, and handed its name to the publish job
If the key secret is missing, what would the publish job use?	This machine's own ssh configuration — on the development VM, the production publish key: a throwaway 9.9.9 would have gone to the real server. The job was cancelled before it ran, and since then a job under Actions without the key refuses, and one with it uses only its own configuration (ssh -F), never the runner's
Does a failing check stop a release build?	Yes: an undocumented workflow left in the tree failed test-workflows, and nothing was built
Can the automatic token create a release, attach an asset, edit the notes?	Yes
Does tools/gitea-release refuse different bytes under a published name, on real Gitea?	Yes
Do github.event.inputs.X and inputs.X both work?	Yes
Do job outputs pass from a job on one runner to a job on another?	Yes
Do vars.* and secrets.* reach a job, and is the secret masked?	Yes, and yes
upload-artifact@v3 / @v4?	v3 works; v4 fails ("not supported on GHES")
Do labels given to act_runner register survive a generated config?	No: the config's labels win
Does a fork's pull request run without approval?	No: held until a maintainer approves
Approved, does it get secrets?	No. It gets variables, a token, and any runner it names

Not measured, because it needs the real update server's key in a real runner: Preview's publish job and Stable's promote job end to end. Their scripts are the ones used by hand since 0.2.6, and tests/test-release-ci.sh runs them against a stand-in server and a stand-in Gitea.