Measured on a throwaway Gitea: with the key secret unset, Preview's publish job would have fallen back to the runner machine's own ssh configuration — on the development VM, the production publish key — and published a throwaway 9.9.9 to the real server. It was cancelled before it ran. And with the key set, the job wrote over the runner account's own ~/.ssh/config and known_hosts. Now, under Actions, a job without RELEASE_SSH_KEY stops; with it, the key, its known_hosts and an ssh configuration live in the job's own directory and every ssh and rsync goes through `ssh -F` (RELEASE_SSH_CONFIG in release-lib), which reads nothing else. The key is deleted when the job ends. By hand, nothing changes. Also records the end-to-end measurements of Release 1 and of Preview's gate job in docs/releasing.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
526 lines
29 KiB
Markdown
526 lines
29 KiB
Markdown
# Releasing Ekklesia Classroom
|
|
|
|
How a change becomes a package, how that package reaches a classroom, and
|
|
who decides each step. Written so it can be followed months from now by
|
|
somebody who was not there when it was set up.
|
|
|
|
The short version: **Gitea is the record, updates.riverofglorymn.org is
|
|
the delivery, and a person makes every decision that reaches a room.**
|
|
|
|
```
|
|
pull request ──► CI (checks, package, container gate) automatic, no secrets
|
|
│
|
|
merge to master or release/X.Y
|
|
│
|
|
Release 1 - build ──► Gitea pre-release vX.Y.Z: .deb, SHA256SUMS, a person runs it
|
|
│ build-info.json, gate report, notes
|
|
Release 2 - Preview ──► the same file on canary; read back; recorded a person runs it
|
|
│
|
|
physical checks on real hardware (docs/testing.md) a person does them
|
|
│
|
|
Release 3 - Stable ──► the same file on stable; read back; recorded a person runs it
|
|
```
|
|
|
|
Nothing is rebuilt after step 1. The file the gate passed is the file the
|
|
laptop installs and the file the desktops install, byte for byte, and its
|
|
SHA-256 is written down at every step.
|
|
|
|
## The tracks
|
|
|
|
| Track | What it is | Where classrooms get it | Who is on it |
|
|
| --- | --- | --- | --- |
|
|
| **Stable** | Approved for production: passed the gate, ran on Preview, and passed the physical checks on real hardware | `https://updates.riverofglorymn.org/stable/` | The three classroom desktops |
|
|
| **Preview** | A finished release candidate, frozen, waiting for its physical checks | `https://updates.riverofglorymn.org/canary/` — Preview **is** the canary channel | The prayer-room laptop |
|
|
| **Nightly** | Development builds of the next version, for lab machines | not yet — see [Nightly](#nightly-designed-not-yet-active) | Nobody yet; never a classroom |
|
|
|
|
"Preview" is a new name for an old channel. The URL, the manifest and the
|
|
laptop's `CHANNEL=canary` are unchanged: renaming a production path would
|
|
strand the one machine on it for no gain. The scripts still say canary.
|
|
|
|
A Gitea release is a **pre-release** while its version is only on Preview
|
|
and becomes a full release when it is promoted to Stable.
|
|
|
|
## Versions
|
|
|
|
As in `tools/release-lib.sh`: `0.2.6` is a feature release, `0.2.6.1` its
|
|
first maintenance release, `0.2.7` the next feature release. Three or four
|
|
numbers, no leading zeros, and ordering is always `dpkg --compare-versions`.
|
|
|
|
**A version is used once.** Once any file of that name has been on the
|
|
update server it is never rebuilt or replaced — 0.2.6 was, and no room that
|
|
had the first one could ever be offered the second. The release tag is
|
|
`v` + the version (`v0.2.6.6`) and is never moved.
|
|
|
|
## Branches
|
|
|
|
| Branch | What it holds | Who merges |
|
|
| --- | --- | --- |
|
|
| `master` | The integration branch: the next version's development (0.3.0 now). Every change arrives by pull request with CI green | Anyone with write access, after review |
|
|
| `release/X.Y` | A maintenance line (`release/0.2`): fixes only, each one a pull request, each release built from here. Created when a release has to be maintained while `master` moves on — which is now | Same |
|
|
| short-lived branches | One change each (`fix/…`, `feat/…`, `docs/…`), deleted after merge | — |
|
|
| `experiment/*` | Work that may never ship. Not built as releases | — |
|
|
|
|
Why this and not more: the repository has always been one line of work on
|
|
`master`, and what it lacked was only a place for 0.2.x to wait while 0.3.0
|
|
starts. One maintenance branch per line, created when it is needed, does
|
|
that. There is no `develop` branch, because `master` already is one, and no
|
|
branch per channel, because a channel is a manifest pointing at a file, not
|
|
a line of code.
|
|
|
|
**Releases are tags, not branches.** `v0.2.6.6` names the exact commit the
|
|
package was built from. Stable and Preview are not branches at all.
|
|
|
|
**Fixes flow forward.** A fix made on `release/0.2` is merged into `master`
|
|
afterwards (`git merge release/0.2`, as a pull request), never cherry-picked
|
|
backwards from `master`. A merge keeps both lines' histories honest and
|
|
makes it impossible to forget a fix in the newer line. If `master` has moved
|
|
too far for a clean merge, the fix is re-done on `master` in its own pull
|
|
request that says so.
|
|
|
|
**A frozen candidate.** 0.2.6.6 is tagged `v0.2.6.6` on `release/0.2`. While
|
|
it waits for its physical checks, 0.3.0 work lands on `master` and cannot
|
|
touch it. If the checks fail, the fix goes to `release/0.2`, becomes 0.2.6.7,
|
|
and is merged forward.
|
|
|
|
### Protecting the branches (Gitea settings)
|
|
|
|
For `master` and `release/*`: no force push, no deletion, and merging only
|
|
from a pull request whose **CI / checks** and **CI / package and container
|
|
gate** statuses are green. See [Gitea administration](#gitea-administration-still-to-do).
|
|
|
|
## Pull requests and commits: CI
|
|
|
|
`.gitea/workflows/ci.yml`, on every push to any branch and every pull
|
|
request, on the `ekklesia-ci` runner, with **no secrets**:
|
|
|
|
1. **checks** — `tools/check.sh --base <where the change started>`:
|
|
- every script's syntax by its `#!` line, every Python module, every JSON
|
|
policy and wallpaper metadata file;
|
|
- the change's hygiene: `git diff --check`, no `.deb`, `dist/`, `repo/` or
|
|
`packaging/root/` committed, and `VERSION` a valid release version;
|
|
- the teacher's manual, rebuilt and checked against itself;
|
|
- every `tests/test-*.sh`: configuration, migration, desktop layout and
|
|
wallpaper convergence, printing (no paper: the queue, the options, the
|
|
picture pipeline), Wonder Ink workspace, Presentations, audio, network
|
|
and Wi-Fi adoption, the updater and its schedule, release versions and
|
|
the release tooling, deployment support, and the Gitea release record.
|
|
The log of each is kept as the `check-logs` artifact, with a
|
|
`summary.md` table.
|
|
2. **package and container gate** — builds the package the commit's
|
|
`VERSION` describes, proves it says that version, and installs it on a
|
|
clean Debian 13 container with `tools/release-gate.sh --ci`: fresh, and as
|
|
an upgrade from whatever canary and stable serve. Between releases
|
|
`master`'s `VERSION` equals what canary already serves, so an upgrade
|
|
that is not possible is recorded as **SKIP**, with the reason, not
|
|
failed. The package contents and the gate report are kept; **the package
|
|
itself is not**, because it is not a release and must never be mistaken
|
|
for one.
|
|
|
|
A test added to `tests/` runs in both jobs, and in every release build, with
|
|
nothing else to edit: `tools/build-release.sh` runs the same
|
|
`tools/check.sh` before it builds.
|
|
|
|
What CI does not do: the GUI lab (`tools/lab/`), which needs the nested
|
|
Plasma session on the development VM, and anything on real hardware. Those
|
|
are run by hand before a release, and their results written into the
|
|
release commit and the Gitea release.
|
|
|
|
## Building a release
|
|
|
|
1. On `master` (a feature release) or `release/X.Y` (maintenance), make
|
|
the release commit: `VERSION` set to the new version and the manual
|
|
rebuilt, nothing else:
|
|
|
|
```
|
|
echo 0.2.6.7 > VERSION && tools/build-manual
|
|
git commit -m "release: 0.2.6.7, for the canary machine only" VERSION docs/manual/build
|
|
```
|
|
|
|
and merge it by pull request like anything else.
|
|
2. Run the GUI lab against a local build (`tools/lab/dirty-upgrade`, from
|
|
the release stable rooms are on and from canary's), and note the result.
|
|
3. In Gitea: **Actions → Release 1 - build → Run workflow**, on that branch,
|
|
with the version (and `0.2.2` or whatever older release rooms are still
|
|
on, to rehearse that upgrade too).
|
|
|
|
`tools/release-ci.sh build` then:
|
|
|
|
- refuses unless the branch is `master` or `release/*`, the checkout is
|
|
clean, and `VERSION` already says the version;
|
|
- refuses a version that has a Gitea release or tag, whose package is on the
|
|
update server, or that is not newer than what canary and stable serve;
|
|
- builds with `SOURCE_DATE_EPOCH` set to the commit's time, so the manual's
|
|
revision date is the commit's and the build changes nothing in the tree —
|
|
and **fails if it did change anything**, because then the package would not
|
|
be the commit;
|
|
- writes `SHA256SUMS` and `build-info.json` (version, package, sha256, full
|
|
commit, branch, time, the run that built it, and the version dpkg reads
|
|
from the file);
|
|
- runs the full gate, `tools/release-gate.sh`, on that file;
|
|
- creates the Gitea **pre-release** `vX.Y.Z` tagged at the commit, with the
|
|
package, `SHA256SUMS`, `build-info.json`, the gate report, the check
|
|
summary, and notes listing the changes since the previous tag.
|
|
|
|
The sha256 it prints is the release's name from then on.
|
|
|
|
## Publishing to Preview
|
|
|
|
**Actions → Release 2 - publish to Preview (canary) → Run workflow**, with the
|
|
version and its sha256 from the Gitea release. Two jobs:
|
|
|
|
- **gate**, on `ekklesia-ci` with no secret: downloads the release's `.deb`,
|
|
proves it is that sha256, that the release's `SHA256SUMS` says so, and that
|
|
dpkg reads the version from it; runs the full gate again, against what
|
|
canary and stable serve *now*; attaches the report to the release under a
|
|
new name.
|
|
- **publish**, on `ekklesia-release` with the update server's key:
|
|
downloads the same file and that report and runs
|
|
`update_canary.sh VERSION PACKAGE` with the report. `update_canary.sh`:
|
|
- refuses a version that has ever been published, or is not newer than
|
|
everything that has (asked of the server itself, over ssh);
|
|
- accepts the gate report only if it names this file's sha256, passed, and
|
|
upgraded from exactly the versions canary and stable serve now — a report
|
|
made before either moved is refused, and the gate has to be run again;
|
|
- uploads the package with `rsync --ignore-existing` and proves the server's
|
|
copy is the same bytes, writes the manifest **after** the package, swaps
|
|
`canary/latest.deb` by rename **after** the manifest, and reads all three
|
|
back over HTTPS the way a classroom does.
|
|
|
|
`tools/release-ci.sh preview` then checks that exactly one file was added
|
|
to the server's packages and none changed, and records on the Gitea release
|
|
that the file went to Preview, with its sha256 and the read-back.
|
|
|
|
The laptop takes it at its next update check (daily at 03:00, caught up after
|
|
a night off) and reboots once nobody is using it.
|
|
|
|
## Physical approval
|
|
|
|
The gate and the lab cannot stand in for a real television, a real lid, a
|
|
real DVD drive, real speakers or a real reboot. `docs/testing.md` lists what
|
|
a person checks for each release on real hardware — for 0.2.6.6, the section
|
|
[Before 0.2.6.6 goes to stable](testing.md#before-0266-goes-to-stable).
|
|
|
|
Only a person does these, and only a person records them. Nothing in CI
|
|
claims a physical check passed.
|
|
|
|
## Promoting to Stable
|
|
|
|
**Actions → Release 3 - promote to Stable → Run workflow**, with the version,
|
|
its sha256, the approval (who, when, which checks, on which machine), and the
|
|
confirmation `PROMOTE X.Y.Z TO STABLE`. On `ekklesia-release`:
|
|
|
|
- it refuses unless the Gitea release records **that sha256** going to
|
|
Preview: only what Preview ran can be promoted;
|
|
- `update_stable.sh VERSION SHA256` refuses any other file, refuses a version
|
|
not newer than stable, refuses unless the server already holds exactly that
|
|
package, then points `stable/manifest.json` and `stable/latest.deb` at it
|
|
and reads them back over HTTPS. **No build, no upload, no new file**;
|
|
- it proves no package on the server changed, records the promotion and the
|
|
approval on the Gitea release, and marks it no longer a pre-release.
|
|
|
|
The desktops take it at their next 03:00 check.
|
|
|
|
Merging code never promotes anything. There is no path from a push to
|
|
Stable that does not pass through a person running this workflow.
|
|
|
|
## Without a runner: the same steps by hand
|
|
|
|
Every workflow step is a script, so a release never depends on a runner
|
|
being up. From a checkout with the update server in `~/.ssh/config` as
|
|
`classroom-control` (as the development VM has always had) and a Gitea token
|
|
with repository write access:
|
|
|
|
```
|
|
export GITEA_URL=https://git.riverofglorymn.org
|
|
export GITEA_REPO=River_of_Glory_Church/Ekklesia_Classroom
|
|
export GITEA_TOKEN=… # never committed
|
|
|
|
tools/release-ci.sh build 0.2.6.7 0.2.2
|
|
tools/release-ci.sh gate 0.2.6.7 <sha256>
|
|
tools/release-ci.sh preview 0.2.6.7 <sha256> <gate-report name>
|
|
tools/release-ci.sh stable 0.2.6.7 <sha256> "physical checks passed on …, by …, on …"
|
|
```
|
|
|
|
And, as before and still true, the underlying tools alone:
|
|
|
|
```
|
|
./update_canary.sh 0.2.6.7 dist/ekklesia-classroom_0.2.6.7_amd64.deb # gate + publish that file
|
|
./update_stable.sh 0.2.6.7 <sha256> # promote that file
|
|
```
|
|
|
|
`update_canary.sh VERSION` with no package still builds one first, for
|
|
compatibility; handing it the file you tested is the better habit.
|
|
|
|
## Immutability: three locks
|
|
|
|
1. **The update server.** `release_check_new_version` refuses a version whose
|
|
file name has ever been used or that is not newer than every version ever
|
|
published; `rsync --ignore-existing` will not overwrite a package even if
|
|
that check were wrong; every upload is read back by sha256.
|
|
2. **The Gitea release.** Gitea itself lets anyone with write access delete an
|
|
asset and upload another of the same name, and Gitea 1.27 has no setting
|
|
to stop it. `tools/gitea-release` is the only thing the workflows use to
|
|
write releases, and it uploads an asset only under a name the release does
|
|
not have; the same bytes again are left alone; different bytes fail
|
|
loudly and change nothing. A tag is never moved. The record under
|
|
"## Record" is only appended to. A person with write access can still
|
|
delete things in the web UI — don't; nothing automated will put them back.
|
|
3. **The checksum everywhere.** The sha256 is in the Gitea release, its
|
|
`SHA256SUMS`, `build-info.json`, the gate reports, the update server's
|
|
manifest and the release commit's follow-up. Preview and Stable both
|
|
require it as an input and refuse a file that does not match.
|
|
|
|
## Gitea Releases and updates.riverofglorymn.org
|
|
|
|
| | Gitea release `vX.Y.Z` | updates.riverofglorymn.org |
|
|
| --- | --- | --- |
|
|
| Purpose | The permanent record: why, from what, how it was checked, what happened | Delivery to classrooms |
|
|
| Read by | People | Each classroom's updater, daily |
|
|
| Holds | The `.deb`, SHA256SUMS, build info, gate reports, notes, the record of Preview and Stable | `packages/*.deb`, and per channel `manifest.json` and `latest.deb` |
|
|
| Decides what a room installs | No | Yes: the channel's manifest |
|
|
|
|
The two always hold the same bytes. The update server stays the only thing
|
|
a classroom talks to: a Gitea outage, a Gitea upgrade or a Gitea migration
|
|
never touches a room.
|
|
|
|
## Rollback and recovery
|
|
|
|
**A classroom never installs an older version.** The updater only takes a
|
|
version newer than its own (it has to: otherwise two channels could
|
|
ping-pong a room), and `update_stable.sh` refuses to move stable backwards,
|
|
because a room would ignore it anyway. So:
|
|
|
|
- **A bad release on Preview only** (the laptop): do not promote it. Fix
|
|
forward: the fix, or a revert, becomes the next version, through the same
|
|
gate and Preview. The bad file stays on the server — it is a version that
|
|
was published, and deleting it would only hide what the laptop is running.
|
|
- **A bad release on Stable**: the same, faster. Build the next version from a
|
|
revert of the bad change (on `release/X.Y`), gate it, Preview it, and
|
|
promote it as soon as the specific failure is shown fixed on real hardware.
|
|
Stable points at a *new* version whose content is the last good one.
|
|
- **One room that must go back now**, before a fix exists: by hand on that
|
|
machine, `sudo apt-get install --allow-downgrades ./ekklesia-classroom_<good>_amd64.deb`
|
|
with the file from its Gitea release, then reboot. It stays there until a
|
|
newer version is published to its channel.
|
|
- **A publish interrupted half way**: the manifest is written after the
|
|
package and `latest.deb` after the manifest, so a room either sees the old
|
|
release or the whole new one. Re-running the same workflow is safe: the
|
|
version checks refuse to publish twice, so check the Gitea release's record
|
|
and `release_verify_channel` before re-running anything.
|
|
- **A lost Gitea**: the update server still has every package, and the tags
|
|
are in every clone. `Release - import an existing release` rebuilds the
|
|
record from the server's files.
|
|
|
|
## Secrets, variables and credentials
|
|
|
|
None of these are in the repository, and none are given to CI.
|
|
|
|
| Name | Kind | Holds | Used by |
|
|
| --- | --- | --- | --- |
|
|
| `CLASSROOM_UPDATES_SSH_KEY` | repository **secret** | The private key the release runner uses to publish to the update server | Release 2 (publish job), Release 3 |
|
|
| `CLASSROOM_UPDATES_ADDRESS` | repository variable | The update server's address from the runner (`10.40.0.2`) | Release 2, 3 |
|
|
| `CLASSROOM_UPDATES_USER` | repository variable | The account that key logs in as | Release 2, 3 |
|
|
| `CLASSROOM_UPDATES_KNOWN_HOSTS` | repository variable | The server's ssh host key line(s), so the job refuses any other host | Release 2, 3 |
|
|
| `RELEASE_API_TOKEN` | repository secret, **optional** | A token that may write releases. Only needed if the automatic Actions token ever cannot; measured on Gitea 1.27.2, it can | Release 1, 2, 3, import |
|
|
| automatic Actions token | provided by Gitea per job | Creates and records releases | Release 1, 2, 3, import |
|
|
|
|
Variables are not secret: a pull request's jobs can read them (measured). Put
|
|
nothing in a variable that would matter if it were printed.
|
|
|
|
**Least privilege for the publish key.** Today the development VM publishes
|
|
as `media@10.40.0.2`, and `media` is in the `sudo` and `docker` groups on that
|
|
host — which also runs Gitea, Home Assistant and Vaultwarden. That key must
|
|
**not** be the one put in Gitea. Before the release workflows are used:
|
|
|
|
1. On 10.40.0.2, create an account used for nothing else (say
|
|
`classroom-publish`), in no group but one that may write
|
|
`/srv/ekklesia/classroom-updates` — not `sudo`, not `docker`.
|
|
2. Generate a new key pair for it on a trusted machine; put the public half
|
|
in its `authorized_keys` with `restrict,from="<release runner address>"`;
|
|
put the private half in `CLASSROOM_UPDATES_SSH_KEY`; delete the local copy.
|
|
3. Set the three variables above.
|
|
|
|
The scripts need `ls`, `test`, `sha256sum`, `cat`, `ln`, `mv` and `rsync`
|
|
there, and nothing else.
|
|
|
|
## Runners
|
|
|
|
| Label | Runs | Needs | Must not have |
|
|
| --- | --- | --- | --- |
|
|
| `ekklesia-ci` | CI on every push and pull request; Release 1; the gate job of Release 2; import | Debian 13, Docker (the container gate), git, python3, dpkg-deb, curl, rsync, nodejs (for `actions/checkout`), the fonts and tools the tests use — in practice, a VM set up like the development VM | Any credential for the update server or a classroom; a route to classrooms it does not need |
|
|
| `ekklesia-release` | Release 2's publish job, Release 3 | ssh, rsync, curl, python3, dpkg-deb, git, nodejs; network to 10.40.0.2 | Docker; any key on its disk; any other label |
|
|
| `classroom-hw-*` (future) | Hardware checks, by hand only | A real classroom computer in the lab | Anything to do with production rooms |
|
|
|
|
**How the separation holds, and where it doesn't** — measured on a local
|
|
Gitea 1.27.2 with act_runner 0.6.1:
|
|
|
|
- A pull request from a **fork** does not run at all until a maintainer
|
|
approves it. Approved, it gets **no secrets**. It does get the variables
|
|
and a token, and it runs on whichever runner its workflow names —
|
|
including `ekklesia-release`, if it asks for that label.
|
|
- A branch pushed to this repository runs with the repository's secrets.
|
|
**Anyone with write access can publish**, by writing a workflow that uses the
|
|
key. Give write access accordingly.
|
|
|
|
The release scripts never use a runner's own ssh keys: under Actions the
|
|
key comes only from the secret, is written to the job's own directory with
|
|
its own `known_hosts` and an ssh configuration used with `ssh -F` (which
|
|
ignores `~/.ssh/config`), and is deleted when the job ends. A job without
|
|
the secret stops. **Never put a runner on a machine that holds the publish
|
|
key by hand** — the development VM does — in case that rule is ever broken.
|
|
|
|
So: the key lives only in Gitea's secret store, never on a runner's disk,
|
|
where an approved fork job could read it. The `ekklesia-release` runner runs
|
|
each job in a fresh container (`ekklesia-release:docker://node:22-trixie`,
|
|
without the Docker socket), so no job can leave anything behind for the
|
|
next. **Never approve a fork's run that changes `.gitea/workflows/`**
|
|
without reading it.
|
|
|
|
`ekklesia-ci` needs Docker for the container gate, and Docker is root on its
|
|
host. Treat that host as something any approved pull request can take over:
|
|
a VM with nothing on it worth taking, and no key to anything else.
|
|
|
|
**Registering a runner** (on the runner's host, as the account it runs as):
|
|
|
|
```
|
|
act_runner generate-config > config.yaml
|
|
# edit runner.labels in config.yaml: [ "ekklesia-ci:host" ]
|
|
# (measured: labels in the config file OVERRIDE --labels at registration;
|
|
# a generated config lists ubuntu-latest etc. and the runner then
|
|
# advertises those, and every job here waits for ever)
|
|
act_runner register --no-interactive -c config.yaml \
|
|
--instance https://git.riverofglorymn.org --token <from Gitea> --name ekklesia-ci-1
|
|
act_runner daemon -c config.yaml # as a systemd service
|
|
```
|
|
|
|
The registration token is under the repository's **Settings → Actions →
|
|
Runners**. Register at repository level, not instance level, so these
|
|
runners take no other repository's jobs.
|
|
|
|
Artifacts: `actions/upload-artifact@v3` works on Gitea 1.27.2; **v4 fails**
|
|
("not supported on GHES", measured). The workflows use v3.
|
|
|
|
## Nightly (designed, not yet active)
|
|
|
|
For installable 0.3.0 development builds, separate from a frozen Preview.
|
|
|
|
- **Source:** `master`, a commit whose CI passed.
|
|
- **When:** once a night if `master` changed since the last nightly, and on
|
|
demand. Not every commit: a lab machine that reboots into a new build every
|
|
hour tests nothing.
|
|
- **Version:** `<next>~nightly.<YYYYMMDD>.<run>`, for example
|
|
`0.3.0~nightly.20261014.3`. The `~` makes it sort **before** `0.3.0` and
|
|
after every 0.2.x, and each night after the last — so a lab machine moves
|
|
to the real 0.3.0 when it ships. The release version rules
|
|
(`tools/release-lib.sh`, and the same pattern in the updater) refuse `~`
|
|
today; they must accept it for the nightly channel only.
|
|
- **Server:** `/nightly/manifest.json`, `/nightly/latest.deb`, and packages in
|
|
`/nightly/packages/` — **not** `/packages/`. The rule that a release must be
|
|
newer than everything ever published reads `/packages/`; a nightly there
|
|
would make 0.2.6.7 "older" than a 0.3.0 nightly and block every
|
|
maintenance release.
|
|
- **Retention:** the last 14 nightlies on the server; older ones deleted from
|
|
`/nightly/packages/` only. No Gitea release per nightly; the build is a
|
|
workflow artifact, kept as long as Gitea keeps artifacts.
|
|
- **Opting in:** a lab machine's `device.conf` says `CHANNEL=nightly`, and its
|
|
first nightly is installed by hand.
|
|
- **Never a classroom:** the nightly updater refuses `CHANNEL=nightly` unless
|
|
`/etc/ekklesia-classroom/lab-machine` exists, and a nightly package refuses
|
|
to install without it. A nightly machine never reads the stable or canary
|
|
manifest, and it cannot drift onto them: its version is newer than any
|
|
0.2.x, so pointed at stable it would simply sit until stable passes it.
|
|
Taking it back to 0.2.x is a reinstall with `--allow-downgrades`, by hand.
|
|
- **Credential:** its own key, allowed to write `/srv/ekklesia/classroom-updates/nightly`
|
|
and nothing else (`CLASSROOM_NIGHTLY_SSH_KEY`).
|
|
|
|
What activating it needs, which is why it is not active: the updater and
|
|
release-lib changes above ship **inside the package**, so they are 0.3.0
|
|
work; plus the server directory, the key, and a `nightly.yml` workflow. That
|
|
is the next infrastructure task, after the backlog audit decides what 0.3.0
|
|
is.
|
|
|
|
## The hardware lab (designed, not yet built)
|
|
|
|
Spare classroom computers, each a runner labelled `classroom-hw-<kind>`
|
|
(`classroom-hw-desktop`, `classroom-hw-laptop`), registered to this
|
|
repository, in a lab network with no path to production rooms, on
|
|
`CHANNEL=nightly` or on whatever a job installs.
|
|
|
|
- **Triggered only by hand** (`workflow_dispatch`) with a version and sha256 —
|
|
never by a pull request or a push.
|
|
- **Installs the exact file** from its Gitea release through apt, proves dpkg
|
|
reports that version, then runs checks from `tools/hw/` (to be written):
|
|
HDMI connect/disconnect (the display count and the desktop's wallpaper
|
|
before and after), HDMI audio (the sink appears, is chosen, plays),
|
|
the DVD drive (seen, a disc read), the printer (queue state; a real page
|
|
only with an explicit input), Wi-Fi recovery (down, up, reconnected with
|
|
nobody typing), sleep/wake (`rtcwake`, and what is standing after), and
|
|
reboot/update (a job that reboots the machine and a second job that reads
|
|
what came back).
|
|
- **Reports** one line per check into `hardware-report-<machine>-<date>.txt`:
|
|
|
|
```
|
|
AUTO PASS hdmi-hotplug 2 -> 1 -> 2 outputs; desktop 1 kept its wallpaper
|
|
AUTO FAIL dvd-read no disc in /dev/sr0
|
|
MANUAL - picture-visible not checked by a machine
|
|
```
|
|
|
|
attached to the Gitea release, with a record line summarising it.
|
|
`AUTO` lines come only from a script that measured something. `MANUAL`
|
|
lines are only ever `-` from a machine; a person's result is written by the
|
|
person, in the Stable workflow's approval, and nowhere else. A machine
|
|
never ticks a physical check.
|
|
- **Does not block** CI or releases. Until it exists, physical checks are
|
|
done by a person from `docs/testing.md`, exactly as now.
|
|
|
|
## Gitea administration still to do
|
|
|
|
Things only a Gitea administrator can do, none of which this repository can
|
|
do for itself:
|
|
|
|
- [ ] Say which host the existing "Ekklesia" runner is on, and its labels. It
|
|
is not on 10.40.0.2, and the Gitea API does not show runners to anyone
|
|
but an administrator. Relabel it `ekklesia-ci:host` (in its config
|
|
file, see above) if it is fit for that role, or register a new one.
|
|
- [ ] Set up an `ekklesia-release` runner as above: container mode, no Docker
|
|
socket, repository level.
|
|
- [ ] Create the `classroom-publish` account and key on 10.40.0.2, and the
|
|
secret and three variables above.
|
|
- [ ] Protect `master` and `release/*`: no force push, no deletion, required
|
|
status checks **CI / checks** and **CI / package and container gate**.
|
|
- [ ] Under the repository's Actions settings, keep approval required for
|
|
runs from forks (the default, measured).
|
|
- [ ] Run **Release - import an existing release** for 0.2.6.6 (version,
|
|
sha256 and commit are in the release commit and in
|
|
`docs/testing.md`) so its Gitea release exists before its promotion.
|
|
|
|
## What was measured
|
|
|
|
On a throwaway Gitea 1.27.2 with act_runner 0.6.1 (the versions production
|
|
runs), on the development VM, 2026-09-29:
|
|
|
|
| Question | Answer |
|
|
| --- | --- |
|
|
| Do all five workflows parse, and plan only on their own events? | Yes: CI on push and pull_request; the release workflows only on a manual run |
|
|
| Does the CI checks job pass on a real runner? | Yes: all 31 checks, and their logs upload as an artifact |
|
|
| Does CI's package job build, and run the gate in its CI mode? | Yes: built, the tree unchanged, fresh install and both upgrades passed, the report marked `mode ci` |
|
|
| Does **Release 1 - build** work end to end? | Yes, at a throwaway version 9.9.9 against what canary (0.2.6.6) and stable (0.2.6.4) really serve: built from the commit with the tree unchanged, full gate passed, pre-release `v9.9.9` tagged at exactly that commit with the package, SHA256SUMS, build-info.json, gate report and check summary |
|
|
| Is a second build of the same version refused? | Yes: "v9.9.9 already has a Gitea release; a version is built once" |
|
|
| Does Preview's gate job work? | Yes: it downloaded the release's file, proved it against the sha256 given and SHA256SUMS, passed the full gate, attached a uniquely named report, recorded it, and handed its name to the publish job |
|
|
| If the key secret is missing, what would the publish job use? | **This machine's own ssh configuration** — on the development VM, the production publish key: a throwaway 9.9.9 would have gone to the real server. The job was cancelled before it ran, and since then a job under Actions without the key refuses, and one with it uses only its own configuration (`ssh -F`), never the runner's |
|
|
| Does a failing check stop a release build? | Yes: an undocumented workflow left in the tree failed `test-workflows`, and nothing was built |
|
|
| Can the automatic token create a release, attach an asset, edit the notes? | Yes |
|
|
| Does `tools/gitea-release` refuse different bytes under a published name, on real Gitea? | Yes |
|
|
| Do `github.event.inputs.X` and `inputs.X` both work? | Yes |
|
|
| Do job outputs pass from a job on one runner to a job on another? | Yes |
|
|
| Do `vars.*` and `secrets.*` reach a job, and is the secret masked? | Yes, and yes |
|
|
| `upload-artifact@v3` / `@v4`? | v3 works; v4 fails ("not supported on GHES") |
|
|
| Do labels given to `act_runner register` survive a generated config? | **No**: the config's labels win |
|
|
| Does a fork's pull request run without approval? | No: held until a maintainer approves |
|
|
| Approved, does it get secrets? | No. It gets variables, a token, and any runner it names |
|
|
|
|
Not measured, because it needs the real update server's key in a real
|
|
runner: Preview's publish job and Stable's promote job end to end. Their scripts are the ones
|
|
used by hand since 0.2.6, and `tests/test-release-ci.sh` runs them against a
|
|
stand-in server and a stand-in Gitea.
|