Zayin/Classroom_Desktop
3
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

CI container gate fails the stable-upgrade test: the previous release, downloaded to the job container's /tmp, reaches the gate as an empty directory #1

New Issue
Open
opened 2026-10-14 04:30:22 +00:00 by ekklesia-bot · 2 comments
ekklesia-bot commented 2026-10-14 04:30:22 +00:00
Member
Copy Link
Copy Source

Summary

CI's package and container gate job can't pass on the Zayin runner as written. tools/validate-trixie-release.sh hands the built .deb to a Debian container with a bind mount:

docker run --rm -i -v "$candidate:/packages/candidate.deb:ro" ... debian:trixie-slim ...

On the Zayin runner, ekklesia-ci jobs run inside a container (docker.gitea.com/runner-images:ubuntu-latest) whose docker CLI talks to the host's Docker daemon. A bind-mount source is resolved on the host, where /workspace/<task>/Zayin/Classroom_Desktop/dist/…deb doesn't exist. Docker then creates an empty directory there, so the gate's container gets /packages/candidate.deb as a directory and installation can't work.

No CI run on Gitea has ever built and gated the package successfully (details below).

Evidence

Reproduced with the same arrangement: a job-style container (docker:27-cli) with the host's Docker socket writes a file, then starts the gate container with -v <that file>:/packages/candidate.deb:ro:

inside the job container:
-rw-r--r--    1 root root 12 … /tmp/claude-bindprobe-14691/pkg.deb
what the gate container sees:
drwxr-xr-x 2 root root 40 … /packages/candidate.deb
dpkg-deb: error: error reading archive magic version number from file /packages/candidate.deb: Is a directory

CI history (all four runs of ci.yml):

Run Branch / commit checks package and container gate
#1 master 14291fd failed (test-release-ci, see #2) skipped
#2 release/0.2 14291fd passed reported success, but its "Build the package" step failed and the gate step was skipped (old git.riverofglorymn.org runner)
#3 master 0bc8851 failed (test-release-ci, see #2) skipped
#4 master 719c0f7 (HEAD) passed, 31/31 built 0.2.6.6, then the gate step failed after 2 s; job shown as cancelled

In run #4 the gate step ran 01:24:19 → 01:24:21. Its output stops at ecc510c1e359: Pulling fs layer, with no error message at all. Then the upload-artifact@v3 step failed five times with Artifact service responded with 500 (Gitea, through Caddy) over 4 minutes, and the run ended cancelled. The checks job's upload in the same run succeeded, so the 500 may be transient or size-related. It's worth checking the Gitea server log for 2026-10-13 01:24–01:28 UTC.

The cause of the 2-second failure isn't in the log, so I can't confirm it was the bind mount rather than the image pull. But the bind mount makes the step fail even once the pull works.

Why it passed before

docs/releasing.md → What was measured says the gate passed on the development VM's runner, and its admin checklist asks for the existing runner to be relabelled ekklesia-ci:host, i.e. host mode, where paths are shared and bind mounts work. The Zayin runner serves ekklesia-ci in container mode.

Options

  1. Make the gate independent of where docker runs (recommended). Create the container, copy the packages in, then start it: 
    id=$(docker create -i debian:trixie-slim bash -s -- ...)
docker cp "$candidate" "$id:/packages/candidate.deb"   # and previous.deb
docker start -ai "$id" <<'CONTAINER' ... CONTAINER
docker rm -f "$id"
    docker cp streams from the CLI's own filesystem, so this works on a host runner, in a container job, and on a developer machine.
  2. Run ekklesia-ci jobs in host mode, as the docs intended. This needs a runner whose label is ekklesia-ci:host. It weakens isolation for code nobody has reviewed yet (CI runs on every push), which the workflow header calls out.
  3. Stage files on the Docker host, as Ekklesia's CI does (staging directories under the host's /tmp). More moving parts than option 1.

Separately:

  • Make the gate step print the underlying error. A gate that fails in 2 s with nothing in the log can't be diagnosed. Check whether docker pull/run stderr is reaching the job log.
  • Consider whether a failed upload should turn the run failed rather than cancelled.

Not checked

The runner host itself (it isn't on the development machine), its act_runner version and config, and whether debian:trixie-slim pulls there at all.

Found by Claude during a review of Classroom_Desktop's CI, 2026-10-14 (found-by-claude).

## Summary CI's **package and container gate** job can't pass on the Zayin runner as written. `tools/validate-trixie-release.sh` hands the built `.deb` to a Debian container with a bind mount: ```bash docker run --rm -i -v "$candidate:/packages/candidate.deb:ro" ... debian:trixie-slim ... ``` On the Zayin runner, `ekklesia-ci` jobs run **inside a container** (`docker.gitea.com/runner-images:ubuntu-latest`) whose `docker` CLI talks to the **host's** Docker daemon. A bind-mount source is resolved on the host, where `/workspace/<task>/Zayin/Classroom_Desktop/dist/…deb` doesn't exist. Docker then creates an empty directory there, so the gate's container gets `/packages/candidate.deb` **as a directory** and installation can't work. No CI run on Gitea has ever built and gated the package successfully (details below). ## Evidence **Reproduced** with the same arrangement: a job-style container (`docker:27-cli`) with the host's Docker socket writes a file, then starts the gate container with `-v <that file>:/packages/candidate.deb:ro`: ``` inside the job container: -rw-r--r-- 1 root root 12 … /tmp/claude-bindprobe-14691/pkg.deb what the gate container sees: drwxr-xr-x 2 root root 40 … /packages/candidate.deb dpkg-deb: error: error reading archive magic version number from file /packages/candidate.deb: Is a directory ``` **CI history** (all four runs of `ci.yml`): | Run | Branch / commit | `checks` | `package and container gate` | |---|---|---|---| | [#1](https://git.zayin.net/Zayin/Classroom_Desktop/actions/runs/89) | master `14291fd` | failed (`test-release-ci`, see #2) | skipped | | [#2](https://git.zayin.net/Zayin/Classroom_Desktop/actions/runs/90) | release/0.2 `14291fd` | passed | **reported success, but its "Build the package" step failed and the gate step was skipped** (old `git.riverofglorymn.org` runner) | | [#3](https://git.zayin.net/Zayin/Classroom_Desktop/actions/runs/95) | master `0bc8851` | failed (`test-release-ci`, see #2) | skipped | | [#4](https://git.zayin.net/Zayin/Classroom_Desktop/actions/runs/96) | master `719c0f7` (HEAD) | **passed, 31/31** | built `0.2.6.6`, then the gate step **failed after 2 s**; job shown as *cancelled* | In run #4 the gate step ran 01:24:19 → 01:24:21. Its output stops at `ecc510c1e359: Pulling fs layer`, **with no error message at all**. Then the `upload-artifact@v3` step failed five times with `Artifact service responded with 500` (Gitea, through Caddy) over 4 minutes, and the run ended *cancelled*. The `checks` job's upload in the same run succeeded, so the 500 may be transient or size-related. It's worth checking the Gitea server log for 2026-10-13 01:24–01:28 UTC. The cause of the 2-second failure isn't in the log, so I can't confirm it was the bind mount rather than the image pull. But the bind mount makes the step fail even once the pull works. ## Why it passed before `docs/releasing.md` → *What was measured* says the gate passed on the development VM's runner, and its admin checklist asks for the existing runner to be relabelled `ekklesia-ci:host`, i.e. **host mode**, where paths are shared and bind mounts work. The Zayin runner serves `ekklesia-ci` in container mode. ## Options 1. **Make the gate independent of where `docker` runs (recommended).** Create the container, copy the packages in, then start it: ```bash id=$(docker create -i debian:trixie-slim bash -s -- ...) docker cp "$candidate" "$id:/packages/candidate.deb" # and previous.deb docker start -ai "$id" <<'CONTAINER' ... CONTAINER docker rm -f "$id" ``` `docker cp` streams from the CLI's own filesystem, so this works on a host runner, in a container job, and on a developer machine. 2. **Run `ekklesia-ci` jobs in host mode**, as the docs intended. This needs a runner whose label is `ekklesia-ci:host`. It weakens isolation for code nobody has reviewed yet (CI runs on every push), which the workflow header calls out. 3. **Stage files on the Docker host**, as Ekklesia's CI does (staging directories under the host's `/tmp`). More moving parts than option 1. Separately: - Make the gate step **print the underlying error**. A gate that fails in 2 s with nothing in the log can't be diagnosed. Check whether `docker pull`/`run` stderr is reaching the job log. - Consider whether a failed upload should turn the run *failed* rather than *cancelled*. ## Not checked The runner host itself (it isn't on the development machine), its act_runner version and config, and whether `debian:trixie-slim` pulls there at all. --- _Found by Claude during a review of Classroom_Desktop's CI, 2026-10-14 (`found-by-claude`)._
ekklesia-bot added the found-by-claude label 2026-10-14 04:40:48 +00:00
ekklesia-bot commented 2026-10-14 04:49:29 +00:00
Author
Member
Copy Link
Copy Source

Correction from a fresh run: it's the previous release's file, not the new package

PR #7 (docs only) ran CI on the current runner (run #5 push, run #6 pull request). This time the image pull worked and the gate got further, which narrows the cause:

== fresh install of 0.2.6.6 (a new machine, on stable)
PASS  fresh install of 0.2.6.6 (a new machine, on stable)  (6 min)
SKIP  upgrade 0.2.6.6 -> 0.2.6.6 on a canary room: canary serves 0.2.6.6, not older than this commit's VERSION (a CI build, not a release)
== upgrade 0.2.6.4 -> 0.2.6.6 on a stable room (stable serves 0.2.6.4)
E: Read error - read (21: Is a directory)
E: Internal error, could not locate member control.tar{.zst,.lz4,.gz,.xz,.bz2,.lzma,}
E: Could not read meta data from /packages/previous.deb
  • The candidate mounts fine. It's built into the workspace (/workspace/<task>/Zayin/Classroom_Desktop/dist/…deb), and the runner evidently shares that workspace with the Docker host at the same path.
  • The previous release doesn't. tools/release-gate.sh:117 downloads it into work="$(mktemp -d)", which is /tmp inside the job container and doesn't exist on the host. Docker creates an empty directory there, and /packages/previous.deb is that directory.

So the fresh-install path already works in CI. Only the upgrade-from-stable path fails, and it fails every time stable is older than the commit, which is always true until the next promotion.

Fix options, revised

  1. Smallest: download into the shared workspace instead of /tmp, e.g. work="$(mktemp -d "$ROOT/.gate.XXXXXX")". Add it to .gitignore, and clean it up on exit; the existing "tree unchanged" test runs before the gate. This depends on the runner keeping its shared workspace.
  2. Most robust (unchanged recommendation): docker create + docker cp + docker start, so no host path is involved at all.

Earlier run #4's 2-second failure with no output was most likely the image pull itself (it never reached the install). This run pulled fine.

Status names CI reports, for branch protection: CI / checks (pull_request), CI / checks (push), CI / package and container gate (pull_request), CI / package and container gate (push).

## Correction from a fresh run: it's the *previous* release's file, not the new package PR #7 (docs only) ran CI on the current runner ([run #5](https://git.zayin.net/Zayin/Classroom_Desktop/actions/runs/300) push, [run #6](https://git.zayin.net/Zayin/Classroom_Desktop/actions/runs/301) pull request). This time the image pull worked and the gate got further, which narrows the cause: ``` == fresh install of 0.2.6.6 (a new machine, on stable) PASS fresh install of 0.2.6.6 (a new machine, on stable) (6 min) SKIP upgrade 0.2.6.6 -> 0.2.6.6 on a canary room: canary serves 0.2.6.6, not older than this commit's VERSION (a CI build, not a release) == upgrade 0.2.6.4 -> 0.2.6.6 on a stable room (stable serves 0.2.6.4) E: Read error - read (21: Is a directory) E: Internal error, could not locate member control.tar{.zst,.lz4,.gz,.xz,.bz2,.lzma,} E: Could not read meta data from /packages/previous.deb ``` - **The candidate mounts fine.** It's built into the workspace (`/workspace/<task>/Zayin/Classroom_Desktop/dist/…deb`), and the runner evidently shares that workspace with the Docker host at the same path. - **The previous release doesn't.** `tools/release-gate.sh:117` downloads it into `work="$(mktemp -d)"`, which is `/tmp` inside the job container and doesn't exist on the host. Docker creates an empty directory there, and `/packages/previous.deb` is that directory. So the fresh-install path already works in CI. Only the upgrade-from-stable path fails, and it fails every time stable is older than the commit, which is always true until the next promotion. ### Fix options, revised 1. **Smallest:** download into the shared workspace instead of `/tmp`, e.g. `work="$(mktemp -d "$ROOT/.gate.XXXXXX")"`. Add it to `.gitignore`, and clean it up on exit; the existing "tree unchanged" test runs before the gate. This depends on the runner keeping its shared workspace. 2. **Most robust (unchanged recommendation):** `docker create` + `docker cp` + `docker start`, so no host path is involved at all. Earlier run #4's 2-second failure with no output was most likely the image pull itself (it never reached the install). This run pulled fine. Status names CI reports, for branch protection: `CI / checks (pull_request)`, `CI / checks (push)`, `CI / package and container gate (pull_request)`, `CI / package and container gate (push)`.
ekklesia-bot changed title from CI container gate cannot pass on the Zayin runner: the .deb bind mount reaches the gate as an empty directory to CI container gate fails the stable-upgrade test: the previous release, downloaded to the job container's /tmp, reaches the gate as an empty directory 2026-10-14 04:49:29 +00:00
ekklesia-bot commented 2026-10-14 04:52:17 +00:00
Author
Member
Copy Link
Copy Source

Priority note (2026-10-14): CI / package and container gate is now a required status check on this repository, so until this is fixed every pull request shows as blocked and needs a maintainer override to merge. That makes this the first CI fix to do (see the order in #6).

**Priority note (2026-10-14):** `CI / package and container gate` is now a **required** status check on this repository, so until this is fixed every pull request shows as blocked and needs a maintainer override to merge. That makes this the first CI fix to do (see the order in #6).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
found-by-claude
Found by Claude while working on another task
No labels found-by-claude
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
Joey Zayin_Admin ekklesia-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Zayin/Classroom_Desktop#1