CI / verify (pull_request) Successful in 59s
CI / infrastructure-smoke (pull_request) Canceled after 0s
CI / migration-smoke (pull_request) Canceled after 0s
CI / image-build (pull_request) Canceled after 0s
CI / deploy-smoke (same-origin) (pull_request) Canceled after 0s
CI / deploy-smoke (split) (pull_request) Canceled after 0s
A migration-smoke stack (PostgreSQL and SeaweedFS) ran on the shared runner for two days. Every CI script removes its stack in an EXIT trap, but a trap never runs when the runner kills the job container: a cancelled job, or the runner's own time limit (Gitea 1.24 ignores timeout-minutes). Reproduced: SIGKILL of the infrastructure smoke mid-run leaves exactly that stack running. Cleanup now has three layers: - the script's own trap, unchanged; - a final `if: always()` step in every job that starts stacks (infrastructure-smoke, migration-smoke, image-build, deploy-smoke) runs `scripts/ci-sweep.sh --job`, which removes everything that job started at any age; - a first step in each of those jobs runs `scripts/ci-sweep.sh --stale`, which removes CI stacks older than three hours: what a killed job left before it could clean up. How stacks are found is defined once, in scripts/lib/ci-project.sh: - every CI Compose project is named `ekklesia-ci-<run>-<attempt>-<job key>-<kind>-<unique>`, and Compose labels its containers, networks and volumes with that name; - the deploy smoke labels everything it starts outside Compose: browser, kiosk and driver containers, the kiosk volume, built images; - images also record when CI built them. An image's own creation time can be its base's (BuildKit, metadata-only builds), and the sweep removed a brand-new image as "old" until it stopped trusting that. The job key is per matrix leg, so one topology's cleanup cannot remove the other's running stack. The sweep never touches anything outside the ekklesia-ci- prefix, disconnects rather than removes a job container attached to a stack's network, and refuses --job outside CI. Network creation times are read as RFC 3339; Docker's default form was unparseable. scripts/test-ci-sweep.sh proves all of that against real Docker, and CI runs it before relying on it. security.md and testing.md document the rule, and that it complements, not replaces, a runner-side reaper. Co-Authored-By: Claude Opus 5.5 <[email protected]>
226 lines
9.8 KiB
YAML
226 lines
9.8 KiB
YAML
name: CI
|
|
|
|
# IMPORTANT — the currently inventoried server is Gitea 1.24.2, which *ignores* several keys used
|
|
# below. Per https://docs.gitea.com/1.24/usage/actions/comparison, its unsupported workflow syntax
|
|
# includes `permissions` / `jobs.<job_id>.permissions`, `concurrency`, and
|
|
# `jobs.<job_id>.timeout-minutes`. They are declared here deliberately, so the workflow is already
|
|
# correct on a supported release and on GitHub Actions, but on this server they are inert:
|
|
#
|
|
# * `permissions:` does NOT restrict the in-job `GITEA_TOKEN`. Job-token permission controls,
|
|
# repository/owner defaults, and server-side maximum-permission clamping arrive in the 1.26
|
|
# feature line (https://docs.gitea.com/1.26/usage/actions/token-permissions). Until the server
|
|
# is upgraded and configured, the token's authority is whatever the server grants — not what
|
|
# this file requests.
|
|
# * `concurrency:` and `timeout-minutes:` are NOT bounded-execution guarantees here. Cancellation,
|
|
# timeout, and teardown must be enforced by the trusted runner provisioner/reaper.
|
|
#
|
|
# `persist-credentials: false` below is unaffected by any of this and is enforced by the action
|
|
# itself. See docs/development/security.md.
|
|
|
|
on:
|
|
pull_request:
|
|
push:
|
|
branches:
|
|
- main
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
concurrency:
|
|
group: ci-${{ github.workflow }}-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
jobs:
|
|
verify:
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 15
|
|
steps:
|
|
- name: Check out repository
|
|
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
with:
|
|
# `actions/checkout` defaults to leaving the job token in `.git/config`, where any later
|
|
# step — including one a pull request added to this file — could read or reuse it. Nothing
|
|
# in this workflow pushes, so the credential is never needed after checkout.
|
|
persist-credentials: false
|
|
|
|
- name: Set up Node.js
|
|
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
|
with:
|
|
node-version-file: .node-version
|
|
|
|
- name: Activate pinned pnpm
|
|
run: |
|
|
corepack enable
|
|
corepack prepare [email protected] --activate
|
|
|
|
- name: Install dependencies
|
|
run: pnpm install --frozen-lockfile
|
|
|
|
- name: Verify repository
|
|
run: pnpm ci:verify
|
|
|
|
infrastructure-smoke:
|
|
needs: verify
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 10
|
|
env:
|
|
# Names this job's stacks, so its final step removes exactly these (scripts/lib/ci-project.sh).
|
|
EKKLESIA_CI_JOB_KEY: infrastructure-smoke
|
|
steps:
|
|
- name: Check out repository
|
|
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
with:
|
|
# `actions/checkout` defaults to leaving the job token in `.git/config`, where any later
|
|
# step — including one a pull request added to this file — could read or reuse it. Nothing
|
|
# in this workflow pushes, so the credential is never needed after checkout.
|
|
persist-credentials: false
|
|
|
|
# A job the runner killed (cancelled, or past the runner's own time limit) never ran its
|
|
# cleanup. Anything a CI job left on this shared daemon more than three hours ago goes first.
|
|
- name: Sweep stale CI stacks from the runner
|
|
run: bash scripts/ci-sweep.sh --stale
|
|
|
|
- name: Prove the sweep removes exactly what it should, against real Docker resources
|
|
run: bash scripts/test-ci-sweep.sh
|
|
|
|
- name: Exercise PostgreSQL and S3 infrastructure
|
|
run: bash scripts/smoke-infrastructure.sh
|
|
|
|
# Whatever happened above, this job's own stacks do not outlive it.
|
|
- name: Remove this job's CI stacks
|
|
if: always()
|
|
run: bash scripts/ci-sweep.sh --job
|
|
|
|
migration-smoke:
|
|
needs: verify
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 15
|
|
env:
|
|
# Names this job's stacks, so its final step removes exactly these (scripts/lib/ci-project.sh).
|
|
EKKLESIA_CI_JOB_KEY: migration-smoke
|
|
steps:
|
|
- name: Check out repository
|
|
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
with:
|
|
# `actions/checkout` defaults to leaving the job token in `.git/config`, where any later
|
|
# step — including one a pull request added to this file — could read or reuse it. Nothing
|
|
# in this workflow pushes, so the credential is never needed after checkout.
|
|
persist-credentials: false
|
|
|
|
# A job the runner killed (cancelled, or past the runner's own time limit) never ran its
|
|
# cleanup. Anything a CI job left on this shared daemon more than three hours ago goes first.
|
|
- name: Sweep stale CI stacks from the runner
|
|
run: bash scripts/ci-sweep.sh --stale
|
|
|
|
- name: Set up Node.js
|
|
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
|
with:
|
|
node-version-file: .node-version
|
|
|
|
- name: Activate pinned pnpm
|
|
run: |
|
|
corepack enable
|
|
corepack prepare [email protected] --activate
|
|
|
|
- name: Install dependencies
|
|
run: pnpm install --frozen-lockfile
|
|
|
|
- name: Exercise transactional migrations against PostgreSQL
|
|
run: pnpm test:migrations:integration
|
|
|
|
# Whatever happened above, this job's own stacks do not outlive it.
|
|
- name: Remove this job's CI stacks
|
|
if: always()
|
|
run: bash scripts/ci-sweep.sh --job
|
|
|
|
image-build:
|
|
needs: verify
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 20
|
|
env:
|
|
# Names this job's stacks, so its final step removes exactly these (scripts/lib/ci-project.sh).
|
|
EKKLESIA_CI_JOB_KEY: image-build
|
|
# Cache mounts in deploy/Dockerfile need BuildKit; this selects it even where the Docker CLI
|
|
# has no buildx plugin.
|
|
DOCKER_BUILDKIT: "1"
|
|
steps:
|
|
- name: Check out repository
|
|
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
with:
|
|
# `actions/checkout` defaults to leaving the job token in `.git/config`, where any later
|
|
# step — including one a pull request added to this file — could read or reuse it. Nothing
|
|
# in this workflow pushes, so the credential is never needed after checkout.
|
|
persist-credentials: false
|
|
|
|
# A job the runner killed (cancelled, or past the runner's own time limit) never ran its
|
|
# cleanup. Anything a CI job left on this shared daemon more than three hours ago goes first.
|
|
- name: Sweep stale CI stacks from the runner
|
|
run: bash scripts/ci-sweep.sh --stale
|
|
|
|
# The platform-only image (ADR 0044): a checkout has no `modules/`, so this proves the image
|
|
# builds, validates, and assembles with zero modules. Build only — no registry login, no
|
|
# credentials, no push, no tag anywhere but this runner's local image store.
|
|
# Tagged per run and attempt, not only per commit: a pull request and a push of the same commit
|
|
# can build at once on one runner's Docker daemon, and neither may remove the other's image.
|
|
# Labeled like every CI stack, so this job's final step removes it even if the build is killed.
|
|
- name: Build the production image without modules
|
|
shell: bash
|
|
run: |
|
|
source scripts/lib/ci-project.sh
|
|
docker build \
|
|
--file deploy/Dockerfile \
|
|
--build-arg REVISION="$GITHUB_SHA" \
|
|
--label "${EKKLESIA_CI_LABEL}=$(ci_project_name image)" \
|
|
--label "${EKKLESIA_CI_CREATED_LABEL}=$(date +%s)" \
|
|
--tag ekklesia-ci:"$GITHUB_SHA-${GITHUB_RUN_ID:-0}-${GITHUB_RUN_ATTEMPT:-1}" \
|
|
.
|
|
|
|
# Whatever happened above, this job's own stacks do not outlive it.
|
|
- name: Remove this job's CI stacks
|
|
if: always()
|
|
run: bash scripts/ci-sweep.sh --job
|
|
|
|
deploy-smoke:
|
|
needs: verify
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 30
|
|
# Both topologies (ADR 0045): same-origin, the default every installation gets, and split, which
|
|
# adds a test-only module and drives a real headless browser across the two origins.
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
topology:
|
|
- same-origin
|
|
- split
|
|
env:
|
|
# Names this job's stacks, so its final step removes exactly these (scripts/lib/ci-project.sh).
|
|
EKKLESIA_CI_JOB_KEY: deploy-smoke-${{ matrix.topology }}
|
|
DOCKER_BUILDKIT: "1"
|
|
EKKLESIA_SMOKE_TOPOLOGY: ${{ matrix.topology }}
|
|
steps:
|
|
- name: Check out repository
|
|
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
with:
|
|
# `actions/checkout` defaults to leaving the job token in `.git/config`, where any later
|
|
# step — including one a pull request added to this file — could read or reuse it. Nothing
|
|
# in this workflow pushes, so the credential is never needed after checkout.
|
|
persist-credentials: false
|
|
|
|
# A job the runner killed (cancelled, or past the runner's own time limit) never ran its
|
|
# cleanup. Anything a CI job left on this shared daemon more than three hours ago goes first.
|
|
- name: Sweep stale CI stacks from the runner
|
|
run: bash scripts/ci-sweep.sh --stale
|
|
|
|
# The reference deployment end to end (ADR 0044): a platform-only image, base + proxy with
|
|
# Caddy's internal CA (test-only), an explicit migration apply against this job's own fresh
|
|
# PostgreSQL, then the API, Dashboard, storage delivery, client-address chain, and verify-full
|
|
# database TLS through the proxy. Every secret is generated for this run and discarded with
|
|
# it; nothing is pushed, published, or logged in to.
|
|
- name: Smoke-test the reference deployment (${{ matrix.topology }})
|
|
run: bash scripts/smoke-deploy.sh
|
|
|
|
# Whatever happened above, this job's own stacks do not outlive it.
|
|
- name: Remove this job's CI stacks
|
|
if: always()
|
|
run: bash scripts/ci-sweep.sh --job
|