fix: run the TLS integration harness under a containerized job

The `migration-smoke` job fails on the Gitea runner at
`docker port ekklesia-database-tls-<pid> 5432` with
`no public port '5432' published`; the later invalid-URL and
`ECONNREFUSED 127.0.0.1:5432` failures all cascade from it.

Gitea executes the job inside a container and drives sibling containers
through the runner's Docker daemon. The harness held two host-only
assumptions, both reproduced in a nested job container and fixed here.

Bind-mount source paths are resolved by the daemon against the runner
host's filesystem. The certificate directory is created in the job
container's `/tmp`, which does not exist there, so the daemon silently
creates an empty directory, the entrypoint's `cp` fails, the container
exits, and `docker port` reports no published port for an exited
container. The material is now transferred with `docker cp` into the
created-but-not-yet-started container, which crosses the daemon boundary
correctly in both topologies, and only the server leaf and its key are
transferred — both CA private keys stay in the harness directory.

A published port lands on the runner host's loopback, a different
network namespace from the job container's; measured, a container the
daemon published at `127.0.0.1:32907` is `ECONNREFUSED` at that address
from inside the job container. Under a containerized job the harness now
publishes nothing and joins both containers to one isolated user-defined
network, reaching PostgreSQL by container DNS on 5432. Direct-host runs
keep a random port pinned to loopback via Docker's `127.0.0.1::5432`
form. `compose_job_network_job_container` exports the topology decision
the integration script already made, so there is still exactly one
detector.

Every TLS assertion is preserved, and the certificate identity is
generated from the host actually connected to — `localhost` directly, the
container's DNS name otherwise. The mismatch and chain-only controls use
the TLS container's address on the isolated network rather than a
hardcoded one, and the certificate carries no IP SAN, so verify-full
still fails with `ERR_TLS_CERT_ALTNAME_INVALID`.

Correcting one measurement in passing: chain-only verification does not
skip the identity check. For an IP host `pg` omits `servername` and hands
`tls.connect` a bare socket with no host, and `net.Socket._host` is null
for an IP literal, so Node checks the fixed fallback name `localhost` —
never the address dialled. A certificate whose only SAN is
`DNS:example.invalid`, reached chain-only over `127.0.0.1`, is rejected
with `Host: localhost. is not in the cert's altnames`. The direct-host
control therefore only ever passed because its SAN was `DNS:localhost`.
The SAN set now always includes that fallback name so the control proves
the same gap in both topologies, and the stale claim is corrected in the
transport doc comment. No production behavior changes.

Teardown also removes the container's anonymous volume, the isolated
network, and the job container's attachment, after success and failure
alike.

Co-Authored-By: Claude Opus 5 <[email protected]>

core/src/database/transport.ts

@@ -191,12 +191,13 @@ export function assertNoConnectionStringTlsParameters(databaseUrl: string): void
  *
  * `verify-full` is **not** simply `ssl: true` or `{ rejectUnauthorized: true }`. Verified directly
  * against a live TLS PostgreSQL instance: those verify the certificate *chain*, but `pg` only sets
- * `options.servername` for non-IP hosts (`pg/lib/connection.js`), and Node performs no identity
- * check at all when `servername` is unset — so a certificate issued for an entirely different name
- * is accepted when connecting to an IP address. Supplying an explicit `checkServerIdentity` pinned
- * to the configured host closes that gap for DNS and IP hosts alike (proven: a certificate valid
- * only for `localhost` is rejected with `ERR_TLS_CERT_ALTNAME_INVALID` when reached via
- * `127.0.0.1`).
+ * `options.servername` for non-IP hosts (`pg/lib/connection.js`) and passes `tls.connect` a bare
+ * socket carrying no host, so for an IP endpoint Node has no identity to check and falls back to
+ * the fixed name `localhost` — never the address actually dialled. A certificate issued for an
+ * entirely different name is therefore accepted when connecting to an IP address. Supplying an
+ * explicit `checkServerIdentity` pinned to the configured host closes that gap for DNS and IP hosts
+ * alike (proven: a certificate valid only for `localhost` is rejected with
+ * `ERR_TLS_CERT_ALTNAME_INVALID` when reached via `127.0.0.1`).
  */
 export function databaseTlsConnectionOptions(
   databaseUrl: string,

core/tests/database-tls.integration.ts

@@ -1,5 +1,7 @@
 import { execFile } from "node:child_process";
-import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
+import { randomBytes } from "node:crypto";
+import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
+import net from "node:net";
 import { tmpdir } from "node:os";
 import path from "node:path";
 import { promisify } from "node:util";
@@ -15,13 +17,48 @@ const run = promisify(execFile);
  *
  * Every certificate, key, and CA used here is generated at test runtime into an isolated temporary
  * directory and deleted in `afterAll` — no private-key material is committed to this repository.
- * The container, its name, and its published loopback port are likewise disposable and removed in
- * `afterAll` even when an assertion fails.
+ * The container, the isolated network, and any published loopback port are likewise disposable and
+ * removed in `afterAll` even when an assertion fails.
+ *
+ * ## Two supported topologies
+ *
+ * `EKKLESIA_TEST_JOB_CONTAINER` is the single topology signal, produced by
+ * `compose_job_network_job_container` in `scripts/lib/compose-job-network.sh` and exported by
+ * `scripts/test-migrations-integration.sh`. This harness never re-derives the topology itself.
+ *
+ * - **Direct host** (signal empty): the container publishes a random port bound to `127.0.0.1`
+ *   only, and the suite connects through `localhost`.
+ * - **Containerized job** (signal set): the job executes inside a container and drives *sibling*
+ *   containers through the runner's Docker daemon. Two host assumptions break there, and both are
+ *   avoided rather than worked around:
+ *     1. A port published by the daemon lands on the **runner host's** loopback, which is a
+ *        different network namespace from the job container's. Proven: a container published at
+ *        `127.0.0.1:32907` by the daemon is `ECONNREFUSED` at that address from inside the job
+ *        container. So no port is published at all; both containers join one isolated user-defined
+ *        network and talk over container DNS on the standard port.
+ *     2. A bind-mount source path is resolved by the daemon against the **runner host's**
+ *        filesystem. A directory created in the job container's `/tmp` does not exist there, so the
+ *        daemon silently creates an empty one, the entrypoint's `cp` fails, the container exits,
+ *        and `docker port` then reports `No public port '5432' published`. So the certificate and
+ *        key are transferred with `docker cp` into the created-but-not-yet-started container, which
+ *        crosses the daemon boundary correctly in both topologies.
  */
-const containerName = `ekklesia-database-tls-${process.pid}`;
+const instanceId = `${process.pid}-${randomBytes(4).toString("hex")}`;
+/** Also the container's DNS name on the isolated network, hence the certificate identity there. */
+const containerName = `ekklesia-database-tls-${instanceId}`;
+const networkName = `ekklesia-database-tls-net-${instanceId}`;
+const jobContainer = (process.env.EKKLESIA_TEST_JOB_CONTAINER ?? "").trim();
+const containerizedJob = jobContainer !== "";
+
 let certificateDirectory: string | undefined;
-let containerStarted = false;
-let boundPort: number | undefined;
+let containerCreated = false;
+let networkCreated = false;
+let jobContainerAttached = false;
+/** The host the certificate is issued for, and the only host that may verify successfully. */
+let identityHost: string | undefined;
+/** A reachable address of the *same* server that the certificate deliberately does not cover. */
+let mismatchHost: string | undefined;
+let databasePort: number | undefined;
 
 class TlsHarnessError extends Error {
   constructor(message: string) {
@@ -40,10 +77,42 @@ async function openssl(args: readonly string[], cwd: string): Promise<void> {
   }
 }
 
-/** Generates a private CA, a server certificate signed by it whose only SAN is `DNS:localhost`,
- * and an unrelated second CA. The deliberately narrow SAN is what makes the hostname-mismatch
- * assertion below meaningful: reaching the very same server through `127.0.0.1` must fail. */
-async function generateCertificates(directory: string): Promise<void> {
+async function docker(args: readonly string[]): Promise<string> {
+  const { stdout } = await run("docker", [...args]);
+  return stdout.trim();
+}
+
+/**
+ * The identity Node checks when nothing supplies one. Measured against the pinned driver, not
+ * assumed: for an IP host `pg` skips `options.servername` (`pg/lib/connection.js`) and hands
+ * `tls.connect` a bare socket with no `host`, and `net.Socket._host` is `null` for an IP literal —
+ * so Node falls back to the fixed string `localhost`. A certificate whose only SAN is
+ * `DNS:example.invalid`, reached chain-only over `127.0.0.1`, is rejected with
+ * `Host: localhost. is not in the cert's altnames` — proof that chain-only verification checks this
+ * name rather than the endpoint.
+ */
+const NODE_FALLBACK_IDENTITY = "localhost";
+
+/**
+ * Generates a private CA, a server certificate signed by it, and an unrelated second CA.
+ *
+ * The certificate's SANs are exactly two DNS names and **no IP SAN**:
+ *
+ * - the host this run actually connects through, so the verify-full success case matches a real
+ *   identity rather than a coincidence;
+ * - {@link NODE_FALLBACK_IDENTITY}, so the chain-only control below reproduces the same outcome in
+ *   both topologies.
+ *
+ * Neither covers the server's IP address, which is what keeps the hostname-mismatch assertion
+ * meaningful: reaching the very same server through its address must fail under the production
+ * policy, whose pinned `checkServerIdentity` demands an `IP:` SAN that is deliberately absent.
+ * On a direct host the two names coincide and the SAN set is the single `DNS:localhost` entry this
+ * suite has always used.
+ */
+async function generateCertificates(directory: string, host: string): Promise<void> {
+  const subjectAltNames = [...new Set([host, NODE_FALLBACK_IDENTITY])]
+    .map((name) => `DNS:${name}`)
+    .join(",");
   await openssl(
     [
       "req",
@@ -71,11 +140,11 @@ async function generateCertificates(directory: string): Promise<void> {
       "-keyout",
       "server.key",
       "-subj",
-      "/CN=localhost",
+      `/CN=${host}`,
     ],
     directory,
   );
-  await writeFile(path.join(directory, "san.cnf"), "subjectAltName=DNS:localhost\n");
+  await writeFile(path.join(directory, "san.cnf"), `subjectAltName=${subjectAltNames}\n`);
   await openssl(
     [
       "x509",
@@ -113,32 +182,40 @@ async function generateCertificates(directory: string): Promise<void> {
     ],
     directory,
   );
+  // Only the server leaf and its key are ever transferred into the container. Both CA private keys
+  // stay in the harness temporary directory and are deleted with it.
+  const material = path.join(directory, "server-material");
+  await mkdir(material);
+  for (const file of ["server.crt", "server.key"]) {
+    await writeFile(path.join(material, file), await readFile(path.join(directory, file)));
+  }
 }
 
-async function docker(args: readonly string[]): Promise<string> {
-  const { stdout } = await run("docker", [...args]);
-  return stdout.trim();
-}
+/**
+ * Creates the container, transfers the runtime-generated material into it, and starts it. The
+ * certificate/key are copied *inside* the container and re-owned before PostgreSQL starts:
+ * PostgreSQL refuses a key file that is not owned by the database user or root, and `docker cp`
+ * lands the files as root.
+ */
+async function startTlsPostgres(directory: string): Promise<void> {
+  const publication = containerizedJob
+    ? // No published port: the runner host's loopback is not the job container's loopback.
+      ["--network", networkName]
+    : // Docker's random-port form, pinned to loopback so PostgreSQL is never offered on any
+      // other interface.
+      ["-p", "127.0.0.1::5432"];
 
-async function startTlsPostgres(directory: string): Promise<number> {
-  // The certificate/key are copied *inside* the container and re-owned before PostgreSQL starts:
-  // PostgreSQL refuses a key file that is not owned by the database user or root, and a bind-
-  // mounted host file keeps the host's ownership.
   await docker([
-    "run",
-    "-d",
+    "create",
     "--name",
     containerName,
+    ...publication,
     "-e",
     "POSTGRES_DB=ekklesia_tls",
     "-e",
     "POSTGRES_USER=ekklesia_tls",
     "-e",
     "POSTGRES_PASSWORD=ekklesia_tls",
-    "-v",
-    `${directory}:/certs-src:ro`,
-    "-p",
-    "127.0.0.1:0:5432",
     "--entrypoint",
     "/bin/sh",
     "postgres:18.4-alpine3.24",
@@ -148,28 +225,73 @@ async function startTlsPostgres(directory: string): Promise<number> {
       "exec docker-entrypoint.sh postgres -c ssl=on -c ssl_cert_file=/tls/server.crt " +
       "-c ssl_key_file=/tls/server.key",
   ]);
-  containerStarted = true;
+  containerCreated = true;
 
-  const published = await docker(["port", containerName, "5432"]);
-  const match = /^127\.0\.0\.1:(\d+)$/u.exec(published.split("\n")[0]?.trim() ?? "");
-  if (!match) {
+  // `docker cp` is executed by the daemon against the container filesystem, so it is correct
+  // whether the source directory lives on the daemon host or only inside the job container. The
+  // destination does not exist, so the source directory's *contents* become `/certs-src`.
+  await docker(["cp", path.join(directory, "server-material"), `${containerName}:/certs-src`]);
+  await docker(["start", containerName]);
+}
+
+/** Resolves the endpoint this run connects through, plus a reachable address of the same server
+ * that the certificate identity deliberately does not cover. */
+async function resolveEndpoint(): Promise<void> {
+  if (!containerizedJob) {
+    const published = await docker(["port", containerName, "5432"]);
+    const match = /^127\.0\.0\.1:(\d+)$/u.exec(published.split("\n")[0]?.trim() ?? "");
+    if (!match) {
+      throw new TlsHarnessError(
+        "The disposable TLS PostgreSQL port is not bound safely to loopback.",
+      );
+    }
+    databasePort = Number(match[1]);
+    mismatchHost = "127.0.0.1";
+    return;
+  }
+
+  if (!(await connectJobContainerToNetwork())) {
+    throw new TlsHarnessError("The TLS test job could not join the isolated PostgreSQL network.");
+  }
+  const address = await docker([
+    "inspect",
+    "--format",
+    `{{ with index .NetworkSettings.Networks "${networkName}" }}{{ .IPAddress }}{{ end }}`,
+    containerName,
+  ]);
+  if (!net.isIPv4(address)) {
     throw new TlsHarnessError(
-      "The disposable TLS PostgreSQL port is not bound safely to loopback.",
+      "The disposable TLS PostgreSQL has no address on the isolated test network.",
     );
   }
-  return Number(match[1]);
+  // The same server, reached by address instead of by name — a genuine identity mismatch, not a
+  // weakened assertion.
+  mismatchHost = address;
+  databasePort = 5432;
+}
+
+async function connectJobContainerToNetwork(): Promise<boolean> {
+  jobContainerAttached = true;
+  try {
+    await docker(["network", "connect", networkName, jobContainer]);
+    return true;
+  } catch {
+    await docker(["network", "disconnect", networkName, jobContainer]).catch(() => undefined);
+    jobContainerAttached = false;
+    return false;
+  }
 }
 
 /** Bounded readiness polling — never an arbitrary sleep. Resolves once the server accepts a TLS
  * connection, or fails after the deadline. */
-async function waitForTlsReadiness(port: number, ca: string): Promise<void> {
+async function waitForTlsReadiness(host: string, port: number, ca: string): Promise<void> {
   const deadline = Date.now() + 60_000;
   let lastError: unknown;
   while (Date.now() < deadline) {
     const client = new Client({
       connectionTimeoutMillis: 3_000,
       database: "ekklesia_tls",
-      host: "localhost",
+      host,
       password: "ekklesia_tls",
       port,
       ssl: { ca, rejectUnauthorized: true },
@@ -223,9 +345,32 @@ async function connectWith(
   }
 }
 
+/** The trusted CA for the disposable server, read from the harness directory. */
+async function readCertificateAuthority(file: "ca.crt" | "other-ca.crt"): Promise<string> {
+  if (certificateDirectory === undefined) {
+    throw new TlsHarnessError("The disposable TLS material was never generated.");
+  }
+  return await readFile(path.join(certificateDirectory, file), "utf8");
+}
+
+function requireEndpoint(): { identity: string; mismatch: string; port: number } {
+  if (identityHost === undefined || mismatchHost === undefined || databasePort === undefined) {
+    throw new TlsHarnessError("The disposable TLS PostgreSQL endpoint was never resolved.");
+  }
+  return { identity: identityHost, mismatch: mismatchHost, port: databasePort };
+}
+
+// Deterministic teardown: every disposable resource this harness created is removed after both
+// successful and failed runs, container first so the network has no remaining endpoint.
 afterAll(async () => {
-  if (containerStarted) {
-    await docker(["rm", "-f", containerName]).catch(() => undefined);
+  if (containerCreated) {
+    await docker(["rm", "-f", "-v", containerName]).catch(() => undefined);
+  }
+  if (jobContainerAttached) {
+    await docker(["network", "disconnect", networkName, jobContainer]).catch(() => undefined);
+  }
+  if (networkCreated) {
+    await docker(["network", "rm", networkName]).catch(() => undefined);
   }
   if (certificateDirectory) {
     // Removes the generated CA, server certificate, and every private key.
@@ -236,63 +381,79 @@ afterAll(async () => {
 describe.sequential("production PostgreSQL TLS policy against a real TLS endpoint", () => {
   it("starts a disposable TLS-enabled PostgreSQL with runtime-generated certificates", async () => {
     certificateDirectory = await mkdtemp(path.join(tmpdir(), "ekklesia-tls-"));
-    await generateCertificates(certificateDirectory);
-    boundPort = await startTlsPostgres(certificateDirectory);
-    const ca = await readFile(path.join(certificateDirectory, "ca.crt"), "utf8");
-    await waitForTlsReadiness(boundPort, ca);
-    expect(boundPort).toBeGreaterThan(0);
+    if (containerizedJob) {
+      await docker(["network", "create", networkName]);
+      networkCreated = true;
+    }
+    // The certificate identity is generated from the host this run will actually connect through.
+    identityHost = containerizedJob ? containerName : "localhost";
+    await generateCertificates(certificateDirectory, identityHost);
+    await startTlsPostgres(certificateDirectory);
+    await resolveEndpoint();
+
+    const endpoint = requireEndpoint();
+    await waitForTlsReadiness(
+      endpoint.identity,
+      endpoint.port,
+      await readCertificateAuthority("ca.crt"),
+    );
+    expect(endpoint.port).toBeGreaterThan(0);
+    expect(endpoint.mismatch).not.toBe(endpoint.identity);
   });
 
   it("connects successfully, and encrypted, when the certificate is trusted and its identity matches the host", async () => {
-    const ca = await readFile(path.join(certificateDirectory as string, "ca.crt"), "utf8");
+    const { identity, port } = requireEndpoint();
+    const ca = await readCertificateAuthority("ca.crt");
     const ssl = databaseTlsConnectionOptions(
-      `postgresql://ekklesia_tls@localhost:${boundPort}/ekklesia_tls`,
+      `postgresql://ekklesia_tls@${identity}:${port}/ekklesia_tls`,
       { certificateAuthority: ca, mode: "verify-full" },
     );
 
-    const { encrypted } = await connectWith("localhost", boundPort as number, ssl);
+    const { encrypted } = await connectWith(identity, port, ssl);
     expect(encrypted).toBe(true);
   });
 
   it("refuses a certificate that does not chain to the configured CA", async () => {
-    const otherCa = await readFile(
-      path.join(certificateDirectory as string, "other-ca.crt"),
-      "utf8",
-    );
+    const { identity, port } = requireEndpoint();
+    const otherCa = await readCertificateAuthority("other-ca.crt");
     const ssl = databaseTlsConnectionOptions(
-      `postgresql://ekklesia_tls@localhost:${boundPort}/ekklesia_tls`,
+      `postgresql://ekklesia_tls@${identity}:${port}/ekklesia_tls`,
       { certificateAuthority: otherCa, mode: "verify-full" },
     );
 
-    await expect(connectWith("localhost", boundPort as number, ssl)).rejects.toMatchObject({
+    await expect(connectWith(identity, port, ssl)).rejects.toMatchObject({
       code: "UNABLE_TO_VERIFY_LEAF_SIGNATURE",
     });
   });
 
   it("refuses a trusted certificate whose identity does not match the host being connected to", async () => {
-    // The server certificate's only SAN is `DNS:localhost`, so reaching the identical server via
-    // its IP address is a genuine hostname mismatch. This is the case the pinned driver does *not*
-    // catch on its own: `pg` sets `servername` only for non-IP hosts, and Node performs no identity
-    // check when it is unset, so `rejectUnauthorized: true` alone would accept this connection.
-    const ca = await readFile(path.join(certificateDirectory as string, "ca.crt"), "utf8");
+    // The certificate carries no IP SAN, so reaching the identical server via its address is a
+    // genuine hostname mismatch. This is the case the pinned driver does *not* catch on its own:
+    // `pg` sets `servername` only for non-IP hosts, and with it unset Node checks
+    // `NODE_FALLBACK_IDENTITY` instead of the endpoint — so `rejectUnauthorized: true` alone would
+    // accept this connection, as the control below demonstrates.
+    const { mismatch, port } = requireEndpoint();
+    const ca = await readCertificateAuthority("ca.crt");
     const ssl = databaseTlsConnectionOptions(
-      `postgresql://ekklesia_tls@127.0.0.1:${boundPort}/ekklesia_tls`,
+      `postgresql://ekklesia_tls@${mismatch}:${port}/ekklesia_tls`,
       { certificateAuthority: ca, mode: "verify-full" },
     );
 
-    await expect(connectWith("127.0.0.1", boundPort as number, ssl)).rejects.toMatchObject({
+    await expect(connectWith(mismatch, port, ssl)).rejects.toMatchObject({
       code: "ERR_TLS_CERT_ALTNAME_INVALID",
     });
   });
 
   it("proves the gap the policy closes: chain-only verification would accept that same mismatch", async () => {
-    const ca = await readFile(path.join(certificateDirectory as string, "ca.crt"), "utf8");
+    const { identity, mismatch, port } = requireEndpoint();
+    const ca = await readCertificateAuthority("ca.crt");
     // Exactly what `ssl: { ca, rejectUnauthorized: true }` alone does — no pinned
-    // `checkServerIdentity`. It connects, which is why the policy supplies one.
-    const { encrypted } = await connectWith("127.0.0.1", boundPort as number, {
-      ca,
-      rejectUnauthorized: true,
-    });
+    // `checkServerIdentity`. It connects to an address the certificate does not cover, because the
+    // only identity Node had to check was `NODE_FALLBACK_IDENTITY`. The endpoint is never verified
+    // at all, which is why the policy supplies its own check.
+    expect(mismatch).not.toBe(identity);
+    expect(net.isIP(mismatch)).toBeGreaterThan(0);
+    const { encrypted } = await connectWith(mismatch, port, { ca, rejectUnauthorized: true });
     expect(encrypted).toBe(true);
   });
 
@@ -303,7 +464,7 @@ describe.sequential("production PostgreSQL TLS policy against a real TLS endpoin
       throw new TlsHarnessError("The plaintext integration database is not configured.");
     }
     const parsed = new URL(plaintextUrl);
-    const ca = await readFile(path.join(certificateDirectory as string, "ca.crt"), "utf8");
+    const ca = await readCertificateAuthority("ca.crt");
     const ssl = databaseTlsConnectionOptions(plaintextUrl, {
       certificateAuthority: ca,
       mode: "verify-full",