feat: deliver module objects directly to browsers through opaque read URLs
ADR 0030 left exactly one way to get an image to a browser: read the
object into the module's process and write it back out of a module HTTP
route. Every byte of every asset travelled storage -> Ekklesia -> browser.
For the slideshow this platform exists to replace that means Ekklesia in
the path of every image on every display, permanently, for bytes that
never change — and it would make Ekklesia responsible for re-implementing
byte ranges, without which video seeking cannot work at all.
A module can now ask a logical bucket for a temporary browser URL:
const { url, expiresAt } = await assets.createReadUrl("slides/welcome.png");
The contract names intent, not mechanism. There is no signing, expiry
format, endpoint, or bucket in it — a host with no S3 anywhere could
satisfy the same method, which is what keeps a future single-module
standalone host able to implement it. The URL is opaque to module code and
a bearer capability to whoever holds it, so a module authorizes the
request in its own route before asking: storage bounds reach, never
identity. Lifetimes run one minute to six hours, fifteen by default, and
out-of-range values are refused rather than clamped, so "expires forever"
is unreachable. Logical keys stay canonical state; URLs are generated on
demand and never persisted or logged.
The provider endpoint and the browser delivery origin are separated
because measurement showed they cannot be collapsed: an S3-compatible
signature covers the request Host, so a URL signed for one origin and
rewritten to another is rejected. The browser-facing origin is therefore a
signing input (S3_DELIVERY_ENDPOINT), defaulting to S3_ENDPOINT — so the
standard deployment configures nothing and delivery just works. The scheme
is not signed, so a reverse proxy may terminate TLS provided it passes the
original Host through; that is pinned by a test rather than described and
hoped for.
Read URLs are stable within a window of one third of their lifetime.
Signing is a pure function of the clock, so naive re-signing yields a new
URL per second — a guaranteed browser cache miss, and the reference client
keys its <img> by URL, so an unchanged image would be torn down and
re-downloaded continuously. Quantizing the signing timestamp makes
repeated calls byte-identical while leaving at least two-thirds of the
lifetime; Cache-Control is set at signing time to the guaranteed
remainder, because the pinned SeaweedFS image sends none of its own.
Verified against live SeaweedFS: byte ranges answered directly with 206
and a correct Content-Range, so no range proxy is needed in Core; signing
is local (200 URLs in 66ms), so no batch API is needed; unsigned requests
for the same object are refused, so delivery never depends on a public
bucket; tampered signatures, swapped keys, and swapped buckets all
rejected. Signed write and delete URLs are deliberately not built.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
9a3b8dab8f
commit
7ff063b87a
+83
-17
@@ -7,6 +7,9 @@ import {
|
||||
export type StorageConfigErrorCode =
|
||||
| "STORAGE_ACCESS_KEY_MISSING"
|
||||
| "STORAGE_BUCKET_PREFIX_INVALID"
|
||||
| "STORAGE_DELIVERY_ENDPOINT_INSECURE_IN_PRODUCTION"
|
||||
| "STORAGE_DELIVERY_ENDPOINT_INVALID"
|
||||
| "STORAGE_DELIVERY_ENDPOINT_WITH_CREDENTIALS"
|
||||
| "STORAGE_ENDPOINT_INSECURE_IN_PRODUCTION"
|
||||
| "STORAGE_ENDPOINT_INVALID"
|
||||
| "STORAGE_ENDPOINT_WITH_CREDENTIALS"
|
||||
@@ -16,6 +19,12 @@ export type StorageConfigErrorCode =
|
||||
const storageConfigMessages: Readonly<Record<StorageConfigErrorCode, string>> = Object.freeze({
|
||||
STORAGE_ACCESS_KEY_MISSING: "A storage access key is required when a storage endpoint is set.",
|
||||
STORAGE_BUCKET_PREFIX_INVALID: "The storage bucket prefix must be lowercase kebab-case.",
|
||||
STORAGE_DELIVERY_ENDPOINT_INSECURE_IN_PRODUCTION:
|
||||
"A production browser delivery endpoint must use https, or browsers will refuse to load from it.",
|
||||
STORAGE_DELIVERY_ENDPOINT_INVALID:
|
||||
"The browser delivery endpoint must be an absolute http or https URL.",
|
||||
STORAGE_DELIVERY_ENDPOINT_WITH_CREDENTIALS:
|
||||
"The browser delivery endpoint must not embed credentials.",
|
||||
STORAGE_ENDPOINT_INSECURE_IN_PRODUCTION:
|
||||
"A production storage endpoint must use https, exactly as the database transport must use TLS.",
|
||||
STORAGE_ENDPOINT_INVALID: "The storage endpoint must be an absolute http or https URL.",
|
||||
@@ -51,6 +60,8 @@ export interface StorageProviderConfigInput {
|
||||
readonly secretKey?: string | undefined;
|
||||
readonly forcePathStyle?: string | boolean | undefined;
|
||||
readonly bucketPrefix?: string | undefined;
|
||||
/** The origin a **browser** reaches object storage on, when that differs from `endpoint`. */
|
||||
readonly deliveryEndpoint?: string | undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -70,6 +81,53 @@ export interface StorageProviderConfig {
|
||||
* deployment does not have. */
|
||||
readonly forcePathStyle: boolean;
|
||||
readonly bucketPrefix: string;
|
||||
/**
|
||||
* The origin browser-facing read URLs are signed for.
|
||||
*
|
||||
* Equal to `endpoint` unless the deployment separates them, which is the common case once Core and
|
||||
* the storage service share a private network: Core reaches the service at something like
|
||||
* `http://object-storage:8333`, a name that means nothing in a browser, while the browser reaches
|
||||
* it through a public name.
|
||||
*
|
||||
* The two roles cannot be collapsed by rewriting a URL after the fact. An S3-compatible signature
|
||||
* covers the request's `Host` header, so a URL signed for one host and then re-pointed at another
|
||||
* fails verification — which is why this is a *signing* input rather than a post-processing step.
|
||||
* What the signature does **not** cover is the scheme, so terminating TLS at a reverse proxy and
|
||||
* forwarding plaintext to the storage service verifies correctly, provided the proxy passes the
|
||||
* original `Host` through unchanged.
|
||||
*/
|
||||
readonly deliveryEndpoint: string;
|
||||
}
|
||||
|
||||
interface EndpointErrorCodes {
|
||||
readonly invalid: StorageConfigErrorCode;
|
||||
readonly withCredentials: StorageConfigErrorCode;
|
||||
readonly insecure: StorageConfigErrorCode;
|
||||
}
|
||||
|
||||
/** Applies one rule set to both endpoints, so the browser-facing origin can never be validated more
|
||||
* loosely than the one Core itself uses. */
|
||||
function normalizeEndpoint(
|
||||
value: string,
|
||||
environment: StorageEnvironment,
|
||||
codes: EndpointErrorCodes,
|
||||
): string {
|
||||
let parsed: URL;
|
||||
try {
|
||||
parsed = new URL(value);
|
||||
} catch {
|
||||
throw new StorageConfigError(codes.invalid);
|
||||
}
|
||||
if (parsed.protocol !== "http:" && parsed.protocol !== "https:") {
|
||||
throw new StorageConfigError(codes.invalid);
|
||||
}
|
||||
if (parsed.username !== "" || parsed.password !== "") {
|
||||
throw new StorageConfigError(codes.withCredentials);
|
||||
}
|
||||
if (environment === "production" && parsed.protocol !== "https:") {
|
||||
throw new StorageConfigError(codes.insecure);
|
||||
}
|
||||
return parsed.origin + (parsed.pathname === "/" ? "" : parsed.pathname.replace(/\/+$/u, ""));
|
||||
}
|
||||
|
||||
const DEFAULT_REGION = "us-east-1";
|
||||
@@ -114,22 +172,25 @@ export function createStorageProviderConfig(
|
||||
return undefined;
|
||||
}
|
||||
|
||||
let parsed: URL;
|
||||
try {
|
||||
parsed = new URL(endpoint);
|
||||
} catch {
|
||||
throw new StorageConfigError("STORAGE_ENDPOINT_INVALID");
|
||||
}
|
||||
if (parsed.protocol !== "http:" && parsed.protocol !== "https:") {
|
||||
throw new StorageConfigError("STORAGE_ENDPOINT_INVALID");
|
||||
}
|
||||
if (parsed.username !== "" || parsed.password !== "") {
|
||||
throw new StorageConfigError("STORAGE_ENDPOINT_WITH_CREDENTIALS");
|
||||
}
|
||||
const environment = options.environment ?? "production";
|
||||
if (environment === "production" && parsed.protocol !== "https:") {
|
||||
throw new StorageConfigError("STORAGE_ENDPOINT_INSECURE_IN_PRODUCTION");
|
||||
}
|
||||
const normalizedEndpoint = normalizeEndpoint(endpoint, environment, {
|
||||
insecure: "STORAGE_ENDPOINT_INSECURE_IN_PRODUCTION",
|
||||
invalid: "STORAGE_ENDPOINT_INVALID",
|
||||
withCredentials: "STORAGE_ENDPOINT_WITH_CREDENTIALS",
|
||||
});
|
||||
|
||||
// Unset means the one endpoint serves both roles, which is correct whenever Core and the browser
|
||||
// reach storage by the same name — the repository's own development stack publishes the service on
|
||||
// loopback, so the ordinary developer configures nothing here at all.
|
||||
const deliveryInput = input?.deliveryEndpoint?.trim();
|
||||
const deliveryEndpoint =
|
||||
deliveryInput === undefined || deliveryInput === ""
|
||||
? normalizedEndpoint
|
||||
: normalizeEndpoint(deliveryInput, environment, {
|
||||
insecure: "STORAGE_DELIVERY_ENDPOINT_INSECURE_IN_PRODUCTION",
|
||||
invalid: "STORAGE_DELIVERY_ENDPOINT_INVALID",
|
||||
withCredentials: "STORAGE_DELIVERY_ENDPOINT_WITH_CREDENTIALS",
|
||||
});
|
||||
|
||||
const region = input?.region?.trim() === "" ? undefined : input?.region?.trim();
|
||||
if (region !== undefined && !REGION_PATTERN.test(region)) {
|
||||
@@ -160,7 +221,8 @@ export function createStorageProviderConfig(
|
||||
bucketPrefix === undefined || bucketPrefix === ""
|
||||
? DEFAULT_STORAGE_BUCKET_PREFIX
|
||||
: bucketPrefix,
|
||||
endpoint: parsed.origin + (parsed.pathname === "/" ? "" : parsed.pathname.replace(/\/+$/u, "")),
|
||||
deliveryEndpoint,
|
||||
endpoint: normalizedEndpoint,
|
||||
forcePathStyle: parseBoolean(input?.forcePathStyle, true),
|
||||
region: region ?? DEFAULT_REGION,
|
||||
secretKey,
|
||||
@@ -177,9 +239,13 @@ export function createStorageProviderConfig(
|
||||
*/
|
||||
export function describeStorageProvider(config: StorageProviderConfig): string {
|
||||
const origin = new URL(config.endpoint).origin;
|
||||
const deliveryOrigin = new URL(config.deliveryEndpoint).origin;
|
||||
// Named only when it differs, so the ordinary single-origin deployment reads exactly as it did
|
||||
// before this distinction existed. Both are host infrastructure names, never credentials.
|
||||
const delivery = deliveryOrigin === origin ? "" : ` · browser delivery ${deliveryOrigin}`;
|
||||
return `S3-compatible endpoint ${origin} · region ${config.region} · ${
|
||||
config.forcePathStyle ? "path-style" : "virtual-host-style"
|
||||
} · bucket prefix "${config.bucketPrefix}"`;
|
||||
} · bucket prefix "${config.bucketPrefix}"${delivery}`;
|
||||
}
|
||||
|
||||
export { STORAGE_BUCKET_PREFIX_MAX_LENGTH };
|
||||
|
||||
@@ -1,15 +1,21 @@
|
||||
import {
|
||||
EMPTY_MODULE_STORAGE,
|
||||
isValidModuleStorageKey,
|
||||
isValidModuleStorageReadUrlFilename,
|
||||
MODULE_STORAGE_READ_URL_DEFAULT_SECONDS,
|
||||
MODULE_STORAGE_READ_URL_MAX_SECONDS,
|
||||
MODULE_STORAGE_READ_URL_MIN_SECONDS,
|
||||
type ModuleStorage,
|
||||
type ModuleStorageBody,
|
||||
type ModuleStorageBucket,
|
||||
ModuleStorageError,
|
||||
type ModuleStorageListOptions,
|
||||
type ModuleStorageListing,
|
||||
type ModuleStorageListOptions,
|
||||
type ModuleStorageObject,
|
||||
type ModuleStorageObjectMetadata,
|
||||
type ModuleStoragePutOptions,
|
||||
type ModuleStorageReadUrl,
|
||||
type ModuleStorageReadUrlOptions,
|
||||
} from "@ekklesia/contracts";
|
||||
import { type ModuleStorageBinding, resolveModuleStorageBindings } from "./binding.js";
|
||||
import type { StorageProvider } from "./provider.js";
|
||||
@@ -58,6 +64,27 @@ function assertValidKey(key: unknown): asserts key is string {
|
||||
throw new ModuleStorageError("STORAGE_KEY_INVALID", echo);
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves a requested read-URL lifetime against the contract's bounds.
|
||||
*
|
||||
* Enforced here rather than in a provider so every provider agrees on what a module may ask for, and
|
||||
* so an out-of-range request is refused rather than quietly clamped — a module that asks for a
|
||||
* week's lifetime has a misunderstanding worth surfacing, not a value worth rounding.
|
||||
*/
|
||||
function resolveExpirySeconds(requested: number | undefined): number {
|
||||
if (requested === undefined) {
|
||||
return MODULE_STORAGE_READ_URL_DEFAULT_SECONDS;
|
||||
}
|
||||
if (
|
||||
!Number.isInteger(requested) ||
|
||||
requested < MODULE_STORAGE_READ_URL_MIN_SECONDS ||
|
||||
requested > MODULE_STORAGE_READ_URL_MAX_SECONDS
|
||||
) {
|
||||
throw new ModuleStorageError("STORAGE_READ_URL_EXPIRY_INVALID");
|
||||
}
|
||||
return requested;
|
||||
}
|
||||
|
||||
function createBucketHandle(
|
||||
provider: StorageProvider,
|
||||
binding: ModuleStorageBinding,
|
||||
@@ -69,6 +96,30 @@ function createBucketHandle(
|
||||
// where the host keeps its data.
|
||||
id: binding.logicalBucketId,
|
||||
|
||||
async createReadUrl(
|
||||
key: string,
|
||||
options: ModuleStorageReadUrlOptions = {},
|
||||
): Promise<ModuleStorageReadUrl> {
|
||||
assertValidKey(key);
|
||||
const expiresInSeconds = resolveExpirySeconds(options.expiresIn);
|
||||
if (
|
||||
options.downloadFilename !== undefined &&
|
||||
!isValidModuleStorageReadUrlFilename(options.downloadFilename)
|
||||
) {
|
||||
throw new ModuleStorageError("STORAGE_READ_URL_FILENAME_INVALID", key);
|
||||
}
|
||||
// The bucket is ensured for the same reason every other operation ensures it: a URL for a
|
||||
// bucket that was never created would resolve to a failure the browser reports rather than one
|
||||
// the module can see. Memoized per binding, so a module signing forty slides pays for it once.
|
||||
await ensure();
|
||||
return provider.createReadUrl(binding.physicalBucket, key, {
|
||||
expiresInSeconds,
|
||||
...(options.downloadFilename === undefined
|
||||
? {}
|
||||
: { downloadFilename: options.downloadFilename }),
|
||||
});
|
||||
},
|
||||
|
||||
async delete(key: string): Promise<void> {
|
||||
assertValidKey(key);
|
||||
await ensure();
|
||||
|
||||
@@ -1,10 +1,11 @@
|
||||
import type {
|
||||
ModuleStorageBody,
|
||||
ModuleStorageListOptions,
|
||||
ModuleStorageListing,
|
||||
ModuleStorageListOptions,
|
||||
ModuleStorageObject,
|
||||
ModuleStorageObjectMetadata,
|
||||
ModuleStoragePutOptions,
|
||||
ModuleStorageReadUrl,
|
||||
} from "@ekklesia/contracts";
|
||||
|
||||
/**
|
||||
@@ -42,6 +43,25 @@ export interface StorageProvider {
|
||||
options: ModuleStoragePutOptions,
|
||||
): Promise<ModuleStorageObjectMetadata>;
|
||||
delete(physicalBucket: string, key: string): Promise<void>;
|
||||
/**
|
||||
* A URL a browser can read one object from directly, valid for `expiresIn` seconds.
|
||||
*
|
||||
* The seconds and the download filename arrive already validated against the contract's bounds —
|
||||
* a provider enforces neither, so two providers can never disagree about what a module is allowed
|
||||
* to ask for.
|
||||
*
|
||||
* A provider that cannot issue such a URL has no business implementing this interface: the
|
||||
* facility is part of the storage contract, not an optional extra a module has to feel around for.
|
||||
* An implementation with no notion of signing could satisfy it by minting its own opaque token.
|
||||
*/
|
||||
createReadUrl(
|
||||
physicalBucket: string,
|
||||
key: string,
|
||||
options: {
|
||||
readonly expiresInSeconds: number;
|
||||
readonly downloadFilename?: string | undefined;
|
||||
},
|
||||
): Promise<ModuleStorageReadUrl>;
|
||||
/** A cheap reachability probe for diagnostics only. Its result is **never** persisted and never
|
||||
* influences a module's lifecycle state — a provider being briefly unreachable is an operational
|
||||
* condition, not an administrative decision (ADR 0030). */
|
||||
|
||||
@@ -9,6 +9,7 @@ import {
|
||||
PutObjectCommand,
|
||||
S3Client,
|
||||
} from "@aws-sdk/client-s3";
|
||||
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
|
||||
import {
|
||||
type ModuleStorageBody,
|
||||
ModuleStorageError,
|
||||
@@ -17,6 +18,7 @@ import {
|
||||
type ModuleStorageObject,
|
||||
type ModuleStorageObjectMetadata,
|
||||
type ModuleStoragePutOptions,
|
||||
type ModuleStorageReadUrl,
|
||||
} from "@ekklesia/contracts";
|
||||
import type { StorageProviderConfig } from "./config.js";
|
||||
import type { StorageProvider } from "./provider.js";
|
||||
@@ -173,6 +175,43 @@ const DEFAULT_TRANSPORT_BOUNDS: S3TransportBounds = Object.freeze({
|
||||
socketIdleTimeoutMs: 5_000,
|
||||
});
|
||||
|
||||
/**
|
||||
* How coarsely a read URL's signing timestamp is rounded down, as a fraction of its lifetime.
|
||||
*
|
||||
* Signing is a pure function of the request and the clock, so signing the same object twice a second
|
||||
* apart yields two different URLs for the same bytes. That is worse than untidy. A browser caches by
|
||||
* URL, so a changed query string is a guaranteed cache miss, and UI frameworks routinely key list
|
||||
* items by URL — the reference consumer this was measured against renders `<img key={url}>`, so a
|
||||
* new URL for an unchanged image tears the element down and re-downloads it. A slideshow rebuilding
|
||||
* its state every few seconds would therefore re-fetch every slide continuously, having gained a
|
||||
* direct path to storage and lost the caching that made it worth having.
|
||||
*
|
||||
* Rounding the signing time down to a window makes repeated calls return a **byte-identical** URL
|
||||
* within that window, so caches hit and keyed elements stay mounted. Dividing by three, rather than
|
||||
* fixing a duration, keeps the guarantee proportional: whatever lifetime is asked for, at least
|
||||
* two-thirds of it always remains when a URL is handed out.
|
||||
*/
|
||||
const READ_URL_STABILITY_DIVISOR = 3;
|
||||
|
||||
interface ReadUrlWindow {
|
||||
readonly signedAt: Date;
|
||||
readonly expiresAt: Date;
|
||||
readonly guaranteedSeconds: number;
|
||||
}
|
||||
|
||||
function readUrlWindow(nowMs: number, expiresInSeconds: number): ReadUrlWindow {
|
||||
const windowSeconds = Math.max(1, Math.floor(expiresInSeconds / READ_URL_STABILITY_DIVISOR));
|
||||
const windowMs = windowSeconds * 1000;
|
||||
const signedAtMs = Math.floor(nowMs / windowMs) * windowMs;
|
||||
return {
|
||||
expiresAt: new Date(signedAtMs + expiresInSeconds * 1000),
|
||||
// Deliberately derived from the window rather than from the current clock: anything that varies
|
||||
// second by second would end up in the signed query string and undo the stability above.
|
||||
guaranteedSeconds: expiresInSeconds - windowSeconds,
|
||||
signedAt: new Date(signedAtMs),
|
||||
};
|
||||
}
|
||||
|
||||
export function createS3StorageProvider(
|
||||
config: StorageProviderConfig,
|
||||
/** Tightened only by the live provider suite, so it can prove the deadline actually fires without
|
||||
@@ -197,6 +236,25 @@ export function createS3StorageProvider(
|
||||
},
|
||||
});
|
||||
|
||||
/**
|
||||
* The client read URLs are signed with.
|
||||
*
|
||||
* An S3-compatible signature covers the request's `Host`, so a URL cannot be signed against the
|
||||
* endpoint Core talks to and then re-pointed at the origin a browser can reach — verification
|
||||
* fails. The delivery origin therefore has to be a *signing input*, which is what this second
|
||||
* client supplies. It never sends a request: presigning is a local computation, so this opens no
|
||||
* socket and is built only when the two origins actually differ.
|
||||
*/
|
||||
const signingClient =
|
||||
config.deliveryEndpoint === config.endpoint
|
||||
? client
|
||||
: new S3Client({
|
||||
credentials: { accessKeyId: config.accessKey, secretAccessKey: config.secretKey },
|
||||
endpoint: config.deliveryEndpoint,
|
||||
forcePathStyle: config.forcePathStyle,
|
||||
region: config.region,
|
||||
});
|
||||
|
||||
return Object.freeze({
|
||||
async ensureBucket(physicalBucket: string): Promise<void> {
|
||||
try {
|
||||
@@ -326,6 +384,41 @@ export function createS3StorageProvider(
|
||||
}
|
||||
},
|
||||
|
||||
async createReadUrl(
|
||||
physicalBucket: string,
|
||||
key: string,
|
||||
options: {
|
||||
readonly expiresInSeconds: number;
|
||||
readonly downloadFilename?: string | undefined;
|
||||
},
|
||||
): Promise<ModuleStorageReadUrl> {
|
||||
const window = readUrlWindow(Date.now(), options.expiresInSeconds);
|
||||
try {
|
||||
const url = await getSignedUrl(
|
||||
signingClient,
|
||||
new GetObjectCommand({
|
||||
Bucket: physicalBucket,
|
||||
Key: key,
|
||||
// The pinned SeaweedFS image returns no `Cache-Control` of its own, which leaves
|
||||
// browsers guessing how long an asset may be reused. Stating it turns the stable URL
|
||||
// above into an actual cache hit. It never outlives the URL itself.
|
||||
ResponseCacheControl: `private, max-age=${window.guaranteedSeconds}`,
|
||||
...(options.downloadFilename === undefined
|
||||
? {}
|
||||
: {
|
||||
ResponseContentDisposition: `attachment; filename="${options.downloadFilename}"`,
|
||||
}),
|
||||
}),
|
||||
{ expiresIn: options.expiresInSeconds, signingDate: window.signedAt },
|
||||
);
|
||||
return Object.freeze({ expiresAt: window.expiresAt, url });
|
||||
} catch (error) {
|
||||
// Signing is local, so a failure here is a malformed request rather than an outage — but it
|
||||
// is translated exactly like any other so no SDK exception can reach a module.
|
||||
throw translateStorageError(error, key);
|
||||
}
|
||||
},
|
||||
|
||||
async probe(): Promise<void> {
|
||||
try {
|
||||
// The cheapest call that proves the endpoint is reachable and the credentials sign
|
||||
@@ -338,6 +431,9 @@ export function createS3StorageProvider(
|
||||
|
||||
async close(): Promise<void> {
|
||||
client.destroy();
|
||||
if (signingClient !== client) {
|
||||
signingClient.destroy();
|
||||
}
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user