Skip to content

Local E2E Multi-Stack Runbook

Use this runbook when running Playwright E2E tests from more than one SyRF worktree. Each worktree receives its own host ports, Compose project, containers, pid files, logs, Lambda build artifacts, runtime web configuration, and browser auth-cookie names. The port resolver uses lsof plus Python 3's standard-library fcntl locking. lsof is available by default on macOS and can be installed from the standard package repositories on Linux; the existing Python 3 prerequisite supplies the same kernel-backed lock on both platforms without a separate flock command.

Reuse and state isolation

Reuse cached Docker image layers; recreate stateful containers and their data for independent acceptance runs. Keeping an iterative stack running is useful for debugging, but it does not prove isolation: MongoDB, SQL Server, RabbitMQ, LocalStack and browser sessions can retain state. Browser acceptance runs must also create fresh contexts and regenerate their authentication storage.

Run bash e2e/scripts/test-resource-isolation.sh from a PR worktree to reproduce the image-reuse check. It recreates that worktree's owned E2E stack three times, seeding canaries in MongoDB, SQL Server, a durable RabbitMQ queue and an S3 bucket. It checks that canaries are absent after normal teardown and after a simulated partial teardown, while image IDs remain unchanged. Each cycle verifies that owned containers and the network were removed; an EXIT trap also cleans up on failure. Logs and container/image identities remain in the printed temporary directory. This is a local real-store check, separate from CI cancellation-hook verification and from the full browser smoke suite.

Dedicated CI recovery

The CI stack runs on pomegranate-01 under a run-scoped identity (see Run-scoped stack identity). Each Compose service, network, volume and LocalStack Lambda executor carries org.syrf.e2e.owner (repository, run, attempt, job and shard, e.g. 123:456:1:e2e-tests:1) and org.syrf.e2e.listener. These labels also connect the workflow to the installed root-owned completion hook's listener-scoped fallback. The previous workflow omitted those labels, so that hook could not recognize containers left by a failed or cancelled bring-up.

After checking the listener prerequisites and checking out the trusted head, ci-resources.py recover inventories all fixed-stack candidates and checks each prior owner against GitHub's run-attempt status. Only completed attempts from the same repository qualify. A live object owned by another shard of the same repository, run and attempt is a sibling: neither recovery nor cleanup removes it or treats it as a blocker. Other active owners, missing ownership, foreign network attachments and volumes used by foreign containers stop recovery before any deletion, with all blockers listed. Owners written before shards existed (repo:run:attempt:e2e-tests) remain recoverable once their attempt completed. Legacy resources without owner labels need separate provenance investigation; a matching name is insufficient.

Normal teardown checks the current owner before removal, retries partial Docker failures and verifies absence. The completion hook handles hard cancellation. Unrelated PR label events have a distinct concurrency group so they cannot cancel an active suite. The additional cleanup step has time reserved in the test budget. The job needs actions: read to prove prior attempts completed; it gains no write permission. No product feature flag is needed for this CI-only repair.

Validate recovery with python3 e2e/scripts/test-ci-resources.py and the workflow contract with bash .github/scripts/test-e2e-concurrency.sh. A green trigger job with a skipped E2E job is not test evidence: record the executed smoke count, report artifacts and cleanup result, followed by a second clean-start run.

Prerequisites

  • Docker
  • .NET 10 SDK
  • Node.js 22 and pnpm
  • Python 3 (also used for atomic reservation retirement and Lambda tooling)
  • lsof
  • Chromium for Playwright: cd e2e && pnpm exec playwright install chromium

Start a stack

From the worktree root:

bash e2e/run-local.sh --spec infrastructure-isolation --iterative

The first run builds the required services. Do not use --skip-build until the worktree has current binaries. --iterative leaves this worktree's Docker infrastructure running for subsequent test runs; the host .NET processes still stop when the command exits.

Direct Playwright commands use the same resolver automatically, but they do not start the API, PM, or Quartz host processes. Use them only while those services are already running for this worktree's resolved allocation:

cd e2e
pnpm exec playwright test tests/infrastructure-isolation.spec.ts --project=@full

Do not run docker compose with the file's fallback defaults. That bypasses the reservation and worktree ownership environment. For infrastructure-only setup, use the ownership-aware script:

bash e2e/scripts/setup.sh

Inspect the allocation

source e2e/scripts/ports.sh
e2e_print_ports

For a pr<N>.<slug> worktree, the preferred allocation is:

slot   = PR number mod 41
offset = slot * 137
port   = service base port + offset

There are 41 slots. The 137-port step prevents ports from different services and stacks overlapping, while keeping the highest possible assigned port below Linux's default ephemeral range. Offset 0 preserves the established CI ports.

Every Compose container is labelled with org.syrf.e2e.worktree. A resolver atomically creates /tmp/syrf-e2e-port-reservations/offset-<offset> before it checks ports or starts Compose. The reservation records the canonical worktree, a stable owner token, and the owning process identity. A second worktree that maps to the same preferred slot therefore advances before either stack binds a port; it cannot observe and accept the same free slot.

A reservation is reusable only by the same resolver lifecycle or by the worktree whose labelled stack is already live. Before allocating, the resolver scans every slot for that worktree's existing stack, so a prior fallback remains stable even if the preferred slot later becomes free. A dead owner is reclaimed only after confirming that it has no matching container and no listener remains in the slot. Release and stale reclamation atomically rename the directory to an inode-specific hidden tombstone; a delayed old owner therefore cannot delete a new reservation created at the same offset. External listeners make the slot unavailable and the resolver releases its new reservation before trying the next slot. An explicit E2E_PORT_OFFSET=<number> must be one of the 41 supported offsets (slot * 137) and fails if that slot is occupied rather than sharing another stack. Stopped deterministic containers also occupy a slot unless their worktree ownership label matches the current checkout.

Run-scoped stack identity (E2E_STACK_ID)

Set E2E_STACK_ID to run more than one stack from a single checkout. Nothing changes for a developer who leaves it unset.

E2E_STACK_ID=proof-a bash e2e/scripts/setup.sh
E2E_STACK_ID=proof-a bash -c 'source e2e/scripts/ports.sh && e2e_print_ports'
E2E_STACK_ID=proof-a bash e2e/scripts/teardown.sh

The id must be lowercase [a-z0-9] groups joined by single hyphens, at most 48 characters and not all digits (that would alias syrf-e2e-<offset>). Anything else is rejected rather than rewritten, so two different ids can never map to one stack. Port probing uses lsof and, where available, ss, because an unprivileged lsof cannot see root-owned docker-proxy listeners. With an id:

  • E2E_STACK is syrf-e2e-<id> whatever offset is allocated, and cookie, pid and log names follow it.
  • The offset is still allocated from the reservation root, starting at the worktree's preferred slot.
  • The reservation owner and each container's org.syrf.e2e.worktree label are stack:<id> (exported as E2E_STACK_OWNER), not the worktree path. Two ids in one checkout therefore never adopt, tear down or release each other's stack.
  • A later resolver with the same id adopts that id's reservation. If the reservation is live but cannot be proved its own (different token, owner still running, no labelled stack yet) the resolver fails instead of allocating a second slot for the same id; a dead owner's reservation is reclaimed in place.
  • The Playwright runtime feature-flag lock is keyed by the id, because runtime flags belong to one API and Mongo instance.

CI sets E2E_STACK_ID=<run_id>-<run_attempt>-<shard>. Its "Allocate run-scoped E2E stack" step sources ports.sh once with the job's Runner.Worker as the reservation owner process (so a dead job's slot becomes reclaimable), then exports the offset, reservation token, stack name, owner and every port to GITHUB_ENV. Later steps and the Playwright resolver adopt that reservation; no step names a literal port or syrf-e2e project. Stack state lives under $RUNNER_TEMP/syrf-e2e-job/<stack-id>, while reservations and toolchain caches are shared one level up. CI copies Compose, mock-oidc/ and the ports, setup, teardown, Lambda and service-env scripts into <stack-id>/exec, patches and runs them there (E2E_REPO_ROOT tells deploy-lambda.sh where the checkout is), and sets E2E_AUTH_DIR and E2E_RESULTS_DIR so Playwright writes auth state, traces and reports under the job root instead of e2e/. Both variables default to the checkout locally. The Angular runtime overlay and build cache still live in the checkout because ng serve runs there. The .NET services receive their endpoints from e2e/scripts/service-env.sh, which run-local.sh also uses.

bash e2e/scripts/prove-two-stacks.sh is the local acceptance proof for two ids on one checkout: both stacks run at once on distinct slots, a runtime flag enabled on one API is not visible on the other, hard-killing one owner's process group leaves the other's containers, network, reservation and API intact, and teardown of the killed id removes only that stack. It starts real Docker stacks, so run it on a workstation rather than a CI listener; --dry-run prints the commands.

Browser session isolation

Browser cookies are scoped by hostname, not port. run-local.sh therefore sets BffAuth:SessionCookieName to syrf-session-$E2E_STACK and BffAuth:PkceCookieName to syrf-pkce-$E2E_STACK. The API uses those same configured names for session authentication and for the complete OIDC PKCE issuance, callback, and deletion lifecycle. Two localhost stacks can consequently remain signed in and complete simultaneous interactive logins in one browser profile without overwriting each other's auth cookies.

Artifacts and cleanup

Generated process files are scoped by the resolved $E2E_STACK:

  • /tmp/$E2E_STACK-{api,pm,quartz}.{pid,log}
  • /tmp/$E2E_STACK-lambda-publish
  • /tmp/$E2E_STACK-lambda.zip

Normal teardown is ownership-aware and must be run from the same worktree. It also enumerates the exact stack network before removing any LocalStack Lambda executor that Docker Compose does not manage. The reservation is released only after the matching labelled stack is gone and both the worktree and owner token match:

source e2e/scripts/ports.sh
bash e2e/scripts/teardown.sh

Do not stop containers or processes by an unscoped name. If diagnosing a stale container, inspect its org.syrf.e2e.worktree label before taking action.

Verify isolation logic

bash e2e/scripts/ports.test.sh

The test covers preferred-slot selection, labelled-stack and prior-fallback reuse, fallback when a slot belongs elsewhere, concurrent same-slot starts, stale-owner recovery, ownership-checked release, external listener fallback, explicit-offset refusal, the cross-stack collision invariant, the ephemeral-port ceiling, and run-scoped ids: id sanitisation, two ids from one checkout on distinct slots, same-id re-adoption, refusal to adopt a live same-id reservation without its token, sibling teardown/release refusal and dead-owner reclamation.

The Pomegranate CI setup accepts the Compose file’s stack-scoped LocalStack identity and the deployment script’s existing bucket-versioning and notifier metadata setup. It only injects missing legacy setup, so native initialization is not duplicated before browser tests.

CI performance work

The measured performance plan separates runtime build reduction from multi-stack CI. Local multi-stack support does not by itself make two shards in one writable checkout safe, or add concurrent runner capacity.