Skip to content

E2E stack identity and two-shard execution

Slice 2 of the E2E throughput plan. Slice 1 (PR #3473) cut the CI backend build from about 200 s to 40 s; the full suite still exceeds the 45-minute contract because 59 tests need roughly 33 minutes of Playwright time and one runner executes them serially. This slice makes two stacks able to coexist and splits the suite into two shards. It does not add runner capacity: both shards run one after the other on the existing pomegranate-01 lane until slice 3 activates a second listener.

No feature flag: this changes CI execution only. Browser coverage, retries, timeouts, performance budgets and trusted-source gates are unchanged.

Why a stack identity

CI today runs the stack at port offset 0 under the literal Compose project syrf-e2e. Every step that inspects, budgets, monitors or cleans the stack names syrf-e2e, syrf-e2e_default, syrf-e2e-<service> and ports 8080, 8081, 8083 and 15673 directly. The owner label is repo:run:attempt:job with no shard component, and ci-resources.py aborts when any live object has a different owner. Two runs on one host therefore cannot start, let alone run.

The local allocator already parameterises most surfaces from E2E_PORT_OFFSET through E2E_STACK. The gap is that CI fixes the offset, E2E_STACK cannot be set independently, the owner has no shard, and several steps rewrite files inside the checkout.

Contract

E2E_STACK_ID = <run_id>-<run_attempt>-<shard> (shard is 1 or 2). It is the only input; everything below derives from it.

Surface Today After
Port offset E2E_PORT_OFFSET=0 fixed in CI Allocated per shard from the reservation root; CI passes the resolved offset to every step through ports.sh exports, never literal numbers
Compose project / network / containers syrf-e2e, syrf-e2e_default, syrf-e2e-* syrf-e2e-<stack-id> and derived names; E2E_STACK accepts an explicit override
Owner label repo:run:attempt:e2e-tests repo:run:attempt:e2e-tests:<shard>; cleanup treats a live sibling shard of the same run and attempt as legitimate and still aborts on unknown owners
Job root $RUNNER_TEMP/syrf-e2e-job (guarded) $RUNNER_TEMP/syrf-e2e-job/<stack-id>; guard updated; shared caches stay one level up
Checkout mutations compose, deploy-lambda.sh, setup.sh, appConfig.env.json, e2e/.auth written in place Compose, mock-oidc/, ports.sh, setup.sh, teardown.sh, deploy-lambda.sh and service-env.sh are copied into <job root>/exec and patched and run from there; browser auth state lives in <job root>/exec/e2e/.auth (E2E_AUTH_DIR). No workflow step writes into the checkout after the build. Exception: ng serve still runs in the checkout, so runtime-config.ts writes the resolved appConfig.env.json overlay (and its backup) there and the Angular build cache stays shared; redirecting either would need src/services/web changes. Each CI job has its own checkout, so shards never share them
.NET service endpoints overlay defaults (27018, 5673, 4566, 9000, 1434, 4200) __ environment overrides from the resolved ports, as run-local.sh already does
Cookie names literal syrf-session-syrf-e2e derived from E2E_STACK (already exported by ports.sh)
Feature-flag lock keyed by GITHUB_RUN_ID keyed by E2E_STACK_ID; flags are per API instance, so the lock must be too
Results, report, artifacts e2e/test-results, e2e/playwright-report, fixed artifact names <job root>/results, artifact names suffixed with the shard
PR comment one result, one duration per-shard row plus merged verdict

Sharding

  • playwright test --project=@full --shard=<n>/2; smoke stays one unsharded job. The config sets fullyParallel: true, so Playwright shards by test group rather than by file: a serial describe group stays inside one shard, other tests of one file may land in different shards.
  • Implemented as the e2e-tests matrix ([1, 2] for full, [1] for smoke, max-parallel: 1, fail-fast: false). Each shard keeps the full 45-minute job contract and uploads e2e-shard-report-<n> (shard.json, the shard's JSON report and an unsharded --list inventory).
  • Not yet done: balancing by the measured healthy first-attempt durations from handover evidence (run 34905067814). Playwright 1.58 splits groups by count only; the Dockview, performance and annotation specs dominate, so the first main run's per-shard durations decide whether a weighted split is needed.
  • merge-e2e-reports (ordinary Juniper route, read-only, no secrets, base-branch merger e2e/scripts/merge-shard-reports.cjs) is red when the job result is not success, any expected shard is missing, failed or cancelled, shards disagree on the inventory, or any inventoried test ran in zero or several shards (retries are results of one entry; the setup project runs in every shard and is excluded). The PR comment shows per-shard status, duration and test count plus the merged verdict, and is green only when both are.

Acceptance

  1. Contract tests reject any literal syrf-e2e name or fixed port in a CI step and prove every surface above derives from E2E_STACK_ID.
  2. Local proof on one checkout: two stacks with different ids run at once; mutating a runtime flag in one does not affect the other; killing one owner mid-run leaves the other's containers, network and reservation intact; cleanup removes only the terminated stack.
  3. Report merger unit tests: missing, failed or cancelled shard is red; duplicate or absent test ids are red.
  4. First main run after merge (the workflow executes from main only): both shards complete their inventory, per-phase timings are reported, and the sum of shard Playwright time is compared with the serial baseline.

Out of scope

Second listener activation and host provisioning (slice 3, coordinated with the server-config owner), performance-lane isolation, prebuilt web bundle and recording policy (slice 4).