Skip to content

E2E throughput: measured improvements and delivery order

Outcome and scope

The owner requested both failure recovery and faster E2E execution, including parallel stacks within a workflow and across workflows. Deliver independently reviewable slices; do not make a timeout increase stand in for fixing failures. The initial implementation is a runtime-only backend build. Multi-stack CI is the next architectural slice, not an already-enabled capability.

No application feature flag is needed for the build-scope change: it changes which projects CI compiles, not runtime product behaviour. Browser coverage, assertions, retries, resource limits, trusted-source gates and cleanup deadlines remain unchanged. Full unit builds and dedicated generated-contract validation remain separate checks. No deployment or live database operation is included.

Evidence, not a healthy baseline

Allocation integration run tested head 3416e70073b5175b9f3a2e52db81d0bc228d1ba3 on 13 September:

Phase Observed duration Interpretation
Preparation before Playwright 6m 41s Includes infrastructure and backend build
Infrastructure startup 2m Lambda publishing already overlaps container startup
Full backend solution build 3m 17s Builds more than the three launched services
Playwright step 22m 49s Interrupted by its shell budget; not a complete suite
Reporting and cleanup About 38s One observation, not a worst-case bound
Allocation/capacity journey 22.2s Passed on this exact head

Four other test cases (three Dockview cases and ordinary AF2 performance) failed on all three attempts, consuming roughly seven minutes. Repair their causes before estimating healthy-suite speed. The final exit was timeout 124; more time would not itself correct these failures. Recent workflow-level success is not proof of browser execution: inspect the heavy job, selected suite and exact commit, because some trigger routes legitimately skip tests.

Ordered delivery

1. Recover correctness and remove redundant build work

  • Reconcile panel/performance failures with current main and active owners. Fix stale assumptions or application regressions from trace evidence, not by weakening assertions, removing retries or raising performance thresholds.
  • Build e2e/runtime.slnf: API, PM and Quartz roots plus their transitive project references. Lambda remains published by infrastructure setup. Do not use the existing service filters unchanged: they include test projects.
  • Pass GenerateOpenApi=false only in this E2E build. Dedicated OpenAPI/client validation owns generation; E2E uses checked-in clients. Keep analyzers enabled.
  • Pin the exact build command and runtime roots in the existing workflow contract, including missing-runtime and accidental-test-root regressions.
  • Acceptance: real Release build succeeds; unchanged full browser suite executes on the candidate head; all selected cases are accounted for; no generated-file mutation; report setup/build/test timings without claiming unmeasured savings.

2. Reusable immutable build bundle and two isolated functional stacks

Build once per exact source commit. Publish runtime binaries, Lambda package and the E2E web bundle as immutable, checksummed artifacts. Runtime overlays must remain external to the common build. Validate that serving a prebuilt E2E bundle retains its test configuration; do not silently switch performance measurements from the development server to production mode and reuse old baselines.

Give each stack an identity containing run, attempt and shard, not merely the PR number or checkout path. Audit and separate all of:

  • Host ports, Compose project/network/container names and ownership labels.
  • Mongo/SQL data, broker queues, LocalStack buckets, Lambda executor networks.
  • PID/log/temp paths, resource monitor state, reservations and cleanup manifests.
  • Browser auth files, cookie names, runtime web overlays and configuration backups.
  • Test results and report artifact names; feature-flag locks scoped per stack.

Status (2026-09-15, PR #3487, contract in e2e-stack-identity-and-shards): run-scoped stack identity is implemented. CI sets E2E_STACK_ID=<run_id>-<run_attempt>-<shard>; ports.sh allocates a free offset from the reservation root, names the stack syrf-e2e-<stack-id>, and records stack:<stack-id> as the reservation and container owner, so two ids from one checkout never adopt each other's stack. Every workflow step reads the resolved ports and names instead of offset zero and literal syrf-e2e names, the owner label carries the shard, cleanup leaves a live sibling shard alone, the job root is $RUNNER_TEMP/syrf-e2e-job/<stack-id>, and the runtime-flag lock is keyed by stack id. .github/scripts/test-e2e-stack-identity.sh rejects any literal stack name or E2E port in the job. CI also patches and runs Compose and the setup/teardown/Lambda scripts from a private <job root>/exec copy, keeps browser auth state there and writes traces and reports to <job root>/results; the contract rejects any step that writes into the checkout after the build. ng serve still runs in the checkout, so the resolved appConfig.env.json overlay and the Angular build cache remain checkout state (one checkout per CI job, so shards never share them). The full suite runs as two Playwright shards (a sequential matrix on the single pomegranate-e2e listener, each shard keeping the 45-minute contract; smoke stays one shard), and merge-e2e-reports produces a fail-closed verdict: missing, failed or cancelled shards, or any test that ran in zero or several shards, are red. Duration-weighted balancing, a second listener and the immutable build bundle are still future work.

Before that change, local port offsets were useful machinery but not a complete CI solution: CI assumed fixed offset zero, and the allocator could reuse a live stack for the same worktree. Two processes sharing one writable checkout can overwrite runtime overlays/auth state. Use separate scratch execution directories per shard, with validated immutable inputs and private writable outputs; these are runtime sandboxes, not additional implementation worktrees.

Start with two functional shards, one Playwright worker per stack. Keep serial test groups intact, and balance by measured healthy first-attempt duration rather than only counting files. Merge reports into a fail-closed aggregate: missing, failed or cancelled required shards cannot become green. Prove each selected test runs exactly once apart from declared retries/setup.

Acceptance: concurrently mutate different flags/data and prove no cross-stack effect; terminate one owner while the other completes; verify ownership-aware cleanup/recovery, no orphan Lambda executors and no deletion of another stack; compare repeated cold and warm timings against the complete serial baseline.

3. Bounded cross-workflow capacity and performance isolation

The inspected routing contract assigns E2E to the exclusive pomegranate-01 listener. A matrix alone will queue, not run multiple jobs on that listener. Additional E2E capacity requires coordination with the server-config owner, actual CPU/memory/disk measurements and revised routing/cleanup contracts. Do not infer capacity from total host RAM or add runner labels manually.

Enforce a total stack admission limit across workflows: two shards multiplied by several workflows must not create unbounded concurrency. Preserve trusted same-repository execution and fork denial. Keep per-PR superseded-run cancellation; do not add a global Actions concurrency group that displaces unrelated pending work.

Performance tests need dedicated resource capacity or an exclusive measurement window. Separate databases do not isolate CPU, memory bandwidth or disk. Keep performance coverage required in the aggregate even if separated from functional shards. Host provisioning/activation remains an explicitly coordinated follow-up.

4. Smaller improvements after measurement

  • Replace fixed sleeps and broad network-idle waits with observed operation completion. For negative assertions, first prove the attempted operation has finished; immediate unchanged data is not evidence that a write was rejected.
  • Reuse expensive setup within a scenario where state ownership is explicit. Keep representative real import/Lambda flows; use setup APIs for unrelated prerequisites without replacing the behaviour under test with mocks.
  • Consider content-addressed package caches only with integrity, trust and cleanup controls. In the inspected run pnpm install took ten seconds and Chromium preparation twenty: lower priority than builds and test execution.
  • Measure trace/video overhead before changing recording policy. Retain adequate first-failure evidence; removing diagnostics can lengthen the overall fix loop.
  • Avoid optional large fixtures in every functional test; keep essential large cases in the required performance lane. Do not substitute smoke-only coverage or unproven path filtering for the full acceptance suite.
  • Revisit reporting reserve only after repeated failure/slow-upload measurements. The current fifteen-minute allowance covers explicitly bounded post-test steps; changing it requires updating the matching deadline contracts, not just YAML.

Validation and reporting

Initial local evidence on 14 September: the runtime-filter Release build passed with zero errors in 72.8 seconds using two MSBuild workers and shared compilation disabled. Four runtime-root/dependency contract tests and the existing E2E concurrency/routing/cleanup contract passed. This is build correctness evidence, not a controlled comparison with the earlier CI build or complete E2E acceptance.

Track queue time separately from setup, first attempts, retries, reporting and cleanup. Report median and tail wall time, slowest-shard imbalance, peak resources, test inventory, flakes and orphan cleanup failures. Do not promise a twofold speedup: fixed preparation, uneven groups and contention constrain the result. The MVP is successful only when correctness is preserved and observed complete run time improves; until CI proves that, describe changes as implemented but performance-unqualified.

Primary references