Skip to content

FEAT-024 Phase 2C: Staging Proof Runbook

Purpose

Phase 2's staging proof is to "dark-write an allowlisted corpus, run shadow parity and load, force Stale/Rebuilding/Incompatible and kill-switch paths, and prove legacy response equivalence" (technical plan, Phase 2). This runbook is the operational half of that: which gates must close first, what to change, what to run, what each response means, and what evidence to keep.

It covers the project screening family only, on one staging project, with no page, SignalR or export consumer. Phase 2C is explicitly "no ordinary page or SignalR cutover", so the only reader of a materialized value during this proof is the administrator-only parity audit.

What this runbook is not

  • It is not an approval. Running it requires the gates in Preconditions to be closed, including an explicit go from the FEAT-024 programme owner.
  • It is not a production procedure. Nothing here may be pointed at production, and the soak evidence it produces is an input to a later separate production-pilot decision, not that decision.
  • It does not authorize a consumer cutover, a second project, a second family, or any legacy retirement.

Preconditions and gates

# Gate Why
1 syrf #3196 merged The parity audit endpoint lives on this pull request. As of 38a272d4c main has only backfill and rebuild; without #3196 there is no way to compare the projection against the authoritative calculation, which is the entire point of the proof.
2 syrf #3232 merged Binds capacity writes to durable-mode transactions.
2a syrf #3371 merged The administrative mode-transition surface (issue #3369). Without it no production code opens the fleet control or a project's narrow gate, so a completed backfill writes rows that can never serve and the parity audit reports Disabled for every scope — steps 3, 5 and 10 are unperformable and the proof establishes nothing.
3 syrf issue #3185 closed The README's pilot gate: until its single-transaction settings write lands, a Project.AgreementThreshold write can commit between a rebuild's pinned snapshot and its publication without advancing the control's source revision, so a freshly rebuilt row can fail the reader's row/control equality. The pilot must not be activated for any project before this closes.
4 Pilot project chosen and its GUID recorded See Choosing the pilot project.
5 Explicit go from the FEAT-024 programme owner The README holds every flag off and the allowlist empty "until the separately authorized single-project staging activation". This runbook does not grant that authorization.
6 An administrator account on staging Both endpoints sit behind ApplicationAuthorization.BatchAdminProjectsPolicy, whose activity BatchAdminProjects is granted to the administrator application group only. An authenticated non-administrator receives 403; an anonymous caller receives 401.

Confirm the deployed staging API and project-management images actually contain #3196 and #3232 before starting. A promoted chartTag in cluster-gitops is the intent; the running pod's image digest is the fact.

Step 1 — enable the pilot in cluster-gitops

The change is prepared as a draft pull request against camaradesuk/cluster-gitops, held closed behind the gates above. It touches two files:

File Change
syrf/environments/staging/api/values.yaml statistics flags into the existing featureFlags: map; allowlist into the existing env: map
syrf/environments/staging/project-management/values.yaml the same block

Both hosts are required. The API is the serving side and hosts the administrative endpoints; project-management is the source-transaction writing side. Enabling one without the other produces a projection that is either written and never read or read and never maintained.

Flag values

Three on, nine off:

Flag Value Note
materializedProjectStatisticsWrites true Global write kill switch.
materializedProjectStatisticsServing true Global serving kill switch; the runtime catalog also requires the write gate.
materializedProjectStatisticsScreening true The project screening family.
materializedProjectStatisticsMembershipScreening false A separate Phase 4 family, not part of the screening family.
materializedProjectStatisticsAnnotation false
materializedProjectStatisticsMembershipAnnotation false
materializedProjectStatisticsQuestionAnswers false
materializedProjectStatisticsSearchPopulation false
materializedProjectStatisticsDerivedSummaries false
materializedProjectStatisticsPages false Consumer gate — stays off for the whole proof.
materializedProjectStatisticsSignalR false Consumer gate — stays off for the whole proof.
materializedProjectStatisticsExports false Consumer gate — stays off for the whole proof.

ProjectStatisticsFlagMap.FamilyFlagKey maps ProjectStatisticsMetricFamily.ProjectScreening to materializedProjectStatisticsScreening and to nothing else, so the screening proof needs no second family gate. A family with no gate is denied rather than defaulted on.

Allowlist

ProjectStatistics:ProjectAllowlist is deployment configuration, not a generated feature flag. ProjectStatisticsAllowlistConfiguration reads it through IConfiguration, so a non-production administrator cannot widen it from the runtime flag admin UI — which matters, because it is the thing that decides which real projects a dark projection may serve.

It has no env-mapping.yaml entry, so it is set through the chart's generic .Values.env map, the same mechanism ASPNETCORE_ENVIRONMENT already uses:

env:
  SYRF__ProjectStatistics__ProjectAllowlist: "<STAGING_PILOT_PROJECT_ID>"

_deployment-dotnet.tpl copies .Values.env verbatim into the container environment, and the hosts call AddEnvironmentVariables("SYRF__") last, which strips the prefix and turns __ into :, yielding exactly ProjectStatistics:ProjectAllowlist.

The value is a comma-separated list of project GUIDs. There is no wildcard and no permissive failure: absent, empty and malformed all admit nothing. The draft therefore ships the literal placeholder <STAGING_PILOT_PROJECT_ID>, which cannot enable any project even if merged by mistake.

Verifying it rendered — read-only

After ArgoCD syncs, confirm the running pods carry the values. All three commands are reads.

# The rendered environment on each host. Expect the three true flags, the nine false ones,
# and the allowlist GUID.
kubectl -n syrf-staging get deploy api -o json \
  | jq -r '.spec.template.spec.containers[0].env[]
           | select(.name | test("MaterializedProjectStatistics|ProjectAllowlist"))
           | "\(.name)=\(.value)"'

kubectl -n syrf-staging get deploy project-management -o json \
  | jq -r '.spec.template.spec.containers[0].env[]
           | select(.name | test("MaterializedProjectStatistics|ProjectAllowlist"))
           | "\(.name)=\(.value)"'

# Prove the pods were actually replaced, rather than the Deployment spec having moved ahead
# of a stuck rollout.
kubectl -n syrf-staging get pods -l app.kubernetes.io/name=api \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].imageID}{"\n"}{end}'

kubectl exec ... env is an acceptable alternative; it is still a read. Do not use kubectl set env, kubectl apply, kubectl edit or helm upgrade to correct anything found here — a discrepancy is a cluster-gitops change, per the repository's GitOps-only policy.

The clearest functional confirmation costs nothing and comes in step 4: a project outside the allowlist answers the backfill endpoint with 409 and failureReason: "NotAllowlisted". If the pilot project answers that way, the allowlist did not reach the API host — a 404 there means something else, namely that the project document does not exist.

Choosing the pilot project

Record the decision and its reasoning in the evidence file. Criteria:

  • Real screening activity. The screening profiles must have something to distribute over: studies with include and exclude decisions, ideally from more than one reviewer, and ideally some studies with none. A project of all-zero counters proves that zero equals zero.
  • Small enough to backfill in minutes. The backfill is synchronous (see step 4), so its duration is a request duration. Prefer hundreds to low thousands of studies for the first run.
  • Not production-critical and not somebody's live work. Staging data is shared. Do not choose a project that a colleague is mid-review on: step 7 requires making a real screening decision on it.
  • Ideally one whose AgreementThreshold is stable. Configuration changes during the proof invalidate the control's configuration digest and complicate the report (ConfigurationMatchesCurrentSettings).

Record: project id, name, study count, reviewer count, screening decision count, and who confirmed it is safe to use. Read those counts from the project's existing UI or an authoritative read; do not write to the database to arrange them.

Step 2 — capture the "before" state

Before enabling anything, capture the project's current authoritative screening statistics through the ordinary application surface, and keep them in the evidence file. This is the legacy-response baseline that "prove legacy response equivalence" is measured against. With the consumer flags off, every page read is served authoritatively for the whole proof, so this baseline should still hold at the end for anything that did not change.

Step 3 — declare the fleet versions and open the fleet gate

POST https://api.staging.syrf.org.uk/api/admin/project-statistics/fleet/mode
Authorization: Bearer <administrator token>
Content-Type: application/json

{ "mode": "Enabled" }

Nothing serves until this runs. The flags in step 1 say what this deployment requests; the durable fleet control says what it is allowed to do, and it defaults to Disabled. Before this endpoint existed no production code created that control at all, so a completed backfill wrote rows that could never be read and the parity audit reported Disabled for every scope. Enabling is deliberately a separate act from backfilling: the rollout order is dark-write, prove parity, then serve, and a backfill that switched serving on would collapse the middle step it exists to make possible.

This one call does three things in one bounded transaction: it creates the singleton pmProjectStatisticsGlobalControl when the fleet has never had one, declares the catalogue/storage/source versions this build's writers produce, records the deployment's effective reviewer mode, and moves the durable gate to Enabled. A first declaration preserves both the write epoch and mode epoch at zero; it does not retire backfilled rows.

Before this first declaration, verify that every API and project-management replica agrees on ActiveReviewerTrackingEnabled && SignalRActive, and record that effective value in the evidence. The endpoint captures that process value once and stores it with the singleton and invalidation slot in one transaction. It does not assume the default false: an existing true deployment must retain its capacity semantics, including outside the pilot allowlist. This endpoint cannot verify other replicas' configuration. Use the coordinated static rollout; the runtime host-consistency gap remains

3360. Once a singleton exists, its reviewer mode is authoritative: a disagreeing enable returns

409 InvalidState and never silently changes the formula or resets its epoch.

The versions are not yours to choose. They are derived constants (ProjectStatisticsFleetVersions.Deployed). The reader applies two independent version equalities — fleet-versus-project-control and row-versus-control — and every published row is stamped from the deployed family writer's own constants, so a fleet declaring anything else would guarantee that no row is ever readable. Today they are catalogue 1, storage 0, source 1.

What the response reports is the durable row, not this build. recordedCatalogueVersion, recordedStorageVersion and recordedSourceVersion are read back from the control row the transition acted on — the fleet singleton here, the project's own control row for a narrow gate. A successful first declaration therefore echoes the constants above, but a transition over a row an earlier deployment enabled reports that row's numbers, so a disagreement with the constants above is real version drift rather than a reporting artefact. They are null only when the transition found no control row at all (a narrow gate refused NotFound, or a fleet transition whose singleton neither existed nor was created).

Run it before the backfill

The project control row copies the fleet's storage version verbatim when a backfill creates it, and the backfill refuses (ControlVersionMismatch) a project whose control disagrees with the fleet. Declaring first means the two agree by construction. It is also what makes the step-6 parity audit capable of reporting anything but Disabled.

Outcomes

Status outcome Means
202 Applied The gate is now Enabled. mode, writeEpoch and clientInvalidationRevision in the body are read back from the durable row.
200 AlreadyApplied The fleet was already Enabled under exactly these versions. Nothing changed and no revision was allocated.
409 InvalidState The recorded reviewer mode disagrees with this host, the fleet is serving under different versions — a version change is a disable/declare/re-enable sequence, never a silent edit under live traffic — or the requested stage is not legal from the current mode.
409 RebuildRequired This fleet control has been through a durable disable, so it re-opens only under rebuild evidence. See step 10.
409 QuarantineNotElapsed Only for a Disabled claim; see step 10.
409 InvalidationSlotUnavailable The fixed global invalidation slot could not be admitted, so the durable mode was deliberately left unchanged: a gate change nobody can be told about is worse than no gate change. Retry.
409 ConcurrencyLost A competing transition or writer won. The transaction was aborted; reload and retry.
400 — mode was not Enabled, Disabling or Disabled.

Both fleet and project transitions retry only an uncertain commit acknowledgement, at most three commit attempts on the same transaction. They never replay the transition body or rebuild sweep. If all acknowledgements remain uncertain, the error remains an outage rather than a false 409 claim that nothing committed; inspect the durable mode before resubmitting. Definitive write conflicts and duplicate-key races instead return 409 ConcurrencyLost after abort. Applied outcomes carry the committed invalidation revision; refusal responses do not allocate one.

Record the whole response body in the evidence file: mode, writeEpoch, clientInvalidationRevision and the three recorded*Version fields. Compare those three against catalogue 1, storage 0, source 1 before moving on: a difference means the durable fleet row was declared by another deployment, and every row published against it is ControlVersionMismatch for the reader.

Step 4 — backfill

POST https://api.staging.syrf.org.uk/api/admin/project-statistics/{projectId}/backfill
Authorization: Bearer <administrator token>

No request body, no query string. The route parameter is deliberately named statisticsProjectId rather than projectId in source, so that the shared authorization handler does not probe for the project and leak its existence to an authenticated non-administrator; on the wire it is just the project GUID.

It is synchronous despite the 202

The work runs inline in the request, on the caller's cancellation token, and the response is returned only after every scope has been attempted. There is no background job, no hosted service, no Quartz schedule, no Location header, no job id and no polling endpoint. Treat 202 exactly as you would 200.

Size the client timeout for the whole rebuild. A client that times out aborts the request's cancellation token mid-run, which leaves scopes Stale and a rebuild lease held under the owner string stats.backfill.screening:admin:<32-hex-user-guid>. Use curl --max-time generously rather than a default.

202 Accepted — the success body

{
  "projectId": "...",
  "metricFamily": "ProjectScreening",
  "forced": false,
  "scopesRebuilt": 1,
  "scopesAlreadyCurrent": 0,
  "scopesStale": 0,
  "scopesAbsent": 0,
  "checkpointStatus": "Succeeded",
  "scopes": [
    {
      "scopeKind": "Project",
      "scopeKey": "project",
      "disposition": "Rebuilt",
      "status": "Succeeded",
      "failureReason": "None",
      "publishedGeneration": 1,
      "detail": null
    }
  ]
}

Screening declares exactly one scope, so scopes has one entry with scopeKind: "Project" and scopeKey: "project". disposition is one of Rebuilt, AlreadyCurrent, Stale, Absent. forced is false on /backfill and true on /rebuild.

A 202 is returned only when no scope is Stale and the bootstrap checkpoint was not refused. An Absent scope does not block a 202 — it means the authoritative calculator reported the scope does not exist and the rebuild returned it to Stale rather than publishing a fabricated zero row.

404 Not Found — empty body

One cause: the project document does not exist, so it declares no screening scope to rebuild. The service's explanatory detail is discarded, because NotFoundResult carries no body.

Note that this is not how backfill answers a project outside the allowlist — that is a 409 with failureReason: "NotAllowlisted" (below). The parity endpoint behaves differently and answers 404 for both; see step 6.

409 Conflict — not admitted

Nothing has been written; the check runs before any storage is touched. Key on the failureReason extension, not on status: Type and Instance are never set, so there is no RFC problem-type URI to match, and the Newtonsoft ProblemDetailsConverter flattens extensions to top level so status appears both as the framework's 409 and as a lifecycle enum name.

failureReason Meaning What to do
NotAllowlisted The project is outside ProjectStatistics:ProjectAllowlist. There is no implicit wildcard. Fix the cluster-gitops value, or the allowlist did not reach this host. This is the expected answer for every project today, since the allowlist is empty everywhere.
WriteDisabled materializedProjectStatisticsWrites is off. A backfill that ignored the global kill switch would be the one write the switch exists to stop. Turn the flag on through cluster-gitops.
FamilyNotServable materializedProjectStatisticsScreening is off. Turn the family flag on through cluster-gitops.
FleetVersionMismatch Either the fleet is serving but never declared its catalogue/source versions, or it declares versions the screening writer cannot produce. Every row published would be stamped at the writer's versions and rejected by the reader. Do not retry. This is a deployment-consistency problem: complete the fleet version declaration or deploy the matching family writer.
ControlVersionMismatch The project's control row carries catalogue/storage/source versions that do not match the writer's. Backfill only — the forced rebuild is exempt. Run POST .../rebuild, which reconciles the control and its current rows together inside the publication's own control compare-and-swap.

409 Conflict — partial failure or refused checkpoint

The title distinguishes three cases:

Title Means
The statistics rebuild could not publish every scope. at least one scope is Stale
The statistics rows are current, but the bootstrap checkpoint is incomplete. rows published, checkpoint refused
The statistics rebuild could not publish every scope, and the bootstrap checkpoint is incomplete. both

The body carries failureReason, status and a summary extension holding the full success DTO, so you can see which scopes did publish. Save it; it is evidence.

Common per-scope reasons and their responses:

failureReason Means Response
StatisticsRebuildBusy Another owner holds the per-scope rebuild lease, or the family guard's sole candidate slot is held. Wait for the lease to expire and retry. If it was your own timed-out client, wait it out rather than forcing.
FamilyFenced An active operation fence covers the family — a bulk, import or definition-rewrite operation. No visible scope can become servable while a fence covers it. Wait for the fencing operation to finish, then retry.
StatisticsInclusionRecalculationInProgress An inclusion recalculation or definition-rewrite fence is live. Wait and retry.
PublicationRaceLost A concurrent publication won the compare-and-swap. Retry.
LeaseLost The lease expired or was taken over mid-run — usually a run that took longer than expected. Retry.
DigestMismatch A replayed operation identity carried different content. Do not retry blindly; capture and escalate.
ScopeAbsent The authoritative calculator says the scope does not exist. Reported as Absent, not Stale; does not fail the run.
CheckpointCapacityExceeded, StatisticsScopeCapacityExceeded, StatisticsPublicationOperationCapacityExceeded A bounded capacity limit was hit. Capture; this is a Phase 1 provisional-limit finding worth reporting.

A refused bootstrap checkpoint on a non-forced run publishes nothing — current rows must not outrun the single bootstrap point they are paired with. The detail says to retry the backfill, and, if another checkpoint occupies the identity, to run a forced rebuild (which publishes and therefore moves the projection revision) and then backfill again to record the bootstrap at the new identity. The forced rebuild is exempt from this short-circuit.

500

Non-development hosts return {"error":"An unexpected server error occurred."}. Capture the timestamp and correlate with the API logs; do not retry in a loop.

Step 4b — stage-annotation backfill

Only needed when the pilot will exercise the Stage Overview annotation pie (materializedProjectStatisticsStageOverview). Run it after step 4 and before any consumer flag is opened: until a stage has a Fresh row, the pie's query falls back and opening its flag changes nothing an operator could observe.

POST https://api.staging.syrf.org.uk/api/admin/project-statistics/{projectId}/stage-annotation/backfill
Authorization: Bearer <administrator token>

Same shape as step 4 in every respect — no body, the same statisticsProjectId route token, the same BatchAdminProjects policy, synchronous despite the 202, /rebuild for the forced variant — and the same 202/404/409 tables apply unchanged, because the same service runs behind both routes. Three things differ, and they are the whole of the difference:

  • metricFamily is StageAnnotation, and the family gate this run checks is materializedProjectStatisticsAnnotation, not materializedProjectStatisticsScreening. A deployment with screening on and annotation off gets 409 FamilyNotServable here while step 4 succeeds, which is the expected answer rather than a fault.
  • One scope per stage — every stage, not only the annotation-mode ones. The legacy aggregation projects an annotation section for each of project.Stages with no review-mode filter, so a screening-mode stage has one too (all zeroes). scopes therefore has one entry per stage with scopeKind: "ProjectStage" and a scopeKey naming the stage. Stages are enumerated from the Project document, not from the projection, so a first run covers every stage. A project with no stage completes with an empty scopes array and a 202; 404 still means the project document does not exist. A stage whose project carries no agreement threshold is reported Absent — the legacy aggregation cannot run without one — rather than failing the whole run.
  • The lease owner string is stats.backfill.screening:admin:<32-hex-user-guid>, unchanged — the operation namespace is shared across families, so a stuck lease is told apart by its scope, not by its owner.

Record the response body in the evidence file alongside step 4's. Re-running it is a no-op: every scope whose published row already matches the authoritative calculation is reported AlreadyCurrent and no generation is churned.

The membership-stage annotation family has no backfill route. Nothing shipped reads its rows, so there is nothing to establish.

Step 5 — open the project narrow gate

POST https://api.staging.syrf.org.uk/api/admin/project-statistics/{projectId}/narrow-gate
Authorization: Bearer <administrator token>
Content-Type: application/json

{ "enabled": true }

The fleet gate from step 3 is necessary and not sufficient: each project carries its own narrow serving gate on its control row, fenced by its own ProjectWriteEpoch, and it also defaults to Disabled. Both must be Enabled before the bundle reader will serve a materialized value or the parity audit can report anything but Disabled.

Run it after the backfill, not before. The narrow gate lives on the project's control row, and only a backfill or a materialized source write creates that row. There is deliberately no create-on-enable here: a control row created by this endpoint would carry no configuration digest and no versions, and the reader would refuse every row published against it for ever. A project with no control row is answered 404.

The route parameter is named statisticsProjectId in source for the same reason as the backfill route — so the shared authorization handler does not probe for the project and leak its existence to an authenticated non-administrator. On the wire it is just the project GUID.

Outcomes

Status outcome Means
202 Applied The narrow gate is now Enabled. The body carries the resulting mode and writeEpoch (the project write epoch), the project control row's own recorded*Version fields and, on a re-enable, scopesRebuilt.
200 AlreadyApplied The gate was already Enabled. Nothing changed.
404 — The project has no statistics control row. Run step 4 first.
409 RebuildRequired The gate has previously been disabled, so it re-opens only under a complete rebuild proof, and the sweep this call ran did not complete. rebuildFailureReason carries the first scope's typed reason — FamilyFenced, StatisticsRebuildBusy, PublicationRaceLost and so on — which is the thing to clear.
409 InvalidState An unfinished Disabling quarantine may not be jumped out of. Complete the disable, then re-enable.
409 QuarantineNotElapsed Only for a { "enabled": false } call claiming Disabled; see step 10.
409 InvalidationSlotUnavailable / ConcurrencyLost As for the fleet gate. Retry.

Confirm it took

Re-run the parity audit (step 6). Before this step it reports Disabled for every scope — the audit reader deliberately keeps every durable gate, overriding only the two flag-level serving switches, so an all-Disabled report is exactly what a shut narrow gate looks like. After it, the report should carry real scopeStates and a real inParity verdict.

An all-Disabled parity report is not a passed gate. It is the report saying nothing was audited. Read fallbackReason before concluding anything from it.

Step 6 — parity audit

GET https://api.staging.syrf.org.uk/api/admin/project-statistics/{projectId}/parity
Authorization: Bearer <administrator token>

This endpoint arrives with #3196 and is not on main as of 38a272d4c. It is read-only on both sides: nothing here creates a control row, family summary, guard or lease, and nothing advances a revision. It answers while serving is off, deliberately — the rollout order is dark-write, then prove parity, then serve.

Outcomes:

  • 200 OK with ProjectStatisticsParityReportDto.
  • 404 — not allowlisted, or the project does not exist. Here the two are deliberately indistinguishable, so the endpoint cannot be used to enumerate the pilot. This differs from backfill, which answers 409 NotAllowlisted for the first case and reserves 404 for the second.
  • 200 OK with fallbackReason: "Disabled" on every scope — the durable gates are shut. The audit reader overrides only the two flag-level serving switches; it deliberately keeps every durable gate, including the fleet control's mode and the project's narrow gate. Run steps 3 and 5. This is a report of nothing having been audited, not a passed gate.
  • 409, type: "urn:syrf:project-statistics:writes-disabled", title "There is no materialized projection to audit." — materializedProjectStatisticsWrites is off, so nothing is maintaining the projection and any report would describe a frozen artefact. Enable writes and backfill first.

Reading the report

{
  "projectId": "...",
  "metricKey": "project-screening",
  "inParity": true,
  "isInconclusive": false,
  "sourceAdvancedDuringAudit": false,
  "materializedAvailable": true,
  "readSource": "...",
  "fallbackReason": "...",
  "capacityFailure": "...",
  "checkpoint": { "checkpointSourceRevision": 0, "checkpointProjectionRevision": 0,
                  "checkpointModeEpoch": 0, "canonical": "..." },
  "watermarks": { "globalClientInvalidationRevision": 0, "sourceInvalidationRevision": 0,
                  "committedProjectionRevision": 0, "clientInvalidationRevision": 0,
                  "modeEpoch": 0, "catalogueVersion": 0, "storageVersion": 0,
                  "sourceVersion": 0, "configurationDigest": "..." },
  "scopeStates": [ { "selection": "...", "scope": "...", "availability": "...", "state": "...",
                     "isTombstone": false, "isPublished": true,
                     "publicationGeneration": 1, "lastChangedRevision": 0 } ],
  "metrics": [ { "metricKey": "screening.sufficientlyScreened",
                 "expected": 0, "actual": 0, "delta": 0, "inParity": true } ],
  "observedAtUtc": "...",
  "controlConfigurationDigest": "...",
  "currentSettingsConfigurationDigest": "...",
  "configurationMatchesCurrentSettings": true
}

inParity is the verdict. It is true only when a materialized value exists and equals the expected one, for every metric. Crucially, a bundle that legitimately refuses to serve is not "in parity" — it is unaudited, inParity is false, materializedAvailable is false, and fallbackReason says why. Read fallbackReason before concluding anything from a false verdict.

The comparison runs over the union of both key sets with absence read as zero, in the catalogue's declaration order. That union matters both ways: a metric the projection never wrote is a legitimate zero, and a counter the projection holds that the authoritative side does not produce is a real defect the report surfaces rather than skips.

isInconclusive means a write landed underneath the comparison, so the deltas may be an artefact of the race rather than real drift. The two halves are separate reads; the project's revision is checked either side of the authoritative half, a move is retried once internally, and a second move reports the run as inconclusive rather than as a divergence an operator would chase. sourceAdvancedDuringAudit reports the same underlying observation.

Retry rule for inconclusive. Re-run the audit unchanged. If it is still inconclusive, the project is under sustained write load and cannot be audited coherently at all — that is the report telling you so, not a transient. Re-run during a quiet window, or choose a quieter pilot project. Do not re-run repeatedly until it happens to look quiet, and never record an inconclusive run as a parity pass or as a divergence. Record every attempt in the evidence file, including the inconclusive ones.

configurationMatchesCurrentSettings is separate from the verdict on purpose:

  • true — the control row is stamped under the project's current settings.
  • false — the control is still stamped under settings the project no longer has, so its counters answer a superseded question even when they agree with today's calculation. This is drift to chase separately, not a divergence in the values compared here. It is also false whenever isInconclusive is true.
  • null — the comparison could not be made. Null never means it passed. It happens when there is nothing to digest, for example a project with no agreement threshold or one that vanished mid-run.

A fresh forced rebuild adopts the digest of the project it read, so it reports a match; the field says nothing about how the digest came to be stamped.

Also capture watermarks and scopeStates verbatim — availability, state, isPublished and publicationGeneration are what distinguish Fresh from Stale, Rebuilding, Incompatible, fenced and tombstoned, and they are the record of which of those paths the proof actually exercised.

Step 7 — exercise a real screening decision

With the projection backfilled and audited, make an actual screening decision on the pilot project through the ordinary UI, as an ordinary reviewer would. This is the point of the proof: the transactional delta path must move the projection in step with the source, without a rebuild.

  1. Note the pre-decision watermarks.committedProjectionRevision and the relevant metric values.
  2. Screen one study — include or exclude — through the normal screening surface. Note the study, the reviewer, the decision and the time.
  3. Re-run the parity audit.
  4. Expect inParity: true with the counters moved by exactly the decision made, and committedProjectionRevision advanced. A false verdict here with materializedAvailable: true and a non-zero delta is the finding the proof exists to catch: capture it in full and stop.
  5. Repeat for a decision of the opposite kind, and for a study that takes a profile from one populated bucket to another, so more than one counter is proven to move.

Worth exercising deliberately while here, since they are named in the Phase 2 staging proof:

  • Kill switch. With the flags turned off in cluster-gitops, confirm reads fall back to the authoritative calculation and the parity endpoint answers 409 writes-disabled. Turn them back on and confirm the projection is still there and still in parity — the rollback preserves data.
  • Stale / Rebuilding. Observe them in scopeStates.state around a rebuild.
  • Forced rebuild. POST .../rebuild and confirm the report is still in parity afterwards, with a new publicationGeneration.

Step 8 — capture evidence

Write one JSON file per proof run to evidence/phase2c-staging/, named phase2c-staging_<YYYY-MM-DD>_<short-run-label>.json. Start from the committed template, phase2c-staging-evidence-template.json, which follows the same shape as the Phase 0 baselines.

Record, at minimum: the date, environment, project id and selection rationale, the exact deployed flag values and allowlist as read from the running pods, the deployed image digests, the backfill outcome, every parity report including inconclusive attempts, the screening decisions made, the fallback reasons observed, and notes. Store the raw response bodies rather than a summary of them — a verdict without its watermarks and scopeStates cannot be re-interpreted later.

Never put reviewer identities, study titles or other project content into the evidence file. Ids, counts and the response envelope are enough.

Step 9 — soak

The README's soak gate and Phase 6's acceptance criteria require, before any production pilot:

  • at least seven actual days of staging soak — calendar days, not seven days' worth of traffic compressed into an afternoon;
  • 10,000 reads;
  • 1,000 relevant mutations;
  • 100% exact counter parity, zero stale materialized serves, 100% injected fallback success, no unresolved rebuild failures, and bounded delta-ledger and storage growth.

Two honest caveats about counting those numbers on a single staging project.

The read and write counters now exist, but nothing scrapes them yet — and they are silent until the flags are on. Since syrf #3482 both hosts register a real Meter and record reads by source and reason, confirmed write commits, rebuild and backfill outcomes and parity results — see Telemetry for the instruments and their tags. Two limits matter for counting. The read counters record only once a consumer actually enters the adapter, which the consumer flags gate, so a dark deployment produces none of them (its startup line is the whole FEAT-024 signal). And there is still no destination: OTEL_EXPORTER_OTLP_ENDPOINT is unset in every environment, so the series live only inside each pod and must be read there with dotnet-counters, or reproduced by a driver that counts its own calls. A driver remains the more defensible option for the controlled 10,000/1,000 figures, since the plan calls them "controlled"; ambient staging traffic on one project will not reach them in seven days. Use the counters to corroborate the driver's own totals and to see the distribution — which reasons, which commit paths — that a driver cannot tell you.

With every consumer flag off, ordinary page reads do not touch the projection at all. They are served authoritatively. So "10,000 reads" against a Phase 2C deployment means 10,000 reads of the projection, which during this phase means the parity/admin surface or a driver exercising the bundle reader — not 10,000 page loads. Decide which you are counting, write it down in the evidence file, and do not let the distinction blur.

Observing fallback reasons

Every gated read that declines to serve a materialized value logs a fallback reason. Watch the distribution across the soak: a fallback that is supposed to be rare (Stale, Rebuilding, Incompatible, an epoch mismatch, an active fence) appearing steadily is the signal the soak is for. The two reason enums, the exact log message templates and the substrings to query them by are in Fallback reasons and log fields below, and the counter that carries the same two reasons as tags is in Telemetry. Read both before planning the soak, and note what neither gives you before step 1: both API consumers check materializedProjectStatisticsPages (and, for the overview, ...ProjectOverview) and return the authoritative answer without entering the adapter at all, so in a fully dark deployment the read counters stay at zero and no per-read line is emitted. The only FEAT-024 series a dark pod produces is the one-per-host startup line. Once the flags are on, the counter is the more reliable of the two because it does not depend on a log level or a log pipeline.

Record the observed set — not just the counts, the set — in the evidence file. A fallback reason nobody expected to see is a finding whether or not parity held.

Step 10 — rollback

Rollback closes the durable gates first, through the API, and then the flags, through cluster-gitops. Both halves matter and they are not interchangeable: the flags stop this deployment from asking to serve, while the durable gates and their write epochs stop any deployment from serving the rows — including a replica whose per-process flag cache has not caught up. The flag cache is explicitly not a correctness authority.

The flag half is a cluster-gitops change, reverted through git and synced by ArgoCD. Never kubectl set env, never helm upgrade.

1. Close the project narrow gate.

POST /api/admin/project-statistics/{projectId}/narrow-gate   { "enabled": false }

Answers 202 with mode: "Disabling". That is not a half-done call — it is the first of two stages. The project write epoch advances at this commit, so a source transaction that observed the gate as Enabled and commits afterwards writes rows carrying the superseded epoch, which are already non-servable. Reads fall back to the authoritative calculation immediately.

2. Claim Disabled once the quarantine has elapsed. Repeat the identical call. Before the deployment-verified maximum transaction lifetime has passed it answers 409 with QuarantineNotElapsed; that interval, not the mode field, is the durable barrier. Afterwards it answers 202 with mode: "Disabled".

3. Close the fleet gate, the same two stages.

POST /api/admin/project-statistics/fleet/mode   { "mode": "Disabled" }

The first call answers 202 with mode: "Disabling" and advances the fleet write epoch; the second, after the quarantine, answers 202 with mode: "Disabled".

4. Turn the flags off. Set materializedProjectStatisticsWrites, materializedProjectStatisticsServing and materializedProjectStatisticsScreening back to false on both hosts.

5. Return SYRF__ProjectStatistics__ProjectAllowlist to the placeholder, or remove the key.

Any one of steps 1, 3, 4 and 5 alone is sufficient to stop the pilot serving — the allowlist admits nothing without a valid GUID, and the flags and the durable gates each gate every read independently — but do all of them, so the deployed state matches the intended state in one reading.

Confirm with the parity audit: after the gates are closed it reports Disabled again, and after the flags are off it answers 409 writes-disabled.

Narrow-gate requests must explicitly supply enabled: true or enabled: false; an omitted or null value returns 400 before any transition. Contention responses report the mode, epoch and invalidation revision observed together after abort. These request-validation and reporting corrections use the existing administrative gates and need no additional feature flag.

Re-enabling after a durable disable

A gate that has been through a disable does not re-open the way it was first opened. Its quarantine retired a whole epoch of rows, so:

  • The narrow gate re-opens only under a complete rebuild proof. { "enabled": true } produces that proof itself — it rebuilds every scope the project's routed families declare and presents the outcomes to the transition — and refuses with RebuildRequired plus the sweep's own rebuildFailureReason if any scope could not be published.
  • A successful re-enable still serves nothing until you rebuild again. The re-enable allocates a new project write epoch, and the sweep that produced the proof necessarily ran before that epoch existed, so its rows carry the superseded tuple and the reader answers EpochMismatch. This is the plan's stated residual — "a re-enable that serves nothing until a rebuild runs, never one that serves stale data" — not a defect. Run POST .../rebuild after the gate is open and confirm the parity report before treating the project as serving again.
  • The fleet gate answers 409 RebuildRequired and is deliberately not re-openable from this surface: a fleet re-enable's evidence would have to span the whole fleet, and no bounded transaction can establish that. Re-enabling a fleet that has been durably disabled is out of scope for the Phase 2C pilot; a pilot that needs it should be torn down and rebuilt from a fresh fleet control rather than coaxed back open.

What rollback leaves behind

Data, deliberately. "Rollback to authoritative reads is immediate and does not require data deletion", and "a flag rollback returns every read to the authoritative screening facets and preserves history". Concretely, after rollback:

  • The pmProjectStatistics* documents written during the proof — current rows, control row, delta ledger, the backfill-observed checkpoint — remain. They are simply never read.
  • Every read returns to the authoritative calculation immediately. There is no drain and no window in which a stale materialized value is served.
  • Source mutations continue committing normally. With writes off they commit the source, mark affected scopes Stale and fall back, so the projection stops tracking the source from the moment the flag goes false. This is why re-enabling requires a fresh backfill or forced rebuild, not just flipping the flags back: the rows are intact but no longer current, and the reader will correctly refuse them as Stale.
  • History is preserved. The bootstrap checkpoint keeps the identity it was captured under and is never rewritten.

Nothing needs to be deleted, and nothing should be. If the pilot is genuinely being abandoned rather than paused, removing the projection documents is a separate, separately approved cleanup — not part of rollback.

Telemetry

Since syrf #3482 the FEAT-024 instruments are real. ProjectStatisticsMetrics (SyRF.ProjectManagement.Core/Telemetry/ProjectStatisticsMetrics.cs) owns one System.Diagnostics.Metrics.Meter named SyRF.ProjectManagement.ProjectStatistics, registered as a Lamar singleton by ProjectStatisticsRegistry and named in AddMeter(...) by both hosts' Program.cs, alongside the existing SyRF.API.BffAuth meter.

Recording is not flagged. It has no behavioural effect, and the whole point is that a dark pilot should be observably dark; gating telemetry on the flags being on would leave the dark period unobservable, which is the state this section used to describe.

The instruments

Every tag value is a bounded enum member rendered snake_case (ProjectNotAllowlisted becomes project_not_allowlisted). No instrument is ever labelled with a project, membership, investigator, question or scope-key identifier — that is a plan rule, and a unit test asserts no tag value parses as a GUID. deployment.environment (development / preview / staging / production / unknown) is on every series, exactly as it is on the BFF meter.

Instrument Kind Tags beyond deployment.environment Recorded by
syrf.project_statistics.requests counter family, scope_type, request_kind, outcome (materialized / authoritative_fallback) ProjectScreeningStatisticsQueryAdapter, once per gated read
syrf.project_statistics.materialized_reads counter family, scope_type, request_kind the same adapter, when the projection served
syrf.project_statistics.materialized_read.duration histogram (ms) family, scope_type, request_kind the same adapter: the bundle read plus its mapping
syrf.project_statistics.authoritative_fallbacks counter family, scope_type, request_kind, reason, reader_reason the same adapter, when it answered authoritatively
syrf.project_statistics.authoritative_aggregations counter as above the same adapter, only when the fallback really ran an aggregation
syrf.project_statistics.authoritative_aggregation.duration histogram (ms) as above the same adapter, on the same condition
syrf.project_statistics.source_operation_receipts.committed counter outcome (point_path / source_only) the source-write owners, after the transaction's commit is confirmed
syrf.project_statistics.source_operation_receipts.deduplicated counter outcome (point_path / source_only) ResolveByReceiptAsync, when a receipt proves the operation already committed
syrf.project_statistics.source_operation_receipts.rejected counter outcome (refused), reason either coordinator entry point, on any typed ProjectStatisticsWriteRejectedException
syrf.project_statistics.rebuilds.requested counter family, operation, scope_type (scope rebuilds only) ProjectStatisticsRebuildService.RebuildScopeAsync and ProjectScreeningBackfillService.BackfillAsync
syrf.project_statistics.rebuilds.completed counter as above the same two, on Succeeded
syrf.project_statistics.rebuilds.failed counter as above plus reason the same two, on every other status
syrf.project_statistics.rebuild.duration histogram (ms) as above plus outcome (succeeded / failed) the same two
syrf.project_statistics.parity.audits counter family, outcome, reader_reason ProjectScreeningParityAuditService.AuditAsync
syrf.project_statistics.parity.mismatches counter as above the same service, only on mismatch

Four conventions are worth knowing before you write a query.

  • reason and reader_reason are different enums and both matter. reason is the consumer's own ProjectScreeningStatisticsFallbackReason — including the decisions taken before any storage was touched, such as writes_disabled or project_not_allowlisted. reader_reason is the bundle reader's ProjectStatisticsFallbackReason, which stays none for those pre-snapshot decisions. reason=project_not_allowlisted means the pilot is not on; reason=bundle_fallback, reader_reason=stale means it is on and the projection is behind. Those are the two answers the soak exists to separate. The same pair appears in the adapter's Debug log line, /-separated. The parity counters publish the reader's enum under reader_reason too, so one enum is never split across two tag names.
  • authoritative_fallbacks and authoritative_aggregations are not the same number, by design. The first counts reads that fell back; the second counts authoritative calculations actually executed. ReviewController.GetFullStats runs the one 15-facet pipeline for the whole response before consulting the adapter and hands the screening section over, so its fallbacks move the first counter and not the second — and contribute no sample to authoritative_aggregation.duration, which is one of the five histograms the plan's p50/p95 gate reads. Only the screening-overview consumer, which passes a delegate that runs the query, moves both. Read the difference between the two counters as "fallbacks that cost nothing extra".
  • source_operation_receipts.committed counts confirmed commits, not attempts. The coordinator writes inside the caller's still-open transaction and a TransientTransactionError aborts and replays it, so the source-write owners record this only after ProjectStatisticsTransaction.CommitWithUnknownResultRetryAsync returns. Refusals (...rejected) are counted where they are raised, because a refusal is terminal for its transaction by contract.
  • operation separates a sweep from a scope. scope_rebuild is one scope's rebuild; backfill and forced_rebuild are the administrative sweeps, which run many scope rebuilds underneath and therefore also produce scope_rebuild rows. A sweep carries no scope_type — it spans every scope kind the family declares, so there is no honest single value — which means a query filtered on scope_type silently drops sweeps. Sum the two separately.
  • parity.audits{outcome} has four values and only one of them is a defect. in_parity (conclusive and equal), mismatch (conclusive and unequal — alert on this), inconclusive (the source moved between the two halves), and unavailable (the projection legitimately refused: Stale, fenced, rebuilding, epoch-mismatched, disabled — which is also what every audit reports before step 5 opens the narrow gate, and after rollback closes it).

Reading them during the soak

There is still no OTLP destination. AddOpenTelemetryConfig exports only when OTEL_EXPORTER_OTLP_ENDPOINT is set, and that variable appears nowhere in the charts or in cluster-gitops. There is no Prometheus scrape endpoint either — neither host calls MapPrometheusScrapingEndpoint. So the series are produced but not shipped, and the two ways to read them are:

  1. dotnet-counters, in the pod. This needs no deployment change and is the baseline. Both Dockerfiles' final stage is mcr.microsoft.com/dotnet/aspnet:10.0 — the runtime, not the SDK — so dotnet tool install is not available in the container. Copy the single-file build in instead:
# once, on your workstation
curl -sSL https://aka.ms/dotnet-counters/linux-x64 -o dotnet-counters && chmod +x dotnet-counters

POD=$(kubectl -n syrf-staging get pod -l app.kubernetes.io/name=api -o name | head -1)
kubectl -n syrf-staging cp dotnet-counters "${POD#pod/}:/tmp/dotnet-counters"
kubectl -n syrf-staging exec -it "${POD#pod/}" -- /tmp/dotnet-counters ps
kubectl -n syrf-staging exec -it "${POD#pod/}" -- /tmp/dotnet-counters monitor \
  --process-id <pid from ps> \
  --counters SyRF.ProjectManagement.ProjectStatistics

Verify this end to end on one pod before the soak starts rather than discovering at day seven that it does not attach: the diagnostic IPC socket lives under /tmp and belongs to the app's user, so a container running as a different user, with a read-only root filesystem, or with DOTNET_EnableDiagnostics=0 will refuse. Substitute the project-management deployment for the write, rebuild and backfill instruments — the API host records those only for operations it initiates. Use collect --format csv --output <file> rather than monitor when you want a series to keep as evidence, and kubectl cp it back out. dotnet-counters reports per process, so a multi-replica deployment needs one session per pod and the totals summed; record the replica count in the evidence file alongside the numbers, and remember a rolling restart resets every counter.

  1. Set OTEL_EXPORTER_OTLP_ENDPOINT in the staging values and point it at a collector. That is a cluster-gitops change and a decision outside this runbook — do not make it as part of running the proof. If it has been made by the time you run the soak, everything above is already flowing and dotnet-counters becomes a cross-check rather than the primary source.

Whichever you use, put the raw counter output in the evidence file, not a summary: a fallback total without its reason/reader_reason breakdown cannot be re-interpreted later, and that breakdown is the finding.

Fallback reasons and log fields

Read this section before planning the soak: what is observable is narrower than it looks.

The metrics exist; the pipe to a dashboard does not

ProjectStatisticsTelemetry.cs used to declare the instrument names and nothing else, so this section used to read "there are no metrics". Since syrf #3482 the meter is real and every instrument in Telemetry records. What is still absent is a destination: AddOpenTelemetryConfig exports only when OTEL_EXPORTER_OTLP_ENDPOINT is set, that variable appears nowhere in the charts or in cluster-gitops, and neither host exposes a Prometheus scrape endpoint.

So you can still not plan a dashboard query — but you can read the counters per pod with dotnet-counters, which is the more reliable of the two sources because it does not depend on a log level or a log pipeline. The logs below remain the way to tie an individual observation to a project id, which no metric carries.

While the pilot is dark, only the startup line is emitted

ReviewController.GetFullStats calls IsMaterializedReadRequested(projectId) first, which evaluates the flag and allowlist gate and logs nothing. If any of materializedProjectStatisticsWrites, ...Serving, ...Screening is off, or the project is outside the allowlist, the adapter is never entered and no per-read FEAT-024 line is emitted.

Since #3482 that is no longer indistinguishable from a misconfigured pod. Each host writes exactly one Information line at startup, from SyRF.ProjectManagement.Core.Telemetry.ProjectStatisticsDarkStateStartupLog:

FEAT-024 materialized project statistics startup state: serving requested {MaterializedStatisticsServingRequested}; flags {@MaterializedStatisticsFlags}; allowlisted projects {MaterializedStatisticsAllowlistSize}; meter {MaterializedStatisticsMeterName}.

MaterializedStatisticsFlags carries all twelve materializedProjectStatistics* values as this process resolves them — through the API's runtime-override-aware adapter in the API host, and the shared FeatureFlags singleton in project-management — with the serving entry already conjoined with writes, because serving a projection nothing maintains is never valid. MaterializedStatisticsServingRequested is the same conjoined value, not the raw catalogue flag. MaterializedStatisticsAllowlistSize is a count: the allowlisted project ids are deliberately never logged, because a rollout log is not an access-controlled surface. MaterializedStatisticsMeterName names the meter to attach dotnet-counters to — and resolving it to write the line is also what forces the meter to exist on a freshly rolled pod, rather than at the first statistics read.

Capture both hosts' lines in the evidence file at the start of the soak and again at the end. They are the record that the deployment was in the state the runbook says it was, and they are what distinguishes "the flags were off" from "the pod never restarted after the cluster-gitops sync". Grep them with FEAT-024 materialized project statistics startup state.

Once the gate passes, the adapter logs on every read as well: exactly one of the materialized-serve or fallback lines per call.

The two fallback enums

They are different types and both appear in one log line, separated by a literal /.

ProjectScreeningStatisticsFallbackReason — the adapter's reason, arriving with #3196:

Member Meaning
None the projection served
WritesDisabled materializedProjectStatisticsWrites off
ServingDisabled materializedProjectStatisticsServing off
FamilyDisabled materializedProjectStatisticsScreening off
ProjectNotAllowlisted project outside the pilot allowlist
NoCaller no caller identity to authorize against
NotAvailable bounded not-available: unauthorized, unknown or foreign selection
BundleFallback the reader decided the whole bundle is authoritative
BundleUnavailable the reader refused — visibility token or durable-mode disagreement
CapacityExceeded a bounded capacity ceiling
ScopeTombstoned the selection landed on an explicit deletion tombstone
ScopeRowMissing the materialized bundle carried no row for the scope
CheckpointUnresolved an all-zero (source revision, projection revision, mode epoch) tuple
ReaderFailed the read path threw

ProjectStatisticsFallbackReason — the reader's reason, already on main: None, Missing, Stale, Rebuilding, Incompatible, Disabled, EpochMismatch, Fenced, SnapshotPredicateFailed, InclusionRecalculationInProgress, DefinitionRewriteInProgress, DurableModeDisagreement, ProjectUnavailable.

Both render as their exact PascalCase member names — no attributes change the string form. The fallbackReason field of the parity report carries the reader enum's string.

For the soak, the reasons that should be rare are the interesting ones: a steady stream of BundleFallback/Stale, BundleFallback/Rebuilding, BundleFallback/Incompatible, .../EpochMismatch or .../Fenced is precisely what the soak is for. WritesDisabled, ServingDisabled, FamilyDisabled and ProjectNotAllowlisted mean the pilot is not actually on.

The log statements

All from ProjectScreeningStatisticsQueryAdapter (category SyRF.ProjectManagement.Core.Services.ProjectStatistics.Families.Screening.ProjectScreeningStatisticsQueryAdapter) except the last, which is from ReviewController.

Level Template
Debug FEAT-024 answered the project-screening section authoritatively for project {ProjectId}: {FallbackReason}/{ReaderFallbackReason}.
Debug FEAT-024 served the project-screening section from the projection for project {ProjectId} at checkpoint {CheckpointId}.
Error FEAT-024 project-screening materialized read failed for project {ProjectId}; answering from the authoritative screening facets.
Warning FEAT-024 declined to substitute the materialized screening section for project {ProjectId} at checkpoint {CheckpointId}: the materialized totals and the authoritative screening values differ; retaining the coherent response. Answering authoritatively.

The fallback line is Debug on purpose: while the flags are off every request takes that path, and a dark rollout that fills the log with warnings gets its warnings ignored.

The Warning line is the parity-divergence alarm. It is the only Warning-level FEAT-024 read-path signal and the one worth alerting on during the soak. A single occurrence is a finding: capture the project id and checkpoint, run the parity audit immediately, and record both.

Naming the query

API and project-management log plain text to stdout, not JSON. Their appsettings.json uses "WriteTo": [{ "Name": "Console" }] with no formatter key, so Serilog's default MessageTemplateTextFormatter renders the {ProjectId} and {FallbackReason} properties into the message string. Only Identity emits JSON.

Two consequences:

  1. Query by message substring, not by field predicate. FallbackReason="Stale" will not match anything; you must match the message text and parse the trailing <Guid>: <Reason>/<ReaderReason>.
  2. logging.format: json in staging.values.yaml is inert — no env var is mapped from logging.format in env-mapping.yaml or _env-blocks.tpl, so it switches nothing. Do not rely on it.

Substrings to query:

Purpose Substring
every gated read (count for the soak) FEAT-024 and the project-screening section
fallbacks, with reason FEAT-024 answered the project-screening section authoritatively for project
materialized serves FEAT-024 served the project-screening section from the projection for project
parity divergence — alert FEAT-024 declined to substitute the materialized screening section
host dark-state at startup (capture at both ends of the soak) FEAT-024 materialized project statistics startup state
read-path exception FEAT-024 project-screening materialized read failed for project

The baseline access is pod stdout, e.g. kubectl -n syrf-staging logs deploy/api --since=24h | grep 'FEAT-024'.

Three preconditions before any of the per-read lines yields output:

  • 3196 merged and deployed — these identifiers do not exist on main;

  • the pilot flags on and the project in the allowlist, or no per-read line is emitted at all (the startup line is emitted regardless, and is Information);
  • the effective Serilog minimum level at Debug, for the two Debug lines.

On that last point: SYRF__Serilog__MinimumLevel comes from .Values.logging.level, and staging sets level: debug in lowercase, whereas production carries the comment "Serilog requires capitalized full word". Serilog's enum parse is case-insensitive so it should bind, but verify it live before depending on the Debug lines — confirm a known-Debug line appears in kubectl logs before starting a soak whose read count comes from them.

What is centrally visible

  • Sentry is enabled in staging (monitoring.sentry.enabled: true, environment: staging), so the Error and Warning lines above surface there. The two Debug lines do not, and the Information startup line is below the event threshold too — read it from pod stdout.
  • Elastic APM is disabled in staging (elasticApm: { enabled: false }).
  • No OTLP, and no Loki/Elasticsearch/Fluent/Promtail/Alloy configuration exists in cluster-gitops. docs/architecture/system-overview.md describes "Aggregation: Loki or CloudWatch / Format: Structured JSON"; that is aspirational and does not match the deployed configuration. Confirm the actual cluster log pipeline out of band before writing a query against a named aggregator.

So, concretely: alert on Sentry for the divergence Warning; once the flags are on, count reads from the syrf.project_statistics.requests counter with dotnet-counters, cross-checked against pod stdout; and take fallback-reason and revision observations from the parity endpoint, whose fallbackReason, readSource, capacityFailure and watermarks fields give the same information without depending on log levels, a log pipeline or a pod's uptime.

Counting mutations

Since syrf #3482 there is a mutation counter: syrf.project_statistics.source_operation_receipts.committed, split by outcome into the point path and the source-only fallback, read per pod with dotnet-counters (see Telemetry). It is recorded only after each transaction's commit is confirmed, so a retried transient abort is not double-counted — but it resets on a restart and is per process, so it is a corroborating series rather than the primary evidence. It also counts only mutations that reached a statistics commit, which while the write flag is off is none of them.

The durable monotonic revisions on the control row remain the authoritative proxy, because they survive restarts and replicas, and the parity report exposes them: watermarks.committedProjectionRevision, watermarks.sourceInvalidationRevision and watermarks.clientInvalidationRevision. The difference between two observations of committedProjectionRevision is a mutation count. Poll the parity endpoint at a fixed cadence through the soak and keep every response; that series is the mutation evidence, and it doubles as the parity-sampling evidence Phase 6B asks for.

A disable request against an already Disabled project reaffirms its intent through the control transaction without advancing its epoch or invalidation revision. A never-enabled project retains its null serving-transition token: its control version orders initial requests, so a later first enable needs no rebuild and preserves backfilled rows. Previously disabled projects rotate their existing transition token. This supersedes an earlier re-enable still rebuilding; that re-enable must refuse if its captured token changed. Rebuild-failure responses report the durable mode and watermarks read after the failed sweep, including any intervening administrative transition. This uses the existing control CAS and administrative gates; no additional feature flag is needed.

A fleet disable also claims or reaffirms the singleton in a transaction. If the singleton was absent, it records Disabled with deployed versions and reviewer mode, leaving write/mode epochs and invalidation at zero. This prevents an earlier pending first enable from committing after the disable reports success. A newly requested first enable remains available; no epoch has been retired.