## Summary
- Removed the skip on `testBackfillWithBufferOneOverlap` - verified
passing on both V1 and CHASM: the deferred backfill start under
`BUFFER_ONE` now correctly re-enables once the running workflow
completes.
- Removed `testUpdateScheduleMemoOnly` and its registration -
`UpdateSchedule` has no partial-update path for the `schedule` field;
every update (including memo/search-attribute-only ones) requires
resending the full schedule, so the test's premise no longer applies.
Other previously-skipped scheduler tests (`TestFailedStart`,
`TestPauseUnpauseBetweenNominalAndJittered`) were checked and still
reproduce their underlying issues, so they remain skipped.
## Test plan
- [x] `go test -tags=test_dep ./tests/ -run
'TestScheduleCHASM/Backfill|TestScheduleV1/Backfill' -count=1 -v`
- [x] `go build -tags=test_dep ./tests/...`
## What changed?
- Added an opt-in recovery path for a child workflow missing from a
standby cluster.
- After the existing resend delay, VerifyFirstWorkflowTaskScheduled
asynchronously fetches the child state from the active cluster, applies
it locally, and verifies it again.
- Added deduplication, per-shard concurrency limits, metrics, namespace
checks, and transition-history gating.
Corrected the discard-time source check to verify the child workflow
rather than the parent.
- Moved the reusable in-flight resend tracker into the shared
workflowresend package.
- Updated the existing XDC parent-child test to assert that the missing
child and its first workflow task are restored.
## Why?
Cross-shard replication may deliver the parent’s
ChildWorkflowExecutionStarted event before the child workflow reaches
the standby cluster. Previously, verification repeatedly returned
NotFound and eventually discarded the standby task, leaving the child
missing.
This adds the child-side symmetric recovery behavior to the parent
resend implemented in #11424 .
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
### Rollout
`history.enableChildWorkflowResend` default to `false`.
### Known
- Enabling this feature introduces additional cross-cluster state-sync
traffic. It is disabled by default and protected by deduplication and a
per-shard concurrency limit.
- Regular replication may race with state sync; duplicate application is
treated as success.
- The resend delay and replication timeout should remain below the
standby task discard delay so recovery has time to complete.
## What changed?
- Atomically return and hold the next parent workflow task after
starting child workflows.
- Release pending intercepted replication tasks during test cleanup.
- Ensure each blocked replication task executes at most once.
## Why?
This is a follow-up to
[#11789](https://github.com/temporalio/temporal/pull/11789).
The test separately scheduled and polled the next workflow task, leaving
a race where a duplicate child outcome could be recorded before the
parent workflow task started. When the assertion failed, blocked
replication tasks were not released, potentially stalling subsequent XDC
tests until CI timed out.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
Ran the affected test with the race detector 10 times:
```bash
go test -tags test_dep -race -count=10 ./tests/xdc \
-run '^TestFuncClustersTestSuite$/^EnableTransitionHistory$/^TestNaturallyBufferedChildWorkflowOutcomesFlushedToLosingBranch$' \
-args -persistenceType=sql -persistenceDriver=sqlite
## What changed?
Stop installing Temporal-specific process-global OTEL error handlers.
OTEL errors now use the default ie stderr.
## Why?
The process-global OTEL handler retained the Temporal server logger
after shutdown; ie it leaked.
## What changed
- Add `worker_deployment_name` and `worker_build_id` labels to Workflow
Task completion and failure metrics.
- Add the split labels to Activity success, failure, cancellation,
timeout, and completion-latency metrics.
- Add the split labels to Workflow Task and Activity schedule-to-start
latency and poll-time task-dispatch latency.
- Continue emitting the existing combined `worker_version` label on
task-dispatch latency for compatibility.
- Reuse `metrics.breakdownByBuildID` as the task-queue-scoped
cardinality gate for the new labels.
## Which metrics changed
Every metric below now emits both `worker_deployment_name` and
`worker_build_id` when `metrics.breakdownByBuildID` is enabled and
deployment attribution is available. When the gate is disabled or
attribution is unavailable—including unversioned tasks and timeouts
before a worker starts—both labels remain present with empty values.
### Workflow Task outcomes
- `workflow_tasks_completed`
- `failed_workflow_tasks`
### Activity outcomes
- `activity_success`
- `activity_fail`
- `activity_task_fail`
- `activity_cancel`
- `activity_timeout`
- `activity_task_timeout`
### Activity latency
- `activity_end_to_end_latency` (deprecated; use
`activity_start_to_close_latency` instead)
- `activity_start_to_close_latency`
- `activity_schedule_to_close_latency`
### Task-routing latency
- `task_schedule_to_start_latency`
- `task_dispatch_latency` (continues to emit the existing combined
`worker_version` label as well)
## Why
- To improve user-experience for worker-versioning by having more
insights
- Worker Deployment name and build ID need to be independently
filterable. Emitting them separately avoids requiring consumers to parse
the combined `worker_version` value. This PR preserves `worker_version`
on `task_dispatch_latency` for compatibility and does not remove or
deprecate it.
## How each metric was tested
The functional tests run real versioned Workflow and Activity workers
with `metrics.breakdownByBuildID` enabled. They assert that emitted
server metrics contain the actual Worker Deployment name and build
ID—not merely that the label keys exist.
| Functional scenario | Metrics verified | What is asserted |
|---|---|---|
| Workflow and Activity task dispatch |
`task_schedule_to_start_latency`, `task_dispatch_latency` | Both
Workflow and Activity task series contain real deployment/build values
when enabled. Disabled and unversioned cases emit empty values. |
| Successful Workflow and Activity | `workflow_tasks_completed`,
`activity_success`, `activity_end_to_end_latency`,
`activity_start_to_close_latency`, `activity_schedule_to_close_latency`
| Successful completion series contain real deployment/build values. |
| Terminal Activity failure | `activity_task_fail`, `activity_fail`,
`activity_end_to_end_latency`, `activity_start_to_close_latency`,
`activity_schedule_to_close_latency` | Both the failed attempt and
terminal-failure series retain real worker attribution. |
| Activity cancellation | `activity_cancel` | The cancellation series is
attributed to the worker that started the Activity. |
| Terminal Activity timeout | `activity_task_timeout`,
`activity_timeout` | Both attempt-level and terminal timeout series
retain the started worker’s deployment/build values. |
| Failed Workflow Task | `failed_workflow_tasks` | A real versioned
Workflow Task is polled and failed; the resulting series contains the
poller’s deployment/build values. |
### Local end-to-end validation
The PR server was also run locally with the Worker Deployment versioning
canary and a `bench-go` workload.
Prometheus was scraped directly to verify that:
- real `worker_deployment_name` and `worker_build_id` values were
emitted;
- versioned and unversioned series were both accepted without
inconsistent-label errors;
- `activity_success`, `activity_task_fail`,
`activity_end_to_end_latency`, `activity_start_to_close_latency`, and
`activity_schedule_to_close_latency` carried the expected real
deployment/build values.
The remaining failure, cancellation, timeout, and failed-Workflow-Task
paths are covered by the functional tests above.
NEW: Also tested each of these 13 metric families in a cloud test cell
with sample metrics pasted here:
https://grafana.tmprl-internal.cloud/d/shdb6gf/new-dashboard?orgId=1&from=2026-08-27T00:00:00.000Z&to=2026-08-27T23:59:59.000Z&timezone=utc
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Broad metrics surface area across History and Matching with
cardinality gated by config; deployment attribution on timeouts and
eager starts changes which tag values appear on existing metric names.
>
> **Overview**
> Adds **`worker_deployment_name`** and **`worker_build_id`** to History
and Matching metrics so worker versioning can be filtered without
parsing the combined **`worker_version`** label.
**`metrics.breakdownByBuildID`** (task-queue scoped) controls whether
those tags carry real values or empty strings;
**`task_dispatch_latency`** still emits **`worker_version`** for
compatibility.
>
> **Workflow tasks:** Completion and failure counters
(`workflow_tasks_completed`, `failed_workflow_tasks`) now go through
shared helpers that attach versioning behavior plus deployment tags from
the poller’s **`DeploymentOptions`**.
>
> **Workflow activities:** Success, failure, cancel, timeout, and
related latency metrics get the same split labels via
**`VersioningMetricContext`** on respond paths (from request deployment
options), timer-driven timeouts (from **`LastDeploymentVersion`** when
an attempt had started), and schedule-to-start latency on
**`RecordActivityTaskStarted`**. Eager activities started during WFT
completion now record the completing worker’s deployment on the started
event.
>
> **Standalone activities (CHASM):** Completion/timeout/cancel paths use
**`completionMetricsHandler`**, which always includes the deployment
label keys with empty values until standalone versioning exists—keeping
Prometheus label-set parity with workflow-embedded activities.
>
> **Other:** **`AddActivityTaskStartedEvent`** clears stored deployment
when the poller is unversioned; Matching **`task_dispatch_latency`**
adds the new tags alongside **`worker_version`**.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
1c60b05f06. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## Summary
- add transition-history XDC E2E coverage for every buffered event that
current production code can emit
- create buffered events naturally through workflow/activity APIs,
workflow commands, transfer and timer tasks, child callbacks, Nexus
callbacks, and conflict reapplication
- hold a workflow task, fail over, let its timeout flush the old active
cluster's buffer, and then release replication to exercise conflict
resolution
- prove every flushed event is persisted with a real event ID on the
non-current losing history branch
- verify cherry-pickable inputs are reapplied to the winning branch and
branch-dependent outcomes are skipped
- reproduce the Nexus #10986 conflict shape: reapply a shared-operation
completion while skipping an operation that exists only on the losing
branch
- keep all test infrastructure in
`buffered_events_replication_helpers_test.go`; no production or testcore
code is changed
## Coverage
The scenarios naturally cover all 30 production-reachable buffered event
types:
- activity started, completed, failed, timed out, and canceled
- timer fired
- workflow cancel requested and signaled
- workflow options updated, paused, and unpaused
- external signal/cancel success and failure callbacks
- child start failure, child started, and all five child terminal
outcomes
- update admitted through real conflict reapplication
- Nexus started, completed, failed, canceled, timed out, and both
cancel-request outcomes
Four values in the buffered-event set cannot be naturally buffered on
current `main` and are intentionally not fabricated:
- `WORKFLOW_EXECUTION_UPDATE_REJECTED`,
`WORKFLOW_PROPERTIES_MODIFIED_EXTERNALLY`, and
`ACTIVITY_PROPERTIES_MODIFIED_EXTERNALLY` have no production emitter
- `WORKFLOW_EXECUTION_TIME_SKIPPING_TRANSITIONED` has an emitter, but
its production precondition rejects workflows with a pending workflow
task
## What the tests prove
- expected events first exist in mutable-state buffering with
`BufferedEventID`
- failover creates a winning branch before the old active cluster's
replication is released
- workflow-task timeout flushes the old active cluster's buffer
- every expected event is found on the non-current losing branch with a
positive, non-buffered event ID
- a naturally buffered marker signal proves the losing batch passed
through conflict reapplication
- signals and updates that are eligible for reapplication reach the
winner
- activity, timer, child, external-command-result, Nexus
cancel-request-result, and losing-only Nexus events remain on the losing
branch when the winner lacks the state needed to apply them
- shared Nexus operation outcomes are reapplied by scheduled event ID,
while losing-only operations are skipped
- current histories converge after the relevant replication tasks are
released
## Determinism
- replication is intercepted and blocked only for the workflow under
test
- the next workflow task is explicitly created and polled before
callbacks are released
- Nexus handlers use response barriers released only after the held
workflow task is confirmed
- losing branches are read directly from persistence and identified by
their expected events
- assertions poll observable conditions instead of relying on fixed
sleeps
- the pause feature flag is overridden only for the test that exercises
pause/unpause
- scenarios skip when transition history is disabled; Nexus conflict
scenarios also skip for the CHASM implementation
## Validation
- repository `gci` formatting and `git diff --check`
- `GOLANGCI_LINT_FIX=false GOLANGCI_LINT_BASE_REV=HEAD~ make lint-code`
- XDC package compilation with `test_dep`
- focused transition-history E2E runs for mixed inputs/update
reapplication, activity outcomes, child outcomes, external workflow
outcomes, and Nexus outcomes
- CI PostgreSQL XDC, unit, integration, mixed-brain, formatting, and all
linter checks pass
## What changed?
- Add the Temporal failure response capability header to Nexus
cancellation requests in both HSM and CHASM executors.
- Respect the `nexusoperation.useNewFailureWireFormat` dynamic config
when adding the header to cancellation requests.
## Why?
Nexus start requests explicitly advertise support for Temporal failure
responses, but cancellation requests relied on the request header map
being mutated elsewhere.
[#10720](https://github.com/temporalio/temporal/pull/10720) and
[#10746](https://github.com/temporalio/temporal/pull/10746) introduced
defensive copies of Nexus request headers. That exposed the missing
capability header on cancellation requests, causing SDK workers to use
the legacy failure wire format for cancel handler failures.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
Cancellation requests now advertise support for Temporal failure
responses by default. The existing dynamic config can disable the
behavior if needed.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What changed?
1. Object leak reports now includes sorted heap addresses for unexpected
retained objects.
2. The leak test can optionally capture a raw heap dump with
`LEAK_HEAP_DUMP=1`.
## Why?
Local debugging; it helps identify the runtime root retaining a reported
object.
## Potential risks
Raw heap diagnostics can be large and may contain sensitive process
data. They are only captured after an object leak failure when
`LEAK_HEAP_DUMP=1` is explicitly set.
## What changed?
**1. Delete execution replication tasks carry a failover version.**
Stamped with the source cluster's
failover version at generation (`shard/context_impl.go:1013`), carried
in the existing
`ReplicationTaskInfo.version` (no proto change), and skipped on apply
when older than the target's
namespace failover version (`executable_delete_execution_task.go:130`).
Still applied: unversioned
tasks (queued pre-upgrade), tasks newer than the target's namespace
entry, and deletions synthesized
by versioned-transition tasks.
**2. `DeleteWorkflowExecution` is rejected on a cluster passive for the
workflow**
(`workflow_handler.go:2513`), with the usual `NamespaceNotActive`. In
the frontend, because the history
path is shared with replication apply, which must delete on passive
clusters — that path, the
delete-namespace worker, and admin force-delete are untouched.
## Why?
An unversioned delete task generated before a failover kept being
applied afterwards, deleting a run
the new active cluster owns.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
```
temporal --address :7233 workflow -n global-ns delete -w pay-invoice-0 -r 019fcf5b-996e-7416-9a2d-4ffb0ae7fc13 --grpc-meta xdc-redirection=false
WARNING: Deleting Workflow Executions in a global Namespace removes them from all replicas. Requests sent to a passive cluster are forwarded to the active cluster by default; to target the passive cluster directly, specify `--grpc-meta xdc-redirection=false`.
Delete Workflow "pay-invoice-0" with Run ID "019fcf5b-996e-7416-9a2d-4ffb0ae7fc13"? y/N y
Error: failed to delete workflow: Namespace: global-ns is active in cluster: cluster-b, while current cluster cluster-a is a standby cluster.
```
## Potential risks
- A deletion issued just before a failover is dropped on targets;
cleanup waits for the new active
cluster's retention timer. Delayed, not leaked — intended trade-off.
- API behavior change: deleting against a passive cluster now fails,
including batch delete
(`batcher/activities.go:704`). No killswitch; admin force-delete is the
escape hatch.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
- Keep attaching completion callbacks to every CHASM scheduler workflow
start, including actions whose resolved overlap policy is `ALLOW_ALL`,
so start requests remain safe across rolling upgrades.
- Exclude scheduler-wide last-completion result/failure input from new
`ALLOW_ALL` actions.
- After `StartWorkflowExecution` succeeds, remove the new `ALLOW_ALL`
`BufferedStart` in the same transaction. Separately, copy its start data
to `ScheduleInfo.RecentActions` as a start-only `RUNNING` history
record.
- Resolve an unspecified pending-start policy while building the
V1-to-V2 migration request, so every V2 dispatch and retry uses the
policy persisted on the buffered start.
- For a pre-existing callback, use the policy stamped on that start
rather than resolving against the schedule's current policy; an
unstamped V1-migrated running workflow remains tracked, as it was in V1.
- Preserve compatibility for previously persisted `ALLOW_ALL` callbacks
without updating last-completion state or `PauseOnFailure`.
- Carry start-only history through CHASM-to-V1 migration.
- Merge start-only and completion-tracked history by actual start time,
retaining the newest ten actions across both sources.
- Re-run generation after recording a start-only action, so a finite or
manual-only schedule rearms its idle timer from that start.
## Behavior
This makes CHASM match modern V1:
- A new `ALLOW_ALL` workflow starts normally, but is absent from
`DescribeSchedule.Info.RunningWorkflows` and does not consume
active-buffer capacity.
- It appears in `DescribeSchedule.Info.RecentActions` (and List's recent
actions) with its start time, workflow execution, and `RUNNING` status.
- Its completion callback remains attached for rolling-upgrade
compatibility. Because the start has already moved out of active
buffered state, the callback is ignored and success or failure cannot
change shared completion input or `PauseOnFailure`; its recent status
therefore remains `RUNNING`.
- For a final `ALLOW_ALL` action, an idle task armed before the start is
invalidated by the newer start time; generation immediately arms its
replacement, so the schedule still closes after `IdleTime`.
- If an older handler retained an `ALLOW_ALL` start, its callback
remains a compatibility path: the terminal action record is retained,
but its completion cannot update shared completion state or pause the
schedule once handled by this version.
- `DescribeSchedule.Info.RecentActions` is ordered by actual start time
and bounded to the newest ten actions across start-only and
completion-tracked history. Newer retained actions evict the oldest;
completion does not remove a start-only action.
- Non-`ALLOW_ALL` actions remain active and appear in `RunningWorkflows`
until their completion is handled.
- A V1 pending start with an unspecified override snapshots the
schedule's effective policy when its V2 migration request is built.
- A V1-migrated `RunningWorkflows` entry has an unspecified policy but
is nevertheless tracked: its completion updates sequential state and may
pause the schedule, regardless of the schedule's current `ALLOW_ALL`
default.
## Why?
`ALLOW_ALL` actions are independent executions. Tracking their
completions made scheduler-wide last result/failure and `PauseOnFailure`
depend on callback arrival order, and retaining them as active could
affect overlap and capacity behavior.
Callbacks remain attached because `StartWorkflowExecution` deduplicates
by request ID without reconciling callback differences. Keeping the
request callback-compatible prevents a mixed-version retry from
retaining a start that waits for a callback the workflow never received.
## How did you test it?
- [x] added unit coverage
- [x] added shared V1/CHASM functional coverage
Commands:
- `go test -tags test_dep ./chasm/lib/scheduler/... -count=1`
- `go test -tags test_dep ./tests -run
'^TestSchedule(CHASM|V1)/TestAllowAllDescribeContract$' -count=1`
- `make fmt-imports`
- `git diff --check`
- `env GOCACHE=/tmp/sch-038-gocache go vet -tags
disable_grpc_modules,,test_dep -vettool=.bin/errortype
-style-check=false ./chasm/lib/scheduler/...`
The functional test uses workflow signals to control completion. It
asserts counters, buffer size, recent-action status/timestamps, active
workflows, and failure/pause isolation across both backends. A CHASM
functional idle-close case verifies that a final `ALLOW_ALL` action
still closes after `IdleTime`. The migration regression and
callback-reason metric test drive real component transactions through
the CHASM test engine.
`make lint-code` currently exits before analysis with `no go files to
analyze` from its `--new-from-rev` filter, despite the Go diff; package
`go vet` is clean.
## Additional observability
Ignored callbacks are tagged as either `unrecognized_request_id` or
`already_completed`. A newly started `ALLOW_ALL` callback is expected to
be unrecognized after its buffered start moves to start-only history. An
already-completed callback is a valid redelivery (for example, after a
workflow reset). ~~Both preserve scheduler state while emitting a
warning and counter increment.~~ Missing request IDs are now metric-only
because they include expected `ALLOW_ALL` callbacks; known
`already_completed` redeliveries still emit the warning, event, and
counter.
## Potential risks
Keeping callbacks attached avoids permanently orphaning buffered starts
when old and new binaries race on the same request ID. During a rolling
upgrade, however, old and new requests still differ in whether
scheduler-wide last-completion result/failure input is included for
`ALLOW_ALL`; the request that wins deduplication determines whether that
workflow receives the legacy input. An older callback handler can also
temporarily apply the legacy completion and pause semantics. These
mixed-version differences end once the rollout completes, and the
callback ensures an older handler cannot wait indefinitely.
New `ALLOW_ALL` terminal status is intentionally not reflected in
schedule Describe/List results, as in modern V1. ~~Its expected late
callback is recorded as `unrecognized_request_id`, which adds callback
delivery plus warning/metric volume compared with omitting callbacks.~~
Its expected late callback remains recorded as
`unrecognized_request_id`, but only as a metric; warning and event
logging are suppressed until these callbacks can be safely omitted.
Migration resolves an unspecified pending-start policy at the V1-to-V2
boundary. If the schedule policy changes while that start remains
pending and the schedule then rolls back to V1, the explicit migrated
policy is preserved instead of inheriting the newer schedule policy.
This is a narrow semantic difference that keeps V2 dispatch and retry
behavior stable.
---------
Co-authored-by: David Porter <david.porter@temporal.io>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
## What changed?
- Added a reusable parent-child XDC test harness for controlling
replication task application, delay, ordering, and omission around
namespace failover.
- Added five functional scenarios covering orphaned children, missing or
incomplete children, task discard, missing parent completion, and parent
resend recovery.
- Added unit tests for the replication gate and
legacy/transition-history task decoding.
Configured two history shards per cluster and placed parent and child
workflows on different shards.
- Updated XDC synchronization checks to support multiple history shards.
## Why?
Parent-child replication failures depend on rare cross-shard ordering
and failover timing, making them difficult to reproduce reliably.
These tests deterministically construct the relevant partial states
while still exercising real Temporal services, persistence, replication,
verification RPCs, task retry/discard behavior, and namespace failover.
The harness also makes future scenarios easier to add and review.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
Go 1.27 prerequisite that applies the `errorsastype` Go fixer and its
required error-interface updates.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Adds generic `await.Rcv` and `await.Snd` helpers that bound blocking
channel operations by the test context and report closure or
cancellation clearly.
Replaces both previous helpers and existing unsafe channel rcv/snd
across tests.
## What changed?
- Moves `activity.linkValidator` into `common/links`.
- Moves `callback.Validator` into `common/callbacks`
In addition, this PR performs some minor refactorings for consistency
and clarity.
- Moved some `links.Validator`-specific tests from
`chasm/lib/activity/validator_test.go` elsewhere.
- Introduced a `callbacks.ValidatorConfig` to bundle all of the specific
settings. (Since we'll need to wire 3+ more parameters when updating the
`callbacks.Validator` to support worker callbacks.)
> The singular package names `common/link` or `common/callback` would be
more consistent. But `common/links` already existed, there are other
pluralized ones like `common/enums` or `common/headers`. And IMHO, the
plural seems a little more applicable since the validations are only on
groupings of links or callbacks.
## Why?
The `activity.linkValidator` and `callback.Validator` types are great,
but they aren't able to be used as across other CHASM components as
easily. Moreover, `callback.Validator` uses types that are exposed from
the CHASM `callback` package, it will lead to circular dependencies in
the future. (I'm hitting this now in PRs for landing worker callbacks.)
Moving the `commonpb` protobuf validation into `common/` means we can
better separate the the distinction between validation logic and the
CHASM executions that rely on it.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
This should just be a standard refactoring. There should not be any new
validation checks enabled on codepaths where they weren't already
present. (Or in test cases, we initialize fields of
`callback.ValidatorConfig` that weren't used before.)
## What changed?
Fixes a build error, seemingly introduced when multiple changes were
merged automatically after approval.
## Why?
Because a broken build stops the flow of spice. And the spice must flow.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None. But there might be other issues, I'm curious to see if there are
any other issues flagged by CI/CD.
## What changed?
Pass one test-owned context through namespace creation, namespace cache
polling, and search attribute registration during functional test setup.
## Why?
Reusing the test context avoids creating independent timeout contexts
for each setup RPC and ties their resources to the test lifecycle.
## What changed?
`GetWorkflowExecutionHistory` and `GetWorkflowExecutionHistoryReverse`
now check `branch_token` in the page token against the token in mutable
state.
## Why?
To confirm if it is still the correct branch after conflict resolution.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
Go 1.27 prerequisite that applies the `slicesbackward` Go fixer for
reverse iteration.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What changed?
Added Temporal Nexus attributes to spans.
## Why?
Domain attributes make Nexus traces more useful.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Adds server support for WorkflowQuery-backed Nexus Operations
## Why?
Part of effort to expose all Temporal primitives as Nexus Operations
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## TODO:
- [x] https://github.com/temporalio/api/pull/842
## What changed?
Adds `testcontext.EnsureRemaining` and has `await` use it so long await
calls can request additional test-scoped context time while still
respecting the test context cap.
## Why?
Await calls can need more time than the default test context has left
(esp after the environment setup). Extending the test timeout in this
way allows for (1) stuck tests to fail earlier than the default test
timeout and (2) legitimately longer running tests to pass without
manually tweaking the test timeout.
---------
Co-authored-by: Sean Kane <sean.kane@temporal.io>
## What changed?
- Clear an activity’s timer-task status when it is unpaused so timeout
tasks are regenerated.
- Make ResetActivity with keepPaused=false fully unpause both scheduled
and running activities, including clearing pause metadata.
- Add and strengthen unit and functional coverage for unpause,
reset-unpause, timer regeneration, and keepPaused=true.
## Why?
Timeout tasks can fire while an activity is paused and be discarded.
Previously, the activity’s timer-task status still indicated that those
tasks existed, preventing them from being recreated after unpause and
potentially making the timeout ineffective.
ResetActivity also bypassed normal unpause handling, and running
activities returned early without clearing their paused state.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
Unpausing now invalidates existing timeout tasks and recreates the next
applicable timer during transaction close. This is correct, but a
behavioral change. Stale queued tasks may still be processed and
discarded through the existing stamp validation.
## What changed?
Makes Standalone Activity conflict updates idempotent by recording the
`requestID` when attaching callbacks or links and recognizing duplicate
request IDs. I needed to add a dedicated CHASM error for
## Why?
This prevents a successful attachment whose response was lost from
failing on retry or duplicating/replacing callbacks and links.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## The bug
`applyUpdatesToSubStateMachine` shadowed the variable it meant to
populate:
```go
var existing V // outer: never assigned
if existing, ok := pendingInfos[key]; ok { // := declares a NEW existing, scoped to the if
...
ms.approximateSize -= existing.Size() + getSizeOfKey(key) // inner one — correct here
}
val := updated
if sanitizeFn != nil {
val = common.CloneProto(updated)
sanitizeFn(existing, val) // outer one — always nil
}
```
`:=` only requires *one* new variable on the left (`ok`), and the `if`
init statement is its own scope, so `existing` is redeclared there
rather than assigned. It compiles because both variables are used, and
`go vet` does not check shadowing by default.
`sanitizeFn` has therefore always received `current == nil`.
## What this activates
Two of the five `sanitizeFn` implementations depend on `current`; the
other three (timers, request cancels, signals) ignore it or are `nil`,
so they are unaffected.
**Activities.** `getActivityTimerTaskStatus` short-circuits to
`TimerTaskStatusNone` when `current` is nil, so every applied activity
update cleared the *entire* timer task mask rather than only the bits
whose deadline moved. The deadline comparison added in #11565 has never
taken effect — that PR is currently a no-op.
**Child executions.** `if current != nil { incoming.Clock =
current.Clock }` never ran. `Clock` is a local shard vector clock that
`sanitizeChildExecutionInfo` strips before replicating, so the incoming
copy always arrives nil — the guard exists precisely to restore the
local value. Without it, every replicated update to an existing child
execution erased it.
## Risk
Both are restorations of intended behavior, not new behavior, and both
previously failed in the safe direction:
- An over-cleared activity mask regenerates a timer task that is already
pending. Duplicates are dropped at execution
(`processSingleActivityTimeoutTask` re-derives the sequence and fires
what expired), so the symptom was extra timer-queue writes, not missed
timeouts.
- A nil child clock is tolerated by callers — see the comment in
`recordchildworkflowcompleted/api.go`: *"it should be fine e.g. that
ci.Clock is nil"*.
The child execution change is a **no-op unless the local cluster
recorded a clock by starting the child itself**, which for a passive
cluster means after a failover. On a standby that never started the
child, both sides are nil.
For activities, preserved bits suppress recreation of a timer task, so
it is worth being explicit about why that is safe on the passive side:
an expired-but-unresolved standby timer returns `ErrTaskRetry` and stays
in the queue rather than being consumed, so there is nothing to
recreate. The one path that does drop a task, past
`StandbyTaskMissingEventsDiscardDelay`, logs a warning and increments
`task_errors_discarded`, so it cannot happen silently.
## Tests
Two new tests at the `applyUpdatesToSubStateMachines` level, one per
activated `sanitizeFn`. **Both fail with the shadowing restored**
(verified: `expected: 13, actual: 0` for the mask; `local child
execution clock was not carried over` for the clock).
The gap they close: the existing `getActivityTimerTaskStatus` tests call
the decision function directly with a non-nil `current`, so no unit test
could observe the *caller* passing nil. Nine passing tests, zero
coverage of the wiring.
Two pre-existing tests needed updating, both informative:
- `TestApplyMutation` / `TestApplySnapshot` failed with *"Unexpected
call to IsVersionFromSameCluster"* — that mock was never needed because
the nil short-circuit made the cluster check unreachable. The gap is
itself evidence the path was dead.
- `verifyActivityInfos` asserted `s.Equal(int32(TimerTaskStatusNone),
actual.TimerTaskStatus)`, encoding the bug and blocking any correct fix.
Replaced with the invariant that actually matters:
```go
s.Zero(actual.TimerTaskStatus&^originStatus, "TimerTaskStatus gained a
bit that was not set locally")
```
Applying an update may carry over or drop locally set bits, but must
never introduce one that was not already set — claiming a timer task
exists when none does is the direction that loses timers. That holds
regardless of which bits a given case preserves, so it will not need
rewriting as the mask logic evolves.
## Related, tracked separately
While auditing whether the mask can be trusted, one path was found that
moves a deadline without invalidating the mask: unpausing an activity.
Paused activities are excluded from the timer sequence, so a timeout
task firing during a long pause is dropped as an invalid task, yet
`unpauseActivityInfo` leaves the bit set. That is pre-existing and
active-side, independent of this PR, and is being tracked as its own
change.
It is worth noting here because this PR removes the passive side's
accidental repair for that class of stale bit: today's blanket wipe
clears any stale bit on the next replicated activity update.
---
`go build ./service/...`, `go vet`, and golangci-lint (repo config,
`--new-from-rev=origin/main`) clean; `./service/history/workflow/...
./service/history/ndc/...` pass across 3 consecutive runs.
Two pre-existing flakes are skipped in those runs and unrelated to this
change (both reproduce on unmodified `main`):
`TestTaskRefresherSuite/TestRefreshSubStateMachineTasks`
(nanosecond-differing HSM deadlines, map iteration order — fails 7/12 on
main) and
`TestMutableStateSuite/*/TestApplyWorkflowExecutionOptionsUpdatedEvent_TimeSkippingConfig`
(wall-clock resolution, ~1/30).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
## What changed?
Validate the retry delay is a valid proto duration.
## Why?
Prevent malformed retry delays from poisoning Nexus completions.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
Potentially users could have been sending an invalid proto duration,
unclear how exactly, and now we would fail their request.
## What changed?
1. Binds httpCaller after setting httpClient
2. Uses TransitionStarted.Possible for complete-before-start
3. Records request time for nexus operation cancel
## Why?
1. Binds httpCaller after setting httpClient
When clusterID isn't set, a non-nil httpCaller with a nil receiver gets
passed to `nexusrpc.NewHTTPClient`. Since the httpCaller is non-nil, it
will skip the check that sets the nil httpCaller to the default caller.
2. Uses TransitionStarted.Possible for complete-before-start
Matches HSM, for if a start response is lost, and a completion lands
while while the operation is in BACKING_OFF.
3. Records request time for nexus operation cancel
This just seems like it wasn't being recorded.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Batch operations now target Paused executions in addition to Running
ones. The filter auto-appended by adjustQueryBatchTypeEnum changed from
ExecutionStatus='Running' to ExecutionStatus='Running' OR
ExecutionStatus='Paused', affecting all workflow batch types
(terminate/signal/cancel/update-options) and activity batch types
(unpause/update-options/reset/terminate/cancel).
## Why?
A paused execution is still non-terminal. Users issuing a batch
terminate/signal/cancel reasonably expect it to apply to paused targets,
but the old Running-only filter silently skipped them.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
- The filter now references ExecutionStatus='Paused' on every batch
operation. This is ok as it's an additive clause to the visibilty query
- This would be changing behavior for batch callers as paused activities
are now affected.
## What changed?
1. perf improvement: no time-skipping task regen on full refresh
2. new functional test: claim Nexus HSM timers are part of in-fight
nexus operation, and won’t be skipped, add functional test
3. new functional test: add a functional test to verify time skipping
won't change retention time
4. trivial bug fix: time-skipping: task regen didn’t read
EnableWorkflowExecutionTimeoutTimer
## Why?
- correctness related No. 4 though it is an edgy case
- perf related No.1
- test coverage No.2,3
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
Wrapped the frontend Nexus dispatch routes with the shared OpenTelemetry
HTTP handler.
## Why?
Nexus HTTP requests need an inbound server span to connect the caller
trace.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Removed blanket timestamp wrapping
- cron/retry/start child wf use virtual time directly
## Why?
there was a double-shifting for CaN but not for other cases
it shall be fixed unifying the clock used by all virtual time
propagating cases, and this PR chooses to unify in a way that all cases
propagate use virtual time
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
Instrumented local frontend, CHASM callback, and external Nexus
operation HTTP clients with the shared OpenTelemetry transport.
Legacy HSM callbacks and cross-cluster forwarding are intentionally out
of scope.
## Why?
Outbound Nexus HTTP calls need to carry trace context so callbacks and
internal frontend calls remain connected to their originating spans.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
To editorialize, the presence and difficulty of spotting these bugs
makes me want to refactor this code more, but I'm going to defer that
for now. Anyway, checked the V1 code's equivalent and also checked that
the integration-test catches the problem before applying fix.
## LLM Summary
- include an existing deferred `BUFFER_ONE` start in overlap resolution
- keep the earliest buffered occurrence and reject later arrivals
- retain catchup-window expiry for an already-deferred occurrence
## Problem
`BUFFER_ONE` permits at most one pending occurrence while an action
workflow is running. CHASM represents buffered-start lifecycle with
`Attempt`:
- `0`: newly enqueued and not processed;
- `-1`: processed but deferred while another workflow is running;
- `1+`: executing or retrying.
Before this change, `InvokerProcessBufferTask` passed only `Attempt ==
0` starts to overlap resolution. This produced the following sequence:
1. A workflow is running.
2. Occurrence A becomes due under `BUFFER_ONE`.
3. A is retained and marked deferred with `Attempt == -1`.
4. Occurrence B becomes due before the workflow closes.
5. Processing ignores A and presents only B to the shared overlap
resolver.
6. The resolver sees no occupied one-element buffer and retains B as
well.
The invoker can therefore contain both A and B, violating `BUFFER_ONE`
and potentially executing an unexpected workflow.
## Fix
Include a deferred start in the pending overlap set when its effective
overlap policy resolves to `BUFFER_ONE`:
```go
return start.Attempt == 0 ||
(start.Attempt == -1 &&
scheduler.resolveOverlapPolicy(start.GetOverlapPolicy()) ==
enumspb.SCHEDULE_OVERLAP_POLICY_BUFFER_ONE)
```
This lets the existing shared V1 `ProcessBuffer` logic see the occupied
buffer, retain the earliest occurrence, and reject later arrivals. The
effective policy is resolved because a buffered start may store
`UNSPECIFIED` and inherit the schedule current policy.
The special handling is limited to `BUFFER_ONE`; other deferred starts
should not be broadly reprocessed merely because a new occurrence
arrived.
## How this was found
The Schedule V1-to-CHASM production-history replay initially reported
5,355 V1 actions versus 4,961 CHASM actions, suggesting CHASM had
skipped 394 workflows.
Tracing the first independently different action showed the opposite:
CHASM emitted one extra occurrence. Because that workflow did not exist
in V1 history, the replay had no completion event to apply to it. It
remained running in simulated CHASM state and blocked hundreds of later
starts. The apparent 394-action deficit was a downstream alternate-chain
cascade, not 394 independent defects.
Inspecting state at the first difference revealed one running workflow,
one deferred `BUFFER_ONE` start (`Attempt == -1`), one new start
(`Attempt == 0`), and only the new start participating in overlap
resolution. A focused unit test reproduced that exact state.
After the fix, the representative converged to 5,355 actions on both
sides with identical workflow identities; only observation-time
differences remained.
## Impact
The direct product impact is one additional retained and potentially
executed workflow. This is high severity because `BUFFER_ONE` explicitly
bounds pending work, and an unexpected workflow may perform externally
visible or non-idempotent actions.
## Testing
- `TestProcessBufferTask_BufferOneKeepsExistingDeferredStart` verifies
that the first deferred occurrence occupies the buffer and the later
occurrence is rejected.
- `TestProcessBufferTask_BufferOneDropsDeferredStartPastCatchupWindow`
verifies that a deferred occurrence is still dropped when its catchup
deadline has expired.
```sh
go test -tags test_dep ./chasm/lib/scheduler \
-run "^TestProcessBufferTask_BufferOne" -count=1
```
## What changed
- move the seven Versioning3 query-routing tests into a dedicated
`Versioning3QuerySuite`
- keep each suite on `RunLegacySequential` so methods remain sequential
while the suites can run concurrently
- centralize shared Versioning3 environment setup and matching-behavior
iteration for reuse by future splits
## Why
This is the next incremental split of the large Versioning3 functional
suite. It improves suite-level parallelization without reintroducing the
method-level flakes seen with `parallelsuite.Run`.
## Validation
- confirmed the combined Versioning3 test inventory remains unchanged at
109 methods
- `GOWORK=off go test -tags test_dep ./tests -run '^$'`
- `GOWORK=off go test -tags test_dep ./tests -run
'^TestVersioning3FunctionalSuite/TestPinnedTask_NoProperPoller$'
-count=1`
- `GOWORK=off go test -tags test_dep ./tests -run
'^TestVersioning3QueryFunctionalSuite/TestQueryWithPinnedOverride_NoSticky$'
-count=1`
- `GOWORK=off go vet -tags test_dep ./tests`
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> Test-only refactor with no production code changes; behavior is
preserved by moving tests and centralizing helpers.
>
> **Overview**
> Splits **Versioning3** functional tests so query-routing coverage
lives in a new **`Versioning3QuerySuite`**
(`versioning_3_query_test.go`), while the main **`Versioning3Suite`**
keeps the rest of the v3 scenarios.
>
> Shared setup is centralized in **`versioning_3_suite_test.go`**:
**`newVersioning3TestEnv`** (dynamic config, deployment limits, short
build IDs) and **`runVersioning3TestWithMatchingBehavior`** (iterate all
matching behaviors). **`Versioning3Suite`**,
**`Versioning3OneTimeOverrideSuite`**, and the new query suite all call
these helpers instead of duplicating env bootstrap.
>
> The moved query tests cover pinned/unpinned queries (sticky and
non-sticky), drained versions with and without pollers, and rollback
from drained state. **`Versioning3OneTimeOverrideSuite`** only switches
**`setupEnv`** to **`newVersioning3TestEnv`**.
>
> Test inventory stays at **109** methods; suites still use
**`RunLegacySequential`** for method ordering while allowing concurrent
suite runs.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
a4b2cb6ffa. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Clear removed tests from the shared cluster's backing slice.
## Why?
`sharedClusterT` shortened `activeTests` without clearing the removed
interface slot. The backing array could therefore retain completed tests
and their functional test state.
## What changed?
Change datetime format to always include nanos component in
Elasticsearch Visibility.
Also added Visibility integration tests with Elasticsearch.
## Why?
With Elasticsearch, comparing dates without explicitly specifying all
components can lead to unexpected results.
Eg: `StartTime = '2023-04-05T06:07:08Z'` would match a workflow with
`StartTime = '2023-04-05T06:07:08.100000000Z'`.
This behavior is unexpected and can mess up with pagination.
This PR fixes it so missing nanos components is always specified (in the
example, it would be filled with 0s).
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
## What changed?
Add process-lifetime baselines to `objectleak`, ignore tiny pointer-free
allocations that the runtime cannot track individually, and allow
asynchronous cleanup the full settle timeout.
## Why?
Make object leak reports actionable by distinguishing process-lifetime
objects from leaks and avoiding runtime-induced false positives.
Replaces #11313.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What changed?
Use the `jobID` parameter in the `deterministicRequestID` function to
allow multiple batch operations to signal the same workflow.
## Why?
Two batch operations that send the same signal to the same workflow/run
ID will only send one signal, the second will be de-duped in the signal
logic because the request ID was hashed from workflow id / run id /
signal name. Including the batch job ID gives proper hashing to prevent
these separate signals from being de-duped
## How did you test it?
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
NA, this is a bug fix.
## What changed?
Only use 4-core ARM runners instead of 8-core; and increase shard size
for test sharding.
## Why?
8-core runners have - at least during peak hours - very long
provisioning time (several minutes). By using 4-core we reduce that
time, but risk OOM kills as they have less memory. To counter that, we
increase the number of test shards so that fewer tests are run per
shard.
## What changed?
When trying to clean up some PRs for worker callbacks, I noticed some
unnecessary duplication and cruft in our testcases for completion
handlers.
Essentially it's this:
<img width="567" height="216" alt="image"
src="https://github.com/user-attachments/assets/c64b759c-c5be-4fc3-a346-92cf77e1f6db"
/>
Rather than having N places where we construct a `completionHandler` and
several implementations of spinning up an `httptest` webserver, we now
just have a single `newNexusCompletionHandler` function that does both.
## Why?
Remove boilerplate, making tests more concise and easier to read.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
If there was some subtle behavioral change this could undermine our
testing.
## What changed
`RPCFactory` and the SDK client factory close the gRPC connections they
own on shutdown, as fx stop hooks. Removes the four gRPC connection
ignores from the leak test.
Also bumps `auto-scaled-workers` to pick up
temporalio/temporal-auto-scaled-workers#108, without which the SDK
connection stays open.
## Why?
Nothing released these connections, so every connection's goroutines and
the membership resolver watching for changes outlived the cluster.
Clients from `NewClient` share the system client's ref-counted
connection, so the SDK closes it only once the last one is closed.
## How did you test it?
- [x] built
- [x] covered by existing tests
`make leak-test` at the CI settings reports `no unexpected goroutines`.
Each change was verified to fail the leak test when reverted.
## What changed?
`parentclosepolicy.Processor` and `scanner.Scanner` keep a reference to
the SDK workers they start and stop them on shutdown, and the batch
activity closes the SDK client it creates per execution. The SDK worker
goleak ignores go with them.
## Why?
Neither component kept a reference before, so nothing could stop the
workers and their pollers ran until the process exited. The batcher
leaked a client reference per activity execution, so that one
accumulated per server process rather than per cluster.
Part of removing the TODO ignores in `tests/leakcheck/leak_test.go` (see
#11322).
## How did you test it?
- [x] built
- [x] covered by existing tests
## What changed?
Add a `__temporal_system_payload = "true"` metadata field to outer
`Payload` returned by system nexus operations when the operations
protobuf result embeds a nested `commonpb.Payload(s)` field.
## Why?
System Nexus operation results are serialized into a single outer
`Payload`. When the underlying proto message itself contains a nested
`Payload(s)` field, that nested payload's bytes get hidden inside the
outer payload's opaque `Data`. Downstream consumers (payload
codec/visitor logic) have no way to know they need to unwrap the outer
payload before they can reach and process what's inside.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA
## What changed
`versioninfo.Caller.Call` closes the response body on every path, not
just the 200 one. The two goroutines reported in the leak test is from
this.
## Why?
The `defer` sat below the status check, so a non-200 returned with the
body open. The transport sets `DisableKeepAlives`, so its read and write
loops stay pinned waiting for a body nothing will close — one connection
leaked per failed check, for the process lifetime.
## How did you test it?
- [x] built
- [x] covered by existing tests
## What changed?
Enable the `activity.enableStandalone` dynamic config on the release
server in the mixed-brain test.
## Why?
omes `throughput_stress` auto-enables standalone-activity load when the
namespace reports support. The current server (1.32) defaults it on, so
omes sends SAA calls; the previous release (v1.31.2) defaults it off, so
calls routed to the release frontend were rejected (`Unimplemented:
"Standalone activity is disabled"`), stalling workflows until the run
timed out. Enabling the DC on the release server matches a real
rolling-upgrade cluster and lets the test exercise SAA across both
versions instead of failing on every branch.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
