## What changed?
Removed the UpdateOptionsAllowedAfterDeferredRestoreSupersededByCancel
functional test and added focused unit coverage verifying that
TransitionCancelRequested clears ResetRestoreOptions.
Also corrected the nearby comment claiming that UpdateOptions was
permitted in RESET_REQUESTED when no restore was pending.
## Why?
PR #11394 disallowed UpdateOptions in CANCEL_REQUESTED and
RESET_REQUESTED. PR #11358 later added a conflicting test expecting an
update to succeed after cancel superseded a deferred restore.
The test was removed instead of changed to expect an error because that
result is already covered by UpdateWhileCancelRequestedFails. It also
could not verify whether the deferred restore flag was cleared: both a
cleared and stale flag produce the same FailedPrecondition while the
activity is CANCEL_REQUESTED.
## How did you test it?
- [X] built
- [] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Update sqlparser to v0.1.0. It's actually no-op since it points to the
same commit.
## Why?
Use tagged version instead of commit.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
* Deduplicate cancel retries before terminal-state validation
* Added tests covering terminal-state retries and run-qualified retries
after operation ID reuse.
## Why?
A delayed retry can arrive after the original activity has closed.
Deduplication must still recognize that retry, and callers must pin
cancellation to a run_id so activity ID reuse cannot
redirect the request to a replacement execution.
See also: https://github.com/temporalio/temporal/pull/11344
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
Moved activity failure retry classification into a shared helper used by
standalone activities, workflow activities, and workflow retries.
Standalone activities now use the same retry classification as workflow
activities for failures reported through
`RespondActivityTaskFailedById`. This includes retryable
`ServerFailure`s, worker-reported start-to-close and heartbeat timeouts,
and otherwise unrecognized failure variants. Schedule-to-start and
schedule-to-close timeouts remain non-retryable.
## Why?
Standalone activities previously treated non-application failures as
non-retryable. Sharing the existing workflow retry classifier keeps
retry behavior consistent across SAA, WFA, and workflow retries.
---------
Co-authored-by: Fred Tzeng <fred.tzeng@temporal.io>
## What changed?
Added user_metadata summary and details size validation to
StartActivityExecution, using the existing namespace-specific limits and
matching standalone Nexus operation behavior.
## Why?
Standalone activities previously persisted user_metadata without
enforcing its configured size limits. This made the limits inconsistent
across top-level executions and allowed oversized metadata to reach
persistence.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Moved `r.Lock()`/`defer r.Unlock()` in `ReaderImpl.AppendSlices` above
the `r.slices.Back()` ordering check, matching the lock-before-read
pattern already used by `MergeSlices` and `ClearSlices`.
## Why?
`AppendSlices` is called from the queue's `processEventLoop` while the
reader's own `eventLoop` mutates `r.slices` under the mutex in
`loadAndSubmitTasks`. Reading `Back()` without the lock races on
`container/list` internals — confirmed with the race detector.
## How did you test it?
- [x] built
- [x] added new unit test(s)
`TestAppendSlices_RaceWithLockedMutation` fails under `-race` on
unpatched `main` (data race: `AppendSlices` → `Back()` vs. locked
`MoveToBack`) and passes after this change.
Commands run:
- `go test -race -tags test_dep -count=1 -run
'TestReaderSuite/TestAppendSlices_RaceWithLockedMutation$'
./service/history/queues/` — fails on unpatched code, passes with the
fix
- `go test -race -tags test_dep -count=1 ./service/history/queues/` —
full package passes
- `make lint-code` — 0 issues
- `make fmt-imports` — clean
Fixes#11352
## Potential risks
Lock is now held slightly longer (covers the cheap `Back()` read + range
compare in addition to `PushBack`). Same pattern as `MergeSlices`; no
API or persistence behavior change.
## What changed?
Reject SAA unpause requests unless an unpause transition is possible.
## Why?
A no-op unpause could be recorded and later retried, obscuring invalid
state transitions.
Context: follow-up to [standalone activity operator-request
idempotency](https://github.com/temporalio/temporal/commit/d8b84b8aac).
## How did you test it?
- [x] covered by existing tests
- [x] added new functional test(s)
## Potential risks
- Clients that unpause non-paused activities now receive
`FailedPrecondition` instead of a successful no-op.
- This also affects clients unpausing activities that *were*
legitimately paused: the request-ID dedup that makes the rejection
retry-safe only works when the client supplies `RequestId`. When it's
omitted, the server mints a fresh UUID per attempt, so a retry never
matches `LastUnpauseRequestId`. Concretely: a client unpauses a `PAUSED`
activity, the mutation commits, the RPC times out on the way back
(activity is now `SCHEDULED`), the client retries — on `main` that retry
was a benign no-op, now it returns `FailedPrecondition`. Worth
confirming the SDK/CLI populate `RequestId` before this ships.
- The same `UnpauseActivityExecution` RPC against a workflow-owned
activity still silently no-ops when the activity isn't paused
(`service/history/api/unpauseactivity/api.go:120`), so behavior now
diverges by activity kind (standalone vs. workflow-owned) for the same
RPC.
- Separately, `UpdateActivityExecutionOptions` now also rejects with
`FailedPrecondition` while a deferred `Reset(RestoreOriginalOptions)` is
pending (previously it would silently apply and then be clobbered when
the deferred restore landed). Callers that previously succeeded here now
fail while the restore is pending.
---------
Co-authored-by: Dan Davison <dandavison7@gmail.com>
## What
Replace the hand-rolled `cloneTaskQueueStats` with `common.CloneProto`
so new `TaskQueueStats` fields propagate automatically. Also add
`RateLimitingActive` to the two struct literals in
`splitTaskQueueStatsByRampPercentage`.
## Why
[#10944](https://github.com/temporalio/temporal/pull/10944) added
`RateLimitingActive` to `TaskQueueStats` but missed updating these
hand-rolled struct constructions. The field was silently zeroed, causing
`DescribeTaskQueue` and `DescribeWorkerDeploymentVersion` to always
report `false`.
## How did you test it?
- **Unit**: tests for `cloneTaskQueueStats` and
`splitTaskQueueStatsByRampPercentage` verifying the field survives both
transforms.
- **Functional**: `TestDescribeTaskQueue_RateLimitingActive` — sets a 1
RPS API rate limit, drives traffic, and asserts `RateLimitingActive ==
true` in the `DescribeTaskQueue` response end-to-end.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
`cherryPickHSMEvent` treats `hsm.ErrStateMachineNotFound` as skippable
again instead of routing the event to the CHASM tree, restoring the
behavior from before #10986. That is the whole behavior change — one
line.
- `service/history/ndc/workflow_resetter.go` — the routing change, plus
comment cleanup on the surrounding cherry-pick helpers.
- `service/history/ndc/workflow_resetter_test.go` — updated the existing
`state machine not found` case, and added
`TestReapplyEventsHSMNotFoundDoesNotConsultChasm`, which asserts the
event is skipped, no error surfaces, and CHASM is **never consulted**.
- `tests/nexus_workflow_test.go` — the post-reset assertion in
`TestNexusOperationAsyncCompletion` now branches on the rail:
`RequireHistoryEvent` on HSM, `RequireNoHistoryEvent` on CHASM. Both
resets still run on both rails, so the
`RESET_REAPPLY_EXCLUDE_TYPE_NEXUS` coverage and the "reset itself still
succeeds" assertion are kept on the CHASM rail.
## Why?
**Full background, root cause, and the shape of the real fix are
captured in #11384.** In short: reapply cannot distinguish "the CHASM
tree owns this Nexus operation" from "no tree owns it", because CHASM
answers a missing operation with a bare `serviceerror.NotFound`. Reapply
is fail-fast and `BackfillWorkflow` commits only on a clean return, so
one such event discards an entire replication batch — including
completions already applied earlier in the same batch.
This was diagnosed against an MCN handover failure in
`TestReconfigureMCNReplicaWithBenchGoOnTestEnv`, where one Nexus
operation's completion was applied at batch index 0 and then discarded
80 times because an unrelated operation at index 8 existed in neither
tree.
The reset path is affected too, not only replication, which rules out
the narrower fix of gating the fallback on `!isReset`. Details in
#11384.
**This PR is the workaround, not the fix.** It restores pre-#10986 skip
semantics so the failures stop; #11384 tracks doing it properly.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
Verified locally on sqlite:
| scope | result |
|---|---|
| `service/history/ndc` unit tests | green |
| Nexus + Callbacks + Reset + Workflow functional suites | 648 PASS / 0
FAIL |
| XDC Nexus state replication, all 3 variants incl. `Chasm` | 19 PASS /
0 FAIL |
| `TestNexusOperationAsyncCompletion`, both HSM and CHASM rails | PASS |
Both new assertions are mutation-checked rather than merely green.
Restoring the fallback:
- fails the unit test on an unexpected `ChasmEnabled()` call, and
- fails the CHASM functional test at the post-reset
`RequireNoHistoryEvent`.
Across the three functional test files touched by CHASM Nexus work since
#10986 (`nexus_workflow_test.go`, `nexus_standalone_test.go`,
`xdc/nexus_state_replication_test.go`), exactly one assertion depended
on the fallback — the one now branched on the rail.
`TestNexusOperationSurvivesResetCrossTree`, both cancellation cross-tree
tests, and `TestNexusOperationChasmReplicatedWithMixedFlag` all pass
unchanged.
## Review iteration
This went through several rounds of review, which changed the test
strategy more than the fix. Worth knowing if you're re-reading after an
earlier round:
- The functional-test guard started as a top-of-test `Skip`, became a
mid-test early `return`, and is now a per-rail branch on the single
assertion that actually differs. The earlier forms dropped passing CHASM
coverage (completion-token validation, the async-completion happy path,
and the exclude-types block) as collateral.
- The new unit test started as a nine-subtest loop over every Nexus
event type, then two subtests over `isReset`, and is now a single case.
Both loops were vacuous: `chasmworkflow.Registry.Register` dedupes by Go
type so only one definition ever registered, and `reapplyEvents` reads
`isReset` only in the hardcoded `CancelRequested` / `Terminated` cases,
which a Nexus event never reaches.
- Several rounds went into comment accuracy around what is and isn't
reachable after this change. Those comments are now short and defer to
#11384 rather than restating the analysis in four places.
## Potential risks
**This does not itself demonstrate the MCN handover failure is fixed.**
The failure is a replication branch-fork under handover, which no local
test constructs; validation needs the bench-go repro (~1 in 2–3 hit
rate). Local results show the fix restores pre-#10986 skip semantics
without collateral damage, nothing more.
Reset reapply of a Nexus completion for a CHASM-tree operation is
silently skipped again — the bug #10986 fixed. Unreachable while CHASM
Nexus operations are not rolled out, but the feature cannot ship until
#11384 is addressed. The CHASM-rail assertion in
`TestNexusOperationAsyncCompletion` pins that regression, so it will
flip to `RequireHistoryEvent` as part of the real fix rather than being
forgotten.
Two related defects are left in place deliberately, since neither is
reachable once the fallback is gone: CHASM's bare `NotFound` on a
missing operation, and `cherryPickChasmEvent` returning
`serviceerror.Internal` when CHASM is disabled. Both are tracked in
#11384, along with the `TODO(follow-up)` about completion-token
resolution that this PR removes from the code.
---------
Co-authored-by: Chris Smith <aChrisSmith@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
Do not permit `UpdateOptions` in `CANCEL_REQUESTED` or `RESET_REQUESTED`
## Why?
- Hard to define semantics: if `UpdateOptions` lands after
`Reset(restore_original_options)` then should the update be silently
overridden when honoring the reset on attempt end?
- We opt to simplify the combinatorial possibilities now and retain the
possibility of evolving the API to allow it in the future.
- It is unclear whether these transitions should be allowed.
## How did you test it?
- [x] covered by existing tests
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes public API semantics for activities with pending cancel/reset;
clients that relied on mid-flight option updates will now get
FailedPrecondition.
>
> **Overview**
> **`UpdateActivityExecutionOptions` is no longer allowed** while an
activity is in **`CANCEL_REQUESTED`** or **`RESET_REQUESTED`**. Those
statuses are now treated like other non-updatable states and return
**`FailedPrecondition`** with the same message pattern as terminal
statuses.
>
> This replaces the prior behavior where options could still be updated
on a running attempt with a pending cancel, and where updates during
**`RESET_REQUESTED`** could bump the attempt stamp and re-issue timeout
tasks. Standalone activity tests were flipped to expect refusal and to
assert timeouts and run state stay unchanged.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
b4c607a9a8. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Added a missing `d.syncSummary()` call in
`syncVersionDataToComputeStatus`, so it now notifies the parent
Deployment workflow after pulling a compute status from WCI.
## Why?
Without this, the pull only updates the Version workflow's own state.
The Deployment workflow (which
`ListWorkerDeployments`/`DescribeWorkerDeployment` actually read from)
is not updated, so `computeStatus` can stay permanently missing from the
API even when the data is available.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
`Activity.HandleFailed` now treats a `RespondActivityTaskFailed` request
with an omitted `Failure` as retryable, matching WFA behavior.
Previously a nil failure was treated as non-retryable and the SAA closed
as `FAILED`.
## Why?
This achieves parity for SAA with workflow activities.
## How did you test it?
- [x] covered by existing tests
- [x] added new functional test(s)
## Potential risks
This is a behavioral change, but was a bug in the original
implementation
## What changed?
- Emit schedule-to-start latency metric when SAA starts
- The metric distribution (which is per-task queue) will now contain
data points from both SAA and WFA. This is reasonable because there's
nothing about SAA that implies that its time in matching backlog should
have a different distribution.
## Why?
- Required metric; WFA parity
## How did you test it?
- [X] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> Adds observability only on the activity-start path after a successful
transition; no auth, persistence, or matching behavior changes. Shared
metric name with workflow tasks is filtered by operation/tags in tests.
>
> **Overview**
> **Standalone activities (SAA)** now emit
`task_schedule_to_start_latency` when History accepts an activity task
start in `HandleStarted`, matching workflow-embedded activity (WFA)
behavior and filling a required metrics gap.
>
> Latency is **started time minus attempt dispatch time** (via
`dispatchTimeForAttempt`), not raw schedule time—so retries measure
backlog from the current attempt’s dispatch, excluding prior attempts
and backoff.
>
> Samples use the same per-task-queue partition scope as WFA
(`HistoryRecordActivityTaskStartedScope`, activity task type,
`MetricsBreakdownByTaskQueue` → real task queue name vs `__omitted__`).
Idempotent `RecordActivityTaskStarted` replays do not record again.
>
> Unit tests in `activity_test` assert sample count and latency for
first start and retry; functional parity tests cover SAA vs WFA for
first attempt and retry across task-queue breakdown settings.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
e702b65ccd. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Added an explicit precondition check for standalone activity
cancellation responses. RespondActivityTaskCanceled now requires the
activity to be in CANCEL_REQUESTED and returns the established
ErrActivityTaskNotCancelRequested error otherwise.
## Why?
Standalone activities previously exposed an internal invalid-transition
error when a worker reported cancellation without a prior cancellation
request. Workflow activities return a stable InvalidArgument API error
for the same scenario.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
See API change https://github.com/temporalio/api/pull/846
## What changed?
- Drop `reset_attempts` and `reset_heartbeat` from
`UnpauseActivityExecution`
## Why?
- We have so far been unable to assign desirable and consistent
semantics to them during implementation: for example if
`Unpause[resetAttempts]` is received during retry backoff it is unclear
whether to honor the remaining delay time, because this is how Unpause
usually behaves, or dispatch immediately, because this is how Reset
behaves.
- No known user demand
- They are confusing: they mix `Unpause` and `Reset` functionality in a
confusing way
- They can be added later
## How did you test it?
- [x] covered by existing tests
## Breaking changes
- This API has always been rejected by the server. When server starts to
accept it, an old client could submit these options and they would be
ignored. Operator API is not GA.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes activity unpause scheduling semantics for the execution API
and drops reset-on-unpause behavior that was only partially implemented;
low user impact if the API was not GA and had no known callers.
>
> **Overview**
> Aligns the server with the **UnpauseActivityExecution** API change:
**`reset_attempts`** and **`reset_heartbeat`** are no longer part of
unpause for standalone (CHASM) activities.
>
> **CHASM activity unpause** no longer resets attempt count, retry
interval, or heartbeat state on unpause, and always considers the
pending retry backoff when scheduling dispatch (the branch that skipped
that when `reset_attempts` was set is removed). Workflow-embedded
unpause forwarding via **`UnpauseActivityExecution`** no longer passes
those fields to the legacy **`UnpauseActivity`** history call (jitter
and identity only).
>
> **`go.temporal.io/api`** is bumped to the revision that removes the
fields from **`UnpauseActivityExecutionRequest`**.
>
> **Tests** are updated so unpause helpers no longer take a reset flag;
reset-on-unpause coverage stays on legacy **`UnpauseActivity`** only
(execution API skips that case). Standalone tests for
**`UnpauseWithResetAttempts`** and **`UnpauseWithResetHeartbeat`** on
**`UnpauseActivityExecution`** are removed.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
7a41d507ca. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Exposed RetryState on standalone activity execution outcomes and
persisted it in activity state. Retry evaluation now records terminal
reasons including retry policy not set, cancellation requested,
non-retryable failure, maximum attempts reached, and timeout.
## Why?
Standalone activities previously collapsed retry decisions into a
boolean, preventing callers from distinguishing why an activity stopped
retrying. This brings standalone activity behavior and observability
into parity with workflow activities.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
During a rolling upgrade, activities closed by an older server may
return RETRY_STATE_UNSPECIFIED. Existing closed activities also remain
unspecified because retry state was not previously persisted. Older
clients safely ignore the new protobuf field.
---------
Co-authored-by: Dan Davison <dandavison7@gmail.com>
## What changed?
- Persist payload sent with attempt failure as last heartbeat details
- Emit metrics associated with that codepath for WFA parity
## Why?
- The first is a relatively bad bug: an activity attempt should be able
to have the latest checkpoint data sent in and persisted with a
retryable failure, but SAA was not persisting it
- SAA vs WFA metrics parity
## How did you test it?
- [x] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes activity failure and retry persistence where workers rely on
checkpoint data; scope is narrow with new SAA/WFA parity tests, but
incorrect handling could affect retries or observability.
>
> **Overview**
> Standalone activities (SAA) now **persist `LastHeartbeatDetails` from
`RespondActivityTaskFailed`** before deciding whether to retry or fail
terminally. Previously that checkpoint lived only on the terminal
`TransitionFailed` path, so **retryable failures dropped the worker’s
final progress payload**.
>
> Heartbeat handling on failure now mirrors a normal heartbeat: update
last-heartbeat state (details, recorded time, count) and record metrics
via a shared **`emitHeartbeatMetrics`** helper used by
**`RecordHeartbeat`** as well.
>
> Trace drivers and parity tests gain **`HasHeartbeatDetails`** on
failed-respond events, plus coverage that WFA and SAA expose the same
stored heartbeat details (including terminal SAA failures).
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
a013d1627f. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Updated the CHASM scheduler's DescribeSchedule response to resolve the
effective catchup window using the namespace's scheduler tweakables.
DescribeSchedule now reports:
- nil, zero, or negative values as `DefaultCatchupWindow`
- positive values below the minimum as `MinCatchupWindow`
- values at or above the minimum unchanged
The resolution is applied to a cloned schedule, leaving persisted
scheduler state unchanged.
## Why?
DescribeSchedule previously used a hard-coded one-year default and did
not consistently report the same effective catchup window used during
schedule processing.
Using the existing catchup-window resolver keeps DescribeSchedule
consistent with runtime behavior and namespace-specific dynamic
configuration.
## V1 and V2 behavior
V1 and V2 currently differ when the configured catchup window is zero or
negative:
- V1 treats zero or negative values as below the minimum and resolves
them to `MinCatchupWindow`.
- V2 treats zero or negative values as unset and resolves them to
`DefaultCatchupWindow`.
- Both implementations clamp positive values below the minimum to
`MinCatchupWindow`.
This PR changes only the CHASM/V2 DescribeSchedule path and does not
modify V1 behavior.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
Commands run:
- `go test -tags test_dep ./chasm/lib/scheduler`
- `go test -tags test_dep ./tests -run
'TestScheduleCHASM/TestDescribeCatchupWindowAfterCreateAndUpdate'
-count=1`
- `make lint-code`
`testBasics` already performs create→describe and update→describe,
including the unset/default case, but it is shared by V1 and V2. Since
zero/negative semantics currently differ between the implementations,
adding those cases there would break V1 coverage. A focused CHASM-only
functional test was added instead.
The CHASM-only test covers create with an unset catchup window, followed
by updates with zero, negative, positive-below-minimum, and
above-minimum values.
Local testing:
1. http://localhost:3000/ + UI side override to allow <10 secs
2. Create schedule with catchup window 0 secs. Load it and verify it
shows as 10 secs.
3. DC changes to switch to CHASM. Repeat(2) to verify its set as 1 year.
## Potential risks
DescribeSchedule now returns the effective catchup window rather than
the raw persisted value for non-positive and below-minimum values. This
matches the value used by CHASM schedule processing.
### V1 and V2 migration
This PR changes only the CHASM DescribeSchedule response. It does not
normalize the persisted schedule policy or change migration payloads, so
the existing migration behavior remains:
- **V1 → V2:** V1 eagerly normalizes zero or negative values to an
explicit `MinCatchupWindow`. Migration copies that positive duration, so
V2 continues using the minimum.
- **V2 → V1:** V2 persists the original zero or negative value and
treats it as unset/default at runtime. Migration currently copies that
raw value. V1 then resolves it to `MinCatchupWindow`, potentially
changing the effective behavior from the V2 default to the V1 minimum.
- **Unset:** The target implementation resolves the unset value using
its own default. Behavior could change if the source and target defaults
differ.
- **Positive below minimum:** The target implementation applies its own
minimum. Behavior could change if the source and target minimums differ.
- **At or above minimum:** The explicit value is preserved across
migration.
Resolving or persisting the effective catchup window during migration is
outside the scope of this PR.
## What changed?
- Port SAA/WFA `CompleteById` tests to declarative framework
## Why?
- We will gain additional test assertions when the declarative tests are
wired up to the spec (model)
- Easier to read and reason about the tests: 225 LOC reduction
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> Changes are limited to test vocabulary, test drivers, and parity
tests; no production activity completion logic is modified.
>
> **Overview**
> Extends the activity parity **trace model** with `CompleteByID`
(`RespondCompletedByIDType`) and teaches the workflow-activity and
standalone-activity drivers to issue `RespondActivityTaskCompletedById`
when that event appears in a trace.
>
> Replaces two long, hand-written parity tests
(`TestCompleteByID_BeforeAnyWorkerStarts` and
`TestCompleteByID_WhilePaused`) with a single table-driven
`TestCompleteByID` that drives the same scenarios via traces
(`CompleteByID` alone, or `Pause` then `CompleteByID`) for both WFA and
SAA. Standalone activity still asserts `LastStartedTime` is set after
force-complete without a worker poll.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
79ecf3099e16db51d593b71237164770064116d8. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Enable standalone activity start delay by default. Remove unnecessary
test overrides.
## Why?
Start delay to be enabled by default for standalone activities GA
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed
- Allow the S3 visibility archiver query parser to accept
`WorkflowType`.
- Keep `WorkflowTypeName` as a deprecated compatibility alias.
- Rename the parsed query field to `workflowType` and update S3
visibility archiver tests.
- Fix the S3 parser StartTime test assertion and StartTime operator
error message.
## Why
- S3 visibility archiver queries only accepted `WorkflowTypeName`, while
filestore, gcloud, and non-archived visibility records use
`WorkflowType`. This keeps old queries working while accepting the
standard field name.
- Fix https://github.com/temporalio/temporal/issues/7821
### Motivation
- The change that updated MaxDeployments behavior and error text caused
flakiness in `main` by making `TestNamespaceDeploymentsLimit` unstable
under shared test namespaces.
- Revert and re-disable the affected functional test to restore stable
test behavior while the underlying flakiness is investigated.
### Description
- Restore the previous worker-deployment limit error wording in
`service/worker/workerdeployment/client.go` (reverting the wording
change that exposed "worker deployments" in the message).
- Re-disable the unstable functional test by adding `s.T().Skip()` in
`tests/worker_deployment_test.go` for `TestNamespaceDeploymentsLimit`
and restore its prior flow/assertions that expect the original error
message.
- Adjust test assertions in
`TestCreateWorkerDeployment_MaxDeploymentsLimit` to match the restored
error text (`"reached maximum deployments in namespace"`).
- Modified files: `service/worker/workerdeployment/client.go` and
`tests/worker_deployment_test.go`.
### Testing
- Ran the targeted functional test with `go test -tags test_dep ./tests
-run 'TestWorkerDeploymentSuite/TestNamespaceDeploymentsLimit$'
-count=1`, which completed successfully.
- Ran `make lint-code`, which failed due to a network error downloading
`golangci-lint` (HTTP 403 from `proxy.golang.org`).
------
[Codex
Task](https://chatgpt.com/codex/cloud/tasks/task_b_6a6cedda33c08324a2863a637b5c00ca)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> User-visible error string reversion and test skip/assertion alignment
only; no limit logic changes.
>
> **Overview**
> Reverts namespace **max deployment** limit errors to say **"reached
maximum deployments in namespace"** instead of **"worker deployments"**,
in both `CreateWorkerDeployment` and auto-create via poll paths in
`client.go`.
>
> **`TestNamespaceDeploymentsLimit`** is skipped again (shared namespace
/ visibility flake) with a TODO on poller error messaging; the in-test
flow is simplified to shared helpers when re-enabled.
**`TestCreateWorkerDeployment_MaxDeploymentsLimit`** now expects the
restored error substring.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
63a7cf00ac. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What
Move ListWorkers, DescribeWorker, and CountWorkers from
`VisibilityAPIToPriority` to `APIToPriority`.
Changed priority to P3 to be aligned with other status Querying APIs.
## Why
Today, these APIs are served by matching and not visibility.
## How did you test it?
Updated unit tests
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
In the CHASM (V2) scheduler, treat a non-positive (zero or negative)
schedule
catchup window the same as unset: return the default catchup window
instead of
clamping up to the minimum. Only a positive value below the minimum is
clamped up.
Change is in `chasm/lib/scheduler/spec_processor.go` (`catchupWindow`).
## Why?
Previously only a `nil` catchup window fell back to the default; a zero
or
negative value slipped through to `max(cw, MinCatchupWindow)` and was
silently
clamped up to the minimum. A non-positive value is effectively
"unset/invalid"
and should resolve to the default.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s)
Added a component level functional test (Generator) for now.
Ideally, we'd add a server-level test that creates schedules with
different catchup window values and verifies the result via
DescribeSchedule. However, that doesn't currently validate the intended
behavior because the describe logic overwrites the catchup window in
some cases (see:
https://github.com/temporalio/temporal/blob/main/chasm/lib/scheduler/scheduler.go#L697-L699).
Will work on this fix next as it needs more plumbing and also add this
specific test in the next PR.
## Potential risks
Behavior change for any schedule that explicitly sets a catchup window
<= 0:
it now resolves to DefaultCatchupWindow instead of MinCatchupWindow.
## What changed?
SAA terminal timeout failures now chain the previous attempt’s failure
as their cause.
## Why?
When retries ended in a timeout, SDK users could see only the timeout
and not the application failure that drove the retries. Preserving the
cause exposes the useful underlying error and matches Workflow Activity
behavior.
## How did you test it?
- [x] added new unit test(s)
- [x] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes activity timeout failure protobuf shape and terminal outcome
logic on a user-visible error path; scope is limited to SAA timeout
handling with strong test coverage.
>
> **Overview**
> **Standalone activity (SAA) terminal timeouts now set `Failure.Cause`
to the last attempt’s stored failure** (typically the application error
that triggered retries), so clients see the underlying error via
`TimeoutError.Unwrap()` instead of only the timeout wrapper—aligned with
workflow-embedded activities.
>
> `TransitionTimedOut` reads `priorAttemptFailure` from
`LastFailureDetails` **before** recording the current timeout, then
passes it into schedule-to-start/close outcome failures and sets `Cause`
on start-to-close and heartbeat terminal failures. When a per-attempt
timeout exhausts the schedule-to-close retry window
(`RETRY_STATE_TIMEOUT`), the final schedule-to-close outcome still
chains that prior failure even though the per-attempt timeout was
written to attempt state first.
>
> Coverage adds a state-machine unit test for the retry-window path and
SAA/WFA parity tests (including a check that **retryable** timeouts do
not chain causes on `LastFailure`). Test helpers use a stable
`TestFailure` application failure type for assertions.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
89ae0251ee. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
- Workflow task completion buffer overflow now fails the WFT with
`WORKFLOW_TASK_FAILED_CAUSE_REQUEST_TOO_LARGE` instead of
`PAYLOADS_TOO_LARGE`.
- `DescribeNamespace` reports
`NamespaceInfo.Limits.workflow_task_completion_size_limit_error`,
sourced from `history.workflowTaskCompletionBufferSizeLimit`.
API PR: https://github.com/temporalio/api/pull/838
## Why?
`PAYLOADS_TOO_LARGE` is misleading here, no single payload is oversized,
the request total is. Exposing the limit lets SDKs page under it instead
of discovering it by failing.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
No
## What changed?
Propagated the activity retry policy into the
PollActivityTaskQueueResponse created for eagerly dispatched activities.
Added a unit-test assertion verifying that eager activity tasks contain
the retry policy from the original schedule command.
## Why?
The eager response construction omitted RetryPolicy, causing SDKs to
report an undefined retry policy in activity context even though the
workflow scheduled the activity with one.
## How did you test it?
- built
- run locally and tested manually
- added new checks to existing unit test
- Verified manually using the TypeScript SDK retry-policy test against
the patched local server.
## What changed?
This PR refactors the validation logic used for SANO from a collection
of loose functions, into methods on an unexported `validator` type.
(Similar to how `chasm/lib/callback/validator.go` is structured.)
The same checks have all been preserved, although I did fix up one error
message string to be consistent with others.
## Why?
The motivation for this refactoring is to make it easier to land the
"worker callbacks" feature. That will require expanding the validation
checks, and bundling all the dependent parameters on the `type validator
struct` is cleaner than needing to wire through a new parameter at every
callsite.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
I am relying on GenAI in its assertion that the existing validation
checks are essentially identical with these changes. Worst case
scenario, this alerts which types of SANO requests are accepted or
rejected.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What changed?
- Deduplicate cancel and pause retries before terminal-state validation.
- Added tests covering terminal-state deduplication and run-qualified
retries after activity ID reuse.
## Why?
A delayed mutation retry can arrive after the original activity has
closed. Deduplication must still recognize that retry, and callers must
pin mutations to a run_id so activity ID reuse cannot redirect the
request to a replacement execution.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## What changed?
Migrated the priority heap in PollerPQ to an instrusive linked list.
## Why?
The priority queue is not exactly needed, we were not really utilizing
the properties of it.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What changed?
- During V2->V1 migration (rollback), ALLOW_ALL starts are now excluded
from the `RunningWorkflows` array.
## Why?
- `RunningWorkflows` in V1 will block a schedule if it isn't set to an
`ALLOW_ALL` policy itself. This ensures that the workflows are copied to
`RecentActions`, but not eligible for `WatchWorkflow` to block on.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Description
`WorkflowHandler.UpdateSchedule` evaluated
`wh.config.EnableSchedules(request.Namespace)` before its own `if
request == nil` guard, so a nil request was dereferenced before it was
validated. The panic is recovered by `log.CapturePanic`, so the process
stays up, but the caller gets a `serviceerror.Internal` plus a logged
stack trace instead of the standard `InvalidArgument` "Request is nil."
that every other schedule handler returns.
The fix is purely guard ordering — move the existing nil check above the
feature-gate call, matching what `CreateSchedule` and `PatchSchedule`
already do.
## Reachability — limited, please read
This is **not** reachable over standard gRPC. Protobuf decoding always
materialises a non-nil (possibly empty) `UpdateScheduleRequest` before
dispatch, so a typed-nil client request arrives at the handler as an
empty request and takes the ordinary feature-gate path. The practical
impact is limited to direct in-process callers and non-standard adapters
that invoke the handler without going through the generated transport.
The regression test is therefore a direct handler call, not a
transport-level test.
## Testing
New `TestUpdateSchedule_ValidationAndErrors`, modelled on
`TestPatchSchedule_ValidationAndErrors`. Committed before the fix; on
unfixed code:
```
--- FAIL: TestWorkflowHandlerSuite/TestUpdateSchedule_ValidationAndErrors/nil_request_should_return_error
Error: Not equal:
expected: *serviceerror.InvalidArgument{Message:"Request is nil."}
actual : *serviceerror.Internal{Message:"runtime error: invalid memory address or nil pointer dereference"}
```
Includes a schedules-disabled subtest confirming the `EnableSchedules`
gate still fires for non-nil requests after the reorder. `go test -tags
test_dep ./service/frontend/... -count=1` passes.
Source: SCH-066, Schedule V2 Bug Review (P1, Confirmed, Qualified).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
Added request-ID-based idempotency for standalone activity unpause,
reset, and update-options operations. Successful request IDs are
persisted in activity state and duplicate requests are handled as
no-ops.
## Why?
These APIs can be retried after timeouts or transient failures.
Persisting the latest successful request ID prevents duplicate
mutations, including delayed unpause retries undoing a later pause.
Workflow-backed activity support will follow separately.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
---------
Co-authored-by: Dan Davison <dandavison7@gmail.com>
## What changed?
`sqlExecutionStore.DeleteWorkflowExecution` now also deletes from
`chasm_node_maps`, alongside the other mutable-state sub-collection
tables.
Added a `MutableStateTableCounts` hook to `ExecutionMutableStateSuite`
(wired from the raw DB by the SQL entrypoints, nil on Cassandra) so
`AssertMissingFromDB` verifies every mutable-state sub-collection table
is empty after delete, not just that the execution is gone. This runs on
the existing delete tests across SQLite/MySQL/PostgreSQL.
## Why?
On SQL backends, deleting a workflow execution left orphaned
`chasm_node_maps` rows: the delete path cleared seven sibling
sub-collection tables plus `executions` but omitted `chasm_node_maps`,
and nothing else reclaims them — an unbounded storage leak on any
deleted CHASM-bearing run. Cassandra is unaffected (CHASM nodes are a
column on the `executions` row). The assertion gap is why this went
unnoticed: `AssertMissingFromDB` only checked `GetWorkflowExecution`,
which returns `NotFound` from the missing `executions` row before ever
reading child tables.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
The augmented `AssertMissingFromDB` fails on `chasm_node_maps` before
the fix and passes after, on the existing SQL delete tests. Verified
manually on SQLite by creating a CHASM scheduler execution and
confirming its `chasm_node_maps` rows are removed when the run is
deleted (via namespace deletion).
## Potential risks
Low. The change adds one delete within the existing
`DeleteWorkflowExecution` transaction (no new persistence surface —
`DeleteAllFromChasmNodeMaps` already exists on all three drivers), keyed
identically to its siblings. Pre-existing orphaned rows from before this
fix are not cleaned up.
## What changed
- Adds SAA payload-size and heartbeat-count metrics.
- Bring SAA metric tags into parity with WFA
- Bring SAA timeout metric behavior into parity with WFA by omitting
start-to-close latency when an attempt times out.
## Why
- Correctness / parity with de-facto correct WFA
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes observable metrics and timeout latency recording in the
activity execution path; parity tests reduce regression risk but
dashboards or alerts keyed on old SAA timeout latency may shift.
>
> **Overview**
> Standalone activity (SAA) metrics are brought in line with
workflow-embedded activity (WFA) behavior and coverage.
>
> **Handler split:** Metrics use a **base** handler (namespace +
`operation` only) for payload-size and heartbeat counters, and an
**enriched** handler (per-activity tags: activity type, task queue,
workflow type, etc.) for success/fail/latency counters. Complete and
fail responses record **`activity_payload_size`** on the base handler
from result/failure serialized size; heartbeats record
**`activity_heartbeat_count`** (with `has_details`) and payload size
when details are present. New standalone activities emit payload size
for schedule input on **`RecordActivityTaskStarted`**.
>
> **Timeout parity:** Terminal and retryable attempt timeouts no longer
emit **`activity_start_to_close_latency`** on the timed-out path
(matching WFA). Attempt-level timeout counters are unchanged.
>
> **Verification:** Integration **`TestWFASAAMetricsParity`** drives
shared event traces against WFA and SAA and compares metric names, tags,
and values; test drivers gain RPCs for heartbeat, complete, terminate,
unpause, and update-options.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
df982df381. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
1. add a max skip field to TimeSkippingConfig
2. add TimeSkippingInfo to DescribeWorkflowExecution (contains virtual
current time and running status)
3. add PollWorkflowExecutionTimeSkipping for fast-forward completion
## Why?
1. a generic mechanism to stop endless retries or schedules
2. to give clients easier access to time skipping state changes
related API change: https://github.com/temporalio/api/pull/835
## Summary
- The `Mixed brain test` job has been consistently failing on `main`
(and on unrelated PRs, e.g. #11276) with:
```
scenario failed: failed to parse scenario configuration: nexus-endpoint
was set but nexus-enabled is false
```
- `tests/mixedbrain` builds Omes fresh from `omes@main` on every run
(see `downloadAndBuildOmes`), so it's exposed to upstream changes.
Omes's `throughput_stress` scenario now validates that `nexus-endpoint`
can't be set unless `nexus-enabled=true` is also set (see
`scenarios/throughput_stress.go`), but our `throughput_stress` scenario
options in `mixed_brain_test.go` only set `nexus-endpoint` without
`nexus-enabled=true`.
- This fixes it by explicitly passing `nexus-enabled=true` alongside
`nexus-endpoint`.
## Test plan
- [x] `go build ./...` and `go vet ./...` in `tests/mixedbrain` pass
- [ ] CI `Mixed brain test` job passes on this PR
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
## What changed?
Bump `github.com/temporalio/ringpop-go` to `v0.1.0` and
`github.com/temporalio/tchannel-go` to `v1.22.1`, replacing the previous
commit pseudo-versions.
Picks up ringpop-go#22 (bounds label and member resource usage on
incoming membership changes, fixes a state-transition timer leak) and
tchannel-go#14 (returns errors instead of panicking on malformed call
frames).
## Why?
Pull the membership/transport robustness fixes from both forks into the
server. Both forks are now tagged, so we can pin real releases instead
of commit hashes.
## How did you test it?
- [x] built
- [x] covered by existing tests
## What changed?
A small documentation update to correct a slightly misleading comment
## Why?
The current comment gives a slightly misleading view that transient
termination / cancellation errors will just result in it being dropped,
whereas the history-client should be provided wrapped with an internal
retrier, so in practice any failure will get a couple of retries before
being dequeued.
As I understand it, this is arguably different than the v1 behaviour
(local activity, retries for quite a while), however, it's not super
clear to me that it warrants a p1/bugfix.
I think there's an interesting question about whether or not this
architectural pattern of a CHASM task doing multiple things in a single
handler is a good idea; I'm quite tempted to say that each individual
RPC should probably be its own task. However, for now imho this is ok
as-is.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Backfill: capacity stall must not skip part of the requested range
### Description
`processBackfill` uses `Attempt > 0` to decide whether to resume from
the high-water mark. But `Attempt` is a buffer-full back-off counter,
not a progress marker: `Execute` increments `Attempt` via a `defer` even
on the buffer-full early-return where no work happens, while
`LastProcessedTime` stays at its creation-time default (set to "now" in
`addBackfiller`).
### User experience
A backfill request (e.g. "re-run the last N hours") returns success, but
if the invoker buffer was full on the backfiller's first task execution,
some or all of the requested actions silently never run. Timing/load
dependent, no error surfaced — a silent data-completeness bug.
### How it occurs
1. `addBackfiller` creates the backfiller with `Attempt=0` and
`LastProcessedTime=now`.
2. The first task sees `limit <= 0` (buffer full), takes the early
return — no range processed, `LastProcessedTime` untouched — but the
`defer` bumps `Attempt` to 1.
3. On retry, `processBackfill` sees `Attempt > 0` and resumes from
`LastProcessedTime` (= creation "now"). For a past range that is
at/after the range end, so the whole range is skipped and the backfiller
completes having produced nothing.
### How it's fixed
Resume from the high-water mark only when it reflects genuine progress
strictly within the requested range (`start < HWM < end`); otherwise
start from the range start. A capacity-only stall (HWM at its
creation-time default) is no longer mistaken for durable progress.
Single-attempt and legitimate mid-range resume behavior are unchanged.
### Test
`TestBackfillCapacityStallDoesNotSkipRange` — fails before the fix,
passes after.
### Risks:
This is actually not backwards compatible, strictly speaking, but I
think the window during which the lastUpdate has not been recorded is
going to be sub-second for any backfill that's unfortunate enough to be
started in a mixed-brain scenario, so my feeling is that the risk is
probably acceptable. see
https://github.com/temporalio/temporal/pull/11162/changes#r3636260022
If a backfill were to be created at precisely the right time as to use
lastUpdatedTime == now, but then use the new codepath (because it hadn't
had a chance to make progress yet) it might skip to the end. But as far
as I am aware this is only a gap between creation and the CHASM task
completion. As soon as a backfill has made any progress, it should be
fine afaik.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What changed?
Separate RPS rate limiters into separate structs.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Address an edge case in VerifyVersionedTransition that causes the task
to DLQ when new CAN run id does not exists on the receiver. Change the
logic to attempt to recover/self-heal reusing the recovery machinery
already in use for a missing/stale parent.
## Why?
This is a valid edge case that could occur under the following sequence
of events.
1. **Source generates the CAN.** The parent's long timer fires on the
source (active) cluster and the workflow continues-as-new. This produces
a `SYNC_VERSIONED_TRANSITION` task carrying the successor's
`NewRunInfo`, bound for the target. Under replication lag it is not yet
applied on the target.
2. **Failover.** The namespace fails over; the target becomes the new
active cluster, on a higher failover version.
3. **Target replays the CAN.** The new active replays the same
still-pending timer and generates its *own* CAN event — a second
successor on its higher-version branch. This branches the parent's
history from the source's. The higher failover version wins, so both
clusters converge on the target's branch (the "winner"); the source's
successor is the "loser".
4. **gRPC stream reset.** The cross-cluster replication stream resets —
for reasons unrelated to the failover (a history shard moved to another
host, a pod restart, a network blip, etc.). The source's CAN SYNC from
step 1 was still never applied on the target.
5. **SYNC → VERIFY downgrade.** On reconnect the source re-reads that
un-acked transition. Its progress cache still marks the transition
"sent" (marked at *build* time, and preserved across stream resets), so
instead of re-shipping the state-bearing SYNC it ships a **content-free
VERIFY** carrying only the loser's `new_run_id`.
6. **Verify lands on the winner branch.** The target processes the
VERIFY. Its current branch is the winner (step 3), so the loser's
transition is on a non-current branch → **case 4**. `verifyNewRunExist`
then looks up the loser's `new_run_id`, which was never created on the
target (it lost the conflict) → `NotFound`.
7. **Fatal step.** `verifyNewRunExist`
(`executable_verify_versioned_transition_task.go`) turns that `NotFound`
into `softassert.UnexpectedDataLoss(...)`. `DataLoss` is non-retryable
(`executable_task.go`), so the task goes straight to the DLQ on the
first miss, with no attempt to recover.
This change should self-heal this scenario. More detail on the edge case
can be found
[here](https://app.notion.com/p/temporalio/s-aw026-ReplicationTaskEnqueuedToDLQ-fix-proposal-VerifyVersionedTransition-3ab8fc5677388053861dcff056ce9de5?source=copy_link)
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What changed?
`RespondActivityTaskCompletedById` can now force-complete an SAA from
the `Scheduled` state.
## Why?
Workflow activities allow force completing an activity before any worker
starts it.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
- Added a new file `tests/activity_parity_test.go` to hold all tests
related to parity of workflow activities and standalone activities.
-
## Potential risks
NA
## What changed?
Previously not tested.
## Why?
Test it.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
None
<!-- CURSOR_SUMMARY -->
---
## What changed?
revive MaxDeployments error test by giving the test its own namespace to
hit the max in
## Why?
test that MaxDeployments error is exposed correctly
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> User-visible error string tweak plus test-only namespace isolation; no
change to limit enforcement logic.
>
> **Overview**
> Clarifies namespace **worker deployment** limit errors and re-enables
coverage for pollers hitting that cap.
>
> **Error text:** `ResourceExhausted` messages in
`CreateWorkerDeployment` and auto-create
(`updateWithStartWorkerDeployment`) now say *reached maximum worker
deployments in namespace* instead of *maximum deployments*, so clients
can tell this limit apart from other deployment concepts.
>
> **Tests:** `TestNamespaceDeploymentsLimit` is no longer skipped. It
registers a dedicated namespace with `MatchingMaxDeployments` set to 1,
creates one deployment via poll, then asserts a second deployment’s poll
fails with the exact new message.
`TestCreateWorkerDeployment_MaxDeploymentsLimit` expects the updated
substring in the API error.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
86cbe01a1e. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Un-skips the CHASM variants of eight Nexus workflow functional tests —
the full remaining set — plus two small changes on the CHASM Nexus
completion path.
**Test-only unskips:**
- `TestNexusOperationCancelation`,
`TestNexusOperationCancelBeforeStarted_CancelationEventuallyDelivered`,
`TestNexusOperationAsyncCompletionInternalAuth`,
`TestNexusAsyncOperationWithNilIO`, `TestNexusOperationSyncNexusFailure`
— the CHASM behavior they exercise has landed.
- `TestNexusAsyncOperationWithMultipleCallers` — only needed
`EnableCHASMSignalBacklinks` in the CHASM env; it is now enabled by
default in `newTestEnv` alongside the other CHASM flags (runtime-gated
on `EnableChasm`).
**Production fixes:**
- **Error rehydration** (`TestNexusAsyncOperationErrorRehydration`):
`completeChasmOperation` converted the whole `nexus.OperationError`
wrapper, so the caller saw a generic "nexus operation completed
unsuccessfully" instead of the handler's original error (and no
canceled/terminated info). It now unwraps the wrapper's cause.
- **Completion after the caller closed**
(`TestNexusCallbackAfterCallerComplete`): a completion for an operation
whose caller has already closed now fails the callback non-retryably.
## Why?
Bring the CHASM Nexus-in-workflow path to parity with HSM so both
produce the same outcomes for Nexus operations, and enable the
functional tests that verify it.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Low. Un-skipping some test cases when chasm is enabled and fixing two
small issues in chasm operation completion path.
## What changed?
Add idempotency tests for `CancelOutstandingWorkerPolls` and
`ShutdownWorker` — unit test and functional test confirming repeated
calls are safe and the shutdown cache remains effective.
## Why?
The SDK may call `ShutdownWorker` multiple times. We should have
explicit coverage for this.
## How did you test it?
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
None — test-only changes.
## What changed?
- Preserve `PAUSE_REQUESTED` in Describe when a running standalone
activity is reset with `keep_paused=true`.
- Add workflow/standalone parity coverage for this state transition.
## Why?
A deferred reset changed the public run state to `STARTED`, hiding the
pending pause despite the activity being guaranteed to land paused.
## How did you test it?
- [x] added new functional test(s)
- [x] covered by existing tests
## Potential risks
Low: only the Describe run-state projection for deferred keep-paused
resets changes.
## What changed?
Removes the `system.enableNamespaceHandoverWait` dynamic config and the
per-namespace gate it controlled in `NamespaceHandoverInterceptor`.
- Deleted the `EnableNamespaceHandoverWait` setting from
`common/dynamicconfig/constants.go`.
- Removed the `enabledForNS` field and its `.Get(dc)` wiring from the
interceptor.
- The handover-wait now applies unconditionally: `Intercept` runs the
wait for any request whose namespace is in `REPLICATION_STATE_HANDOVER`
(methods in the allowed-during-handover set remain exempt).
Net effect: the feature graduates from a per-namespace toggle (OSS
default `false`) to always-on.
## Why?
The namespace-handover wait is a correctness mechanism that briefly
holds requests while a namespace transitions replication state, so
callers don't hit a namespace mid-handover. It is already enabled in
production, and gating it behind a dynamic config adds configuration
surface without a real use case for turning it off. Making it
unconditional removes a foot-gun and simplifies the interceptor.
## How did you test it?
- [x] built (`go build ./common/...`, `go vet`, `gofmt` clean)
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
- **Behavior change for OSS deployments**: the config previously
defaulted to `false`, so OSS clusters that never set it will now perform
the handover wait. This only affects requests to namespaces actively in
`REPLICATION_STATE_HANDOVER`, and the wait is bounded by the request
deadline / namespace cache refresh interval, but it is a behavioral
change worth calling out.
- No way to disable the wait remains; if an operator needs an escape
hatch, that would need to be reintroduced.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
