## What changed?
- Added an opt-in recovery path for a child workflow missing from a
standby cluster.
- After the existing resend delay, VerifyFirstWorkflowTaskScheduled
asynchronously fetches the child state from the active cluster, applies
it locally, and verifies it again.
- Added deduplication, per-shard concurrency limits, metrics, namespace
checks, and transition-history gating.
Corrected the discard-time source check to verify the child workflow
rather than the parent.
- Moved the reusable in-flight resend tracker into the shared
workflowresend package.
- Updated the existing XDC parent-child test to assert that the missing
child and its first workflow task are restored.
## Why?
Cross-shard replication may deliver the parent’s
ChildWorkflowExecutionStarted event before the child workflow reaches
the standby cluster. Previously, verification repeatedly returned
NotFound and eventually discarded the standby task, leaving the child
missing.
This adds the child-side symmetric recovery behavior to the parent
resend implemented in #11424 .
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
### Rollout
`history.enableChildWorkflowResend` default to `false`.
### Known
- Enabling this feature introduces additional cross-cluster state-sync
traffic. It is disabled by default and protected by deduplication and a
per-shard concurrency limit.
- Regular replication may race with state sync; duplicate application is
treated as success.
- The resend delay and replication timeout should remain below the
standby task discard delay so recovery has time to complete.
> **Part 3 of a 5-PR series** building to replication stream namespace
isolation (a restructuring of #10147): read buffer → reader group → lane
protocol → isolation manager → sender isolation.
> #11302 (reader group) has merged, so this PR's diff is now standalone
against `main`. · **Next in series: #11304** (isolation manager).
## What changed?
The wire-level building blocks for per-namespace lane isolation,
receiver side only:
- **Proto**: `SyncReplicationState` gains `throttle_high_namespace_ids`
(namespaces the receiver reports as overwhelming the HIGH lane — the
priority lives in the field name, so a future LOW extension adds its own
field rather than widening this one), `isolated_lane_states` (per-lane
applied watermarks keyed by namespace ID — with the documented caveat
that a missing key is ambiguous between "not tracked yet" and "retired
and drained", so a sender must never treat absence alone as drain
proof), and `supports_namespace_isolation` (capability advertisement, so
a sender never emits lane-tagged traffic to a receiver that would
misroute it). `WorkflowReplicationMessages` gains
`isolated_namespace_id` — when set, the batch belongs to that
namespace's dedicated lane — and `retire_isolated_lane`, marking a
lane's final message.
- **Receiver**: lane-tagged batches route to lazily-created
per-namespace task trackers. Each lane is its own monotonic stream for
the life of the connection — there is no rewind or rotation machinery,
because the sender-side design (later in the series) gives every lane a
single owner cursor that never goes backwards. Member-lane watermarks
fold into the overall ack minimum (cleanup safety) and are reported per
lane; member-lane backlogs count toward HIGH flow control. Lane
lifecycle is defensive about ordering: a batch's tasks are tracked
BEFORE its retire flag is applied (so the concurrent ack loop can never
delete a lane whose final batch is mid-track), a retiring lane is only
dropped once it is drained AND has tracked at least one batch, and
non-retire traffic arriving on a retiring lane revives it (the sender
re-isolated the namespace before the lane drained). Lane-tagged traffic
at any priority other than HIGH is a protocol violation and fails the
stream rather than silently mis-acking (isolation splits the HIGH lane
only). Lanes created concurrently with `Stop()` are pre-cancelled so no
tasks run after shutdown.
- **`NamespaceThrottler`** interface (default: noop, via fx) observes
per-namespace HIGH-priority task load and decides which namespaces to
report.
The sender does not tag lanes yet, so this is inert until the
sender-side isolation lands.
## Why?
Isolation needs a wire contract before the sender can use it: capability
advertisement, per-lane routing and progress reporting, and the
throttled-namespace feedback channel. Landing the receiver first makes
mixed-version clusters safe by construction. Compared to #10147, lanes
are per-namespace rather than shared per severity tier — which is what
eliminates that design's cursor rewinds and the
watermark-regression/tracker-rotation protocol this PR previously needed
to compensate for them.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s) — lane routing (priority routing when
unset, per-namespace tracker identity, non-HIGH rejection), retirement
lifecycle (drop once drained, never-tracked retiring lane survives the
ack snapshot, revive on re-isolation traffic, fresh lane after drop),
and post-Stop lane creation being pre-cancelled
- [x] added new functional test(s) — exercised end-to-end by the xdc
test in the final PR of the series
## Potential risks
Inert until a sender emits `isolated_namespace_id`, which is gated
behind both a config flag and the capability advertisement.
Receiver-side lane state is bounded by the sender's isolation cap (final
PR).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches replication ack watermarks and stream failure paths on the
passive cluster; lane mis-handling could stall cleanup or mis-ack,
though lane-tagged traffic is not sent until follow-up sender work.
>
> **Overview**
> Adds the **wire contract and receiver behavior** for per-namespace
HIGH-lane isolation before the sender starts tagging traffic.
>
> **Proto:** `SyncReplicationState` now carries
`throttle_high_namespace_ids`, per-namespace `isolated_lane_states`, and
`supports_namespace_isolation`. `WorkflowReplicationMessages` adds
`isolated_namespace_id` and `retire_isolated_lane` so batches can be
routed and retired on dedicated lanes.
>
> **Receiver:** Lane-tagged HIGH batches use lazily created
per-namespace task trackers (monotonic per connection). Member-lane
watermarks fold into the overall ack minimum and are reported per
namespace; member backlog counts toward HIGH flow control. Acks include
shard-scoped throttled namespace IDs via a new **`NamespaceThrottler`**
(noop by default). Lane lifecycle handles retire/drain/revive and
rejects non-HIGH lane traffic.
>
> Sender emission of lane tags is not in this PR, so behavior stays
inert until a later change gates on capability advertisement.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
688a5173f0. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
## What changed?
- `WithRequestID` now applies to `UpdateComponent`, enabling
execution-level idempotency guarding via request ID.
- When a request ID is passed as part of an `UpdateComponent` call (via
API handler), it is persisted upon successful updateFn call. If it is
already present, instead, `UpdateComponent` fails with a
`FailedPrecondition`.
- When a request ID is not passed in, a generated ID is still created
for error tracing purposes, but it is not written to mutable state.
- On transaction close, mutable state will sweep the oldest RequestIDs
(with an `attach_time`) upon hitting the configured limit.
- This will sweep both entries below a configurable max age, as well as
past a certain hard length limit.
## Why?
- Scheduler's `UpdateSchedule` and `PatchSchedule` are implemented as
handlers that persist a signal in V1. V1 signals provide idempotency via
their request IDs. Scheduler V2 doesn't make use of signals, so instead,
it must record request IDs explicitly.
- We reuse the existing map within mutable state.
- We *must* fail with an explicit error (`FailedPrecondition`) instead
of simply returning a zero value (as Signals would on repeated
successful requests). This is because `UpdateComponent` can apply to API
models that include response values (which we don't record, therefore,
we can't return on subsequent calls).
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: Fred Tzeng <fred.tzeng@temporal.io>
## What changed?
- Check all non-drained version queues before marking a partition
drained.
- Scaler Describe calls only inspect loaded partitions and do not
refresh queue liveness.
## Why?
- `AllActive` missed unloaded per-version backlog
- Loading unloaded partitions from the scaler caused reload cycles.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Scale-down may wait until an unloaded partition is loaded by real
traffic. Explicit version queues may still be loaded when their
partition is already loaded.
## What changed?
1. add a max skip field to TimeSkippingConfig
2. add TimeSkippingInfo to DescribeWorkflowExecution (contains virtual
current time and running status)
3. add PollWorkflowExecutionTimeSkipping for fast-forward completion
## Why?
1. a generic mechanism to stop endless retries or schedules
2. to give clients easier access to time skipping state changes
related API change: https://github.com/temporalio/api/pull/835
## What changed?
Clean up obsolete buf.yaml ignore
## Why?
Remove temporary ignore that was needed for a known breaking change.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
This branch implements the activity operator commands feature — a set of
server-initiated control APIs (PauseActivityExecution,
UnpauseActivityExecution, ResetActivityExecution,
UpdateActivityExecutionOptions) for both workflow-embedded and
standalone activities.
- Pause, Unpause, Reset and UpdateOptions for standalone activities plus
idempotency via RequestId
- common/activityoptions package (common/activityoptions/merge.go):
extracted mergeActivityOptions from the update-options handler into a
shared package (now also used by CHASM activity component).
- Metric renames: ActivityPauseRequests → ActivityPause,
ActivityResetRequests → ActivityReset, ActivityUnpauseRequests →
ActivityUnpause, ActivityUpdateOptionsRequests → ActivityUpdateOptions.
- RPC boilerplate, proto generation, and matching/frontend wiring for
the new APIs.
## Why?
Activity operator APIs existed for workflow-embedded activities but were
not wired up for standalone activities.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
Minimal, this is a new feature so it won't break users.
---------
Co-authored-by: Dan Davison <dan.davison@temporal.io>
Co-authored-by: Fred Tzeng <fred.tzeng@temporal.io>
## What changed?
Include backlog counts (quanitzed) in partition scale state and info,
and use
them in simple scaler.
## Why?
Scaling based on large backlogs, and later load balancing based on
backlog too.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
1. remove event ID from and add archetypeID to the time skipping timer
task
2. add VersionedTransition in time skipping timer task/time skipping
info for cross/intra cell task validation
## Why?
1. chasm executions don't have event ID and need archetypeID for loading
ctx
2. VersionedTransition.FailOver can help find out stale tasks during
failover and failback
4. VersionedTransition.TransitionCount can help find out stale tasks
when fast-forward is overwritten
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## What
Add `TaskQueueKind` to the `NexusTask` token proto so the frontend can
pass the correct kind to matching on
`RespondNexusTaskCompleted`/`Failed`, instead of hardcoding
`TASK_QUEUE_KIND_NORMAL`.
## Why
To distinguish internal nexus invocations from user facing ones.
[#10141](https://github.com/temporalio/temporal/pull/10141) switched
`nexus_task_requests` `is_internal` detection from task queue name
prefix to `TaskQueueKind`, but the frontend hardcodes `NORMAL` when
constructing the matching request for Respond operations because the
token didn't carry the kind. This means `is_internal` is always `false`
for Respond on worker-commands queues.
## How did you test it?
Unit tests:
- Frontend handler: `RespondNexusTaskCompleted` and
`RespondNexusTaskFailed` preserve `TaskQueueKind` from token
(WORKER_COMMANDS preserved, NORMAL preserved, UNSPECIFIED defaults to
NORMAL)
- Matching engine: `PollNexusTaskQueue` sets the correct `TaskQueueKind`
in the serialized task token for both NORMAL and WORKER_COMMANDS
partitions
- Backwards compatible: old tokens without the field default to
`TASK_QUEUE_KIND_UNSPECIFIED`, treated as `NORMAL`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What
Move poll cancellation fan-out from frontend to matching. Frontend sends
one RPC to matching root partition; matching root does flat fan-out —
computes all partitions, groups by host, sends one
`CancelOutstandingWorkerPollsPartition` RPC per remote host. Local
partitions are processed directly (no self-RPC).
## Why
Avoids frontend needing to know about partition topology. Enables
host-level batching (one RPC per host instead of one per partition), and
ensures `removePollerFromHistory` runs for all co-located partitions.
## How did you test it?
- Unit tests: flat fan-out groups partitions by host correctly, local
partitions handled without RPC, remote host errors don't block other
hosts, fallback to individual RPCs when routing client unavailable
- Partition API tests: empty partitions/workers, cancel + shutdown cache
population
- Functional test: `TestShutdownWorkerCancelsOutstandingPolls`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
Add FirstExecutionRunID to start response
## Why?
closes https://github.com/temporalio/temporal/issues/8537
>Today, when SDK gets StartWorkflowExecutionResponse with started as
false or WorkflowExecutionAlreadyStartedFailure, it doesn't include the
first execution run ID that we can bind successive calls to. It only
provides the run ID. If you start a workflow today successfully, SDK
uses the run ID as the first execution run ID on successive calls like
cancel and signal and such. But if you start a workflow today w/
conflict policy of use existing and it doesn't start, that run ID is not
acceptable for first execution run ID.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## What changed?
Adding a PartitionedSnapshot function to the ChasmTree interface so
snapshots can be made that separate user data from the cluster-local
metadata. Also adds a way to recombine a the user data with the metadata
to reconstruct the original tree state.
## Why?
This allows for additional flexibility for replication-related work,
where all the cluster-local metadata is not needed.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Low risk, adding to interface.
## What changed?
`DescribeWorkerDeployment` now returns compute status per version,
checking whether Temporal can successfully interact with the version's
compute resource. `ListWorkerDeployments` also returns compute status on
the current, ramping, and latest version summaries, fetched in parallel.
When connectivity changes (e.g. Lambda becomes unreachable or is
restored), WCI signals the version workflow, which propagates the update
to the deployment workflow memo. The list view reads from the memo —
versions that have been validated since deployment will show their
status immediately; others will appear once the first validation runs.
## Why?
Allows customers to see whether Temporal can successfully interact with
their compute resource directly from the Worker Deployments list and
detail views, without navigating into each individual version.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
1. Active cluster — capture time-skipping state changes for replication
using `timeSkippingInfoUpdated`, `ms.isStateDirty()`,
`cleanupTransaction` and
`TimeSkippingInfo.LastUpdateVersionedTransition`.
2. Active & passive — idempotent timer-task regeneration
- Active: only when a skip transition was emitted this transaction.
- Passive (PartialRefresh, state-based replication): only when
TimeSkippingInfo.LastUpdateVersionedTransition >=
minVersionedTransition.
3. Passive cluster — time-skipping timer-task executor
The standby's regenerated TimeSkippingTimerTask fires through
executeTimeSkippingTimerTask. If the associated FastForwardInfo is still
the active, the standby awaits the replicated transition rather than
driving it.
## Why?
the goal is to make sure time skipping works correctly during failovers
with state-based replication
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
Add simplePartitionScaler as a default partition scaler.
Default settings are not enabled, so still no change in behavior by
default.
## Why?
Next part of dynamic partitioning.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s) - covered by functional tests to be
added in future PR
## What changed and why
1) make server compatible to breaking api change
https://github.com/temporalio/api/pull/786
2) change the propagation policy of time skipping to
- for the same execution (a chain of runs) all time skipping state and
config are shared using StatePropagation
- for child executions
virtual time is always propagated to have causal time
but the fast forward action is never propagated
configuration propagation is controlled by a separate flag
3) rename bound to fast-forward
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
the sequence of this pr
1. review this pr first 2. merge api 3. update this pr with public api
dependency and merge
## What
Add `CountWorkers` RPC to count workers matching a query filter without
retrieving full worker details.
## Why
The UI needs to display a worker count in places where listing isn't
necessary. A dedicated count API follows the existing pattern
(`CountWorkflowExecutions`, `CountSchedules`).
## How did you test it?
- [x] new unit tests: count all, count with query filter, count with no
matches, invalid query error
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
Add StartTime and IsRetentionDelete to delete visibility task
## Why?
Additional info for delete visibility task that might be useful for
implementing VisibilityStore.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
- Fix non-atomic two-step sync in handleSetCurrent and setRamp where
step 2 (demoting old version) failure leaves routing config uncommitted,
causing orphaned CURRENT versions and burned revision numbers.
- New flow: commit routing config immediately after step 1 (promote new
version) succeeds, then fire-and-forget signal to old version instead of
blocking sync activity. Gated behind workflow.GetVersion for NDE safety.
- The only thing that I don't have in this PR are new tests to test this
out. Happy to hear ideas if someone has any, but the core idea was that
the current ones should be passing and testing the code paths.
## Why?
- Reliability IMO
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
- I would appreciate a very careful review on this one!
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **High Risk**
> Changes core worker-deployment routing and task-queue propagation in
async mode; demotion is now signal-based and eventually consistent, with
revision tracking on signal failure.
>
> **Overview**
> Fixes a reliability bug in **async** `set current` / `set ramp` where
promoting the new version succeeded but **demoting** the previous
current or ramping version could fail, leaving deployment routing
uncommitted and inconsistent summaries.
>
> When `workflow.GetVersion("commit-routing-first")` is enabled in async
mode, the deployment workflow **commits** `pendingRoutingConfig` to
local state right after the promote step, then **signals** the old
version workflow via new **`demote-version`** (`DemoteVersionSignalArgs`
carrying full `RoutingConfig`) instead of a blocking `syncVersion`
activity. Version workflows handle the signal (gated by
`demote-version-signal`) by deriving status from routing config, syncing
task queues, and starting drainage when needed.
**`signalDemoteVersion`** tracks propagating revision numbers until
`PropagationComplete`; failed signal delivery untracks the revision.
**`setVersionSummaryDraining`** updates deployment-side version
summaries immediately in the new path. Sync mode and workflows without
the version gates keep the prior behavior.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
68698ec0cc. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## What changed?
Add scale manager component.
## Why?
Part of dynamic partitioning.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s) - to come in future PRs
## Potential risks
No behavior changes yet.
## What changed
- Adds a focused repro for a failed transient WFT that observes
`targetWorkerDeploymentVersionChanged=true` but never durably commits a
started event. Thus, in the next workflow task, we don't set the
`targetWorkerDeploymentVersionChanged=true` which results in errors with
respect to trampolining
- Now, to fix this, we just have the LastNotifiedTargetVersion present
in the MS to be the source of truth when filling in this flag.
## Why
A transient/speculative WFT can expose the target-version-changed flag
to the SDK before its started event is durable. If that WFT fails or is
discarded, we still should be sending the
`targetWorkerDeploymentVersionChanged=true` to our next workflow task.
## Verification
- `go test -tags test_dep ./tests -run
'TestVersioning3FunctionalSuite/TestPinnedCaN_FailedTransientNotificationDoesNotBecomeDeclinedOnPlainCaN'
-count=1`\n- `go test -tags test_dep ./tests -run
'TestVersioning3FunctionalSuite/TestPinnedCaN_FailedTransientNotificationDoesNotBecomeDeclinedOnPlainCaN'
-count=3`\n\nNote: broader existing trampolining functional tests hit
local test-cluster worker-deployment/user-data setup failures even from
a clean worktree at `c2c4bc1928`; CI should be the source of truth for
those.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches pinned versioning signals, persistence schema (reserved
field), and continue-as-new metadata; behavior change is intentional but
affects worker SDK upgrade notifications.
>
> **Overview**
> Fixes a **pinned worker-deployment** bug where a
**transient/speculative** workflow task could tell the SDK
`target_worker_deployment_version_changed=true` and set
`LastNotifiedTargetVersion`, but a failed/discarded task left the next
poll without that flag because it was no longer stored on mutable state.
>
> **`target_worker_deployment_version_changed` on `WorkflowTaskStarted`
is now derived at emit time** from `LastNotifiedTargetVersion` (plus
config and **PINNED** behavior) via
`targetWorkerDeploymentVersionChangedForStartedEvent()`, instead of
persisting `workflow_task_target_worker_deployment_version_changed` on
`WorkflowExecutionInfo` / `WorkflowTaskInfo`. Proto field **112** is
**reserved**; apply/rebuild paths and mocks drop the old parameter.
>
> **Continue-as-new:** `computeDeclinedTargetVersionUpgrade` only
propagates declined-upgrade metadata when the new run **inherits a
pinned version**; otherwise it returns nil.
>
> Adds
**`TestPinnedCaN_FailedTransientNotificationRefiresDespiteStaleMatching`**
(failed transient notification, mutable state still has
`LastNotifiedTargetVersion`, next WFT re-fires the signal even if
matching rolls back).
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
dfaf225b71. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Add per-request links (keyed by request_id) and user_metadata fields to
ChasmComponentAttributes so any CHASM component can store them uniformly
without each library defining its own proto field. This first PR just
adds support, I will open up follow up PRs to update any existing
components use these new fields.
## Why?
Per the discussion
[here](https://github.com/temporalio/temporal/pull/10368#discussion_r3312238761)
we think links and user metadata will be common components across CHASM
components so it makes sense for the framework to handle this for us vs
each component implementing it themselves.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Matching server propagates partition counts in ephemeral data.
- Matching server validates partition counts sent by client, and rejects
if they are too far.
## Why?
Next part of dynamic partitioning.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s) - covered by future integration tests
## Potential risks
No change in behavior at this point, partition scale info is always nil,
so nothing will be rejected. The server will start sending grpc trailers
though.
## What changed?
Add a dedicated DeleteChasmExecution RPC to the history service for
deleting CHASM executions by namespace, execution key, and archetype ID.
Uses the CHASM engine's DeleteExecution path (terminate-if-running +
async DeleteExecutionTask), replacing the ForceDeleteWorkflowExecution
workaround in the delete namespace activity which bypassed the engine.
Also adds NewComponentRefByArchetypeID to chasm/ref.go to construct a
ComponentRef from a runtime archetype ID without a compile-time type
parameter.
## Why?
Remove dependency on force deletion API.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## What changed
- Replaced per-key trailer format with a single protobuf
`ContextMetadata` message serialized into `contextmetadata-bin` trailer
key
- gRPC automatically base64-encodes the `-bin` value, making arbitrary
bytes (including HTTP/2-unsafe control chars) transport-safe
- Writer emits both proto format and legacy per-key format for backward
compatibility during rolling deploys
- Reader prefers proto key, falls back to legacy per-key format for old
writers
- Wired `TrailerToContextMetadataInterceptor` in test server to match
production behavior
## Why
Workflow type names containing control characters (newlines, NUL, etc.)
cause the gRPC HTTP/2 framer to reject trailer values. A single proto
message in a `-bin` key is simpler than per-key `-bin` suffixes: one
trailer key, one serialization, no key naming constraints, cleaner
backward compat removal.
## How tested
- Unit tests for proto round-trip, dual-format emission, reader
preference, legacy fallback, HTTP/2 safety
- Integration test suite (TestWorkflowTypeEncodingSuite) with control
chars, UTF-8, long names, -bin suffix workflow types
- All existing tests pass
## Risks
- During rolling deploy, old writers emit only legacy keys. New readers
handle this via fallback path. No data loss.
- After full rollout, legacy key emission can be removed in a follow-up.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes how context metadata is encoded/decoded in gRPC trailers,
which can affect cross-version compatibility and observability of
propagated metadata. Backward-compatible legacy fallback and extensive
unit/integration tests reduce the rollout risk.
>
> **Overview**
> **Switches context-metadata propagation in gRPC trailers to a single
proto-encoded payload.** Server-side `ContextMetadataInterceptor` now
serializes all context metadata into a new `ContextMetadata` protobuf
and emits it under `contextmetadata-bin`, avoiding HTTP/2-unsafe control
characters in values.
>
> **Maintains rolling-deploy compatibility.** Writers still emit legacy
per-key trailers (skipping unsafe values), and the client-side
`TrailerToContextMetadataInterceptor` now *prefers* the proto trailer
and falls back to legacy keys (including unprefixed well-known keys)
when needed.
>
> Adds the new `contextpropagation/v1` proto + generated Go types, plus
unit tests around proto/legacy behavior and an integration suite
(`WorkflowTypeEncodingSuite`) covering control characters, UTF-8, long
names, and `-bin` suffix workflow types.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
e1d772fc3f. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## What changed?
### **High level**
With https://github.com/temporalio/api/pull/761 to add the linking on
the Signal and Signal-with-Start responses, This PR adds logic from the
server that:
* Adds `requestID` from Signal and Signal-with-Start requests to the
CHASM workflow tree under a new map field `IncomingSignals`, and event
store, so these requestIDs stay in buffer
* Return a backlink in the response that references the `requestID`
* On buffer flush to the DB transaction, attach these `requestID` to a
concrete `eventID`, which would allow users to later know which event
correlated w/ this request. We will wire the concrete event ID to the
signal request IDs stored in the workflow component CHASM tree
(`IncomingSignals` map)
> [!NOTE]
> Feature is gated behind a new dynamicconfig
`EnableCHASMSignalBacklinks`, which implicitly is only checked if
`EnableChasm` is enabled.
## Why?
This will enable the caller of the signal to have a backlink to the
cross-namespace signal invoked, which will become more relevant for
Nexus SDK ergonomics.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
In functional tests, I augmented existing tests for Signal and
Signal-with-Start to:
* Ensure that backlink is returned via the responses
* Later use `DescribeWorkflow` to ensure that we get a concrete EventID
(mapped when buffer flushed)
* Multiple signals with the same `requestID` gets de-duped
```
$ go test ./tests/ -run TestLinksTestSuite
ok go.temporal.io/server/tests 1.486s
```
```
$ go test ./tests/ -run 'TestNexusWorkflowTestSuite' -count=1
ok go.temporal.io/server/tests 4.714s
```
## Potential risks
Need to test end-to-end to see that the link shows up correctly in the
Web UI.
Feature is gated behind dynamicconfig since it requires CHASM-based
workflow to be enabled.
## What changed?
Add CHASM lifecycle paused state. Invalidates all pending logical tasks.
## Why?
There are duplicate implementations in Libraries where authors have
implemented their own statuses to act as a PAUSE state, preventing all
tasks from executing.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
(1) elapsed-duration / skipped-duration **bounds** that auto-disable
time-skipping
(2) a new `TimeSkippingTimerTask` that wakes the workflow when an
elapsed bound is reached
(3) time-skipping support for backoff timers (start-with-delay / cron /
retry / CaN-with-backoff) plus an exclusion for child workflows that
haven't scheduled their first WT
## Why?
the completion of the feature
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed and Why?
Paired with [api
change](https://github.com/temporalio/api/pull/770/changes)
Define the default time-skipping propagation behavior for current
features: continue-as-new (CAN), child workflows, retry, and reset.
**Group 1 — Inherit both from the current execution:**
- Continue-as-new (CAN): inherits the current config and accumulated
skipped duration; the configured bound is shared across both the
inherited skipped duration and any duration skipped by the new run.
- Child workflows: same behavior as CAN
This design is because CAN is a technical reason to start a new run of
current run, so logically they can be viewed as a same "run". For Child
WFs they can be viewed as an extension of previous workflows or separate
workflows, and in either case, they shall inherit the skipped duration
so that the virtual time doesn't rewind back, and the default behavior
is designed to treat them as an extension of the parent and share the
config. We only consider adding new config to provide flexibility on
demand.
**Group 2 — Inherit from a specific point in history:**
- Retry: inherits the config and skipped duration recorded in the
StartWorkflowExecutionEvent of the current workflow, since retry is
defined as restarting execution from that event
- Cron: same with Retry
**Group3 -- Both inherit from a specific point and catch up of config
change to current**
- Reset: retains the current time-skipping config, since reset is
designed to replay all events up to the reset point and apply any
UpdateWorkflowExecutionOptions changes made after that point — with no
option to exclude them (covered by tests only)
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
Support completion-before-start.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Summary
- Extends `CheckTaskQueueVersionMembership` response with two new
fields: `is_version_active_or_draining` (bool) and `revision_number`
(int64). Matching populates both from its deployment data.
- Reactivation signals are **skipped** when matching reports the target
version as CURRENT/RAMPING/DRAINING; otherwise they are sent with a
**deterministic UUID v5 RequestId** derived from `revision_number`.
- Replaces the old TTL-based `ReactivationSignalCache` with a per-pod
**revision-based dedup LRU** on the worker-deployment client: each entry
records the highest revision this pod has successfully signaled for a
given version, so older or equal signals are skipped.
## What changed on the wire (matching → history)
`CheckTaskQueueVersionMembershipResponse` now has two new flat fields
(no wrapper message):
```proto
bool is_version_active_or_draining = 2; // true when status is CURRENT/RAMPING/DRAINING
int64 revision_number = 3; // from WorkerDeploymentVersionData.revision_number; 0 if unknown / legacy
```
Matching's `CheckTaskQueueVersionMembership` fills both via the helper
`worker_versioning.IsVersionActiveOrDraining(deploymentData, dep, build)
(bool, int64)`.
Naming choice — we picked `is_version_active_or_draining` (negative
polarity) rather than something like `supports_reactivation` so the
proto zero value (`false`) maps to the safe default ("send the signal").
Old matching binaries and runtime "version not found" both produce the
zero value, and history correctly falls through.
## Where `revision_number` flows
- **Matching**: populates the response field from the version's tracked
revision.
- **History-side helper/caches**:
`ValidateVersioningOverrideAndGetReactivationEligibility` returns
`(isVersionActiveOrDraining bool, revisionNumber int64, err)`.
`VersionMembershipAndReactivationStatusCache` stores both.
- **History signaler plumbing**: `VersionReactivationSignalerFn`,
`ReactivateVersionWorkflowIfPinned`, and all five call sites
(`startworkflow`, `signalwithstartworkflow`, `updateworkflowoptions`,
`resetworkflow`, `multioperation`) carry `revisionNumber int64`.
`resetworkflow.validatePostResetOperationInputs` returns parallel slices
`([]bool, []int64, error)` for per-operation inputs.
- **Signal RequestId**: `ClientImpl.SignalVersionReactivation` composes
`requestID = uuid.NewSHA1(uuid.NameSpaceOID,
[]byte("reactivation-signal:" + revisionNumber)).String()` — a
deterministic UUID v5 derived from the revision alone. Cassandra's
`signal_requested set<uuid>` column requires UUID-formatted RequestIds.
## Why revision-based dedup
History is sharded on `(namespaceID, workflowID)`. N concurrent
`StartWorkflow` calls pinned to the same drained version fan out across
potentially every history pod in the fleet. Before this PR each pod
independently fired a reactivation signal at the version workflow,
producing up to N `WorkflowExecutionSignaled` events — directly at odds
with the version workflow's design (it intentionally keeps history
minimal and CaNs aggressively, see `version_workflow.go:68-74`).
Per-pod caches alone can't fix this because they don't coordinate. What
we need is a **cluster-wide-deterministic dedup key** so all pods
converge on the same value for the same reactivation cycle. The
version's `revision_number` — incremented in `syncTaskQueuesAsync` on
every status change — is exactly that signal. Every pod reads the same
revision from matching, every pod composes the same UUID RequestId, and
Temporal's built-in `mutableState.pendingSignalRequestedIDs` dedup (see
`service/history/api/signalworkflow/api.go:40`) collapses concurrent
signals into exactly one event on the version workflow.
The per-pod map is a local optimization on top of that: it prevents a
single pod from re-sending the same-or-older-revision signal once it has
successfully sent one, cutting RPC volume.
## How the new caches look
### 1. `VersionMembershipAndReactivationStatusCache` (read-side,
per-pod)
Caches matching's `CheckTaskQueueVersionMembership` response so repeated
pinned-override validations on the same task queue don't re-hit
matching.
- **Key**: `(namespaceID, taskQueue, taskQueueType, deploymentName,
buildID)`
- **Value**: `(isMember bool, isVersionActiveOrDraining bool,
revisionNumber int64)`
- **Eviction**: `VersionMembershipCacheTTL` (1s default; 5s in
functional tests).
### 2. `highestRevSignaledToVersionWf` (write-side dedup, per-pod)
A field on `ClientImpl` in `service/worker/workerdeployment/client.go`.
For each target version workflow, stores the highest revision this pod
has successfully signaled. Subsequent calls at the same-or-lower
revision skip the RPC.
- **Key**: `reactivationVersionKey{namespaceID, deploymentName,
buildID}`
- **Value**: `int64` (highest revision successfully signaled)
- **Eviction**: LRU, bounded by `VersionReactivationSignalCacheMaxSize`.
The previous TTL-based `ReactivationSignalCache` module (in
`common/worker_versioning/`) has been deleted along with its provider
and `VersionReactivationSignalCacheTTL` config.
## Backwards/forwards compatibility
- **Old matching → new history**: old binaries don't set
`is_version_active_or_draining` or `revision_number`; both default to
proto zero values. `false` on the active bool → history falls through →
signal fires (safe default). `revisionNumber = 0` flows through as-is.
- **New matching → old history**: new fields on the response are ignored
by old history → identical to pre-PR behavior.
- **New matching → new history**: signal fires only when the version is
not active/draining; cross-pod fires converge on one UUID RequestId and
fold into one `WorkflowExecutionSignaled` event.
## Test plan
- [x] Unit tests for `IsVersionActiveOrDraining` covering all status
cases (CURRENT, RAMPING, DRAINING, DRAINED, INACTIVE, UNSPECIFIED), new
vs. old format, deleted and not-found versions.
- [x] Unit tests for
`ValidateVersioningOverrideAndGetReactivationEligibility` (cache
hit/miss, RPC with/without eligibility, Unimplemented fallback).
- [x] Unit tests for the per-pod dedup on
`ClientImpl.SignalVersionReactivation`: same-rev dedups, newer-rev
fires, older-rev skipped, different version isolated, signal-failure
allows retry.
- [x] Unit test for RequestId format (UUID v5, deterministic across
calls with the same revision).
- [x] Functional tests (all pass on SQLite and cass-es):
- `TestStartWorkflowExecution_ReactivateVersionOnPinned`
-
`TestStartWorkflowExecution_ReactivateVersionOnPinned_WithConflictPolicy`
- `TestSignalWithStartWorkflowExecution_ReactivateVersionOnPinned`
- `TestUpdateWorkflowExecutionOptions_ReactivateVersionOnPinned`
- `TestResetWorkflowExecution_ReactivateVersionOnPinned`
(The four `TestReactivationSignalCache_Deduplication_*` functional tests
from an earlier iteration were deleted — their coverage moved to unit
tests.)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes matching↔history API and reactivation signaling semantics by
skipping signals for active/draining versions and introducing
revision-based dedup via deterministic RequestIds; issues could affect
version workflow state transitions or signal fan-out during upgrades.
>
> **Overview**
> Matching’s `CheckTaskQueueVersionMembershipResponse` is extended with
`should_skip_reactivation` and `revision_number`, and matching now
populates both from per-task-queue deployment data.
>
> History-side versioning validation is refactored to return and cache
reactivation eligibility + revision, and reactivation signaling paths
(`StartWorkflow`, `SignalWithStart`, `UpdateWorkflowExecutionOptions`,
`ResetWorkflow`, multi-op) now **skip signals** when matching reports
the version as *CURRENT/RAMPING/DRAINING*.
>
> The old TTL-based `ReactivationSignalCache` is removed
(configs/metrics/providers updated), and the worker-deployment client
now performs **revision-based per-pod dedup** plus receiver-side dedup
by sending signals with a deterministic UUIDv5-like `RequestId` derived
from `revision_number`. Tests are updated/added to cover status
evaluation, new plumbing, and dedup behavior.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
a1ec5e93db. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Matching now sends the real `targetDeploymentRevisionNumber` for
pinned workflows instead of hardcoded 0
- Added `revision_number` field to `LastNotifiedTargetVersion`
(server-internal) and `DeclinedTargetVersionUpgrade` (public API)
- Case 4 in the target version change switch now uses
`targetRevisionNumber <= declined.RevisionNumber` instead of version
string comparison, preventing trampolining when stale matching
partitions send outdated target versions
## Problem
When a pinned workflow CaNs and declines a target version upgrade, a
stale matching partition can send an older target version. The old case
4 compared version strings (`declined.buildId == target.buildId`), which
didn't match the stale version — causing the workflow to re-signal, CaN
again, and trampoline indefinitely between stale and up-to-date
partitions.
## Test plan
- [x] `TestStalePartition_RevisionSuppressesTrampolining` — integration
test that simulates a stale partition via `rollbackTaskQueueToVersion`
and verifies:
- Stale partition (revision 0) is suppressed after declining at a higher
revision
- Genuinely new version (v4 at higher revision) correctly fires the
signal
- [x] Verified test **fails** when matching + history changes are
reverted (matching sends 0, old string comparison)
## API repo dependency
- api: `temporalio/api@trampolining-rev-number`
- api-go: `temporalio/api-go@trampolining-rev-number`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches pinned-workflow versioning logic across matching, history, and
persistence; a bug here could change when workflows are signaled to
continue-as-new or upgrade, affecting routing behavior.
>
> **Overview**
> Prevents pinned workflows from repeatedly continue-as-new
“trampolining” when task queue partitions have stale routing data by
**tracking and comparing target routing-config revision numbers**.
>
> Matching now propagates the real `targetDeploymentRevisionNumber` for
pinned workflow tasks, and history threads this through
`AddWorkflowTaskStartedEvent` to persist
`LastNotifiedTargetVersion.revision_number` and carry it into
`DeclinedTargetVersionUpgrade` on continue-as-new. The pinned
target-change decision in `workflow_task_state_machine` switches case-4
suppression from deployment version string equality to
`targetRevisionNumber <= declined.RevisionNumber`, and adds an
integration test (`TestStalePartition_RevisionSuppressesTrampolining`)
covering stale vs fresh partition behavior.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
4114662b39. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## What changed?
Support Nexus async completion in CHASM.
PS: completion-before-start will be handled in a follow-up PR.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
Is behind feature flag.
## What
Add support for `TASK_QUEUE_KIND_WORKER_COMMANDS` in the matching
service. This is a new task queue kind for server-to-worker
communication (e.g. activity cancellations). It is represented by a new
`WorkerCommandsPartition` type in the `tqid` package — analogous to how
`StickyPartition` represents sticky queues.
Also adds a distinct `partition` metric tag for worker-commands queues
so their traffic can be distinguished from user-facing normal queues in
metrics.
Depends on [#9900](https://github.com/temporalio/temporal/pull/9900).
## Why
Worker_Commands queue kind has different properties compared to a normal
user defined queue. So we created a separate kind. This allows us to
distinguish task queue metrics based on whether they are internal or
user created queue.
## How did you test it?
Unit tests covering metric tag generation for normal, sticky, and
worker-commands partitions.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: David Reiss <david@temporal.io>
## What changed?
Add client-side of dynamic partition scaling: `partitionCache`,
`PartitionCounts`, `StalePartitionCounts` error,
`invokeWithPartitionCounts`.
The intended behavior is:
For partition-aware calls (add+poll tasks), the client caches the
partition count for active task queues, and uses those partition counts
for load balancing (no change to load balancing algorithm yet). The
client sends its cached partition counts to the server in a grpc header,
and receives updated partition counts with the response in a grpc
trailer. If the updated counts are different, it updates its cache it
for subsequent requests. If the server indicates that the client's view
of partition count is stale, it returns a special `StalePartitionCounts`
error and then client makes one immediate retry with the newly received
counts.
With just this PR by itself, the server will never send counts, the
cache will always be empty, and the client will always fall back to
dynamic config for partition counts, so there's no change in behavior
yet.
## Why?
Implement half of dynamic partition scaling.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s) – tests are in future PRs
## What changed?
1) `taskGenerator`:
- add regenerate tasks for time skipping
2) `mutableState`:
- new methods: add key methods to transition time
- `IsStateDirty`: captures changes of time-skipping related fields
- `closeTransaction`: add closeTransactionHandlerTimeSkipping
3) event: new time-skipping runtime event added
4) persistence: new runtime data added to execution info
## Why?
add the foundational mechanism of how time skipping works in the runtime
of a workflow execution
**for reviewers:**
this is a foundation of t-s runtime, so it works without any granular
and extended features as bound, replication, external-transfer tasks,
etc. **Over 50% of lines are tests/mocks/pb-gen. can focus on code
first, then functional tests, and last ut.**
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What
Dispatches worker commands (starting with activity cancellation) to
workers via their Nexus control queue. When the outbound queue processes
a `WorkerCommandsTask`, the dispatcher sends an `ExecuteCommands` Nexus
operation to the worker's control queue via `DispatchNexusTask`.
- Retries are capped at 3 attempts since these commands are best-effort
(the activity will eventually time out anyway). Uses the retryable
matching client so transient RPC errors (ShardOwnershipLost,
Unavailable) are retried at the RPC layer without consuming queue-level
attempts.
- Stores the clock used to generate the task token in the ActivityInfo.
This is needed to reconstruct the same task token when dispatching to
the worker. Otherwise, it will not match the token expected by the sdk.
Suggested review order: `worker_commands_task_dispatcher.go` →
`nexus_dispatch_response.go` → `recordactivitytaskstarted/api.go`.
## Why
To support activity cancellation without activity heartbeat. This is the
dispatch leg of the flow:
1. [#9231] Store `worker_control_task_queue` in `ActivityInfo` at
activity start.
2. [#9232] On `RequestCancelActivityTask`, batch commands by control
queue into `WorkerCommandsTask` outbound tasks.
3. **[This PR]** Dispatch each task as a Nexus `ExecuteCommands`
operation to the worker, with a 3-attempt retry cap.
4. [SDK] Worker receives the cancel command and cancels the running
activity.
Gated by dynamic config `EnableCancelActivityWorkerCommand` (default:
off).
## How did you test it?
- **Unit tests** cover all dispatch outcomes (success, RPC error,
timeout, worker error, feature-flag-off, max-attempts-exceeded) and
response-to-error conversion paths.
- **Functional test** verifies end-to-end: cancel request → Nexus
dispatch → correct payload arrives on the control queue, and asserts
that the cancel command's task token matches the one from the original
activity poll response.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: rkannan82 <5853897+rkannan82@users.noreply.github.com>
## What changed?
Add time-skipping configuration support.
(1) Feature flag in dynamic config
- frontend.TimeSkippingEnabled (namespace bool, default false) gates the
feature
(2) Frontend validation (validateTimeSkippingConfig)
- StartWorkflowExecution
- SignalWithStartWorkflowExecution
- ExecuteMultiOperation (via the shared prepareStartWorkflowRequest
path)
(3) Persistence
- TimeSkippingInfo proto added to WorkflowExecutionInfo
- MutableState stores the config when a workflow starts or options are
updated
(4) Tests
- New timeskipping functional tests in tests/timeskipping_test.go
## Why?
first step of the time skipping project
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
---------
Co-authored-by: Yichao Yang <yycptt@gmail.com>
## What changed?
New outbound task type (`WorkerCommandsTask`) that carries worker
commands to be dispatched to workers via Nexus. Uses the generic
`WorkerCommand` proto (not cancel-activity-specific), so this task type
can carry any future command types.
Suggested review order: proto changes → `worker_commands_task.go` →
`task_generator.go` → `workflow_task_completed_handler.go`
Key pieces:
- **Proto**: `TASK_TYPE_WORKER_COMMANDS` enum, `WorkerCommandsTask` in
`OutboundTaskInfo` with `repeated WorkerCommand`.
- **Task definition**: `worker_commands_task.go` — implements outbound
`Task` and `HasDestination` interfaces.
- **Task creation** (`workflow_task_completed_handler.go`,
`task_generator.go`): When `RequestCancelActivityTask` is processed for
a started activity whose worker has a control queue, collects a
`CancelActivityCommand` with the activity's task token. Commands are
batched by destination control queue and flushed as one
`WorkerCommandsTask` per queue at the end of WFT processing.
- **Serialization**: `task_serializers.go` for persistence
round-tripping.
Dispatch is a no-op here — handled in #9233. Gated by dynamic config
`EnableCancelActivityWorkerCommand` (default: off).
## Why?
To support proactive activity cancellation without waiting for
heartbeat. This is the task creation leg of the flow.
1. [#9231] Store `worker_control_task_queue` in `ActivityInfo` at
activity start.
2. **[This PR]** On `RequestCancelActivityTask`, batch commands by
control queue into `WorkerCommandsTask` outbound tasks.
3. [#9233] Dispatch each task as a Nexus `ExecuteCommands` operation to
the worker, with a 3-attempt retry cap.
4. [SDK] Worker receives the cancel command and cancels the running
activity.
Gated by dynamic config `EnableCancelActivityWorkerCommand` (default:
off).
## How did you test it?
**Unit tests** cover task generation, command batching (including
multi-queue batching), task serialization round-tripping, and the
feature-flag-off path.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
As part of RecordActivityTaskStarted flow, store
worker_control_task_queue for an activity in the mutable state
(ActivityInfo).
Main changes:
- executions.proto: Added the new worker_control_task_queue field.
- mutable_state_impl.go: Update mutable state.
- matching/forwarder.go: Propagate worker_control_task_queue when polls
get forwarded. Otherwise, RecordActivityTaskStarted request will not
have it set when invoked from a forwarded poll.
## Why?
To support activity cancellation without activity heartbeat.
Overall flow:
- [This PR] Store worker attributes in ActivityInfo as part of
RecordActivityTaskStarted call.
- [#9232] When user cancels a workflow, create 1 or more tasks. Group
all activities belonging to a worker into the task (for efficiency).
- [#9233] Lookup the Nexus task queue for each worker, and send a Nexus
operation for each transfer task.
- [SDK] Worker will receive this cancel task and cancel the running
activities.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
Add `CONTINUE_AS_NEW_VERSIONING_BEHAVIOR_USE_RAMPING_VERSION` as a
`ContinueAsNewInitialVersioningBehavior` in `InheritedAutoUpgradeInfo`
and `VersioningInfo`, which if set, forces the first task of an
AutoUpgrade workflow to go to the Ramping Version of its task queue
instead of basing the Current/Ramping version selection on ramp
percentage and workflow id. If there is no Ramping Version at the time
of workflow dispatch, the workflow will use the Current Version.
This initial behavior comes from the continue-as-new command, and when
history sends workflow tasks to matching, is converted into an
internally defined `UseRampingVersionInitialTask bool` such that _only_
the first workflow task of an AutoUpgrade workflow will change its
Target Version selection. Retries of this workflow will start with the
same behavior, but child workflows and future continue-as-new workflows
initiated by this workflow will not inherit the initial behavior.
## Why?
To enable more fine grained control of upgrade-on-continue-as-new
upgrade stages before doing percentage based ramp. To use this, users
should set their promotion candidate version to the ramping version with
a ramp percentage of zero, and manually signal a certain cohort of
workflows to upgrade-on-continue-as-new with this option.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes workflow-task routing for AutoUpgrade executions by adding a
new directive flag that can bypass ramp-percentage hashing for the
initial task. Moderate risk because it affects matching/history version
selection and ContinueAsNew/retry inheritance semantics.
>
> **Overview**
> Adds a new `use_ramping_version` field to `TaskVersionDirective` and
threads it through history → matching so an AutoUpgrade workflow can
*force its initial workflow task* to route to the task queue’s ramping
deployment version (falling back to current when no ramping version
exists), bypassing the normal ramp-percentage/workflow-id hash.
>
> History now persists and evaluates the ContinueAsNew-requested initial
behavior via `MutableState.GetShouldUseRampingVersion()`, ensures it
applies only to the first workflow task, propagates it across retries,
and explicitly does **not** propagate it to child workflows or
subsequent ContinueAsNew hops. Updates tests (unit +
`versioning_3_test`) and bumps `go.temporal.io/api` to pick up the new
proto/enum support.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
923602f0c2. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
- Add compute config summary for latest and all other version so it is
exposed in `ListWorkerDeployments` and `DescribeWorkerDeployment` APIs.
## Why?
Mainly for UI to show compute provider types.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None
## What changed?
Remove guards to allow breaking changes in two files.
## Why?
Safety.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA
## What changed?
Add a new replication task type `DeleteExecutionReplicationTask` that
replicates workflow deletion from the active cluster to passive/standby
clusters. Gated by feature flag
`history.enableDeleteWorkflowExecutionReplication` (default: false).
#### **Key changes across the replication pipeline:**
1. Proto enums: `TASK_TYPE_REPLICATION_DELETE_EXECUTION` (34),
`REPLICATION_TASK_TYPE_DELETE_EXECUTION_TASK` (13)
2. Replication task is associated with a new stage in
`ShardContext.DeleteWorkflowExecution`, bundled with delete visibility
task.
3. ~Engine interface: added `ForceDeleteWorkflowExecution` so the task
can invoke the `ForceDeleteWorkflowExecution`.~
## Why?
Today, when a user delete workflow execution in source cluster, this
operation will not replicate to the standby/target clusters. When a
namespace failover to a target cluster, those deleted workflow may
resurrected.
<details>
<summary>Race condition analysis</summary>
**Before this change:**
1. **Cross-cluster resurrection:** Active deletes workflow → standby
untouched → failover → workflow reappears.
2. **Termination event silently dropped:** Deleting a running workflow
terminates it first, generating a `HistoryReplicationTask`. But the
async `CloseExecutionTask` may delete mutable state before the stream
sender converts that task.
The converter calls `getBranchToken()` → `NotFound` → task silently
dropped. The standby never sees the termination or the deletion.
**After this change:**
Race 1 is fixed — `DeleteExecutionReplicationTask` explicitly tells the
standby to delete.
Race 2 is mitigated — even if the termination event's replication task
is dropped, the delete replication task ensures the standby cleans up.
- If the workflow is still running (termination not yet replicated), the
`DeleteExecutionTask` reschedules itself until the workflow closes.
- If the termination event arrives later, the workflow closes normally,
then the delete proceeds.
- If the workflow is already deleted (e.g., by retention), the task is a
no-op (`NotFound` treated as success).
</details>
<details>
<summary>Deletion paths</summary>
| Path | Replication task? |
|------|-------------------|
| User deletes workflow (active, running or closed) | Yes |
| User deletes on passive (DC forwarding ON) | Forwarded to active → yes
|
| User deletes on passive (no forwarding) | No — `ActiveInCluster` check
skips |
| Retention expiry (with or without archival) | No — stage pre-marked as
processed |
| Admin ForceDelete (tdbg) | No — bypasses `DeleteWorkflowExecution` |
</details>
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
Before change:
<img width="1507" height="163" alt="Screenshot 2026-03-26 at 11 59
51 PM"
src="https://github.com/user-attachments/assets/118cc50e-b69d-468a-9e45-5f49e4e4b9d1"
/>
After change:
<img width="1507" height="135" alt="Screenshot 2026-03-27 at 12 00
07 AM"
src="https://github.com/user-attachments/assets/8ccb7a11-2cb4-48b5-af89-b4a60ddb6333"
/>
## Potential risks
n/a
## What changed?
Adds `buf format` as a Makefile target; and integrates it into `make
fmt`.
All `.proto` changes are from running `make fmt`.
## Why?
Consistent protobuf file style.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
