## What
Dispatches worker commands (starting with activity cancellation) to
workers via their Nexus control queue. When the outbound queue processes
a `WorkerCommandsTask`, the dispatcher sends an `ExecuteCommands` Nexus
operation to the worker's control queue via `DispatchNexusTask`.
- Retries are capped at 3 attempts since these commands are best-effort
(the activity will eventually time out anyway). Uses the retryable
matching client so transient RPC errors (ShardOwnershipLost,
Unavailable) are retried at the RPC layer without consuming queue-level
attempts.
- Stores the clock used to generate the task token in the ActivityInfo.
This is needed to reconstruct the same task token when dispatching to
the worker. Otherwise, it will not match the token expected by the sdk.
Suggested review order: `worker_commands_task_dispatcher.go` →
`nexus_dispatch_response.go` → `recordactivitytaskstarted/api.go`.
## Why
To support activity cancellation without activity heartbeat. This is the
dispatch leg of the flow:
1. [#9231] Store `worker_control_task_queue` in `ActivityInfo` at
activity start.
2. [#9232] On `RequestCancelActivityTask`, batch commands by control
queue into `WorkerCommandsTask` outbound tasks.
3. **[This PR]** Dispatch each task as a Nexus `ExecuteCommands`
operation to the worker, with a 3-attempt retry cap.
4. [SDK] Worker receives the cancel command and cancels the running
activity.
Gated by dynamic config `EnableCancelActivityWorkerCommand` (default:
off).
## How did you test it?
- **Unit tests** cover all dispatch outcomes (success, RPC error,
timeout, worker error, feature-flag-off, max-attempts-exceeded) and
response-to-error conversion paths.
- **Functional test** verifies end-to-end: cancel request → Nexus
dispatch → correct payload arrives on the control queue, and asserts
that the cancel command's task token matches the one from the original
activity poll response.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: rkannan82 <5853897+rkannan82@users.noreply.github.com>
## What changed?
Add time-skipping configuration support.
(1) Feature flag in dynamic config
- frontend.TimeSkippingEnabled (namespace bool, default false) gates the
feature
(2) Frontend validation (validateTimeSkippingConfig)
- StartWorkflowExecution
- SignalWithStartWorkflowExecution
- ExecuteMultiOperation (via the shared prepareStartWorkflowRequest
path)
(3) Persistence
- TimeSkippingInfo proto added to WorkflowExecutionInfo
- MutableState stores the config when a workflow starts or options are
updated
(4) Tests
- New timeskipping functional tests in tests/timeskipping_test.go
## Why?
first step of the time skipping project
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
---------
Co-authored-by: Yichao Yang <yycptt@gmail.com>
## What changed?
New outbound task type (`WorkerCommandsTask`) that carries worker
commands to be dispatched to workers via Nexus. Uses the generic
`WorkerCommand` proto (not cancel-activity-specific), so this task type
can carry any future command types.
Suggested review order: proto changes → `worker_commands_task.go` →
`task_generator.go` → `workflow_task_completed_handler.go`
Key pieces:
- **Proto**: `TASK_TYPE_WORKER_COMMANDS` enum, `WorkerCommandsTask` in
`OutboundTaskInfo` with `repeated WorkerCommand`.
- **Task definition**: `worker_commands_task.go` — implements outbound
`Task` and `HasDestination` interfaces.
- **Task creation** (`workflow_task_completed_handler.go`,
`task_generator.go`): When `RequestCancelActivityTask` is processed for
a started activity whose worker has a control queue, collects a
`CancelActivityCommand` with the activity's task token. Commands are
batched by destination control queue and flushed as one
`WorkerCommandsTask` per queue at the end of WFT processing.
- **Serialization**: `task_serializers.go` for persistence
round-tripping.
Dispatch is a no-op here — handled in #9233. Gated by dynamic config
`EnableCancelActivityWorkerCommand` (default: off).
## Why?
To support proactive activity cancellation without waiting for
heartbeat. This is the task creation leg of the flow.
1. [#9231] Store `worker_control_task_queue` in `ActivityInfo` at
activity start.
2. **[This PR]** On `RequestCancelActivityTask`, batch commands by
control queue into `WorkerCommandsTask` outbound tasks.
3. [#9233] Dispatch each task as a Nexus `ExecuteCommands` operation to
the worker, with a 3-attempt retry cap.
4. [SDK] Worker receives the cancel command and cancels the running
activity.
Gated by dynamic config `EnableCancelActivityWorkerCommand` (default:
off).
## How did you test it?
**Unit tests** cover task generation, command batching (including
multi-queue batching), task serialization round-tripping, and the
feature-flag-off path.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
As part of RecordActivityTaskStarted flow, store
worker_control_task_queue for an activity in the mutable state
(ActivityInfo).
Main changes:
- executions.proto: Added the new worker_control_task_queue field.
- mutable_state_impl.go: Update mutable state.
- matching/forwarder.go: Propagate worker_control_task_queue when polls
get forwarded. Otherwise, RecordActivityTaskStarted request will not
have it set when invoked from a forwarded poll.
## Why?
To support activity cancellation without activity heartbeat.
Overall flow:
- [This PR] Store worker attributes in ActivityInfo as part of
RecordActivityTaskStarted call.
- [#9232] When user cancels a workflow, create 1 or more tasks. Group
all activities belonging to a worker into the task (for efficiency).
- [#9233] Lookup the Nexus task queue for each worker, and send a Nexus
operation for each transfer task.
- [SDK] Worker will receive this cancel task and cancel the running
activities.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
Add `CONTINUE_AS_NEW_VERSIONING_BEHAVIOR_USE_RAMPING_VERSION` as a
`ContinueAsNewInitialVersioningBehavior` in `InheritedAutoUpgradeInfo`
and `VersioningInfo`, which if set, forces the first task of an
AutoUpgrade workflow to go to the Ramping Version of its task queue
instead of basing the Current/Ramping version selection on ramp
percentage and workflow id. If there is no Ramping Version at the time
of workflow dispatch, the workflow will use the Current Version.
This initial behavior comes from the continue-as-new command, and when
history sends workflow tasks to matching, is converted into an
internally defined `UseRampingVersionInitialTask bool` such that _only_
the first workflow task of an AutoUpgrade workflow will change its
Target Version selection. Retries of this workflow will start with the
same behavior, but child workflows and future continue-as-new workflows
initiated by this workflow will not inherit the initial behavior.
## Why?
To enable more fine grained control of upgrade-on-continue-as-new
upgrade stages before doing percentage based ramp. To use this, users
should set their promotion candidate version to the ramping version with
a ramp percentage of zero, and manually signal a certain cohort of
workflows to upgrade-on-continue-as-new with this option.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes workflow-task routing for AutoUpgrade executions by adding a
new directive flag that can bypass ramp-percentage hashing for the
initial task. Moderate risk because it affects matching/history version
selection and ContinueAsNew/retry inheritance semantics.
>
> **Overview**
> Adds a new `use_ramping_version` field to `TaskVersionDirective` and
threads it through history → matching so an AutoUpgrade workflow can
*force its initial workflow task* to route to the task queue’s ramping
deployment version (falling back to current when no ramping version
exists), bypassing the normal ramp-percentage/workflow-id hash.
>
> History now persists and evaluates the ContinueAsNew-requested initial
behavior via `MutableState.GetShouldUseRampingVersion()`, ensures it
applies only to the first workflow task, propagates it across retries,
and explicitly does **not** propagate it to child workflows or
subsequent ContinueAsNew hops. Updates tests (unit +
`versioning_3_test`) and bumps `go.temporal.io/api` to pick up the new
proto/enum support.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
923602f0c2. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
- Add compute config summary for latest and all other version so it is
exposed in `ListWorkerDeployments` and `DescribeWorkerDeployment` APIs.
## Why?
Mainly for UI to show compute provider types.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None
## What changed?
Add a new replication task type `DeleteExecutionReplicationTask` that
replicates workflow deletion from the active cluster to passive/standby
clusters. Gated by feature flag
`history.enableDeleteWorkflowExecutionReplication` (default: false).
#### **Key changes across the replication pipeline:**
1. Proto enums: `TASK_TYPE_REPLICATION_DELETE_EXECUTION` (34),
`REPLICATION_TASK_TYPE_DELETE_EXECUTION_TASK` (13)
2. Replication task is associated with a new stage in
`ShardContext.DeleteWorkflowExecution`, bundled with delete visibility
task.
3. ~Engine interface: added `ForceDeleteWorkflowExecution` so the task
can invoke the `ForceDeleteWorkflowExecution`.~
## Why?
Today, when a user delete workflow execution in source cluster, this
operation will not replicate to the standby/target clusters. When a
namespace failover to a target cluster, those deleted workflow may
resurrected.
<details>
<summary>Race condition analysis</summary>
**Before this change:**
1. **Cross-cluster resurrection:** Active deletes workflow → standby
untouched → failover → workflow reappears.
2. **Termination event silently dropped:** Deleting a running workflow
terminates it first, generating a `HistoryReplicationTask`. But the
async `CloseExecutionTask` may delete mutable state before the stream
sender converts that task.
The converter calls `getBranchToken()` → `NotFound` → task silently
dropped. The standby never sees the termination or the deletion.
**After this change:**
Race 1 is fixed — `DeleteExecutionReplicationTask` explicitly tells the
standby to delete.
Race 2 is mitigated — even if the termination event's replication task
is dropped, the delete replication task ensures the standby cleans up.
- If the workflow is still running (termination not yet replicated), the
`DeleteExecutionTask` reschedules itself until the workflow closes.
- If the termination event arrives later, the workflow closes normally,
then the delete proceeds.
- If the workflow is already deleted (e.g., by retention), the task is a
no-op (`NotFound` treated as success).
</details>
<details>
<summary>Deletion paths</summary>
| Path | Replication task? |
|------|-------------------|
| User deletes workflow (active, running or closed) | Yes |
| User deletes on passive (DC forwarding ON) | Forwarded to active → yes
|
| User deletes on passive (no forwarding) | No — `ActiveInCluster` check
skips |
| Retention expiry (with or without archival) | No — stage pre-marked as
processed |
| Admin ForceDelete (tdbg) | No — bypasses `DeleteWorkflowExecution` |
</details>
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
Before change:
<img width="1507" height="163" alt="Screenshot 2026-03-26 at 11 59
51 PM"
src="https://github.com/user-attachments/assets/118cc50e-b69d-468a-9e45-5f49e4e4b9d1"
/>
After change:
<img width="1507" height="135" alt="Screenshot 2026-03-27 at 12 00
07 AM"
src="https://github.com/user-attachments/assets/8ccb7a11-2cb4-48b5-af89-b4a60ddb6333"
/>
## Potential risks
n/a
## What changed?
Adds `buf format` as a Makefile target; and integrates it into `make
fmt`.
All `.proto` changes are from running `make fmt`.
## Why?
Consistent protobuf file style.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Ensure the TQs receive and apply the right version data after revive.
- Made delete propagation to always happen serial to other propagations.
It ensures all other propagations are cancelled before starting delete
propagation.
- Deprecate the `deleted` flag in version data and the GC logic around
it. Now we use the good old forgetVersion path which immediately removes
the version data from TQ.
- Ensure version state is reset after revive, in case the recreation
happened before workflow close.
- Also, now workflows CaN based on SDK suggestion if no pending Signal
or Update is present.
## Why?
The version could stuck at deleted state from TQ POV if revived before
the (now deprecated) GC logic cleans it up.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
None
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## What changed?
Add drained field to InternalTaskQueueStatus.
## Why?
This is useful to tell if a queue is empty. Will be used for automatic
partition scaling.
## How did you test it?
- [ ] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed and why?
Ephemeral data was meant to be separate per task queue type (instead of
for all types together like task queue user data), but the
implementation didn't match that exactly: activity/other queues did
propagate ephemeral data from the workflow queue. This fixes that to not
pass ephemeral data along that edge.
Also reduce the `returning user data` log to debug level.
## How did you test it?
- [ ] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Introduce proto types for Visibility:
- `ChasmExecutionInfo`
- `ListChasmExecutionsRequest`
- `ListChasmExecutionsResponse`
Changed VisibilityManager API to use these protos.
## Why?
Make VisibilityManager API more accessible for potential external usage.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
- Accept archetypeID in admin API requests
## Why?
- With this we no longer need to register chasm components to worker
service, which currently performs an archetypeID to name conversion
before calling admin apis.
- For backward compatibility, we can only stop registering chasm
components to worker service starting from next cloud release.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Nexus callbacks are attached to running workflows after migration from
a V1 workflow-backed schedule. This is done through a new side-effect
task.
## Why?
- Drives workflow completion events.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## What changed?
When a workflow is reset, `ApplyWorkflowExecutionStartedEvent`
re-registers the start-event callbacks using the reset operation's
request ID. `HandleNexusCompletion` cannot find a matching
`BufferedStart` and discards the completion. The `resetRequestID` param
is removed from `WorkflowResetter.ResetWorkflow` and the original
request ID is used. `findStartRequestID` reads the original request ID
back from `WorkflowExecutionInfo.RequestIds` by finding the
`EVENT_TYPE_WORKFLOW_EXECUTION_STARTED` entry.
## Why?
CHASM scheduler relies on callback `request_id` to match WF completions
to originating `BufferedStart` entries. When it cannot be found the
scheduler is permanently stuck with the workflow marked as still
running.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
-
`TestScheduledWorkflowDoubleReset_SchedulerSeesCompletion_{HSM,CHASM}Callbacks`:
create a schedule, trigger immediately, reset the workflow twice, signal
the completion to complete, poll `ListSchedules` until scheduler shows
`COMPLETED`.
## Potential risks
CHASM scheduler has not been enabled in production yet, the blast radius
should be minimal.
## What changed?
Emit task dispatch latency metric in matching_engine with the following
improvements:
- latency includes history calls
- latency is not reset in case of sync-match forwards
- metric is not lost for backlogged task in new matcher
- origin partition is preserved during forward and used at the partition
tag in the metric (keeping existing behavior)
- task and poll forwarding do not cause duplicate emits, keeping
existing behavior
Also, fixed the following unrelated bugs that surfaced while testing the
metric:
- Query priority is not lost when forwarded.
## Why?
Fixes bugs.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
None.
## What changed?
- Consists of the change to prevent any Pinned workflows, that may have
forgotten to have the initial CAN Behaviour as AU, from CAN'ing
infinitely.
- Also allows trampolining of a Pinned workflow onto the Unversioned if
un-versioned is the current version of the worker deployment at that
point in time. Note, the effective behaviour of the workflow would later
then be unversioned.
## Why?
- Worker-Versioning correctness.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- Pre-existing workflows (started before this fix) have nil
TargetVersionOnStart. On the first WFT after deployment, "" !=
"build-v2" → spurious targetDeploymentVersionChanged=true. This is a
one-time false-positive regression for those Pinned workflows.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches generated protobuf API surface used by internal services;
while the change is additive, mismatched proto versions across services
could cause integration/compatibility issues.
>
> **Overview**
> Adds `declined_target_version_upgrade` to
`StartWorkflowExecutionRequest` (HistoryService API) so
continue-as-new/retry chains can carry forward the SDK-declined target
deployment version and avoid pinned-workflow trampolining loops.
>
> Regenerates protobuf Go bindings, updating import/type references
across `request_response.pb.go`, and adds missing
`Marshal`/`Unmarshal`/`Size`/`Equal` helpers for the persistence
`LastNotifiedTargetVersion` message.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
0af06d0141. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed
Adds an admin RPC (`MigrateSchedule`) that migrates workflow-backed (V1)
schedulers to CHASM (V2) schedulers.
The flow:
1. Admin API signals the V1 scheduler workflow with `migrate-to-chasm`
2. V1 workflow runs a local activity that snapshots state and calls
`CreateFromMigrationState` on the CHASM scheduler service
3. On success, the V1 workflow closes. On failure, a `PendingMigration`
flag persists so the next run loop iteration retries automatically
Key details:
- `CreateSchedulerFromMigration` initializes the full CHASM scheduler
tree (generator, invoker, backfillers, visibility) from V1 state
- Running/completed workflows are converted to buffered starts; ongoing
backfills are preserved
- Migration is idempotent -- if the CHASM schedule already exists, it's
treated as success
- Metrics: `schedule_migration_started`, `schedule_migration_completed`,
`schedule_migration_failed` (with direction tag)
- Local activity uses 10m schedule-to-close timeout; retries happen at
the workflow level via the persistent flag
## Why
Needed for migrating existing V1 schedulers to CHASM without downtime or
data loss.
## Follow-up items
- V2 to V1 migration (rollback path)
- Sentinel key handling when `EnableCHASMSchedulerCreation` was
previously enabled
- Attach completion callbacks to running workflows after migration
## What changed?
Introduce api category into history APIs
## Why?
There are some APIs we want to exclude from health check. Using API
category is a more protective way to manage this group.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Build `WorkerListInfo` in matching handler and pass through to
frontend
- Update `go.temporal.io/api` to include merged WorkerListInfo proto
## Why?
`WorkerListInfo` contains only static worker attributes for efficient
listing. Building it in matching (closer to the data source) enables
eventually dropping the deprecated `WorkersInfo` field from the internal
response.
## How did you test it?
- [x] built
- [x] added new unit test for field coverage
- [x] covered by existing functional tests
## Potential risks
None - backward compatible. Deprecated `WorkersInfo` field continues to
be populated.
## What changed?
The counts of some top keys are persisted in task queue metadata so
they're preserved on queue movement or reloads.
## Why?
More accurate fairness
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Re-does #9138 which was incidentally merged.
Include transient and speculative WFT events in
`GetWorkflowExecutionHistoryReponse` response, unless UI or CLI made
request.
* Adds `transient_or_speculative_events` back to
`GetMutableStateResponse`
* Reserve `transient_workflow_task` in `HisotryCOntinuation` token
* Add validation helpers
* Add query-compare-query for transient events at request start and end
Re-implements #7732
## Why?
Fix "premature end of stream" errors when workers request history after
cache eviction w/ transient/speculative workflow tasks present. This
adds transient & speculative WFT events in `GetWorkflowExecutionHistory`
(already in `PollWorkflowTask`). Worker cache eviction w/ speculative
workflow tasks causes the expected and actual event counts to be
different. #7732 passed transient events through continuation tokens,
which could become stale during pagination. This PR implements mutable
state querying at both start and end of pagination and compares
transient event IDs to detect if WFT state changed during pagination and
return a retryable error.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Same risks from #7732
## Overview
This feature introduces a special internal Nexus endpoint called
`__temporal_system` that enables adding functionality to workflows
**without requiring new workflow commands and events**. Operations on
this endpoint are routed internally within Temporal's history service
rather than via external HTTP calls.
## Key Components
### 1. System Endpoint Infrastructure
- **Endpoint Name**: `__temporal_system` (constant in
`common/nexus/constants.go:6`)
- **Callback URL**: `temporal://system` for internal routing
- **New History Service RPCs**
(`proto/internal/temporal/server/api/historyservice/v1/service.proto:433-437`):
- `StartNexusOperation` - Starts operations on the system endpoint
- `CancelNexusOperation` - Cancels operations on the system endpoint
### 2. Operation Processor Framework
A new processor pattern (`chasm/nexus_operation_processor.go`) that
allows CHASM libraries to:
- **Validate and transform input**: Processors can validate operation
inputs and set default values
- **Determine routing**: Each processor returns a routing key that
determines which history shard handles the operation
- **Re-serialize input**: Mutated inputs can be re-serialized to persist
default values
**Routing strategies**:
- `NexusOperationRoutingKeyExecution` - Routes to the shard owning a
specific workflow execution
- `NexusOperationRoutingKeyRandom` - Routes to a random shard
### 3. CHASM Library Integration
CHASM libraries can now provide (`chasm/library.go:16-19`):
- **`NexusServices()`**: Regular Nexus service handlers (implement the
actual operation logic)
- **`NexusServiceProcessors()`**: Input processors for validation and
routing
Example from test library (`chasm/lib/tests/nexus_service.go`):
```go
// Service handler - implements the actual operation
TestOperation = nexus.NewSyncOperation("TestOperation",
func(ctx context.Context, input string, options nexus.StartOperationOptions) (string, error) {
return "Hello, " + input, nil
})
// Processor - validates input and determines routing
func (o testOperationProcessor) ProcessInput(ctx chasm.NexusOperationProcessorContext, input string)
(*chasm.NexusOperationProcessorResult, error) {
return &chasm.NexusOperationProcessorResult{
RoutingKey: chasm.NexusOperationRoutingKeyExecution{
NamespaceID: ctx.Namespace.ID().String(),
BusinessID: input, // Route based on input
},
}, nil
}
```
### 4. Execution Flow
When a workflow schedules a Nexus operation on `__temporal_system`
(`components/nexusoperations/executors.go:233-238`):
1. **Input Processing**: The processor validates input and determines
routing
2. **Internal RPC**: Instead of HTTP, calls
`HistoryClient.StartNexusOperation` with the target shard ID
3. **Handler Execution**: The history service invokes the registered
Nexus handler (`service/history/handler.go:2707-2768`)
4. **Result Handling**: Supports both sync (immediate result) and async
(operation token) responses
5. **Workflow Completion**: Results flow back through the same
completion path as external Nexus operations
### 5. Benefits
✅ **No schema changes**: Add functionality without new commands/events
in workflow history
✅ **Consistent API**: Uses existing Nexus operation semantics
(sync/async, callbacks, links)
✅ **Proper routing**: Operations are intelligently routed to the correct
shard
✅ **Input validation**: Type-safe input validation and default value
handling
✅ **Future extensibility**: Foundation for direct client invocation (not
yet implemented)
### 6. Technical Details
- **Error handling** (`components/nexusoperations/executors.go:444`):
Non-retryable service errors are properly handled and fail operations
immediately
- **Metrics**: System operations are tracked separately with
`DestinationTag` set to the endpoint name
- **Link conversion**: Helper functions convert between Nexus SDK links
and protobuf links (`common/nexus/util.go:17-46`)
- **Operation token handling**: Moved link converters to common package
for reuse (`common/nexus/link_converter.go`)
### 7. Current Limitations
- Only accessible from workflows (via `ScheduleNexusOperation` command)
- Direct client invocation not yet implemented
- Headers not supported for system endpoint operations
## Test Coverage
New test (`tests/nexus_workflow_test.go:2763-2843`) demonstrates:
- Scheduling operation on `__temporal_system` endpoint
- Synchronous operation completion
- Result propagation back to workflow
## Architecture
This architecture provides a clean, extensible way to add internal
functionality while maintaining compatibility with Temporal's existing
workflow execution model. The system endpoint acts as a bridge between
workflows and internal CHASM components, enabling:
- **Extensibility**: New operations can be added by implementing CHASM
libraries
- **Type safety**: Input validation happens before operations are routed
- **Scalability**: Intelligent routing ensures operations land on the
correct shard
- **Consistency**: Same execution model as external Nexus operations
## What changed?
Include transient and speculative WFT events in
`GetWorkflowExecutionHistoryReponse` response, unless UI or CLI made
request.
* Adds `transient_or_speculative_events` back to
`GetMutableStateResponse`
* Reserve `transient_workflow_task` in `HisotryCOntinuation` token
* Add validation helpers
* Add query-compare-query for transient events at request start and end
Re-implements #7732
## Why?
Fix "premature end of stream" errors when workers request history after
cache eviction w/ transient/speculative workflow tasks present. This
adds transient & speculative WFT events in `GetWorkflowExecutionHistory`
(already in `PollWorkflowTask`). Worker cache eviction w/ speculative
workflow tasks causes the expected and actual event counts to be
different. #7732 passed transient events through continuation tokens,
which could become stale during pagination. This PR implements mutable
state querying at both start and end of pagination and compares
transient event IDs to detect if WFT state changed during pagination and
return a retryable error.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Same risks from #7732
## What changed?
Eagerly remove worker from pollerHistory during graceful shutdown so
DescribeTaskQueue doesn't show stale pollers.
**Note:** This PR subtly changes how pollerHistory is updated `before`
and `after` a poll request.
Before:
- UpdatePollerInfo called at poll START
- defer UpdatePollerInfo called at poll END (always, regardless of how
poll ended)
After:
- UpdatePollerInfo called at poll START
- UpdatePollerInfo called at poll END only if ctx.Err() !=
context.Canceled
Skipped on cancellation (shutdown/disconnect) to avoid re-adding entry
after RemovePoller
The defer was originally added (PR #2811) to keep timestamps fresh, but
HasPollerAfter() first checks currentPolls > 0 for active polls, so the
timestamp is a secondary check
**Main changes**
API
- request_response.proto: Added worker_identity field to
CancelOutstandingWorkerPollsRequest
Implementation
- matching_engine.go: As part of CancelOutstandingWorkerPolls, also
invoke RemovePoller on the task queue partition manager.
- task_queue_partition_manager.go: Forward the removal to default and
versioned task queues.
## Why?
Previously, pollers lingered in pollerHistory until TTL expired (~5
min), showing workers that already shut down.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Controlled by dynamic config: EnableCancelWorkerPollsOnShutdown
## What changed?
Replace the limited Nexus HandlerError proto with full Temporal Failure
objects for transmitting errors across Nexus operation boundaries. This
preserves failure chains, stack traces, and typed failure info
(ApplicationFailure, CanceledFailure, etc.) end-to-end.
Care was taken to ensure Temporal->Temporal communication works as
expected as well as support for non-Temporal Nexus implementations (to
maintain support for external endpoints).
This ended up being more refactoring that I had originally anticipated
but I think the unification of error serialization across the codebase
and the more straightforward conversion logic was worth it.
The code was validated across variations of new and old caller and
handler workers and servers using a
[harness](https://github.com/bergundy/nexus-error-compat-tests) that was
built specifically to validate the behavior. The harness was also run
with an SDK setup that encoded failure attributes to confirm failures
are properly transmitted over the different boundaries.
Proto changes:
- Add `failure` field to DispatchNexusTaskResponse carrying a Failure
with NexusHandlerFailureInfo, deprecate `handler_error`
- Add `capabilities` field to nexus Request for feature negotiation
Failure conversion (`common/nexus/failure.go`):
- Rewrite TemporalFailureToNexusFailure and
NexusFailureToTemporalFailure with support for recursive cause chains
and stack traces
- Add special-case handling for NexusHandlerFailureInfo (serialized as
nexus.HandlerError type) and OperationError (mapped to
CanceledFailure/ApplicationFailure based on state)
Nexus SDK layer (`common/nexus/nexusrpc/`):
- Inline the failure converter from the upstream SDK with a custom
FailureConverter interface that round-trips Temporal failure metadata
through Nexus Failure objects
- Add a completion client to abstract away HTTP and provide consistent
failure conversion across all APIs
- Support `unwrap-error` metadata marker so Temporal-to-Temporal calls
can unwrap the OperationError envelope and recover the original cause
Frontend and backend handlers:
- Route responses through new failure path when caller sends
`temporal-nexus-failure-support` header
- Handle both new `failure` and deprecated `handler_error` response
types in nexus_handler.go and nexus_http_handler.go
- Update completion handling in history handler and mutable state to
produce Temporal Failures instead of Nexus HandlerErrors
- Update matching engine to forward the new failure field
Dependencies:
- Bump `nexus-rpc/sdk-go` to pre-release with StackTrace, Cause, and
OriginalFailure fields on Failure and OperationError
## Why?
Part of getting Nexus to GA in all SDKs, this change fixes a couple of
notable issues with errors in Nexus+Temporal applications:
- Nexus SDK errors did not have a way to set an error message, which
made them difficult to use in various languages
- The protocol used a custom failure format that is diverges from proxy
expectations and may result in failures not being encrypted
## Potential risks
- Metric label values for callback outcomes now use Nexus handler error
values instead of HTTP response codes. Alerts that check these outcomes
will need adjustment.
## What changed?
Set TargetVersionChanged instead of SuggestCaN when TargetVersionChanged
https://github.com/temporalio/api/pull/709
## Why?
Setting SuggestContinueAsNew=true for Pinned workflows whenever their is
a new Target Version available for that workflow causes Pinned workflows
to hit that condition much more frequently than they expect. Users who
are currently doing: if workflow_info.suggestContinueAsNew{ do
continue-as-new } in their Pinned workflow code would need to change
that code to protect themselves from running into an infinite-CaN-loop,
because the default CaN behavior for a Pinned workflow is to stay
Pinned.
We should not force users to protect themselves from such a situation.
Because upgrading on continue-as-new is opt-in, receiving the suggestion
to continue-as-new-onto-new-target-version should be opt-in as well. If
people are forced to check the new suggest-continue-as-new-reasons field
to "opt out," that is unsafe, because inevitably some people will forget
to do so or misunderstand, and then get hit by this unexpected footgun.
Much safer and still ergonomical to let upgrade-on-can be opt-in on both
fronts, as proposed here. With this change, the people who are currently
doing if workflow_info.suggestContinueAsNew{ do continue-as-new } won't
see any change in semantics, regardless of their versioning behavior.
People who consciously know that they want to do upgrade-on-can /
Trampolining will have to change their CaN options anyway, so it's easy
enough to teach them to pay attention to this new
TargetWorkerDeploymentVersionChanged flag.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
None
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches workflow task started event generation/persistence and
versioning-related signaling, which can affect worker behavior and
history compatibility; changes are gated by dynamic config and covered
by tests.
>
> **Overview**
> Stops using `SuggestContinueAsNew` (and its reason tags) to signal
pinned workflows that a newer target worker deployment version exists,
and instead introduces an explicit
`TargetWorkerDeploymentVersionChanged` boolean on `WorkflowTaskStarted`
events and persisted `WorkflowExecutionInfo`.
>
> Adds namespace dynamic config `EnableSendTargetVersionChanged`
(default on) and a new metric `workflow_target_version_changed_count`
emitted when this flag is set; updates the workflow task state machine,
mutable state plumbing/mocks, proto/pb persistence, and functional tests
accordingly. Also bumps `go.temporal.io/api` to pick up the new event
attribute.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
3886491826. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Adds infrastructure to cancel all outstanding polls for a worker
instance during shutdown.
Key Changes:
1. request_response.proto / service.proto
- New CancelOutstandingWorkerPolls RPC that cancels polls by
worker_instance_key instead of individual poller_id
2. matching_engine.go:
- Added workerInstancePollers map to track pollers by worker instance
key
- pollTask() now registers pollers in both outstandingPollers (by
pollerID) and workerInstancePollers (by worker key)
- CancelOutstandingWorkerPolls() cancels all pollers for a worker
instance and returns count
## Why?
To support eager cancellation of outstanding polls when worker call
ShutdownWorker.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None. No changes to existing functionality.
## What changed?
Added proto definitions and conversion functions for V1 to V2 schedule
migration.
## Why?
This is the foundation for the V1→V2 schedule migration feature. The
`MigrateScheduleRequest` proto captures all scheduler state needed to
recreate a schedule in the CHASM architecture.
## What changed?
This PR extends the raw history optimization to pass raw history bytes
from History Service → Matching Service → Frontend without
deserialization in Matching Service.
**Key changes:**
1. History Service:
- When `SendRawHistoryBetweenInternalServices` is enabled, sets
`RawHistoryBytes` (field 21) with raw proto-encoded history batches
2. Matching Service:
- Passes raw history bytes through to frontend via
`PollWorkflowTaskQueueResponseWithRawHistory`
- Uses wire-compatible proto messages so gRPC auto-deserializes
`[][]byte` → `History` on the client side
3. Frontend:
- Receives raw history in `RawHistory` field (auto-deserialized by gRPC)
- Processes search attributes for raw history since it bypasses history
service's normal processing
4. Proto definitions:
- Added `raw_history_bytes` (field 21) to
`RecordWorkflowTaskStartedResponse`
- Added `PollWorkflowTaskQueueResponseWithRawHistory` message with
wire-compatible layout
- Added `raw_history` (field 22) to `PollWorkflowTaskQueueResponse`
## Why?
When `history.sendRawHistoryBetweenInternalServices` is enabled, the
previous implementation only avoided deserialization from persistence →
History Service. However, Matching Service was still deserializing
history events (via gRPC auto-deserialization) and re-serializing them
when forwarding to Frontend.
This change eliminates that unnecessary serialization/deserialization
cycle in Matching Service by:
1. Having History Service send raw bytes directly
2. Having Matching Service forward these raw bytes without parsing
3. Having Frontend receive the bytes which gRPC auto-deserializes
This reduces CPU usage in Matching Service for workflows with large
histories.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s) (`tests/workflow_task_test.go`)
## Potential risks
SendRawHistoryBetweenInternalServices must be disabled when rolling back
from this version to an older version.
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## What changed?
Implement detached component as a Field option and Registrable Component
option.
Add detached boolean value to ComponentAttributes persistence proto
definition.
## Why?
Allow detached components to continue updates and task execution even if
parent node lifecycle is closed.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- WISOTT
## Why?
- Operational readiness. There could be a world where we want to "reset"
a user's workflow state given that these workflows CAN so frequently.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- None
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Introduces optional state override when force-continuing-as-new.
>
> - New proto messages `ForceCANDeploymentSignalArgs` and
`ForceCANVersionSignalArgs` with `override_state` fields; generated
helper/pb code updated
> - Deployment and Version workflows now receive `forceCAN` signals with
args and, if provided, apply `override_state` before continue-as-new
> - Added tests verifying override is honored (e.g., manager identity
and metadata) after continue-as-new
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
1fd847da1c. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
- Use plain go struct inside migration workflow activity input output
- Customize the json encoding/decoding for migration execution info and
make it backward & forward compatible.
- This will temporary break cloud and the fix is in
https://github.com/temporalio/temporal/pull/9097
- This PR needs to be part of oss v1.30 release.
## Why?
- Backward compatibility and less confusion over how the encoding will
be done.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
This reverts commit 58449b9d80.
## What changed?
This reverts Nexus schedule to start and start to close timeouts (with
fixes) (#9033)
## Why?
Causes failures in nightly pipeline
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
When buffered events are applied, like when applying signals after
workflow task failures occur, preserve the workflow task attempt so the
`TemporalReportedProblems` search attribute is still properly added.
## Why?
Without this change the `TemporalReportedProblems` search attribute will
never be added if a workflow is consistently queried.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
Minimal, this adds a new int to the workflow task execution, but it's
only used in one place.
## What changed?
Reapplied #9010.
The original PR that introduced these timeouts did not populate the
operation-timeout header or set the call context timeout correctly. This
PR fixes the logic.
## What changed?
- Unifies CHASM scheduler's Scheduler.Info.{RecentActions,
RunningWorkflows} into BufferedStarts.
- Computes the RecentActions and RunningWorkflows views on-demand from
BufferedStarts.
- Removes the old requestIDtoWorkflowID map.
## Why?
- Simplifies the data structure; much easier to reason about
concurrently, as related records are no longer spread between distinct
data structures.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Add "ephemeral data" propagated alongside user data.
- Include information about which priority levels have significant
backlog in ephemeral data (currently only propagated down the tree, not
up)
- When sticky partitions see that normal partitions have significant
backlog, set up poll forwarders to forward to those partitions.
- The poll forwarders use min priority to ensure they only get
higher-priority tasks than available local tasks.
## Why?
Without this functionality, sticky queues interfere with priority
dispatch by keeping most of the pollers working on sticky tasks and
potentially starving high-priority tasks that appear on the normal
queue. With the default values of the new settings, sticky pollers
should notice high-priority normal tasks within 10 seconds.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- Clients have to accept that polls on sticky may return normal tasks. I
believe all SDKs are fine with this.
- Increased overhead of user data + ephemeral data.
## Overview
This commit implements two new granular timeout types for Nexus
operations, allowing callers to have fine-grained control over different
phases of operation execution:
- **Schedule-to-Start Timeout**: Maximum time to wait for an operation
to be started (or completed if synchronous) by the handler
- **Start-to-Close Timeout**: Maximum time to wait for an asynchronous
operation to complete after it has been started
These timeouts complement the existing **Schedule-to-Close Timeout** to
provide better control and diagnostics for Nexus operation execution.
See the corresponding API PR:
https://github.com/temporalio/api/pull/695.
## What changed?
Populated standalone activity Run ID and scheduled time when returning
PollActivityTaskQueueResponse. Updated API deps. Added test specific to
validating PollActivityTaskQueueResponse.
## Why?
Standalone activity executions have Run IDs that should be returned to
the task poller. The response scheduled time should also be populated as
the Temporal SDKs depend on it.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
## What changed?
Add batch workflow refresh tasks Admin API.
Add tdbg commands to invoke `StartAdminBatchOperation`.
## Details
AdminBatchOperations are executed in the Batched System Worker, similar
to existing Frontend BatchOperations. The worker activity will fetch
pages of workflow executions to perform operations on, and periodically
heartbeat its results. For RefreshWorkflowTasks, each individual Refresh
call will go through the admin handler.
Refreshing Workflow Tasks regenerates all pending tasks of an execution
given its mutable state.
## CLI usage
```
tdbg workflow batch-refresh-tasks \
--namespace <ns> \
--query "WorkflowType='MyType'" \
--reason "fixing stuck workflows" \
[--job-id <optional-job-id>] \
[--archetype <optional-archetype>]
```
## Why?
Unblock new Matcher migration and allow for general use case batch Admin
calls. Currently, only supports `BatchOperationRefreshWorkflowTasks`.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## What changed?
Add a new dynamic config for Auto Enabling fairness and priority if we
see the relevant tasks coming in.
## Why?
Seamlessly start to transition users who start using the fields over to
the new code path.
## How did you test it?
- [X] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
Due to storing this in the userdata we are using that interface a bit
more, we also need to change the initialization such that we start it
before being able to substantiate the defaultQ, this change in
initialization might have unintended side effects that I'm not currently
seeing.
## What changed?
Keep the total number and the size of the external payloads per the
workflow execution
## Why?
We are working on building the support for external payloads in SDK,
which are stored outside of Temporal. We'd like to be able to show the
total size and the number of external payloads in the given workflow
execution.
## How did you test it?
- [ ] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
N/A
