## What changed?
Add drained field to InternalTaskQueueStatus.
## Why?
This is useful to tell if a queue is empty. Will be used for automatic
partition scaling.
## How did you test it?
- [ ] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed and why?
Ephemeral data was meant to be separate per task queue type (instead of
for all types together like task queue user data), but the
implementation didn't match that exactly: activity/other queues did
propagate ephemeral data from the workflow queue. This fixes that to not
pass ephemeral data along that edge.
Also reduce the `returning user data` log to debug level.
## How did you test it?
- [ ] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Introduce proto types for Visibility:
- `ChasmExecutionInfo`
- `ListChasmExecutionsRequest`
- `ListChasmExecutionsResponse`
Changed VisibilityManager API to use these protos.
## Why?
Make VisibilityManager API more accessible for potential external usage.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
- Accept archetypeID in admin API requests
## Why?
- With this we no longer need to register chasm components to worker
service, which currently performs an archetypeID to name conversion
before calling admin apis.
- For backward compatibility, we can only stop registering chasm
components to worker service starting from next cloud release.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Nexus callbacks are attached to running workflows after migration from
a V1 workflow-backed schedule. This is done through a new side-effect
task.
## Why?
- Drives workflow completion events.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## What changed?
When a workflow is reset, `ApplyWorkflowExecutionStartedEvent`
re-registers the start-event callbacks using the reset operation's
request ID. `HandleNexusCompletion` cannot find a matching
`BufferedStart` and discards the completion. The `resetRequestID` param
is removed from `WorkflowResetter.ResetWorkflow` and the original
request ID is used. `findStartRequestID` reads the original request ID
back from `WorkflowExecutionInfo.RequestIds` by finding the
`EVENT_TYPE_WORKFLOW_EXECUTION_STARTED` entry.
## Why?
CHASM scheduler relies on callback `request_id` to match WF completions
to originating `BufferedStart` entries. When it cannot be found the
scheduler is permanently stuck with the workflow marked as still
running.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
-
`TestScheduledWorkflowDoubleReset_SchedulerSeesCompletion_{HSM,CHASM}Callbacks`:
create a schedule, trigger immediately, reset the workflow twice, signal
the completion to complete, poll `ListSchedules` until scheduler shows
`COMPLETED`.
## Potential risks
CHASM scheduler has not been enabled in production yet, the blast radius
should be minimal.
## What changed?
Emit task dispatch latency metric in matching_engine with the following
improvements:
- latency includes history calls
- latency is not reset in case of sync-match forwards
- metric is not lost for backlogged task in new matcher
- origin partition is preserved during forward and used at the partition
tag in the metric (keeping existing behavior)
- task and poll forwarding do not cause duplicate emits, keeping
existing behavior
Also, fixed the following unrelated bugs that surfaced while testing the
metric:
- Query priority is not lost when forwarded.
## Why?
Fixes bugs.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
None.
## What changed?
- Consists of the change to prevent any Pinned workflows, that may have
forgotten to have the initial CAN Behaviour as AU, from CAN'ing
infinitely.
- Also allows trampolining of a Pinned workflow onto the Unversioned if
un-versioned is the current version of the worker deployment at that
point in time. Note, the effective behaviour of the workflow would later
then be unversioned.
## Why?
- Worker-Versioning correctness.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- Pre-existing workflows (started before this fix) have nil
TargetVersionOnStart. On the first WFT after deployment, "" !=
"build-v2" → spurious targetDeploymentVersionChanged=true. This is a
one-time false-positive regression for those Pinned workflows.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches generated protobuf API surface used by internal services;
while the change is additive, mismatched proto versions across services
could cause integration/compatibility issues.
>
> **Overview**
> Adds `declined_target_version_upgrade` to
`StartWorkflowExecutionRequest` (HistoryService API) so
continue-as-new/retry chains can carry forward the SDK-declined target
deployment version and avoid pinned-workflow trampolining loops.
>
> Regenerates protobuf Go bindings, updating import/type references
across `request_response.pb.go`, and adds missing
`Marshal`/`Unmarshal`/`Size`/`Equal` helpers for the persistence
`LastNotifiedTargetVersion` message.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
0af06d0141. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed
Adds an admin RPC (`MigrateSchedule`) that migrates workflow-backed (V1)
schedulers to CHASM (V2) schedulers.
The flow:
1. Admin API signals the V1 scheduler workflow with `migrate-to-chasm`
2. V1 workflow runs a local activity that snapshots state and calls
`CreateFromMigrationState` on the CHASM scheduler service
3. On success, the V1 workflow closes. On failure, a `PendingMigration`
flag persists so the next run loop iteration retries automatically
Key details:
- `CreateSchedulerFromMigration` initializes the full CHASM scheduler
tree (generator, invoker, backfillers, visibility) from V1 state
- Running/completed workflows are converted to buffered starts; ongoing
backfills are preserved
- Migration is idempotent -- if the CHASM schedule already exists, it's
treated as success
- Metrics: `schedule_migration_started`, `schedule_migration_completed`,
`schedule_migration_failed` (with direction tag)
- Local activity uses 10m schedule-to-close timeout; retries happen at
the workflow level via the persistent flag
## Why
Needed for migrating existing V1 schedulers to CHASM without downtime or
data loss.
## Follow-up items
- V2 to V1 migration (rollback path)
- Sentinel key handling when `EnableCHASMSchedulerCreation` was
previously enabled
- Attach completion callbacks to running workflows after migration
## What changed?
Introduce api category into history APIs
## Why?
There are some APIs we want to exclude from health check. Using API
category is a more protective way to manage this group.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Build `WorkerListInfo` in matching handler and pass through to
frontend
- Update `go.temporal.io/api` to include merged WorkerListInfo proto
## Why?
`WorkerListInfo` contains only static worker attributes for efficient
listing. Building it in matching (closer to the data source) enables
eventually dropping the deprecated `WorkersInfo` field from the internal
response.
## How did you test it?
- [x] built
- [x] added new unit test for field coverage
- [x] covered by existing functional tests
## Potential risks
None - backward compatible. Deprecated `WorkersInfo` field continues to
be populated.
## What changed?
The counts of some top keys are persisted in task queue metadata so
they're preserved on queue movement or reloads.
## Why?
More accurate fairness
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Re-does #9138 which was incidentally merged.
Include transient and speculative WFT events in
`GetWorkflowExecutionHistoryReponse` response, unless UI or CLI made
request.
* Adds `transient_or_speculative_events` back to
`GetMutableStateResponse`
* Reserve `transient_workflow_task` in `HisotryCOntinuation` token
* Add validation helpers
* Add query-compare-query for transient events at request start and end
Re-implements #7732
## Why?
Fix "premature end of stream" errors when workers request history after
cache eviction w/ transient/speculative workflow tasks present. This
adds transient & speculative WFT events in `GetWorkflowExecutionHistory`
(already in `PollWorkflowTask`). Worker cache eviction w/ speculative
workflow tasks causes the expected and actual event counts to be
different. #7732 passed transient events through continuation tokens,
which could become stale during pagination. This PR implements mutable
state querying at both start and end of pagination and compares
transient event IDs to detect if WFT state changed during pagination and
return a retryable error.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Same risks from #7732
## Overview
This feature introduces a special internal Nexus endpoint called
`__temporal_system` that enables adding functionality to workflows
**without requiring new workflow commands and events**. Operations on
this endpoint are routed internally within Temporal's history service
rather than via external HTTP calls.
## Key Components
### 1. System Endpoint Infrastructure
- **Endpoint Name**: `__temporal_system` (constant in
`common/nexus/constants.go:6`)
- **Callback URL**: `temporal://system` for internal routing
- **New History Service RPCs**
(`proto/internal/temporal/server/api/historyservice/v1/service.proto:433-437`):
- `StartNexusOperation` - Starts operations on the system endpoint
- `CancelNexusOperation` - Cancels operations on the system endpoint
### 2. Operation Processor Framework
A new processor pattern (`chasm/nexus_operation_processor.go`) that
allows CHASM libraries to:
- **Validate and transform input**: Processors can validate operation
inputs and set default values
- **Determine routing**: Each processor returns a routing key that
determines which history shard handles the operation
- **Re-serialize input**: Mutated inputs can be re-serialized to persist
default values
**Routing strategies**:
- `NexusOperationRoutingKeyExecution` - Routes to the shard owning a
specific workflow execution
- `NexusOperationRoutingKeyRandom` - Routes to a random shard
### 3. CHASM Library Integration
CHASM libraries can now provide (`chasm/library.go:16-19`):
- **`NexusServices()`**: Regular Nexus service handlers (implement the
actual operation logic)
- **`NexusServiceProcessors()`**: Input processors for validation and
routing
Example from test library (`chasm/lib/tests/nexus_service.go`):
```go
// Service handler - implements the actual operation
TestOperation = nexus.NewSyncOperation("TestOperation",
func(ctx context.Context, input string, options nexus.StartOperationOptions) (string, error) {
return "Hello, " + input, nil
})
// Processor - validates input and determines routing
func (o testOperationProcessor) ProcessInput(ctx chasm.NexusOperationProcessorContext, input string)
(*chasm.NexusOperationProcessorResult, error) {
return &chasm.NexusOperationProcessorResult{
RoutingKey: chasm.NexusOperationRoutingKeyExecution{
NamespaceID: ctx.Namespace.ID().String(),
BusinessID: input, // Route based on input
},
}, nil
}
```
### 4. Execution Flow
When a workflow schedules a Nexus operation on `__temporal_system`
(`components/nexusoperations/executors.go:233-238`):
1. **Input Processing**: The processor validates input and determines
routing
2. **Internal RPC**: Instead of HTTP, calls
`HistoryClient.StartNexusOperation` with the target shard ID
3. **Handler Execution**: The history service invokes the registered
Nexus handler (`service/history/handler.go:2707-2768`)
4. **Result Handling**: Supports both sync (immediate result) and async
(operation token) responses
5. **Workflow Completion**: Results flow back through the same
completion path as external Nexus operations
### 5. Benefits
✅ **No schema changes**: Add functionality without new commands/events
in workflow history
✅ **Consistent API**: Uses existing Nexus operation semantics
(sync/async, callbacks, links)
✅ **Proper routing**: Operations are intelligently routed to the correct
shard
✅ **Input validation**: Type-safe input validation and default value
handling
✅ **Future extensibility**: Foundation for direct client invocation (not
yet implemented)
### 6. Technical Details
- **Error handling** (`components/nexusoperations/executors.go:444`):
Non-retryable service errors are properly handled and fail operations
immediately
- **Metrics**: System operations are tracked separately with
`DestinationTag` set to the endpoint name
- **Link conversion**: Helper functions convert between Nexus SDK links
and protobuf links (`common/nexus/util.go:17-46`)
- **Operation token handling**: Moved link converters to common package
for reuse (`common/nexus/link_converter.go`)
### 7. Current Limitations
- Only accessible from workflows (via `ScheduleNexusOperation` command)
- Direct client invocation not yet implemented
- Headers not supported for system endpoint operations
## Test Coverage
New test (`tests/nexus_workflow_test.go:2763-2843`) demonstrates:
- Scheduling operation on `__temporal_system` endpoint
- Synchronous operation completion
- Result propagation back to workflow
## Architecture
This architecture provides a clean, extensible way to add internal
functionality while maintaining compatibility with Temporal's existing
workflow execution model. The system endpoint acts as a bridge between
workflows and internal CHASM components, enabling:
- **Extensibility**: New operations can be added by implementing CHASM
libraries
- **Type safety**: Input validation happens before operations are routed
- **Scalability**: Intelligent routing ensures operations land on the
correct shard
- **Consistency**: Same execution model as external Nexus operations
## What changed?
Include transient and speculative WFT events in
`GetWorkflowExecutionHistoryReponse` response, unless UI or CLI made
request.
* Adds `transient_or_speculative_events` back to
`GetMutableStateResponse`
* Reserve `transient_workflow_task` in `HisotryCOntinuation` token
* Add validation helpers
* Add query-compare-query for transient events at request start and end
Re-implements #7732
## Why?
Fix "premature end of stream" errors when workers request history after
cache eviction w/ transient/speculative workflow tasks present. This
adds transient & speculative WFT events in `GetWorkflowExecutionHistory`
(already in `PollWorkflowTask`). Worker cache eviction w/ speculative
workflow tasks causes the expected and actual event counts to be
different. #7732 passed transient events through continuation tokens,
which could become stale during pagination. This PR implements mutable
state querying at both start and end of pagination and compares
transient event IDs to detect if WFT state changed during pagination and
return a retryable error.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Same risks from #7732
## What changed?
Eagerly remove worker from pollerHistory during graceful shutdown so
DescribeTaskQueue doesn't show stale pollers.
**Note:** This PR subtly changes how pollerHistory is updated `before`
and `after` a poll request.
Before:
- UpdatePollerInfo called at poll START
- defer UpdatePollerInfo called at poll END (always, regardless of how
poll ended)
After:
- UpdatePollerInfo called at poll START
- UpdatePollerInfo called at poll END only if ctx.Err() !=
context.Canceled
Skipped on cancellation (shutdown/disconnect) to avoid re-adding entry
after RemovePoller
The defer was originally added (PR #2811) to keep timestamps fresh, but
HasPollerAfter() first checks currentPolls > 0 for active polls, so the
timestamp is a secondary check
**Main changes**
API
- request_response.proto: Added worker_identity field to
CancelOutstandingWorkerPollsRequest
Implementation
- matching_engine.go: As part of CancelOutstandingWorkerPolls, also
invoke RemovePoller on the task queue partition manager.
- task_queue_partition_manager.go: Forward the removal to default and
versioned task queues.
## Why?
Previously, pollers lingered in pollerHistory until TTL expired (~5
min), showing workers that already shut down.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Controlled by dynamic config: EnableCancelWorkerPollsOnShutdown
## What changed?
Replace the limited Nexus HandlerError proto with full Temporal Failure
objects for transmitting errors across Nexus operation boundaries. This
preserves failure chains, stack traces, and typed failure info
(ApplicationFailure, CanceledFailure, etc.) end-to-end.
Care was taken to ensure Temporal->Temporal communication works as
expected as well as support for non-Temporal Nexus implementations (to
maintain support for external endpoints).
This ended up being more refactoring that I had originally anticipated
but I think the unification of error serialization across the codebase
and the more straightforward conversion logic was worth it.
The code was validated across variations of new and old caller and
handler workers and servers using a
[harness](https://github.com/bergundy/nexus-error-compat-tests) that was
built specifically to validate the behavior. The harness was also run
with an SDK setup that encoded failure attributes to confirm failures
are properly transmitted over the different boundaries.
Proto changes:
- Add `failure` field to DispatchNexusTaskResponse carrying a Failure
with NexusHandlerFailureInfo, deprecate `handler_error`
- Add `capabilities` field to nexus Request for feature negotiation
Failure conversion (`common/nexus/failure.go`):
- Rewrite TemporalFailureToNexusFailure and
NexusFailureToTemporalFailure with support for recursive cause chains
and stack traces
- Add special-case handling for NexusHandlerFailureInfo (serialized as
nexus.HandlerError type) and OperationError (mapped to
CanceledFailure/ApplicationFailure based on state)
Nexus SDK layer (`common/nexus/nexusrpc/`):
- Inline the failure converter from the upstream SDK with a custom
FailureConverter interface that round-trips Temporal failure metadata
through Nexus Failure objects
- Add a completion client to abstract away HTTP and provide consistent
failure conversion across all APIs
- Support `unwrap-error` metadata marker so Temporal-to-Temporal calls
can unwrap the OperationError envelope and recover the original cause
Frontend and backend handlers:
- Route responses through new failure path when caller sends
`temporal-nexus-failure-support` header
- Handle both new `failure` and deprecated `handler_error` response
types in nexus_handler.go and nexus_http_handler.go
- Update completion handling in history handler and mutable state to
produce Temporal Failures instead of Nexus HandlerErrors
- Update matching engine to forward the new failure field
Dependencies:
- Bump `nexus-rpc/sdk-go` to pre-release with StackTrace, Cause, and
OriginalFailure fields on Failure and OperationError
## Why?
Part of getting Nexus to GA in all SDKs, this change fixes a couple of
notable issues with errors in Nexus+Temporal applications:
- Nexus SDK errors did not have a way to set an error message, which
made them difficult to use in various languages
- The protocol used a custom failure format that is diverges from proxy
expectations and may result in failures not being encrypted
## Potential risks
- Metric label values for callback outcomes now use Nexus handler error
values instead of HTTP response codes. Alerts that check these outcomes
will need adjustment.
## What changed?
Set TargetVersionChanged instead of SuggestCaN when TargetVersionChanged
https://github.com/temporalio/api/pull/709
## Why?
Setting SuggestContinueAsNew=true for Pinned workflows whenever their is
a new Target Version available for that workflow causes Pinned workflows
to hit that condition much more frequently than they expect. Users who
are currently doing: if workflow_info.suggestContinueAsNew{ do
continue-as-new } in their Pinned workflow code would need to change
that code to protect themselves from running into an infinite-CaN-loop,
because the default CaN behavior for a Pinned workflow is to stay
Pinned.
We should not force users to protect themselves from such a situation.
Because upgrading on continue-as-new is opt-in, receiving the suggestion
to continue-as-new-onto-new-target-version should be opt-in as well. If
people are forced to check the new suggest-continue-as-new-reasons field
to "opt out," that is unsafe, because inevitably some people will forget
to do so or misunderstand, and then get hit by this unexpected footgun.
Much safer and still ergonomical to let upgrade-on-can be opt-in on both
fronts, as proposed here. With this change, the people who are currently
doing if workflow_info.suggestContinueAsNew{ do continue-as-new } won't
see any change in semantics, regardless of their versioning behavior.
People who consciously know that they want to do upgrade-on-can /
Trampolining will have to change their CaN options anyway, so it's easy
enough to teach them to pay attention to this new
TargetWorkerDeploymentVersionChanged flag.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
None
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches workflow task started event generation/persistence and
versioning-related signaling, which can affect worker behavior and
history compatibility; changes are gated by dynamic config and covered
by tests.
>
> **Overview**
> Stops using `SuggestContinueAsNew` (and its reason tags) to signal
pinned workflows that a newer target worker deployment version exists,
and instead introduces an explicit
`TargetWorkerDeploymentVersionChanged` boolean on `WorkflowTaskStarted`
events and persisted `WorkflowExecutionInfo`.
>
> Adds namespace dynamic config `EnableSendTargetVersionChanged`
(default on) and a new metric `workflow_target_version_changed_count`
emitted when this flag is set; updates the workflow task state machine,
mutable state plumbing/mocks, proto/pb persistence, and functional tests
accordingly. Also bumps `go.temporal.io/api` to pick up the new event
attribute.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
3886491826. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Adds infrastructure to cancel all outstanding polls for a worker
instance during shutdown.
Key Changes:
1. request_response.proto / service.proto
- New CancelOutstandingWorkerPolls RPC that cancels polls by
worker_instance_key instead of individual poller_id
2. matching_engine.go:
- Added workerInstancePollers map to track pollers by worker instance
key
- pollTask() now registers pollers in both outstandingPollers (by
pollerID) and workerInstancePollers (by worker key)
- CancelOutstandingWorkerPolls() cancels all pollers for a worker
instance and returns count
## Why?
To support eager cancellation of outstanding polls when worker call
ShutdownWorker.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None. No changes to existing functionality.
## What changed?
Added proto definitions and conversion functions for V1 to V2 schedule
migration.
## Why?
This is the foundation for the V1→V2 schedule migration feature. The
`MigrateScheduleRequest` proto captures all scheduler state needed to
recreate a schedule in the CHASM architecture.
## What changed?
This PR extends the raw history optimization to pass raw history bytes
from History Service → Matching Service → Frontend without
deserialization in Matching Service.
**Key changes:**
1. History Service:
- When `SendRawHistoryBetweenInternalServices` is enabled, sets
`RawHistoryBytes` (field 21) with raw proto-encoded history batches
2. Matching Service:
- Passes raw history bytes through to frontend via
`PollWorkflowTaskQueueResponseWithRawHistory`
- Uses wire-compatible proto messages so gRPC auto-deserializes
`[][]byte` → `History` on the client side
3. Frontend:
- Receives raw history in `RawHistory` field (auto-deserialized by gRPC)
- Processes search attributes for raw history since it bypasses history
service's normal processing
4. Proto definitions:
- Added `raw_history_bytes` (field 21) to
`RecordWorkflowTaskStartedResponse`
- Added `PollWorkflowTaskQueueResponseWithRawHistory` message with
wire-compatible layout
- Added `raw_history` (field 22) to `PollWorkflowTaskQueueResponse`
## Why?
When `history.sendRawHistoryBetweenInternalServices` is enabled, the
previous implementation only avoided deserialization from persistence →
History Service. However, Matching Service was still deserializing
history events (via gRPC auto-deserialization) and re-serializing them
when forwarding to Frontend.
This change eliminates that unnecessary serialization/deserialization
cycle in Matching Service by:
1. Having History Service send raw bytes directly
2. Having Matching Service forward these raw bytes without parsing
3. Having Frontend receive the bytes which gRPC auto-deserializes
This reduces CPU usage in Matching Service for workflows with large
histories.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s) (`tests/workflow_task_test.go`)
## Potential risks
SendRawHistoryBetweenInternalServices must be disabled when rolling back
from this version to an older version.
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## What changed?
Implement detached component as a Field option and Registrable Component
option.
Add detached boolean value to ComponentAttributes persistence proto
definition.
## Why?
Allow detached components to continue updates and task execution even if
parent node lifecycle is closed.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- WISOTT
## Why?
- Operational readiness. There could be a world where we want to "reset"
a user's workflow state given that these workflows CAN so frequently.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- None
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Introduces optional state override when force-continuing-as-new.
>
> - New proto messages `ForceCANDeploymentSignalArgs` and
`ForceCANVersionSignalArgs` with `override_state` fields; generated
helper/pb code updated
> - Deployment and Version workflows now receive `forceCAN` signals with
args and, if provided, apply `override_state` before continue-as-new
> - Added tests verifying override is honored (e.g., manager identity
and metadata) after continue-as-new
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
1fd847da1c. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
- Use plain go struct inside migration workflow activity input output
- Customize the json encoding/decoding for migration execution info and
make it backward & forward compatible.
- This will temporary break cloud and the fix is in
https://github.com/temporalio/temporal/pull/9097
- This PR needs to be part of oss v1.30 release.
## Why?
- Backward compatibility and less confusion over how the encoding will
be done.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
This reverts commit 58449b9d80.
## What changed?
This reverts Nexus schedule to start and start to close timeouts (with
fixes) (#9033)
## Why?
Causes failures in nightly pipeline
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
When buffered events are applied, like when applying signals after
workflow task failures occur, preserve the workflow task attempt so the
`TemporalReportedProblems` search attribute is still properly added.
## Why?
Without this change the `TemporalReportedProblems` search attribute will
never be added if a workflow is consistently queried.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
Minimal, this adds a new int to the workflow task execution, but it's
only used in one place.
## What changed?
Reapplied #9010.
The original PR that introduced these timeouts did not populate the
operation-timeout header or set the call context timeout correctly. This
PR fixes the logic.
## What changed?
- Unifies CHASM scheduler's Scheduler.Info.{RecentActions,
RunningWorkflows} into BufferedStarts.
- Computes the RecentActions and RunningWorkflows views on-demand from
BufferedStarts.
- Removes the old requestIDtoWorkflowID map.
## Why?
- Simplifies the data structure; much easier to reason about
concurrently, as related records are no longer spread between distinct
data structures.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Add "ephemeral data" propagated alongside user data.
- Include information about which priority levels have significant
backlog in ephemeral data (currently only propagated down the tree, not
up)
- When sticky partitions see that normal partitions have significant
backlog, set up poll forwarders to forward to those partitions.
- The poll forwarders use min priority to ensure they only get
higher-priority tasks than available local tasks.
## Why?
Without this functionality, sticky queues interfere with priority
dispatch by keeping most of the pollers working on sticky tasks and
potentially starving high-priority tasks that appear on the normal
queue. With the default values of the new settings, sticky pollers
should notice high-priority normal tasks within 10 seconds.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- Clients have to accept that polls on sticky may return normal tasks. I
believe all SDKs are fine with this.
- Increased overhead of user data + ephemeral data.
## Overview
This commit implements two new granular timeout types for Nexus
operations, allowing callers to have fine-grained control over different
phases of operation execution:
- **Schedule-to-Start Timeout**: Maximum time to wait for an operation
to be started (or completed if synchronous) by the handler
- **Start-to-Close Timeout**: Maximum time to wait for an asynchronous
operation to complete after it has been started
These timeouts complement the existing **Schedule-to-Close Timeout** to
provide better control and diagnostics for Nexus operation execution.
See the corresponding API PR:
https://github.com/temporalio/api/pull/695.
## What changed?
Populated standalone activity Run ID and scheduled time when returning
PollActivityTaskQueueResponse. Updated API deps. Added test specific to
validating PollActivityTaskQueueResponse.
## Why?
Standalone activity executions have Run IDs that should be returned to
the task poller. The response scheduled time should also be populated as
the Temporal SDKs depend on it.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
## What changed?
Add batch workflow refresh tasks Admin API.
Add tdbg commands to invoke `StartAdminBatchOperation`.
## Details
AdminBatchOperations are executed in the Batched System Worker, similar
to existing Frontend BatchOperations. The worker activity will fetch
pages of workflow executions to perform operations on, and periodically
heartbeat its results. For RefreshWorkflowTasks, each individual Refresh
call will go through the admin handler.
Refreshing Workflow Tasks regenerates all pending tasks of an execution
given its mutable state.
## CLI usage
```
tdbg workflow batch-refresh-tasks \
--namespace <ns> \
--query "WorkflowType='MyType'" \
--reason "fixing stuck workflows" \
[--job-id <optional-job-id>] \
[--archetype <optional-archetype>]
```
## Why?
Unblock new Matcher migration and allow for general use case batch Admin
calls. Currently, only supports `BatchOperationRefreshWorkflowTasks`.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## What changed?
Add a new dynamic config for Auto Enabling fairness and priority if we
see the relevant tasks coming in.
## Why?
Seamlessly start to transition users who start using the fields over to
the new code path.
## How did you test it?
- [X] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
Due to storing this in the userdata we are using that interface a bit
more, we also need to change the initialization such that we start it
before being able to substantiate the defaultQ, this change in
initialization might have unintended side effects that I'm not currently
seeing.
## What changed?
Keep the total number and the size of the external payloads per the
workflow execution
## Why?
We are working on building the support for external payloads in SDK,
which are stored outside of Temporal. We'd like to be able to show the
total size and the number of external payloads in the given workflow
execution.
## How did you test it?
- [ ] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
N/A
## What changed?
Add concurrency test and refine some rough edges.
## Why?
testing is good!
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
None
## What changed?
Give the option to upgrade-on-Continue-as-New with a continue-as-new
option.
Ensure that Pinned Overrides are inherited across the entire
continue-as-new chain, regardless of the initial versioning behavior of
the continue-as-new.
## Why?
To unlock "Trampolining"
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Introduces explicit Continue-as-New tracking and version-targeting
across APIs and server internals.
>
> - Protos: add `target_deployment_version` to
`RecordWorkflowTaskStartedRequest`; add repeated
`workflow_task_suggest_continue_as_new_reasons` to persistence
`WorkflowExecutionInfo`; regenerate bindings and enum imports
> - History: plumb `target_deployment_version` through
`AddWorkflowTaskStartedEvent`; include `suggest_continue_as_new_reasons`
in `WorkflowTaskStarted` events; choose non-nil `VersioningOverride`
when starting executions (user or inherited)
> - Metrics: remove `workflow_update_continue_as_new_suggestions`; add
`workflow_continue_as_new_count` and
`workflow_suggest_continue_as_new_count`; introduce tags
`continue_as_new_versioning_behavior` and per-reason suggest flags; tag
helpers added
> - Interfaces/tests: extend
`MutableState.Add/ApplyWorkflowTaskStartedEvent` and `WorkflowTaskInfo`
with suggest reasons; update call sites and tests; update
`common/util_test.go` scan; bump `go.temporal.io/api`
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
849687bd0c. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Allow new option for matching poll request to specify a minimum priority
task to match, and also to request not blocking. These are intended to
be used together. These options are only on the internal poll rpcs, they
can't be set externally.
## Why?
This will be used for priority backlog poll forwarding, to avoid
matching with unintended lower-priority tasks. It could also be used
eventually as a building block for activity task preemption.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s) (in future PRs, not this one)
## Potential risks
Using min priority without nowait could cause excessive cpu usage since
the find-match algorithm isn't optimized for it yet. With nowait and
backoff on empty response, it shouldn't be a problem.
## What changed?
Add LastCurrentTime to version info to tell if it ever became current
## Why?
So that the controller can accelerate rollout if it detects that the
target version was previously Current (aka it's actually a roll back)
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Introduce `last_current_time` on version state/summary, populate it
whenever a version becomes Current, tweak routing update time for
draining, and update tests accordingly.
>
> - **API/Proto**:
> - Add `last_current_time` to `VersionLocalState` and
`WorkerDeploymentVersionSummary` in `proto/internal/.../message.proto`
and generated `api/deployment/v1/message.pb.go`.
> - Wire getters/fields and descriptor indexes for new timestamp.
> - **Workflows/Logic**:
> - Set `last_current_time` when a version becomes Current in both sync
path and `updateStateFromRoutingConfig`.
> - Adjust draining `routing_update_time` to consider
`ramping_version_changed_time`.
> - Include `last_current_time` in version summaries/signals.
> - **Client/Conversion**:
> - Propagate `last_current_time` through `client.go` summary mapping.
> - **Tests**:
> - Update functional tests to assert `last_current_time` behavior
(including demote/promote scenarios) and refine timestamp range checks.
> - **Deps**:
> - Bump `go.temporal.io/api` version in `go.mod/go.sum`.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
3915e090f3. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Added standalone activity completion and failure handling. Refactored
existing timeout failure handling. Refactored existing check for retry
method.
Needed to support standalone activities full operation.
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
---------
Co-authored-by: Dan Davison <dan.davison@temporal.io>
Added standalone activity Chasm tasks. Added handling of start activity
and e2e implementation of standalone activity start execution with
existing services. Updated protos related to standalone activities.
The Chasm tasks are needed to kick off standalone activity execution via
the existing services. Proto changes needed to so that the component ref
can be passed and handled via service stack.
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: Roey Berman <roey.berman@gmail.com>
Co-authored-by: Dan Davison <dan.davison@temporal.io>
## What changed?
Add revision number to Worker Deployment Version Data that is synced to
Task Queues.
## Why?
This allows making task queue registration also async, but more
importantly, it prevent's race conditions between concurrent
registrations, setCurrent/Ramping, drainage, and deletion.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None.
## What changed?
Remove unused code about versioning deprecated pre-release APIs (only
v0.3.0).
## Why?
The code is not used by the current implementation. The APIs previously
referring to the code has been returning the following error for the
last 6+ months:
```
Deployments are deprecated and no longer supported, use Worker Deployments instead
```
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None
## What changed?
- Added a new RPC method, scoped to matching only, which shall check if
a specific version exists in the userData of a task queue.
## Why?
- Currently, versioning has several places in history which call the
method GetTaskQueueUserData to validate if a version exists in a task
queue or not.
- However, with [this](https://github.com/temporalio/temporal/pull/8722)
going in very soon, one of the biggest changes put forward by it is the
fact that a task queue's userData could go on to become big (if a task
queue was part of a large number of versions). The main reason for this
is because we keep an entry for versions that are deleted inside the
task-queue's user data so that we can ensure read-after-write
consistencies.
- Thus, to prevent large number of bytes flowing across the wire, it's
best to just make this new RPC that does the check in matching and only
sends a boolean across.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- No risk honestly I think this is an optimization
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Introduce a matching-only RPC to check if a task queue is part of a
specific deployment version, wire it through server/clients, and update
worker versioning to use it.
>
> - **API/Protos**:
> - Add `CheckTaskQueueVersionMembershipRequest`/`Response` messages and
RPC in `matchingservice` (`request_response.proto`, `service.proto`).
> - Regenerate Go bindings (`request_response.pb.go`, `service.pb.go`,
`service_grpc.pb.go`).
> - **Server**:
> - Implement `CheckTaskQueueVersionMembership` in matching engine to
read TQ user data and return `is_member`.
> - Wire handler and register gRPC method; update logtag interceptor.
> - **Clients**:
> - Add `CheckTaskQueueVersionMembership` to generated, metric, and
retryable matching clients.
> - Update mocks to include the new RPC.
> - **Worker Versioning**:
> - Replace `GetTaskQueueUserData` check with
`CheckTaskQueueVersionMembership` in `worker_versioning.go`.
> - **Tests**:
> - Add functional test validating membership check behavior in
`tests/versioning_3_test.go`.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
99fedcfe7e. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
…Connection
## What changed?
Rename to be consistent from
https://github.com/temporalio/temporal/pull/8658
## Why?
The inconsistent naming looks wack
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA, field unused
## What changed?
- Add revision number mechanics for child workflows and CAN workflows.
- Additionally, this PR introduces the following change conceptually:
Child workflows of AutoUpgrade parents and CAN workflow executions of
AutoUpgrade workflows have their first task as AutoUpgrade by default.
Previously, it was unversioned.
## Why?
- Resolving eventual consistency issues.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- None since this feature is gated by a dc
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Adds `inherited_auto_upgrade_info` propagation and
revision-number-based dispatch so child and continue-as-new workflows
start with AutoUpgrade behavior when appropriate; updates
matching/history logic and tests.
>
> - **API/Proto**:
> - Add `StartWorkflowExecutionRequest.inherited_auto_upgrade_info` (and
getter) to carry deployment version and revision from parent/previous
run.
> - **History Service**:
> - Populate/propagate `InheritedAutoUpgradeInfo` when starting child
workflows and during ContinueAsNew; avoid when pinned inheritance
applies or namespaces differ.
> - On start, set execution `VersioningInfo` (deployment, behavior
AutoUpgrade) and revision number from inherited info.
> - Minor fixes: use request context for version checks; guard
`SetVersioningRevisionNumber` against nil.
> - **Matching**:
> - Add test hook `MatchingIgnoreRoutingConfigRevisionCheck` and allow
bypassing routing-config revision gate in updates.
> - Minor cleanup in partition manager; keep revision-based queue
selection logic.
> - **Tests**:
> - Add/rename functional tests covering child/CAN inheritance, cross-TQ
behavior, retry no-bounce with revision lag, and propagation helpers;
introduce rollback utility for task queue user data.
> - **Deps**:
> - Bump `go.temporal.io/api` to latest prerelease.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
8a342ab287. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
