## What changed?
Add health checks for docker compose dependencies.
## Why?
Seeing connection timeouts for cass_es8
[[example](https://github.com/temporalio/temporal/actions/runs/22005015596/job/63897748984?pr=9292)]
that are failing tests.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Might make tests a little bit slower due to extra wait time.
## What changed?
WISOTT
## Why?
A few tests have issues not yielding in the allotted 40 minute test
time, causing timeouts and CI failures. This addresses two of those.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA, tests only
## Overview
This feature introduces a special internal Nexus endpoint called
`__temporal_system` that enables adding functionality to workflows
**without requiring new workflow commands and events**. Operations on
this endpoint are routed internally within Temporal's history service
rather than via external HTTP calls.
## Key Components
### 1. System Endpoint Infrastructure
- **Endpoint Name**: `__temporal_system` (constant in
`common/nexus/constants.go:6`)
- **Callback URL**: `temporal://system` for internal routing
- **New History Service RPCs**
(`proto/internal/temporal/server/api/historyservice/v1/service.proto:433-437`):
- `StartNexusOperation` - Starts operations on the system endpoint
- `CancelNexusOperation` - Cancels operations on the system endpoint
### 2. Operation Processor Framework
A new processor pattern (`chasm/nexus_operation_processor.go`) that
allows CHASM libraries to:
- **Validate and transform input**: Processors can validate operation
inputs and set default values
- **Determine routing**: Each processor returns a routing key that
determines which history shard handles the operation
- **Re-serialize input**: Mutated inputs can be re-serialized to persist
default values
**Routing strategies**:
- `NexusOperationRoutingKeyExecution` - Routes to the shard owning a
specific workflow execution
- `NexusOperationRoutingKeyRandom` - Routes to a random shard
### 3. CHASM Library Integration
CHASM libraries can now provide (`chasm/library.go:16-19`):
- **`NexusServices()`**: Regular Nexus service handlers (implement the
actual operation logic)
- **`NexusServiceProcessors()`**: Input processors for validation and
routing
Example from test library (`chasm/lib/tests/nexus_service.go`):
```go
// Service handler - implements the actual operation
TestOperation = nexus.NewSyncOperation("TestOperation",
func(ctx context.Context, input string, options nexus.StartOperationOptions) (string, error) {
return "Hello, " + input, nil
})
// Processor - validates input and determines routing
func (o testOperationProcessor) ProcessInput(ctx chasm.NexusOperationProcessorContext, input string)
(*chasm.NexusOperationProcessorResult, error) {
return &chasm.NexusOperationProcessorResult{
RoutingKey: chasm.NexusOperationRoutingKeyExecution{
NamespaceID: ctx.Namespace.ID().String(),
BusinessID: input, // Route based on input
},
}, nil
}
```
### 4. Execution Flow
When a workflow schedules a Nexus operation on `__temporal_system`
(`components/nexusoperations/executors.go:233-238`):
1. **Input Processing**: The processor validates input and determines
routing
2. **Internal RPC**: Instead of HTTP, calls
`HistoryClient.StartNexusOperation` with the target shard ID
3. **Handler Execution**: The history service invokes the registered
Nexus handler (`service/history/handler.go:2707-2768`)
4. **Result Handling**: Supports both sync (immediate result) and async
(operation token) responses
5. **Workflow Completion**: Results flow back through the same
completion path as external Nexus operations
### 5. Benefits
✅ **No schema changes**: Add functionality without new commands/events
in workflow history
✅ **Consistent API**: Uses existing Nexus operation semantics
(sync/async, callbacks, links)
✅ **Proper routing**: Operations are intelligently routed to the correct
shard
✅ **Input validation**: Type-safe input validation and default value
handling
✅ **Future extensibility**: Foundation for direct client invocation (not
yet implemented)
### 6. Technical Details
- **Error handling** (`components/nexusoperations/executors.go:444`):
Non-retryable service errors are properly handled and fail operations
immediately
- **Metrics**: System operations are tracked separately with
`DestinationTag` set to the endpoint name
- **Link conversion**: Helper functions convert between Nexus SDK links
and protobuf links (`common/nexus/util.go:17-46`)
- **Operation token handling**: Moved link converters to common package
for reuse (`common/nexus/link_converter.go`)
### 7. Current Limitations
- Only accessible from workflows (via `ScheduleNexusOperation` command)
- Direct client invocation not yet implemented
- Headers not supported for system endpoint operations
## Test Coverage
New test (`tests/nexus_workflow_test.go:2763-2843`) demonstrates:
- Scheduling operation on `__temporal_system` endpoint
- Synchronous operation completion
- Result propagation back to workflow
## Architecture
This architecture provides a clean, extensible way to add internal
functionality while maintaining compatibility with Temporal's existing
workflow execution model. The system endpoint acts as a bridge between
workflows and internal CHASM components, enabling:
- **Extensibility**: New operations can be added by implementing CHASM
libraries
- **Type safety**: Input validation happens before operations are routed
- **Scalability**: Intelligent routing ensures operations land on the
correct shard
- **Consistency**: Same execution model as external Nexus operations
## What changed?
Add additional standalone activity request validations. Renamed blob
error tags to be in line with existing workflow tags. Code cleanup.
Refactor tests into grouped subtests.
## Why?
After an audit of the standalone activity frontend validations, this PR
provides additional coverage of missing corners.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
## What changed?
Use only ARM runners for running tests.
## Why?
They are faster and cheaper and are available faster.
<img width="390" height="249" alt="Screenshot 2026-02-12 at 8 02 58 PM"
src="https://github.com/user-attachments/assets/0de5865e-971a-4d7c-8dd1-a57860412f12"
/>
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
Tested explicitly on all databases.
## Summary
- Bump Alpine base image tag default to 3.23.3 for server/admin-tools
images.
- Update defaults in docker build bake args and manual build workflows.
## What Changed
Exposes the `EnableCancelWorkerPollsOnShutdown` dynamic config as a
namespace capability in `DescribeNamespace` response.
## Why
SDKs need to know whether the server supports server-side poll
completion on shutdown before relying on it. This capability allows SDKs
to check via `DescribeNamespace` and decide:
- If `true`: Send `WorkerInstanceKey` in polls and rely on server to
complete polls on shutdown
- If `false`: Handle poll completion client-side (existing behavior)
## How
- `service/frontend/namespace_handler.go`: Added
`WorkerPollCompleteOnShutdown` to capabilities
- `service/frontend/namespace_handler_test.go`: Added test coverage for
the new capability
## Dependencies
- Bumped `go.temporal.io/api` to `v1.62.2-0.20260213194545-c89ebac64f01`
to include new proto field from temporalio/api#719
## What changed?
Change duplicate task error type to already exists
## Why?
This should not be a unknown error and log it in every request
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Include transient and speculative WFT events in
`GetWorkflowExecutionHistoryReponse` response, unless UI or CLI made
request.
* Adds `transient_or_speculative_events` back to
`GetMutableStateResponse`
* Reserve `transient_workflow_task` in `HisotryCOntinuation` token
* Add validation helpers
* Add query-compare-query for transient events at request start and end
Re-implements #7732
## Why?
Fix "premature end of stream" errors when workers request history after
cache eviction w/ transient/speculative workflow tasks present. This
adds transient & speculative WFT events in `GetWorkflowExecutionHistory`
(already in `PollWorkflowTask`). Worker cache eviction w/ speculative
workflow tasks causes the expected and actual event counts to be
different. #7732 passed transient events through continuation tokens,
which could become stale during pagination. This PR implements mutable
state querying at both start and end of pagination and compares
transient event IDs to detect if WFT state changed during pagination and
return a retryable error.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Same risks from #7732
## What changed?
Implement CHASM UpdateWithStartExecution.
## Why?
Allows callers to update any current execution and also start a new
execution if none already exists for a given ExecutionKey.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
minor: fixes unit test verification for setting deferred pointers.
## What changed?
- [This](https://github.com/temporalio/temporal/pull/9300) PR is the
part 1 of this effort.
- PR #9300 added a dynamic config with the hope that simply making it a
0 would revert the backlog count (and age) metric to be emitted from the
old location. However, I quickly realized that it was not doing that.
This PR fixes that by first checking if that dynamic config is enabled
or not. If it is enabled, we emit the newly created *physical level
backlog* metrics. If it is not emitted, we emit the backlog metrics the
way they used to before #9300 went in.
- Additionally, cursor bot had this
[comment](https://github.com/temporalio/temporal/pull/9300#discussion_r2801945963)
pointing out that our use of `Describe` task queue to get the backlog
metrics to emit would cause task queues to remain "alive" indefinitely
and never be unloaded, potentially causing strain on Matching service.
This PR fixes that by skipping the "mark alive" step when `Describe` is
called internally for metrics use, and only marking it alive when it's
called externally.
## Why?
- Safety and correctness.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
- these are risky, but i have a dynamic config flag now to quickly
revert things in prod if things go wrong.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes affect Matching backlog metrics and task-queue liveness
behavior; incorrect gating could break dashboards/alerts or impact queue
unloading behavior under load.
>
> **Overview**
> Adjusts backlog gauge emission to **conditionally switch** between the
new physical backlog metrics and the legacy `approximate_backlog_*`
metrics based on `BacklogMetricsEmitInterval`: when attribution is
enabled, only the unversioned queue emits
`physical_approximate_backlog_*`; when disabled, all applicable queues
fall back to emitting the original `approximate_backlog_*` series.
>
> Refactors `Describe` into an internal `describe(..., skipMarkAlive)`
variant and uses it for periodic logical backlog metric emission so
internal metric collection no longer calls `MarkAlive`, avoiding
unintended prevention of idle task queue unloading.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
e79f933135. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Carly de Frondeville <carly.defrondeville@temporal.io>
Co-authored-by: Carly de Frondeville <cdefrondeville@berkeley.edu>
## What changed?
When an activity attempt times out and there is not enough time to
schedule the next attempt, record `ActivityTaskTimedOut` with
schedule-to-close timeout type and provide a clear message.
## Why?
Noticed a regression of the behavior while running Java SDK tests with
server 150. This change fixes the regression.
I added tests in the repo for this:
e747980e0e/tests/activity_test.go (L361-L377)
Verified that these added tests fail since
https://github.com/temporalio/temporal/pull/9132. This PR reverts the
behavior and adds a bit more information in the error message.
## How did you test it?
- [x] added new functional test(s)
## What changed?
Eagerly remove worker from pollerHistory during graceful shutdown so
DescribeTaskQueue doesn't show stale pollers.
**Note:** This PR subtly changes how pollerHistory is updated `before`
and `after` a poll request.
Before:
- UpdatePollerInfo called at poll START
- defer UpdatePollerInfo called at poll END (always, regardless of how
poll ended)
After:
- UpdatePollerInfo called at poll START
- UpdatePollerInfo called at poll END only if ctx.Err() !=
context.Canceled
Skipped on cancellation (shutdown/disconnect) to avoid re-adding entry
after RemovePoller
The defer was originally added (PR #2811) to keep timestamps fresh, but
HasPollerAfter() first checks currentPolls > 0 for active polls, so the
timestamp is a secondary check
**Main changes**
API
- request_response.proto: Added worker_identity field to
CancelOutstandingWorkerPollsRequest
Implementation
- matching_engine.go: As part of CancelOutstandingWorkerPolls, also
invoke RemovePoller on the task queue partition manager.
- task_queue_partition_manager.go: Forward the removal to default and
versioned task queues.
## Why?
Previously, pollers lingered in pollerHistory until TTL expired (~5
min), showing workers that already shut down.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Controlled by dynamic config: EnableCancelWorkerPollsOnShutdown
## Why?
It's a failed precondition error that isn't translated properly to Nexus
error semantics and becomes a non retryable error.
This is only relevant for the priority matcher, the "old" matcher
already avoided this.
## What changed?
- WISOTT
- In simpler terms, the backlog count metric that is emitted today has a
discrepancy with the value that is returned from the DescribeVersionAPI.
The main discrepancy is that the backlog count *metric* of an
un-versioned queue does not take into consideration the current/ramping
version, which means that it over-counts and the ramping/current version
emit under-counts. This PR changes that by resolving the discrepancy.
- Additionally, I have added new metrics called
*physical_backlog_count/age*. These metrics are basically going to emit
the values that the current backlog metrics (before this PR) were
emitting. The main purpose of these newly added metrics is to add
operational value.
- Moreover, I moved the metric emission to the task queue partition
manager since it felt more appropriate.
## Why?
- Correctness!
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
- yes, in the sense that I am altering values that a metric that is in
prod and customer facing.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes semantics and emission paths of customer-facing backlog
gauges, which may affect dashboards/alerts and could introduce stale or
missing series if tagging/config edge cases are overlooked.
>
> **Overview**
> Backlog metrics emission is split into **logical
(version-attributed)** vs **physical (raw queue)** metrics to align
`approximate_backlog_count`/`approximate_backlog_age_seconds` with
`DescribeTaskQueue`’s versioning attribution.
>
> A new task-queue dynamic config `matching.backlogMetricsEmitInterval`
drives periodic emission from the partition manager, including zeroing
logical gauges on queue unload to avoid staleness. The existing DB-level
backlog gauges are renamed and re-targeted to new
`physical_approximate_backlog_count`/`physical_approximate_backlog_age_seconds`
metrics (default/unversioned queues only), and unit tests are added to
validate attribution behavior (current/ramping/disabled build-id
breakdown).
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
a5c81c0eb8. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Adds `--decode` flag to tdbg's `wofkflow show` to decode event history
proto payloads to JSON.
## Why?
Make on-call investigation easier.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
The decoder ignores errors; so if it fails, it behaves as before.
## What changed?
- activate drained/inactive versions to draining when they get a
workflow started on it
- also added a history cache, per history node, so that we don't bombard
our version workflows with signals that shall change the drainage status
of these workflows.
## Why?
- versioning correctness, in the sense that if someone were to move a
workflow on to a version that is drained, the drainage status should be
updated to draining (since it now has one open workflow working on it)
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- Sure, this change is lowkey risky. Would appreciate a thorough review.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches history start/reset/update flows and adds asynchronous
signaling into worker-deployment workflows; mis-wiring or cache/config
issues could cause missing or excessive reactivation signals and
unexpected version state churn under load.
>
> **Overview**
> When workflows are **pinned to a specific deployment version**,
history now triggers a fire-and-forget `reactivate-version` signal to
the corresponding worker-deployment *version workflow* so versions in
`DRAINED/INACTIVE` transition back to `DRAINING`.
>
> This wiring is applied across start paths (`StartWorkflowExecution`
incl. conflict handling, `SignalWithStart`, multi-op start), option
changes (`UpdateWorkflowExecutionOptions` after persistence), and
`ResetWorkflowExecution` post-reset operations. A new per-history-node
`ReactivationSignalCache` (TTL/max-size + metrics tags) deduplicates
signals, and a new dynamic config flag
(`history.enableVersionReactivationSignals`) plus cache settings
control/limit load.
>
> Worker-deployment adds `Client.SignalVersionReactivation` and the
version workflow gains a version-gated handler for `reactivate-version`
to update drainage/status and sync summaries; extensive functional tests
cover reactivation and cache dedup behavior.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
15a5ac69b2. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Carly de Frondeville <cdefrondeville@berkeley.edu>
## What changed?
- Avoid double map look up when getting registableComponent/Task by ID
## Why?
- Minor performance improvement.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Add xdc functional test for chasm retention timer
## Why?
- Add test coverage.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## What changed?
- When checking waiters, only check waiter with higher or equal
priority.
- Allow specifying priority in TryAcquire()
## Why?
- If there's all waiters have lower priority, we don't really need to
create a new waiter and later unblock it.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- We reserve keys on the opposite stack to account for a fleet's
inconsistent view of dynamic config.
- This is similar to the [dummy workflow
PR](https://github.com/temporalio/temporal/pull/9201) for V1 stack.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Prevent terminated executions from updating execution state which will
result in an error
- Handle terminated executions when checking for access rules
- Wherever the existing terminated flag is checked, also check the
persisted execution state in mutable state.
## Why?
- Current checks uses the in-memory terminated field on a node which
will be lost after reload or replication.
- No real impact today as terminated execution don't really need any
updates today, but that won't be true in the future when chasm
executions can have callbacks.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What
Adds unit tests to verify `ShutdownWorker` API handles poll cancellation
errors gracefully.
## Why
During rolling upgrades, frontend may be updated before matching. If
frontend calls `CancelOutstandingWorkerPolls` on an old matching node,
it returns `Unimplemented`. The `ShutdownWorker` API should still
succeed (graceful degradation).
## How
- `TestShutdownWorkerWithCancellationError`: All cancellation calls
fail, verifies ShutdownWorker succeeds
- `TestShutdownWorkerWithPartialCancellationFailure`: Mixed
success/failure (2 succeed, 2 fail), verifies ShutdownWorker succeeds
## What changed?
- Adds a new dummy workflow type to the system worker.
## Why?
- See the comments on the workflow. This will be used to block a key in
the
workflow ID space from being written to during the process of creating a
CHASM
schedule, preventing a logical duplicate across separate ID spaces.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Set `component.nexusoperations.useSystemCallbackURL` to true by default.
## Why?
It's now safe to turn this on since server v.130 supports this format
and is a valid rollback target.
## What changed?
Replace the limited Nexus HandlerError proto with full Temporal Failure
objects for transmitting errors across Nexus operation boundaries. This
preserves failure chains, stack traces, and typed failure info
(ApplicationFailure, CanceledFailure, etc.) end-to-end.
Care was taken to ensure Temporal->Temporal communication works as
expected as well as support for non-Temporal Nexus implementations (to
maintain support for external endpoints).
This ended up being more refactoring that I had originally anticipated
but I think the unification of error serialization across the codebase
and the more straightforward conversion logic was worth it.
The code was validated across variations of new and old caller and
handler workers and servers using a
[harness](https://github.com/bergundy/nexus-error-compat-tests) that was
built specifically to validate the behavior. The harness was also run
with an SDK setup that encoded failure attributes to confirm failures
are properly transmitted over the different boundaries.
Proto changes:
- Add `failure` field to DispatchNexusTaskResponse carrying a Failure
with NexusHandlerFailureInfo, deprecate `handler_error`
- Add `capabilities` field to nexus Request for feature negotiation
Failure conversion (`common/nexus/failure.go`):
- Rewrite TemporalFailureToNexusFailure and
NexusFailureToTemporalFailure with support for recursive cause chains
and stack traces
- Add special-case handling for NexusHandlerFailureInfo (serialized as
nexus.HandlerError type) and OperationError (mapped to
CanceledFailure/ApplicationFailure based on state)
Nexus SDK layer (`common/nexus/nexusrpc/`):
- Inline the failure converter from the upstream SDK with a custom
FailureConverter interface that round-trips Temporal failure metadata
through Nexus Failure objects
- Add a completion client to abstract away HTTP and provide consistent
failure conversion across all APIs
- Support `unwrap-error` metadata marker so Temporal-to-Temporal calls
can unwrap the OperationError envelope and recover the original cause
Frontend and backend handlers:
- Route responses through new failure path when caller sends
`temporal-nexus-failure-support` header
- Handle both new `failure` and deprecated `handler_error` response
types in nexus_handler.go and nexus_http_handler.go
- Update completion handling in history handler and mutable state to
produce Temporal Failures instead of Nexus HandlerErrors
- Update matching engine to forward the new failure field
Dependencies:
- Bump `nexus-rpc/sdk-go` to pre-release with StackTrace, Cause, and
OriginalFailure fields on Failure and OperationError
## Why?
Part of getting Nexus to GA in all SDKs, this change fixes a couple of
notable issues with errors in Nexus+Temporal applications:
- Nexus SDK errors did not have a way to set an error message, which
made them difficult to use in various languages
- The protocol used a custom failure format that is diverges from proxy
expectations and may result in failures not being encrypted
## Potential risks
- Metric label values for callback outcomes now use Nexus handler error
values instead of HTTP response codes. Alerts that check these outcomes
will need adjustment.
## What changed?
Do not convert transient task to normal in start deployment transition.
## Why?
The conversion was not handling event generation well all the time. Also
the conversion is not strictly needed. it can be done later as a user
experience improvement.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
None
## What changed?
Set TargetVersionChanged instead of SuggestCaN when TargetVersionChanged
https://github.com/temporalio/api/pull/709
## Why?
Setting SuggestContinueAsNew=true for Pinned workflows whenever their is
a new Target Version available for that workflow causes Pinned workflows
to hit that condition much more frequently than they expect. Users who
are currently doing: if workflow_info.suggestContinueAsNew{ do
continue-as-new } in their Pinned workflow code would need to change
that code to protect themselves from running into an infinite-CaN-loop,
because the default CaN behavior for a Pinned workflow is to stay
Pinned.
We should not force users to protect themselves from such a situation.
Because upgrading on continue-as-new is opt-in, receiving the suggestion
to continue-as-new-onto-new-target-version should be opt-in as well. If
people are forced to check the new suggest-continue-as-new-reasons field
to "opt out," that is unsafe, because inevitably some people will forget
to do so or misunderstand, and then get hit by this unexpected footgun.
Much safer and still ergonomical to let upgrade-on-can be opt-in on both
fronts, as proposed here. With this change, the people who are currently
doing if workflow_info.suggestContinueAsNew{ do continue-as-new } won't
see any change in semantics, regardless of their versioning behavior.
People who consciously know that they want to do upgrade-on-can /
Trampolining will have to change their CaN options anyway, so it's easy
enough to teach them to pay attention to this new
TargetWorkerDeploymentVersionChanged flag.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
None
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches workflow task started event generation/persistence and
versioning-related signaling, which can affect worker behavior and
history compatibility; changes are gated by dynamic config and covered
by tests.
>
> **Overview**
> Stops using `SuggestContinueAsNew` (and its reason tags) to signal
pinned workflows that a newer target worker deployment version exists,
and instead introduces an explicit
`TargetWorkerDeploymentVersionChanged` boolean on `WorkflowTaskStarted`
events and persisted `WorkflowExecutionInfo`.
>
> Adds namespace dynamic config `EnableSendTargetVersionChanged`
(default on) and a new metric `workflow_target_version_changed_count`
emitted when this flag is set; updates the workflow task state machine,
mutable state plumbing/mocks, proto/pb persistence, and functional tests
accordingly. Also bumps `go.temporal.io/api` to pick up the new event
attribute.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
3886491826. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
- Puts an upper bound on how many concurrent backfillers can be active
in a scheduler.
## Why?
- Prevents a case of unbounded growth.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- I've set too low a limit, and on rollout, we break someone expecting a
higher number of concurrent backfillers. But I'd be pretty surprised if
anyone has a use case for 100+ backfillers..
## What changed?
Fix logic error in pri matcher: the task validator was being treated
like a forwarder and matching was blocked on the root (since the root
can't forward).
## Why?
Task validation mechanism should work on root.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Remove `(retry 1/2)` from test name for more accurate reporting.
## Why?
More accurate reporting.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Reporting only
## What changed?
1. Extended existing Go test memory monitor to include goroutine
profile.
2. Changed snapshot to be of moment with _highest_ memory usage (instead
of latest).
3. Unified report into a single one (both printing and disk snapshot).
4. (bonus) added monitor to unit and integration test jobs.
## Why?
Inspect where high goroutine count comes from.
## How did you test it?
Example:
https://github.com/temporalio/temporal/actions/runs/21695994550/job/62566409323?pr=9162#step:10:17
## What changed?
Fairness counters can use a count-min sketch that can automatically grow
on contention. But when it grows it drops all values. This adds a
mechanism to preserve counts of the top 100 keys.
## Why?
More accurate fairness
## How did you test it?
- [x] added new unit test(s)
## What changed?
Increase the time out and retries for TestTransientWorkflowTaskTimeout
which has been flaking consistently. Ran this 50 times on my local
machine and had 0 failures.
## Why?
Prevent future flakes.
## How did you test it?
- [ ] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Slightly longer test times.
## What changed?
This bumps the version of Go used to 1.26.0
## Why?
https://go.dev/doc/go1.26
## How did you test it?
Existing tests
## Potential risks
None that I'm aware of.
## What changed?
Problem: When a worker sends ShutdownWorker, it may still have
outstanding poll requests waiting in matching. If a task is dispatched
to one of these polls, we risk losing the task if the worker were to
restart, or the poll times out. Since the worker is anyway wanting to
shutdown, might as well not deliver any more tasks to it.
Solution: ShutdownWorker now fans out CancelOutstandingWorkerPolls to
all task queue partitions, causing outstanding polls to return empty
before tasks can be dispatched to them.
Key Changes:
1. workflow_handler.go:
- cancelOutstandingWorkerPolls() fans out cancellation to all partitions
in parallel
2. service.go / constants.go: EnableCancelWorkerPollsOnShutdown dynamic
config flag (default: false)
Flow:
- Worker calls ShutdownWorker with worker_instance_key and task_queue
- Frontend fans out CancelOutstandingWorkerPolls to all partitions
(workflow + activity)
- Matching engine cancels all polls for that worker, returning empty
responses
- Worker receives empty responses and completes shutdown cleanly
## Why?
Avoid dispatching tasks to workers that are shutting down by proactively
cancelling their outstanding polls. This way we reduce the chances of
losing a task due to worker restart/poll timeout.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None. New feature disabled by default.
## What changed?
- WISOTT
- Plus, added some more refactorings in that file so that the code is
cleaner.
## Why?
- this was missed when implementing this cache.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
- None, we are just adding a cache with the implementation quite similar
to the existing use cases.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches History request-handling and transfer-queue execution paths by
changing constructor signatures and injecting a shared cache; main risk
is nil/incorrect cache wiring or subtle behavior changes in membership
checks, though logic is largely additive and covered by new unit tests.
>
> **Overview**
> Adds `VersionMembershipCache` support to
`GetIsWFTaskQueueInVersionDetector`, making it consult/populate the
cache around task-queue version membership checks and refactoring the
matching call into `checkTaskQueueVersionMembership` with the existing
*Unimplemented → user-data* fallback.
>
> Wires the cache through History components that call the detector
(workflow task completion/ContinueAsNew flow and transfer queue
child-workflow inheritance checks), including constructor signature
updates and Fx wiring via `transferQueueFactory`. Expands unit coverage
to verify cache-hit/no-RPC behavior, cache population on misses, and
error/no-cache semantics.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
94314a63f9. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Adds infrastructure to cancel all outstanding polls for a worker
instance during shutdown.
Key Changes:
1. request_response.proto / service.proto
- New CancelOutstandingWorkerPolls RPC that cancels polls by
worker_instance_key instead of individual poller_id
2. matching_engine.go:
- Added workerInstancePollers map to track pollers by worker instance
key
- pollTask() now registers pollers in both outstandingPollers (by
pollerID) and workerInstancePollers (by worker key)
- CancelOutstandingWorkerPolls() cancels all pollers for a worker
instance and returns count
## Why?
To support eager cancellation of outstanding polls when worker call
ShutdownWorker.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None. No changes to existing functionality.
## What changed?
1. Exposed metadata functionality:
- Made `SetContextMetadata()` public in MutableState interface
- Added constants for metadata keys (`MetadataKeyWorkflowType`,
`MetadataKeyWorkflowTaskQueue`)
- Added `ContextHasMetadata()` helper function to check if context has
metadata support
2. Fixed readonly operations:
- `QueryWorkflow`: Now explicitly calls `SetContextMetadata()` since
queries never close transactions (readonly operation)
- `RespondWorkflowTaskCompleted`: Calls `SetContextMetadata()` when
there are only messages (e.g., update rejections) and no commands, since
these also don't close transactions
3. Added tests for the new `ContextHasMetadata()` helper and
`SetContextMetadata()` functionality
## Why?
Context metadata (workflow type, task queue) is automatically populated
when mutable state transactions close. However, readonly operations
never close transactions, so the metadata was missing from their
contexts.
This change ensures that successful readonly operations (queries, update
rejections) also have workflow metadata populated in their contexts
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches core history API request paths and mutable state interfaces;
while behavior is additive (context tagging), incorrect invocation or
key usage could affect observability/metrics and tests across multiple
workflows.
>
> **Overview**
> Ensures workflow context metadata (workflow type/task queue) is
consistently populated even for readonly paths that don’t close
mutable-state transactions.
>
> This exposes `MutableState.SetContextMetadata(ctx)` (renaming the
internal helper), standardizes metadata keys via
`contextutil.MetadataKeyWorkflowType`/`MetadataKeyWorkflowTaskQueue`,
adds `ContextHasMetadata()` for debugging, and explicitly calls
`SetContextMetadata` in `QueryWorkflow` and in
`RespondWorkflowTaskCompleted` when `Commands` is empty (e.g., update
rejections / heartbeats). Tests are expanded to assert metadata presence
across success and several error/edge cases.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
69c3140233. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Bump API dep to v1.62.1
## Why?
Fix an API break in wire compatibility
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> Dependency-only change with no application code modifications; risk is
limited to potential wire/proto compatibility or behavior changes
introduced by the new API version.
>
> **Overview**
> Updates the `go.temporal.io/api` dependency from a pinned
pseudo-version to the released `v1.62.1`, and refreshes `go.sum`
accordingly to match the new module checksum entries.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
535061a147. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
fix flaky test
## Why?
flakes no good
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
none
## What changed?
- WISOTT
## Why?
- During the a rolling upgrade, there could be a world where history
(the caller of the matching RPC) is on a higher version than matching
(the service that has the RPC implemented)
- In this world, we want to fallback on an already implemented matching
RPC (GetTaskQueueUserData) so that the functionality does not differ.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
- None, cause the fallback RPC has been in matching since the start of
time.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes behavior in task-queue version membership checks and pinned
override validation during RPC errors; incorrect fallback parsing or
user-data assumptions could allow/deny pinned versions incorrectly
during upgrades.
>
> **Overview**
> Adds a rolling-upgrade-safe fallback when
`matching.CheckTaskQueueVersionMembership` is unavailable: on
`serviceerror.Unimplemented`, the code now calls `GetTaskQueueUserData`
and locally checks whether the deployment version exists.
>
> This fallback is applied both to workflow task-queue membership
detection (`GetIsWFTaskQueueInVersionDetector`) and pinned-version
validation (`validatePinnedVersionInTaskQueue`), including caching the
fallback result and preserving the existing failed-precondition error
when the version is not a member. New unit tests cover the
`Unimplemented` fallback paths for both override validation and the
detector.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
0c26b0cc66. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Add a cause to flow info.
## Why?
Allows accurate logging of the reason for a pause, now that we have
multiple.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> Behavior is largely unchanged (still pauses/resumes under the same
conditions) and the change is confined to flow-control plumbing plus
logging/test updates.
>
> **Overview**
> Enhances replication stream receiver flow control to return a
structured `FlowControlInfo` containing both the pause/resume command
and a human-readable *cause* when pausing (e.g., outstanding task count
over limit or recent slow submissions).
>
> Updates `ackMessage` to log pause events using this cause while still
sending only the `FlowControlCommand` in replication state, and adjusts
mocks/tests to assert on the new return type and cause content.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
77cdfec8ee. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Instead of storing the test sharing salt in a repo var, store it in a
file.
## Why?
https://github.com/temporalio/temporal/pull/9236 tried to use the repo's
variable for storing the test sharding salt; but that is not accessible
to PRs from forks.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
PRs could fail due to flaky tests. But then the next run will delete the
branch (thereby closing the PR) and open a new one.
