## What changed?
Deleted `"system.enableNexus"`.
## Why?
Nexus has been GA since Dec 2024.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Add chasm logical task type dynamic config filter and standby task
discard delay dynamic config flag.
## Why?
Standby tasks have a configurable dynamic config (per task type) that
specifies the timeout for discarding the task and running the
post-discard function. Currently, this is not configurable for CHASM
logical tasks. By default, CHASM tasks need a higher discard timeout
since they are not regenerated from pending tasks in MS, which can lead
to abandoned tasks. For tasks that can be safely dispatched to Matching,
they can be configured with a lower value. (See Standalone activity
tasks).
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Fix a race condition in GetWorkflowExecutionHistory that caused SDK
workers to receive incomplete history and fail with "premature end of
stream". On the last page of a paginated `GetWorkflowExecutionHistory`
response, re-query mutable state to detect events that were committed to
the DB between the first and last page fetches. If `freshNextEventId >
continuationToken.NextEventId`, the gap is fetched from persistence and
appended to the response before transient/speculative events are added.
The continuation token is updated with the fresh boundary so
`appendTransientTasks` validates against the correct `NextEventId`. If
the re-query itself fails, the request returns an error so the client
retries.
Also adds a `nil` check in `ValidateTransientWorkflowTaskEvents`,
preventing a possible nil-pointer dereference.
## Why?
A speculative WFT is converted to normal (e.g., by an incoming signal),
committing 1–2 new events. The continuation token from page 1 points to
NextEventId=8; the DB range [6, 8) on page 2 returns only events 6–7,
missing the newly-committed events 8 and 9. `appendTransientTasks` finds
no transient tasks (speculative was committed), so the assembled history
is missing 2 events.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
- The re-query on the last history page adds one extra GetMutableState
RPC per paginated `GetWorkflowExecutionHistory` call. This is bounded to
final-page responses only and the existing path already made this call
inside `appendTransientTasks`, so the net overhead is one additional
call specifically when a gap is detected.
- Returning an error when the fresh mutable-state re-query fails changes
the previous behavior of silently continuing. Clients will retry, which
is correct, but retry storms are possible if persistence is consistently
unavailable mitigated by the client's existing backoff.
## What changed?
Add a ExecutionQueueScheduler to serialize tasks from a busy workflow
execution. Currently, all workflow tasks go through FIFO scheduler. We
have a per-workflow-execution lock that must be aquired in each history
task. But if a single execution has large number of tasks, each of these
tasks compete for this lock and create large number of retries. By
Adding this new scheduler, we can serialize task processing from such
busy workflows. These tasks are routed to a new WorkflowQueueScheduler
when lock contention is detected in a workflow. This will create a new
queue for that workflow. Additional tasks for this workflow will be then
routed to this new queue. This queue will be cleaned up after a few
seconds of inactivity from that workflow execution. We have added a
ExecutionAwareScheduler which will manage this routing of workflow tasks
to either FIFO scheduler or this new WQ Scheduler.
```
Task → InterleavedWeightedRoundRobinScheduler
↓
ExecutionAwareScheduler
↓
┌─────┴─────┐
↓ ↓
FIFOScheduler ExecutionQueueScheduler
(normal path) (contended executions)
```
This new scheduler is only enabled when
history.taskSchedulerEnableExecutionQueueScheduler is enabled. The
number of workflow queues created in this scheduler will be controlled
by config history.taskSchedulerWorkflowQueueSchedulerQueueSize.
Tasks are routed to FIFOScheduler(Like the way it was before this
change) if number of queues reaches this value.
A new set of goroutines is spawned for each queue in this new scheduler.
This is fine here as we don’t expect more than a few hundred hot
workflows per history host. This simplifies the design for this
scheduler.
## Why?
To reduce workflow lock contention and wasted history CPU when tasks are
competing for workflow lock.
## Benchmark Results
Collected by running 2,000 parallel activities from a single workflow
with a 5ms lock timeout to trigger contention. EQS queue concurrency =
2, max queues = 500.
### Task Routing & Failures
| Metric | **EQS ENABLED (C=2)** | **EQS DISABLED** | Improvement |
| --- | --- | --- | --- |
| EQS Tasks Submitted | 1,953 | 0 | - |
| EQS Tasks Completed | 1,940 | 0 | - |
| EQS Tasks Failed | 13 | 0 | - |
| EQS Tasks Aborted | 0 | 0 | - |
| EQS Submit Rejected | 0 | 0 | - |
| FIFO Tasks Completed | 65 | 2,051 | - |
| **Total Failures** | **66** | **6,382** | **97x fewer failures** |
### End-to-End Task Latency (task_latency_queue)
| Percentile | **EQS ENABLED (C=2)** | **EQS DISABLED** | Improvement |
| --- | --- | --- | --- |
| Average | 426ms | 908ms | **2.1x faster** |
| P90 | 668ms | **1,599ms** | **2.4x faster** |
| P99 | 729ms | **3,982ms** | **5.5x faster** |
| Max | 736ms | **6,773ms** | **9.2x faster** |
### Runtime
| Metric | **EQS ENABLED (C=2)** | **EQS DISABLED** | Improvement |
| --- | --- | --- | --- |
| Test Time | 4.6s | 8.9s | **1.9x faster** |
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
Added `TEMPORAL_TEST_DATA_ENCODING` to change DataBlob encoding from
"proto3" to "json".
## Why?
Observability and debugging. The ability to see payloads decoded in
debugger and OTEL traces is valuable.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
End users should never use this. The env var name therefore includes
`TEST_`.
## What changed?
- activate drained/inactive versions to draining when they get a
workflow started on it
- also added a history cache, per history node, so that we don't bombard
our version workflows with signals that shall change the drainage status
of these workflows.
## Why?
- versioning correctness, in the sense that if someone were to move a
workflow on to a version that is drained, the drainage status should be
updated to draining (since it now has one open workflow working on it)
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- Sure, this change is lowkey risky. Would appreciate a thorough review.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches history start/reset/update flows and adds asynchronous
signaling into worker-deployment workflows; mis-wiring or cache/config
issues could cause missing or excessive reactivation signals and
unexpected version state churn under load.
>
> **Overview**
> When workflows are **pinned to a specific deployment version**,
history now triggers a fire-and-forget `reactivate-version` signal to
the corresponding worker-deployment *version workflow* so versions in
`DRAINED/INACTIVE` transition back to `DRAINING`.
>
> This wiring is applied across start paths (`StartWorkflowExecution`
incl. conflict handling, `SignalWithStart`, multi-op start), option
changes (`UpdateWorkflowExecutionOptions` after persistence), and
`ResetWorkflowExecution` post-reset operations. A new per-history-node
`ReactivationSignalCache` (TTL/max-size + metrics tags) deduplicates
signals, and a new dynamic config flag
(`history.enableVersionReactivationSignals`) plus cache settings
control/limit load.
>
> Worker-deployment adds `Client.SignalVersionReactivation` and the
version workflow gains a version-gated handler for `reactivate-version`
to update drainage/status and sync summaries; extensive functional tests
cover reactivation and cache dedup behavior.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
15a5ac69b2. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Carly de Frondeville <cdefrondeville@berkeley.edu>
## What changed?
Set TargetVersionChanged instead of SuggestCaN when TargetVersionChanged
https://github.com/temporalio/api/pull/709
## Why?
Setting SuggestContinueAsNew=true for Pinned workflows whenever their is
a new Target Version available for that workflow causes Pinned workflows
to hit that condition much more frequently than they expect. Users who
are currently doing: if workflow_info.suggestContinueAsNew{ do
continue-as-new } in their Pinned workflow code would need to change
that code to protect themselves from running into an infinite-CaN-loop,
because the default CaN behavior for a Pinned workflow is to stay
Pinned.
We should not force users to protect themselves from such a situation.
Because upgrading on continue-as-new is opt-in, receiving the suggestion
to continue-as-new-onto-new-target-version should be opt-in as well. If
people are forced to check the new suggest-continue-as-new-reasons field
to "opt out," that is unsafe, because inevitably some people will forget
to do so or misunderstand, and then get hit by this unexpected footgun.
Much safer and still ergonomical to let upgrade-on-can be opt-in on both
fronts, as proposed here. With this change, the people who are currently
doing if workflow_info.suggestContinueAsNew{ do continue-as-new } won't
see any change in semantics, regardless of their versioning behavior.
People who consciously know that they want to do upgrade-on-can /
Trampolining will have to change their CaN options anyway, so it's easy
enough to teach them to pay attention to this new
TargetWorkerDeploymentVersionChanged flag.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
None
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches workflow task started event generation/persistence and
versioning-related signaling, which can affect worker behavior and
history compatibility; changes are gated by dynamic config and covered
by tests.
>
> **Overview**
> Stops using `SuggestContinueAsNew` (and its reason tags) to signal
pinned workflows that a newer target worker deployment version exists,
and instead introduces an explicit
`TargetWorkerDeploymentVersionChanged` boolean on `WorkflowTaskStarted`
events and persisted `WorkflowExecutionInfo`.
>
> Adds namespace dynamic config `EnableSendTargetVersionChanged`
(default on) and a new metric `workflow_target_version_changed_count`
emitted when this flag is set; updates the workflow task state machine,
mutable state plumbing/mocks, proto/pb persistence, and functional tests
accordingly. Also bumps `go.temporal.io/api` to pick up the new event
attribute.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
3886491826. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Disable Suggest-ContinueAsNew-on-new target version#9222
## Why?
Users of Pinned workflows who listen to SuggestContinueAsNew are at risk
of continuing-as-new way too frequently.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> Small, well-scoped behavior gating behind a new dynamic config flag
with default-off behavior; main risk is unintended change in CaN
recommendation behavior when the flag is enabled per-namespace.
>
> **Overview**
> Adds a new namespace-level dynamic config flag,
`system.enableSuggestCaNOnNewTargetVersion`, to **disable by default**
suggesting Continue-As-New to pinned workflows when a newer target
worker deployment version becomes available.
>
> History’s workflow task started event logic now only emits the
`SuggestContinueAsNew` recommendation/reason for target-version changes
when this flag is enabled, and the versioning v3 integration tests are
updated to cover both enabled and disabled behavior.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
9fd0483efb. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
<!-- BUGBOT_STATUS --><sup><a
href="https://cursor.com/dashboard?tab=bugbot">Cursor Bugbot</a>
reviewed your changes and found no issues for commit
<u>9fd0483</u></sup><!-- /BUGBOT_STATUS -->
## What changed?
Enabled the behavior by default.
## Why?
It has been supported for a SDK and server versions already and was
originally put in temporarily.
## What changed?
Cache the result of `GetTaskQueueUserData` that history makes to
Matching when an activity wants to start a deployment version
transition.
## Why?
This protects the matching root partition from being hammered by
requests when a lot of AutoUpgrade workflows want to start
activity-initiated transitions. Activity initiated transitions happen in
one of the following cases:
1) Target version changed while activity was backlogged.
2) Target version changed while activity was in retry backoff
3) Target version changed in some edge cases involving parallel
activities
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
The cache can potentially increase history mem usage but there are knobs
to adjust size and ttl.
## What changed?
This PR extends the raw history optimization to pass raw history bytes
from History Service → Matching Service → Frontend without
deserialization in Matching Service.
**Key changes:**
1. History Service:
- When `SendRawHistoryBetweenInternalServices` is enabled, sets
`RawHistoryBytes` (field 21) with raw proto-encoded history batches
2. Matching Service:
- Passes raw history bytes through to frontend via
`PollWorkflowTaskQueueResponseWithRawHistory`
- Uses wire-compatible proto messages so gRPC auto-deserializes
`[][]byte` → `History` on the client side
3. Frontend:
- Receives raw history in `RawHistory` field (auto-deserialized by gRPC)
- Processes search attributes for raw history since it bypasses history
service's normal processing
4. Proto definitions:
- Added `raw_history_bytes` (field 21) to
`RecordWorkflowTaskStartedResponse`
- Added `PollWorkflowTaskQueueResponseWithRawHistory` message with
wire-compatible layout
- Added `raw_history` (field 22) to `PollWorkflowTaskQueueResponse`
## Why?
When `history.sendRawHistoryBetweenInternalServices` is enabled, the
previous implementation only avoided deserialization from persistence →
History Service. However, Matching Service was still deserializing
history events (via gRPC auto-deserialization) and re-serializing them
when forwarding to Frontend.
This change eliminates that unnecessary serialization/deserialization
cycle in Matching Service by:
1. Having History Service send raw bytes directly
2. Having Matching Service forward these raw bytes without parsing
3. Having Frontend receive the bytes which gRPC auto-deserializes
This reduces CPU usage in Matching Service for workflows with large
histories.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s) (`tests/workflow_task_test.go`)
## Potential risks
SendRawHistoryBetweenInternalServices must be disabled when rolling back
from this version to an older version.
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## What changed?
Change the EnableTransitionHistory dynamic config from a global setting
to a namespace-scoped setting, allowing it to be configured per
namespace.
Also update the default value from false to true.
## Why?
To better control the feature.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
no risk.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## What changed?
Pause replication streams when the scheduler is under pressure.
## Why?
This allows us to apply backpressure to replication streams when we
cannot keep up with the load. The schedulers are shared amongst streams
so just keep tracking of tasked tasks is not enough. We use a timer to
notify us if a submit is taking too long rather than recording the time
after the fact so that the backpressure is more reactive.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Backpressure for replication streams**
>
> - Add slow-submission flow control: `stream_receiver` tracks
per-priority slow submission timestamps and
`stream_receiver_flow_controller` pauses when within
`ReplicationReceiverSlowSubmissionWindow` (in addition to outstanding
task count)
> - New dynamic configs:
`history.ReplicationReceiverSubmissionLatencyThreshold`,
`history.ReplicationReceiverSlowSubmissionWindow`,
`history.EnableReplicationReceiverSlowSubmissionFlowControl`, wired
through `configs.Config`
> - Refactor `StreamReceiver`: split scheduler selection into
`getTaskSchedulerPriority`/`getTaskScheduler`, measure `Submit` latency,
and feed `lastSlowSubmission` into flow control signals
> - Tests: expand flow controller tests to cover slow-submission window
logic; adjust stream receiver tests and minor robustness checks
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
9e6ec6ed81. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Update replication task rate limiter priority based on active/standby
state
## Why?
The replication task should have higher priority than standby task
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
There is a task processing priority change requires review
## What changed?
Enable RateLimitedScheduler in all cases. Pass NoopRatelimiter when
TaskSchedulerEnableRateLimiter is disabled. Also disable
TaskSchedulerEnableRateLimiterShadowMode when
TaskSchedulerEnableRateLimiter disabled. Shadow mode is not relevant
when NoopRatelimiter is used.
## Why?
It makes it easy to inject different TaskSchedulerRateLimiter without
modifying these configs.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Keep the total number and the size of the external payloads per the
workflow execution
## Why?
We are working on building the support for external payloads in SDK,
which are stored outside of Temporal. We'd like to be able to show the
total size and the number of external payloads in the given workflow
execution.
## How did you test it?
- [ ] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
N/A
## What changed?
- WISOTT
- Also added a cache per history host so that we don't overburden
matching with these calls.
- Also added a whole new unit test testing the function
`ValidateVersioningOverride`
- TODO in a follow-up PR: add metrics for this cache. Doing this as a
follow-up in the interest of time but have it tracked in JIRA.
## Why?
- Versioning correctness.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- I don't think this is risky
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Validate versioning overrides by ensuring pinned versions exist in the
task queue via matching RPC with a per-host cache, refactor validation
into history APIs, and add configs and tests.
>
> - **Worker Versioning / Validation**:
> - Add `ValidateVersioningOverride(ctx, ...)` to verify pinned versions
exist in a task queue via `matching.CheckTaskQueueVersionMembership`.
> - Introduce per-host `versionMembershipCache` to cache membership
results; reject with `FailedPrecondition` if not present.
> - Remove frontend/batcher inline override validation; perform it in
history layer (start, signal-with-start, update options, reset
post-ops).
> - **History Service Wiring**:
> - Thread `matchingClient` and `versionMembershipCache` through history
engine, starter, multi-op, signal-with-start, reset, and
update-workflow-options APIs.
> - Mark batch UpdateWorkflowOptions non-retryable for "Pinned version
is not present in the task queue".
> - **Config / Dynamic Config**:
> - Add `history.versionMembershipCacheTTL` and
`history.versionMembershipCacheMaxSize`; provide cache in `fx` with
lifecycle management.
> - **Testing**:
> - Add unit tests for `ValidateVersioningOverride` covering cache
hits/misses and v0.31/v0.32 paths.
> - Extend functional tests to assert membership checks, cache behavior,
batch update failures, and reset with post-reset options.
> - Test helpers: `TestVars.WithDeploymentSeries`, `WithBuildID`, and
utilities to ensure versions are present via matching RPC.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
d8b755273f. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
- The `enableChasm` dynamic config flag is now applied per-namespace.
## Why?
- For this release, we only want CHASM enabled in canary codepaths, for
both callbacks and scheduler.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
- This reduces the risk by reducing `enableChasm`'s blast radius.
However, we'll need to be wary of writing tests that rely on
`enableChasm` being set to `false` (current default), since in the
future it'll be enabled everywhere.
## What changed?
Integrated chasm/lib/callback into MutableState and updated
callback-related APIs to use CHASM implementation. Added integration
tests following the scheduler pattern.
## Why?
Porting callback functionality from HSM to CHASM as part of the ongoing
HSM-to-CHASM migration effort.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
This will need to be tested more, but is not being turned on with this
PR
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Integrates CHASM-based callbacks into mutable state and history APIs,
adds configs and error types, updates routing/state transitions, and
includes migration tests.
>
> - **CHASM Callback Integration**:
> - Add CHASM callback module (FX wiring, HTTP caller provider, request
routing) and metrics shared with HSM implementation.
> - Implement `ToAPICallback`, invocation flow changes, and refined
error wrapping.
> - **History/MutableState**:
> - Expose `ChasmEnabled`, `ChasmWorkflowComponent{,ReadOnly}`; use
CHASM for adding/processing completion callbacks when enabled.
> - Describe API now builds callback info from both CHASM and HSM trees.
> - Continue-as-new/retry path aggregates callbacks from both
implementations.
> - **State Machine/Executors**:
> - Switch to `queues/errors` types; move `NamespaceIDAndDestination` to
`queues/common`.
> - Adjust transitions: generate `BackoffTask` with scheduled time;
include destination on reschedule; no tasks on success/fail.
> - Simplify retry result (policy passed separately) and validation
signatures.
> - **Config**:
> - Add `EnableCHASMCallbacks` and `MaxCHASMCallbacksPerWorkflow`; wire
through history configs.
> - **Tests**:
> - Add migration and CHASM-enabled functional tests; update unit tests
to new error/types and behaviors.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
e3b51a931d. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Roey Berman <roey@temporal.io>
## What changed?
- Add a new flag that controls whether clusters setup replication
streams to each other
- Feature flag to gate the new flag for compatibility
- Functional test to vet the changes
## Why?
Optimization to avoid excessive network activity when we only want
namespace replication.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
---------
Co-authored-by: Claude <noreply@anthropic.com>
## What changed?
- Bump stamp on workflow task retry
- Propagate stamp on matching workflow task forwarding.
- Add functional test
- Refactor test interceptors
## Why?
Workflow tasks can pile up on passive side due to stamps not
incrementing, meaning they never become stale and never become eligible
for invalidation. This ensures tasks can be invalidated due to
staleness.
We need to ensure we forward the stamp in matching, otherwise prior to
dispatch the task will be deemed invalid as the stamp does not match
what is stored in mutable state.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## What changed?
Unify Visibility query converters.
The new query converter is the struct `query.QueryConverter[ExprT]`. The
generic parameter specify the output type of the converter (in ES is
`elastic.Query`, in SQL is `sqlparser.Expr`).
The query converter has a `query.StoreQueryConverter[ExprT]`. This is
the Visibility store specific implementation for building the output
query.
For example, the store needs to implement `BuildAndExpr(exprs)` to tell
the query converter how to build the `AND` expression. In SQL, it's
`*sqlparser.AndExpr{exprs}`. In Elasticsearch, it's
`elastic.NewBoolQuery().Filter(exprs)`.
The store query converter for SQL needs a plugin query converter called
`sqlplugin.VisibilityQueryConverter` which is implemented by each plugin
(MySQL, PostgreSQL, SQLite). This interface is needed to build DB
specific syntax for `KeywordList` and `Text` searches as well as to
build the final `SELECT` statements.
Added dynamic config `system.VisibilityEnableUnifiedQueryConverter` that
acts as switch between the legacy and the unified query converters.
## Why?
There are two query converters: one for Elasticsearch, and another for
SQL.
They are somewhat similar, and have quite a few code duplications for
validation, modifying the query, etc.
Unifying the query converter ensures that we provide the exact same
behavior across different Visibility stores.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
I've added a lot of unit tests for the new code with coverage close to
100%.
Note that there has been basically no changes to functional tests, which
suggests that the change works well.
## Potential risks
There might be some edge cases that either query converters that this
unified query converter might deal differently.
## What changed?
- This PR adds revision number mechanics to handle task dispatch
inconsistencies that could arise since our versioning API's are becoming
eventually-consistent.
## Why?
- Making our versioning API's eventually consistent.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
- There are risks but they are gated by a dynamic config.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Adds revision-number semantics to task dispatch/versioning and
introduces a new per-deployment TQ user-data schema, updating APIs,
persistence, and server logic (DC-gated) with comprehensive tests.
>
> - **APIs (proto changes)**:
> - Add `revision_number` to `taskqueue.v1.TaskVersionDirective` and
propagate `task_dispatch_revision_number` in History
`Record*TaskStarted` requests.
> - Matching `SyncDeploymentUserDataRequest`: new fields
`deployment_name`, `update_routing_config`, `upsert_versions_data`,
`forget_versions`; response includes `routing_config_changed`. Deprecate
old `update_version_data`/`forget_version` oneof.
> - Deployment API: add `WorkerDeploymentVersionData`; mark
`DeploymentVersionData` deprecated where applicable.
> - **Persistence (task queue user data)**:
> - Extend `DeploymentData` with `deployments_data` (map of deployment →
`WorkerDeploymentData` holding `RoutingConfig` and per-build version
data). Deprecate legacy `versions` and `unversioned_ramp_data` fields.
> - **Server logic**:
> - Matching/History: compute target version using revision numbers;
start workflow deployment transitions using revision-aware decisions;
support mixed old/new schemas when calculating current/ramping.
> - Matching: new helpers to migrate/clean old-format entries, apply
routing-config updates atomically, and prevent query blackholes using
status.
> - Task dispatch carries and records revision number; internal task
struct/plumbing updated.
> - **Dynamic config**:
> - Add `system.useRevisionNumberForWorkerVersioning` (plumbed through
History/Matching) to gate new behavior.
> - **Tests**:
> - Add/expand unit and functional tests covering new schema,
routing-config updates, revision-number behavior, and propagation.
> - **Misc**:
> - Describe/Stats paths updated to read both schemas; minor build/deps
update (api-go replace).
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
e84f484506. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: ShahabT <shahab.tajik@temporal.io>
## What changed?
- Gating stamp increment on activity retries from
https://github.com/temporalio/temporal/pull/8536/files behind a feature
flag
- Returning err when dropping stale activities on passive so we have
metrics emitted
## Why?
- We need stamp increments to only occur after the change from the PR
has fully rolled out to all clusters, else we risk compatibility issues
and dropped tasks.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- CHASM: Best effort pure task deletion
## Why?
- Performance improvement, prevent invalid physical pure tasks from
firing.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Drop repeated workflow task failure reports
## Why?
To avoid busy loop of repeated workflow task failures
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
No
## What changed?
Add a new search attribute `TemporalReportedProblems` when a workflow
task fails or timeouts N consecutive times
## Why?
Enables users to easily discover workflows that are not making progress.
After a workflow task fails or times out N consecutive times a Search
Attribute, `TemporalReportedProblems` a `KeywordList`, will be added
with two entries, a `cause` and a `category`. These search attributes
will be queryable by users with queries like:
* `TemporalReportedProblems IS NOT NULL`
* `TemporalReportedProblems IN ('category=WorkflowTaskFailed')` or
`TemporalReportedProblems IN ('category=WorkflowTaskTimedout')`
* `'TemporalReportedProblems IN ("cause=UnhandledApplicationFailure")'
OR 'TemporalReportedProblems IN ("cause=ScheduleToCloseTimeout")'`
## How did you test it?
- [ ] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
- [x] added new xdc test(s)
## Potential risks
Flapping _could_ happen with this change, but is unlikely. There's a
possibility of overloading the visibility system, but these changes are
protected with a dynamic config.
---------
Co-authored-by: Roey Berman <roey@temporal.io>
## What changed?
Optimize workflow task timeouts by completing them when the workflow
task finishes.
This optimization is enabled when the dynamic config
system.enableDeleteHistoryTasksOnUpdate is turned on.
It is disabled by default, as it can cause performance degradation in
cassandra based persistence.
## Why?
A large fraction of all timer tasks processed is workflow task timeouts.
In most of the cases, these are no-ops.
We can complete these tasks when the corresponding workflow task
finishes. This will reduce the number of timer tasks processes.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Introduced a new "move (task) group" action replacing the "incorrect"
slice-predicate action. See inline comment for how it works.
- Improve task Predicate AND/OR calculation to reduce predicate size.
- Skip empty slices when adding queue slices to readers.
## Why?
- Better namespace isolation.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
- The new action can be disabled via dc.
## What changed?
Make replicate stream liveness check dynamic configurable
## Why?
Easy to adjust the liveness check internal without restart servers.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Moving StreamSender and ExecutableTask in our replication stack to use
dynamic config.
## Why?
Gives us the ability to dynamically modify the retry config of the
replication stream on source cluster and retry config during application
on the target side.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Added 2 new namespace specific dynamic configs that allows use to
control the rollout of the new CHASM Scheduler.
## Why?
Once CHASM Scheduler is feature complete we need the ability to control
whether or not it's enabled.
---------
Co-authored-by: alex stanfield <chaptersix@users.noreply.github.com>
## What changed?
Add metrics for dataloss events that containe workflowID, runID and
namespaceID.
This metric will only be emitted if config
`system.enableDataLossMetrics` is enabled.
## Why?
To find affected workflow runs if there are any dataloss errors.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Remove Eager Refresh Namespace Dynamic Config
## Why?
The feature is enabled for long time and we can enable it by default and
no dc is required.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
no risk.
## What changed?
This adds support to the lru cache to actively expire entries older than
the TTL, by spawning a background goroutine that periodically deletes
old entries. A dynamic config, off by default, is added that can enable
this feature for the workflow cache.
## Why?
This can reduce the memory usage, and associated Go GC resource usage,
for workflow entries that won't be utilized since they are past their
TTL.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
Also running this in test setups to verify expected memory reduction.
## Potential risks
## What changed?
Deprecate ndc_history_resender code
## Why?
It is replaced by Resend Handler which support resend local generated
events.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
no risk. The dynamic config is enabled long time in prod.
## What changed?
Follow-up to https://github.com/temporalio/temporal/pull/7921 which
introduced a server-side retry for an UwS that was aborted due to a
closing workflow. This changed the status code so it's retryable by the
client as well.
## Why?
If an UwS didn't perform a start and was aborted due to a closing
workflow, we want to retry the UwS.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
Added a dynamic config to turn off the new behavior if unforeseen issues
arise.
## What changed?
Added a server-side retry for when the Update of Update-with-Start was
aborted due to a closing workflow.
## Why?
Users expect it to retry.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
In case the change goes badly (for technical or user reasons), there's a
dynamic config to disable it again.
## What changed?
Add config flag for replication rate limiter
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Add monitor in stream sender on sync status
## Why?
Receiver send replication status per seconds.
Sender should monitor on this signal and assume the receiver side is
stop if it loses this sync status.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Add replication heartbeat to sync replication state when no task
## Why?
We want to have this to monitor stream liveness.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
This adds an interceptor to the handler service that logs latency and
errors from handlers, and can be reused in other services.
Adds the interceptor to the history service, that takes a health
aggregator provider. This provider is also added to the handler such
that we can read these values in the DeepHealthCheck handler within the
history service.
A future PR will add this check to the other services.
## What changed?
- Remove shard level workflow cache logic and always use host level
workflow cache.
## Why?
- Simplify code. Host level workflow cache has been enabled by default
for a long time.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Added a log for every MultiOperation error in history.
## Why?
Track down a race condition.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Since it's only logging errors, I expect the log volume to be
reasonable/low.
## What changed?
1. Remove max retry times for replication stream
2. Improve log msg
## Why?
We have separate the ReplicationService error and ReplicationStream
error. It is not necessary to disconnect the stream if it is service
error.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
No risk. We have proper metrics and alert on it.
## What changed?
Removed config option `FollowReusePolicyAfterConflictPolicyTerminate`.
## Why?
It was [put in place](https://github.com/temporalio/temporal/pull/7099)
to have the ability to roll back a behavior change. It's been 4+ months
and it's okay to remove it now.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Resend parent for standby child completion verification.
## Why?
In the case that standby fails to verify child completion when parent
workflow is not found or not ready, standby should request resend parent
from active so standby can finish verification earlier.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
New calls will be made to active. But this will only happen after the
new config maxLocalParentWorkflowVerificationDuration. So impacts to
active will be low.
## What changed?
- Implement chasm engine Update/Read Component method
## Why?
- CHASM work stream
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
<!-- Describe what has changed in this PR -->
Dynamic config to enable generating request id reference links
## Why?
<!-- Tell your future self why have you made these changes -->
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
